2024-03-11T21:51:15.495 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T21:51:15.495 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591957 --description orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} --verbose --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591957/orig.config.yaml 2024-03-11T21:51:15.529 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T21:51:15.606 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591957 branch: squid description: orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591957' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: 94a539707733325322964c6a187c6a1b41be1daf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 3176 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 14/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 targets: smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL8HCKAiUE5i57nH3luyguokdWX60uLTvKIIIoEY6+6zrotNR7XmIqhZMcMq9WIpidu+vlY72/Egm3+g/dj5fII= smithi144.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEeSpawdyKHIPZnjnCakl3KMpXLEPUzEfx4UmuNci2etZ/uYwCAVLxMp2K//Tpoq62Ob4NrV0XeGWhHbNOZ3KE4= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T21:51:15.606 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa; will attempt to use it 2024-03-11T21:51:15.607 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks 2024-03-11T21:51:15.607 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T21:51:15.608 INFO:teuthology.task.internal:Checking packages... 2024-03-11T21:51:15.632 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-11T21:51:15.632 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T21:51:15.632 INFO:teuthology.packaging:ref: None 2024-03-11T21:51:15.632 INFO:teuthology.packaging:tag: None 2024-03-11T21:51:15.632 INFO:teuthology.packaging:branch: squid 2024-03-11T21:51:15.632 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:51:15.633 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-11T21:51:15.778 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-11T21:51:15.780 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T21:51:15.787 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T21:51:15.787 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T21:51:15.804 INFO:teuthology.task.internal:Saving configuration 2024-03-11T21:51:15.817 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T21:51:15.823 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:51:15.849 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591957', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:46:21.651062', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL8HCKAiUE5i57nH3luyguokdWX60uLTvKIIIoEY6+6zrotNR7XmIqhZMcMq9WIpidu+vlY72/Egm3+g/dj5fII='} 2024-03-11T21:51:15.872 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591957', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:46:21.652621', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEeSpawdyKHIPZnjnCakl3KMpXLEPUzEfx4UmuNci2etZ/uYwCAVLxMp2K//Tpoq62Ob4NrV0XeGWhHbNOZ3KE4='} 2024-03-11T21:51:15.872 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T21:51:15.878 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-11T21:51:15.878 INFO:teuthology.task.internal:roles: ubuntu@smithi144.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-11T21:51:15.878 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T21:51:15.941 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4193ab20d0>, signals=[15]) 2024-03-11T21:51:15.941 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T21:51:15.947 INFO:teuthology.task.internal:Opening connections... 2024-03-11T21:51:15.947 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-03-11T21:51:15.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:51:16.053 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi144.front.sepia.ceph.com 2024-03-11T21:51:16.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:51:16.145 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T21:51:16.151 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-03-11T21:51:16.168 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-03-11T21:51:16.169 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-03-11T21:51:16.226 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2024-03-11T21:51:16.226 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="9" 2024-03-11T21:51:16.226 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2024-03-11T21:51:16.226 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:51:16.226 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="9" 2024-03-11T21:51:16.226 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:51:16.226 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:51:16.227 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2024-03-11T21:51:16.227 INFO:teuthology.orchestra.run.smithi045.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:51:16.227 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:51:16.227 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:51:16.227 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:51:16.227 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:51:16.227 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:51:16.229 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-03-11T21:51:16.250 DEBUG:teuthology.orchestra.run.smithi144:> uname -m 2024-03-11T21:51:16.267 INFO:teuthology.orchestra.run.smithi144.stdout:x86_64 2024-03-11T21:51:16.267 DEBUG:teuthology.orchestra.run.smithi144:> cat /etc/os-release 2024-03-11T21:51:16.323 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="CentOS Stream" 2024-03-11T21:51:16.324 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="9" 2024-03-11T21:51:16.324 INFO:teuthology.orchestra.run.smithi144.stdout:ID="centos" 2024-03-11T21:51:16.324 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:51:16.324 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="9" 2024-03-11T21:51:16.324 INFO:teuthology.orchestra.run.smithi144.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:51:16.324 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:51:16.324 INFO:teuthology.orchestra.run.smithi144.stdout:ANSI_COLOR="0;31" 2024-03-11T21:51:16.324 INFO:teuthology.orchestra.run.smithi144.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:51:16.324 INFO:teuthology.orchestra.run.smithi144.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:51:16.324 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:51:16.324 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:51:16.324 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:51:16.325 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:51:16.325 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2024-03-11T21:51:16.341 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T21:51:16.350 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T21:51:16.357 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T21:51:16.357 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:51:16.360 DEBUG:teuthology.orchestra.run.smithi144:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:51:16.378 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T21:51:16.384 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T21:51:16.384 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:51:16.417 DEBUG:teuthology.orchestra.run.smithi144:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:51:16.533 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T21:51:16.582 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T21:51:16.596 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T21:51:16.597 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T21:51:16.597 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:51:16.597 DEBUG:teuthology.orchestra.run.smithi144:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:51:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:16.614 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-03-11T21:51:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:51:16.615 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-03-11T21:51:16.671 INFO:teuthology.orchestra.run.smithi144.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:51:16.671 INFO:teuthology.task.kernel:Running kernel on smithi144: 5.14.0-361.el9.x86_64 2024-03-11T21:51:16.671 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum install -y kernel 2024-03-11T21:51:16.672 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:51:16.673 INFO:teuthology.task.kernel:Running kernel on smithi045: 5.14.0-361.el9.x86_64 2024-03-11T21:51:16.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum install -y kernel 2024-03-11T21:51:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T21:51:18.863 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-11T21:51:20.281 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-11T21:51:20.546 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-11T21:51:23.495 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.0 MB 00:02 2024-03-11T21:51:31.853 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 732 kB/s | 8.0 MB 00:11 2024-03-11T21:51:33.613 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-11T21:51:38.207 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 6.2 MB/s | 6.1 MB 00:00 2024-03-11T21:51:39.647 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 15 kB 00:00 2024-03-11T21:51:41.234 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-11T21:51:46.323 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-03-11T21:51:48.547 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:51:48.547 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:51:48.642 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T21:51:48.653 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T21:51:48.653 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-03-11T21:51:48.654 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T21:51:48.654 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T21:51:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:51:48.654 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-11T21:51:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:51:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:51:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:51:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:51:48.654 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T21:51:48.654 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T21:51:48.654 INFO:teuthology.orchestra.run.smithi045.stdout:Install 4 Packages 2024-03-11T21:51:48.654 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:51:48.655 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 100 M 2024-03-11T21:51:48.655 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 123 M 2024-03-11T21:51:48.655 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T21:51:50.333 INFO:teuthology.orchestra.run.smithi045.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 3.7 MB/s | 6.0 MB 00:01 2024-03-11T21:51:50.801 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - AppStream 736 kB/s | 19 MB 00:26 2024-03-11T21:51:51.485 INFO:teuthology.orchestra.run.smithi045.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 7.5 MB/s | 21 MB 00:02 2024-03-11T21:51:51.992 INFO:teuthology.orchestra.run.smithi045.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 12 MB/s | 39 MB 00:03 2024-03-11T21:51:57.393 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.1 MB 00:03 2024-03-11T21:51:58.777 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 15 kB 00:00 2024-03-11T21:51:59.300 INFO:teuthology.orchestra.run.smithi045.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 3.7 MB/s | 33 MB 00:08 2024-03-11T21:51:59.300 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:51:59.300 INFO:teuthology.orchestra.run.smithi045.stdout:Total 9.4 MB/s | 100 MB 00:10 2024-03-11T21:51:59.786 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-11T21:52:00.221 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T21:52:00.692 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T21:52:00.692 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T21:52:02.072 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T21:52:02.073 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T21:52:04.794 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-11T21:52:04.936 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T21:52:05.641 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:52:05.775 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:52:07.009 INFO:teuthology.orchestra.run.smithi144.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:52:07.009 INFO:teuthology.orchestra.run.smithi144.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:52:07.106 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T21:52:07.114 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T21:52:07.114 INFO:teuthology.orchestra.run.smithi144.stdout: Package Architecture Version Repository Size 2024-03-11T21:52:07.114 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T21:52:07.114 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T21:52:07.114 INFO:teuthology.orchestra.run.smithi144.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:52:07.114 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-03-11T21:52:07.114 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:52:07.114 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:52:07.114 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:52:07.114 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:52:07.114 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T21:52:07.115 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T21:52:07.115 INFO:teuthology.orchestra.run.smithi144.stdout:Install 4 Packages 2024-03-11T21:52:07.115 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:52:07.115 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 100 M 2024-03-11T21:52:07.115 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 123 M 2024-03-11T21:52:07.115 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T21:52:07.546 INFO:teuthology.orchestra.run.smithi144.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 22 MB/s | 6.0 MB 00:00 2024-03-11T21:52:07.888 INFO:teuthology.orchestra.run.smithi144.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 34 MB/s | 21 MB 00:00 2024-03-11T21:52:08.463 INFO:teuthology.orchestra.run.smithi144.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 33 MB/s | 39 MB 00:01 2024-03-11T21:52:08.727 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:52:08.780 INFO:teuthology.orchestra.run.smithi144.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 27 MB/s | 33 MB 00:01 2024-03-11T21:52:08.780 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:52:08.780 INFO:teuthology.orchestra.run.smithi144.stdout:Total 60 MB/s | 100 MB 00:01 2024-03-11T21:52:08.926 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:52:09.660 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T21:52:10.053 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T21:52:10.053 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T21:52:11.688 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T21:52:11.689 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T21:52:14.030 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:52:14.146 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:52:14.625 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T21:52:15.298 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:52:15.424 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:52:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:52:18.376 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:52:19.276 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:52:23.094 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:52:23.328 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:52:28.062 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:52:50.009 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:52:50.009 INFO:teuthology.orchestra.run.smithi045.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:52:50.009 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:52:50.097 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:52:50.537 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:52:50.538 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:52:50.538 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:52:50.538 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:52:51.003 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:52:51.003 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:52:51.003 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T21:52:51.003 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:52:51.003 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:52:51.003 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:52:51.003 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:52:51.003 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:52:51.003 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T21:52:51.535 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:52:51.687 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:52:51.687 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:52:51.687 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:52:51.687 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:52:51.687 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:52:51.687 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T21:52:51.688 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:52:51.688 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:52:51.814 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:52:51.814 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:52:51.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.06 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 68.el9 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:Group : Unspecified 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 8264254 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:52:51.852 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : builder@centos.org 2024-03-11T21:52:51.853 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2024-03-11T21:52:51.853 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:52:51.853 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2024-03-11T21:52:51.853 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2024-03-11T21:52:51.853 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:52:51.853 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:52:51.853 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:52:51.853 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:52:51.853 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2024-03-11T21:52:51.853 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:52:51.853 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:52:51.854 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:52:51.854 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:52:51.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:52:52.536 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-03-11T21:52:54.063 INFO:teuthology.orchestra.run.smithi045.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:52:54.095 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-03-11T21:52:54.097 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-03-11T21:52:54.113 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.5eaqx9DxsO 2024-03-11T21:52:54.113 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5eaqx9DxsO 2024-03-11T21:52:54.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.5eaqx9DxsO 2024-03-11T21:52:54.325 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.5eaqx9DxsO is 6KB 2024-03-11T21:52:54.373 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.5eaqx9DxsO 2024-03-11T21:52:54.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:52:54.456 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:52:54.456 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:52:54.456 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:52:54.456 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:52:54.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:52:54.613 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-03-11T21:52:57.341 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:52:57.342 INFO:teuthology.orchestra.run.smithi144.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:52:57.342 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:52:57.426 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:52:57.926 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:52:57.926 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:52:57.926 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:52:57.926 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:52:58.353 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:52:58.353 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:52:58.353 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T21:52:58.353 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:52:58.353 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:52:58.353 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:52:58.353 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:52:58.353 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:52:58.354 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T21:52:58.865 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:52:59.006 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:52:59.006 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:52:59.006 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:52:59.007 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:52:59.007 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:52:59.007 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T21:52:59.007 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi144.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:52:59.007 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:52:59.127 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:52:59.127 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:52:59.127 DEBUG:teuthology.orchestra.run.smithi144:> sudo rpm -qi grub2-tools 2024-03-11T21:52:59.161 INFO:teuthology.orchestra.run.smithi144.stdout:Name : grub2-tools 2024-03-11T21:52:59.161 INFO:teuthology.orchestra.run.smithi144.stdout:Epoch : 1 2024-03-11T21:52:59.161 INFO:teuthology.orchestra.run.smithi144.stdout:Version : 2.06 2024-03-11T21:52:59.161 INFO:teuthology.orchestra.run.smithi144.stdout:Release : 68.el9 2024-03-11T21:52:59.161 INFO:teuthology.orchestra.run.smithi144.stdout:Architecture: x86_64 2024-03-11T21:52:59.161 INFO:teuthology.orchestra.run.smithi144.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:52:59.161 INFO:teuthology.orchestra.run.smithi144.stdout:Group : Unspecified 2024-03-11T21:52:59.161 INFO:teuthology.orchestra.run.smithi144.stdout:Size : 8264254 2024-03-11T21:52:59.161 INFO:teuthology.orchestra.run.smithi144.stdout:License : GPLv3+ 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout:Packager : builder@centos.org 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout:Vendor : CentOS 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout:Summary : Support tools for GRUB. 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout:Description : 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:52:59.162 INFO:teuthology.orchestra.run.smithi144.stdout:hardware devices. 2024-03-11T21:52:59.163 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:52:59.163 INFO:teuthology.orchestra.run.smithi144.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:52:59.163 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:52:59.163 INFO:teuthology.task.kernel:Updating grub on smithi144 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:52:59.164 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:52:59.782 INFO:teuthology.orchestra.run.smithi144.stderr:Generating grub configuration file ... 2024-03-11T21:53:01.050 INFO:teuthology.orchestra.run.smithi144.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:53:01.083 INFO:teuthology.orchestra.run.smithi144.stderr:done 2024-03-11T21:53:01.085 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2024-03-11T21:53:01.106 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.y2HDXgqErc 2024-03-11T21:53:01.106 DEBUG:teuthology.orchestra.run.smithi144:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.y2HDXgqErc 2024-03-11T21:53:01.175 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.y2HDXgqErc 2024-03-11T21:53:01.314 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.y2HDXgqErc is 6KB 2024-03-11T21:53:01.362 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.y2HDXgqErc 2024-03-11T21:53:01.376 DEBUG:teuthology.orchestra.run.smithi144:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:53:01.450 INFO:teuthology.orchestra.run.smithi144.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:53:01.450 INFO:teuthology.orchestra.run.smithi144.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:53:01.450 INFO:teuthology.orchestra.run.smithi144.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:53:01.450 INFO:teuthology.orchestra.run.smithi144.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:53:01.451 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:53:01.600 DEBUG:teuthology.orchestra.run.smithi144:> sudo shutdown -r now 2024-03-11T21:53:24.638 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T21:53:24.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-11T21:53:24.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:53:31.609 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T21:53:31.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-03-11T21:53:31.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:54:24.693 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T21:54:31.618 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T21:54:33.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-11T21:54:33.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:54:34.014 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-03-11T21:54:34.400 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-11T21:54:34.401 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:54:34.401 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-03-11T21:54:34.459 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:54:34.459 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:54:34.459 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:54:34.459 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:54:35.461 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T21:54:35.461 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:54:35.547 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2024-03-11T21:54:35.614 DEBUG:teuthology.parallel:result is None 2024-03-11T21:54:40.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-03-11T21:54:40.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:54:41.064 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-03-11T21:54:41.463 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-03-11T21:54:41.463 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:54:41.463 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-03-11T21:54:41.477 INFO:teuthology.orchestra.run.smithi144.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:54:41.477 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:54:41.477 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:54:41.477 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:54:42.478 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T21:54:42.479 DEBUG:teuthology.orchestra.run.smithi144:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:54:42.539 INFO:teuthology.orchestra.run.smithi144.stdout:ttyS1 2024-03-11T21:54:42.574 DEBUG:teuthology.parallel:result is None 2024-03-11T21:54:42.575 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T21:54:42.581 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T21:54:42.582 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:54:42.584 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:54:42.600 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T21:54:42.606 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T21:54:42.667 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T21:54:42.667 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:54:42.670 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:54:42.698 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T21:54:42.704 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T21:54:42.704 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:54:42.739 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:54:42.768 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:54:42.774 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:54:42.783 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:54:42.787 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:54:42.789 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T21:54:42.796 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T21:54:42.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:54:42.831 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:54:42.858 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T21:54:42.867 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T21:54:42.868 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:54:42.904 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:54:42.918 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:54:43.003 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:54:43.058 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:54:43.142 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:54:43.189 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T21:54:43.190 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:54:43.260 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:54:43.301 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:54:43.350 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:54:43.429 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:54:43.468 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-03-11T21:54:43.468 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:54:43.533 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-03-11T21:54:43.535 DEBUG:teuthology.orchestra.run.smithi144:> sudo service rsyslog restart 2024-03-11T21:54:43.598 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:54:43.630 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:54:44.140 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T21:54:44.148 INFO:teuthology.task.internal:Starting timer... 2024-03-11T21:54:44.148 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T21:54:44.158 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T21:54:44.165 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T21:54:44.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2024-03-11T21:54:44.245 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2024-03-11T21:54:44.247 DEBUG:teuthology.orchestra.run.smithi144:> sudo service auditd rotate 2024-03-11T21:54:44.315 INFO:teuthology.orchestra.run.smithi144.stdout:Rotating logs: 2024-03-11T21:54:44.317 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T21:54:44.317 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2024-03-11T21:54:44.341 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2024-03-11T21:54:44.341 DEBUG:teuthology.orchestra.run.smithi144:> /usr/sbin/getenforce 2024-03-11T21:54:44.367 INFO:teuthology.orchestra.run.smithi144.stdout:Permissive 2024-03-11T21:54:44.367 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi045.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi144.front.sepia.ceph.com': 'permissive'} 2024-03-11T21:54:44.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T21:54:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:54:44.414 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T21:54:44.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:54:44.442 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T21:54:44.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:54:44.501 DEBUG:teuthology.orchestra.run.smithi144:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:54:44.536 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T21:54:44.546 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T21:54:44.561 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T21:54:44.563 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi045.front.sepia.ceph.com,smithi144.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T21:58:56.168 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi144.front.sepia.ceph.com')] 2024-03-11T21:58:56.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-11T21:58:56.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:58:56.283 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-03-11T21:58:56.361 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-03-11T21:58:56.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-03-11T21:58:56.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:58:56.474 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-03-11T21:58:56.550 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-03-11T21:58:56.550 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T21:58:56.561 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T21:58:56.561 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:58:56.562 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:58:56.564 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:58:56.564 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:58:56.594 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:58:56.611 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:58:56.624 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:58:56.640 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2024-03-11T21:58:56.642 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:58:56.654 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2024-03-11T21:58:56.671 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:58:56.673 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: ntpd: command not found 2024-03-11T21:58:56.687 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:58:56.687 INFO:teuthology.orchestra.run.smithi144.stdout:506 Cannot talk to daemon 2024-03-11T21:58:56.704 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:58:56.720 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:58:56.741 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:58:56.744 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:58:56.744 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-03-11T21:58:56.744 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:58:56.744 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:58:56.744 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:58:56.745 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:58:56.770 INFO:teuthology.orchestra.run.smithi144.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:58:56.773 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:58:56.774 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2024-03-11T21:58:56.774 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:58:56.774 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:58:56.774 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:58:56.774 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:58:56.775 INFO:teuthology.run_tasks:Running task pexec... 2024-03-11T21:58:56.784 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-11T21:58:56.784 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi045.front.sepia.ceph.com 2024-03-11T21:58:56.784 DEBUG:teuthology.orchestra.run.smithi045:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T21:58:56.785 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi144.front.sepia.ceph.com 2024-03-11T21:58:56.785 DEBUG:teuthology.orchestra.run.smithi144:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T21:58:57.374 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:47 ago on Mon 11 Mar 2024 09:58:10 PM UTC. 2024-03-11T21:58:57.390 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:48 ago on Mon 11 Mar 2024 09:58:09 PM UTC. 2024-03-11T21:58:57.529 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T21:58:57.530 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T21:58:57.530 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-03-11T21:58:57.530 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T21:58:57.530 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T21:58:57.530 INFO:teuthology.orchestra.run.smithi045.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T21:58:57.530 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:58:57.530 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T21:58:57.530 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T21:58:57.530 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-11T21:58:57.531 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:58:57.535 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T21:58:57.537 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T21:58:57.537 INFO:teuthology.orchestra.run.smithi144.stdout: Package Architecture Version Repository Size 2024-03-11T21:58:57.537 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T21:58:57.537 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T21:58:57.537 INFO:teuthology.orchestra.run.smithi144.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T21:58:57.537 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:58:57.537 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T21:58:57.538 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T21:58:57.538 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-03-11T21:58:57.538 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:58:57.546 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 3.1 M 2024-03-11T21:58:57.546 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 9.8 M 2024-03-11T21:58:57.546 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T21:58:57.549 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 3.1 M 2024-03-11T21:58:57.549 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 9.8 M 2024-03-11T21:58:57.549 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T21:58:59.128 INFO:teuthology.orchestra.run.smithi045.stdout:runc-1.1.12-2.el9.x86_64.rpm 2.0 MB/s | 3.1 MB 00:01 2024-03-11T21:58:59.128 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:58:59.128 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.0 MB/s | 3.1 MB 00:01 2024-03-11T21:58:59.167 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T21:58:59.173 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T21:58:59.174 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T21:58:59.258 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T21:58:59.258 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T21:58:59.476 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T21:58:59.567 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:59:00.059 INFO:teuthology.orchestra.run.smithi144.stdout:runc-1.1.12-2.el9.x86_64.rpm 1.4 MB/s | 3.1 MB 00:02 2024-03-11T21:59:00.059 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:59:00.059 INFO:teuthology.orchestra.run.smithi144.stdout:Total 1.2 MB/s | 3.1 MB 00:02 2024-03-11T21:59:00.096 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T21:59:00.103 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:59:00.103 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T21:59:00.103 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T21:59:00.184 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T21:59:00.184 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T21:59:00.400 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:59:00.400 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:59:00.400 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T21:59:00.401 INFO:teuthology.orchestra.run.smithi045.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T21:59:00.401 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:59:00.401 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T21:59:00.403 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T21:59:00.512 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:59:00.688 DEBUG:teuthology.parallel:result is None 2024-03-11T21:59:01.096 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:59:01.417 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T21:59:01.418 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:59:01.418 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T21:59:01.418 INFO:teuthology.orchestra.run.smithi144.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T21:59:01.418 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:59:01.418 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T21:59:01.689 DEBUG:teuthology.parallel:result is None 2024-03-11T21:59:01.689 INFO:teuthology.run_tasks:Running task install... 2024-03-11T21:59:01.697 DEBUG:teuthology.task.install:project ceph 2024-03-11T21:59:01.698 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'}} 2024-03-11T21:59:01.698 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T21:59:01.698 INFO:teuthology.task.install:Using flavor: default 2024-03-11T21:59:01.703 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-11T21:59:01.703 INFO:teuthology.task.install:extra packages: [] 2024-03-11T21:59:01.703 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-11T21:59:01.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:59:01.706 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-11T21:59:01.706 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T21:59:02.090 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-11T21:59:02.091 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-11T21:59:02.094 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/ 2024-03-11T21:59:02.094 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1607.g94a53970 2024-03-11T21:59:02.203 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T21:59:02.203 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-03-11T21:59:02.203 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T21:59:02.208 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T21:59:02.208 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T21:59:02.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T21:59:02.231 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-11T21:59:02.232 DEBUG:teuthology.orchestra.run.smithi144:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T21:59:02.240 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-11T21:59:02.240 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/94a539707733325322964c6a187c6a1b41be1daf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T21:59:02.304 DEBUG:teuthology.orchestra.run.smithi144:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T21:59:02.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T21:59:02.384 DEBUG:teuthology.orchestra.run.smithi144:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T21:59:02.393 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T21:59:02.451 INFO:teuthology.orchestra.run.smithi144.stdout:check_obsoletes = 1 2024-03-11T21:59:02.453 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum clean all 2024-03-11T21:59:02.461 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2024-03-11T21:59:02.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2024-03-11T21:59:02.697 INFO:teuthology.orchestra.run.smithi144.stdout:74 files removed 2024-03-11T21:59:02.710 INFO:teuthology.orchestra.run.smithi045.stdout:74 files removed 2024-03-11T21:59:02.724 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-radosgw 2024-03-11T21:59:02.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw 2024-03-11T21:59:03.356 INFO:teuthology.orchestra.run.smithi144.stdout:ceph packages for x86_64 262 kB/s | 82 kB 00:00 2024-03-11T21:59:03.542 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 170 kB/s | 82 kB 00:00 2024-03-11T21:59:04.048 INFO:teuthology.orchestra.run.smithi144.stdout:ceph noarch packages 16 kB/s | 10 kB 00:00 2024-03-11T21:59:04.059 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-11T21:59:04.524 INFO:teuthology.orchestra.run.smithi144.stdout:ceph source packages 4.3 kB/s | 1.9 kB 00:00 2024-03-11T21:59:04.927 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-11T21:59:04.984 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-11T21:59:05.487 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-11T21:59:06.576 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-03-11T21:59:06.954 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-11T21:59:07.359 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-11T21:59:10.428 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 8.0 MB 00:03 2024-03-11T21:59:12.179 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-03-11T21:59:12.251 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - AppStream 4.9 MB/s | 19 MB 00:03 2024-03-11T21:59:16.810 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - CRB 5.6 MB/s | 6.1 MB 00:01 2024-03-11T21:59:18.719 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 15 kB 00:00 2024-03-11T21:59:20.681 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-11T21:59:25.612 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-11T21:59:27.137 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 553 kB/s | 6.1 MB 00:11 2024-03-11T21:59:28.191 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T21:59:28.194 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T21:59:28.194 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-03-11T21:59:28.194 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T21:59:28.194 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T21:59:28.194 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-11T21:59:28.194 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrading: 2024-03-11T21:59:28.194 INFO:teuthology.orchestra.run.smithi144.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-11T21:59:28.194 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T21:59:28.195 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-03-11T21:59:28.195 INFO:teuthology.orchestra.run.smithi144.stdout: boost-program-options 2024-03-11T21:59:28.195 INFO:teuthology.orchestra.run.smithi144.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T21:59:28.195 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-11T21:59:28.195 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-11T21:59:28.195 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-11T21:59:28.195 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T21:59:28.195 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T21:59:28.195 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T21:59:28.195 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-11T21:59:28.195 INFO:teuthology.orchestra.run.smithi144.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T21:59:28.195 INFO:teuthology.orchestra.run.smithi144.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T21:59:28.195 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T21:59:28.196 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-11T21:59:28.196 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T21:59:28.196 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-11T21:59:28.196 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T21:59:28.196 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T21:59:28.196 INFO:teuthology.orchestra.run.smithi144.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T21:59:28.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse 2024-03-11T21:59:28.196 INFO:teuthology.orchestra.run.smithi144.stdout: x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-11T21:59:28.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-11T21:59:28.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-11T21:59:28.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout:Installing weak dependencies: 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout:Install 30 Packages 2024-03-11T21:59:28.197 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrade 2 Packages 2024-03-11T21:59:28.198 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:59:28.198 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 58 M 2024-03-11T21:59:28.198 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T21:59:28.913 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2024-03-11T21:59:29.614 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 36 MB/s | 21 MB 00:00 2024-03-11T21:59:29.946 INFO:teuthology.orchestra.run.smithi144.stdout:(1/32): ceph-base-19.0.0-1607.g94a53970.el9.x86 5.0 MB/s | 5.3 MB 00:01 2024-03-11T21:59:29.988 INFO:teuthology.orchestra.run.smithi144.stdout:(2/32): ceph-selinux-19.0.0-1607.g94a53970.el9. 608 kB/s | 25 kB 00:00 2024-03-11T21:59:30.097 INFO:teuthology.orchestra.run.smithi144.stdout:(3/32): libcephfs2-19.0.0-1607.g94a53970.el9.x8 6.7 MB/s | 742 kB 00:00 2024-03-11T21:59:30.189 INFO:teuthology.orchestra.run.smithi144.stdout:(4/32): libradosstriper1-19.0.0-1607.g94a53970. 5.3 MB/s | 498 kB 00:00 2024-03-11T21:59:30.497 INFO:teuthology.orchestra.run.smithi144.stdout:(5/32): librgw2-19.0.0-1607.g94a53970.el9.x86_6 17 MB/s | 5.1 MB 00:00 2024-03-11T21:59:30.547 INFO:teuthology.orchestra.run.smithi144.stdout:(6/32): python3-ceph-argparse-19.0.0-1607.g94a5 907 kB/s | 45 kB 00:00 2024-03-11T21:59:30.589 INFO:teuthology.orchestra.run.smithi144.stdout:(7/32): python3-ceph-common-19.0.0-1607.g94a539 3.1 MB/s | 131 kB 00:00 2024-03-11T21:59:30.631 INFO:teuthology.orchestra.run.smithi144.stdout:(8/32): python3-cephfs-19.0.0-1607.g94a53970.el 3.8 MB/s | 161 kB 00:00 2024-03-11T21:59:30.682 INFO:teuthology.orchestra.run.smithi144.stdout:(9/32): python3-rados-19.0.0-1607.g94a53970.el9 6.3 MB/s | 320 kB 00:00 2024-03-11T21:59:30.732 INFO:teuthology.orchestra.run.smithi144.stdout:(10/32): python3-rbd-19.0.0-1607.g94a53970.el9. 5.8 MB/s | 298 kB 00:00 2024-03-11T21:59:30.774 INFO:teuthology.orchestra.run.smithi144.stdout:(11/32): python3-rgw-19.0.0-1607.g94a53970.el9. 2.3 MB/s | 99 kB 00:00 2024-03-11T21:59:30.907 INFO:teuthology.orchestra.run.smithi144.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 794 kB/s | 106 kB 00:00 2024-03-11T21:59:30.966 INFO:teuthology.orchestra.run.smithi144.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-11T21:59:30.999 INFO:teuthology.orchestra.run.smithi144.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-11T21:59:31.100 INFO:teuthology.orchestra.run.smithi144.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-03-11T21:59:31.141 INFO:teuthology.orchestra.run.smithi144.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-11T21:59:31.192 INFO:teuthology.orchestra.run.smithi144.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 823 kB/s | 41 kB 00:00 2024-03-11T21:59:31.617 INFO:teuthology.orchestra.run.smithi144.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 510 kB/s | 216 kB 00:00 2024-03-11T21:59:31.717 INFO:teuthology.orchestra.run.smithi144.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 349 kB/s | 35 kB 00:00 2024-03-11T21:59:31.818 INFO:teuthology.orchestra.run.smithi144.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 444 kB/s | 44 kB 00:00 2024-03-11T21:59:32.343 INFO:teuthology.orchestra.run.smithi144.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 588 kB/s | 309 kB 00:00 2024-03-11T21:59:33.143 INFO:teuthology.orchestra.run.smithi144.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 4.4 MB 00:00 2024-03-11T21:59:33.344 INFO:teuthology.orchestra.run.smithi144.stdout:(23/32): ceph-common-19.0.0-1607.g94a53970.el9. 4.4 MB/s | 20 MB 00:04 2024-03-11T21:59:33.369 INFO:teuthology.orchestra.run.smithi144.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 115 kB/s | 26 kB 00:00 2024-03-11T21:59:33.469 INFO:teuthology.orchestra.run.smithi144.stdout:(25/32): ceph-radosgw-19.0.0-1607.g94a53970.el9 2.1 MB/s | 9.8 MB 00:04 2024-03-11T21:59:33.494 INFO:teuthology.orchestra.run.smithi144.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 323 kB/s | 49 kB 00:00 2024-03-11T21:59:33.519 INFO:teuthology.orchestra.run.smithi144.stdout:(27/32): libunwind-1.6.2-1.el9.x86_64.rpm 448 kB/s | 67 kB 00:00 2024-03-11T21:59:33.628 INFO:teuthology.orchestra.run.smithi144.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 838 kB 00:00 2024-03-11T21:59:33.678 INFO:teuthology.orchestra.run.smithi144.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-11T21:59:33.854 INFO:teuthology.orchestra.run.smithi144.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.7 MB/s | 1.6 MB 00:00 2024-03-11T21:59:33.954 INFO:teuthology.orchestra.run.smithi144.stdout:(31/32): librados2-19.0.0-1607.g94a53970.el9.x8 10 MB/s | 3.4 MB 00:00 2024-03-11T21:59:34.647 INFO:teuthology.orchestra.run.smithi144.stdout:(32/32): librbd1-19.0.0-1607.g94a53970.el9.x86_ 3.2 MB/s | 3.1 MB 00:00 2024-03-11T21:59:34.650 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:59:34.650 INFO:teuthology.orchestra.run.smithi144.stdout:Total 9.0 MB/s | 58 MB 00:06 2024-03-11T21:59:34.685 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T21:59:34.722 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T21:59:34.722 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T21:59:34.897 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-11T21:59:35.194 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T21:59:35.195 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T21:59:36.633 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T21:59:36.725 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-11T21:59:36.759 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-11T21:59:36.784 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-11T21:59:36.807 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T21:59:36.823 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T21:59:37.017 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T21:59:37.070 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T21:59:37.405 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-11T21:59:37.452 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 7/34 2024-03-11T21:59:37.538 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 7/34 2024-03-11T21:59:37.565 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T21:59:37.567 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T21:59:37.567 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-11T21:59:37.568 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T21:59:37.568 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T21:59:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 9.8 M 2024-03-11T21:59:37.568 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-03-11T21:59:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.4 M 2024-03-11T21:59:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T21:59:37.568 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-11T21:59:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options 2024-03-11T21:59:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T21:59:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.3 M 2024-03-11T21:59:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 20 M 2024-03-11T21:59:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:19.0.0-1607.g94a53970.el9 ceph 25 k 2024-03-11T21:59:37.568 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 742 k 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 498 k 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:19.0.0-1607.g94a53970.el9 ceph 5.1 M 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse 2024-03-11T21:59:37.569 INFO:teuthology.orchestra.run.smithi045.stdout: x86_64 2:19.0.0-1607.g94a53970.el9 ceph 45 k 2024-03-11T21:59:37.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:19.0.0-1607.g94a53970.el9 ceph 131 k 2024-03-11T21:59:37.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:19.0.0-1607.g94a53970.el9 ceph 161 k 2024-03-11T21:59:37.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T21:59:37.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:19.0.0-1607.g94a53970.el9 ceph 320 k 2024-03-11T21:59:37.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 298 k 2024-03-11T21:59:37.570 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:19.0.0-1607.g94a53970.el9 ceph 99 k 2024-03-11T21:59:37.570 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T21:59:37.570 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T21:59:37.570 INFO:teuthology.orchestra.run.smithi045.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T21:59:37.570 INFO:teuthology.orchestra.run.smithi045.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T21:59:37.570 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2024-03-11T21:59:37.570 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T21:59:37.571 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:59:37.571 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T21:59:37.571 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T21:59:37.571 INFO:teuthology.orchestra.run.smithi045.stdout:Install 30 Packages 2024-03-11T21:59:37.571 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 2 Packages 2024-03-11T21:59:37.571 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:59:37.571 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 58 M 2024-03-11T21:59:37.572 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T21:59:37.625 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 8/34 2024-03-11T21:59:37.663 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 9/34 2024-03-11T21:59:37.742 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 9/34 2024-03-11T21:59:37.770 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-11T21:59:37.805 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9. 11/34 2024-03-11T21:59:37.874 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 12/34 2024-03-11T21:59:37.898 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_6 13/34 2024-03-11T21:59:37.944 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_6 13/34 2024-03-11T21:59:37.971 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-11T21:59:38.079 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-11T21:59:38.120 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-11T21:59:38.163 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-11T21:59:38.196 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-11T21:59:38.225 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-11T21:59:38.258 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-11T21:59:38.332 INFO:teuthology.orchestra.run.smithi045.stdout:(1/32): ceph-base-19.0.0-1607.g94a53970.el9.x86 8.3 MB/s | 5.3 MB 00:00 2024-03-11T21:59:38.382 INFO:teuthology.orchestra.run.smithi045.stdout:(2/32): ceph-selinux-19.0.0-1607.g94a53970.el9. 506 kB/s | 25 kB 00:00 2024-03-11T21:59:38.474 INFO:teuthology.orchestra.run.smithi045.stdout:(3/32): libcephfs2-19.0.0-1607.g94a53970.el9.x8 7.9 MB/s | 742 kB 00:00 2024-03-11T21:59:38.568 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-11T21:59:38.575 INFO:teuthology.orchestra.run.smithi045.stdout:(4/32): libradosstriper1-19.0.0-1607.g94a53970. 4.9 MB/s | 498 kB 00:00 2024-03-11T21:59:38.596 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 22/34 2024-03-11T21:59:38.668 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 22/34 2024-03-11T21:59:38.700 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 23/34 2024-03-11T21:59:38.744 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-11T21:59:39.108 INFO:teuthology.orchestra.run.smithi045.stdout:(5/32): librgw2-19.0.0-1607.g94a53970.el9.x86_6 9.6 MB/s | 5.1 MB 00:00 2024-03-11T21:59:39.198 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-11T21:59:39.223 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 26/34 2024-03-11T21:59:39.294 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 26/34 2024-03-11T21:59:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:(6/32): ceph-common-19.0.0-1607.g94a53970.el9.x 12 MB/s | 20 MB 00:01 2024-03-11T21:59:39.317 INFO:teuthology.orchestra.run.smithi045.stdout:(7/32): python3-ceph-argparse-19.0.0-1607.g94a5 216 kB/s | 45 kB 00:00 2024-03-11T21:59:39.342 INFO:teuthology.orchestra.run.smithi045.stdout:(8/32): python3-ceph-common-19.0.0-1607.g94a539 3.1 MB/s | 131 kB 00:00 2024-03-11T21:59:39.363 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 27/34 2024-03-11T21:59:39.367 INFO:teuthology.orchestra.run.smithi045.stdout:(9/32): python3-cephfs-19.0.0-1607.g94a53970.el 3.1 MB/s | 161 kB 00:00 2024-03-11T21:59:39.392 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x8 28/34 2024-03-11T21:59:39.393 INFO:teuthology.orchestra.run.smithi045.stdout:(10/32): python3-rados-19.0.0-1607.g94a53970.el 6.2 MB/s | 320 kB 00:00 2024-03-11T21:59:39.418 INFO:teuthology.orchestra.run.smithi045.stdout:(11/32): python3-rbd-19.0.0-1607.g94a53970.el9. 5.8 MB/s | 298 kB 00:00 2024-03-11T21:59:39.443 INFO:teuthology.orchestra.run.smithi045.stdout:(12/32): python3-rgw-19.0.0-1607.g94a53970.el9. 1.9 MB/s | 99 kB 00:00 2024-03-11T21:59:39.577 INFO:teuthology.orchestra.run.smithi045.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 668 kB/s | 106 kB 00:00 2024-03-11T21:59:39.602 INFO:teuthology.orchestra.run.smithi045.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-11T21:59:39.619 INFO:teuthology.orchestra.run.smithi045.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-11T21:59:39.686 INFO:teuthology.orchestra.run.smithi045.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-11T21:59:39.711 INFO:teuthology.orchestra.run.smithi045.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-11T21:59:39.728 INFO:teuthology.orchestra.run.smithi045.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 988 kB/s | 41 kB 00:00 2024-03-11T21:59:40.028 INFO:teuthology.orchestra.run.smithi045.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 116 kB/s | 35 kB 00:00 2024-03-11T21:59:40.237 INFO:teuthology.orchestra.run.smithi045.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 213 kB/s | 44 kB 00:00 2024-03-11T21:59:40.312 INFO:teuthology.orchestra.run.smithi045.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 360 kB/s | 216 kB 00:00 2024-03-11T21:59:40.588 INFO:teuthology.orchestra.run.smithi045.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 882 kB/s | 309 kB 00:00 2024-03-11T21:59:40.655 INFO:teuthology.orchestra.run.smithi045.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-03-11T21:59:40.738 INFO:teuthology.orchestra.run.smithi045.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 584 kB/s | 49 kB 00:00 2024-03-11T21:59:40.838 INFO:teuthology.orchestra.run.smithi045.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-03-11T21:59:41.081 INFO:teuthology.orchestra.run.smithi045.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 4.4 MB 00:00 2024-03-11T21:59:41.106 INFO:teuthology.orchestra.run.smithi045.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2024-03-11T21:59:41.156 INFO:teuthology.orchestra.run.smithi045.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-11T21:59:41.315 INFO:teuthology.orchestra.run.smithi045.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-03-11T21:59:41.482 INFO:teuthology.orchestra.run.smithi045.stdout:(30/32): librados2-19.0.0-1607.g94a53970.el9.x8 10 MB/s | 3.4 MB 00:00 2024-03-11T21:59:41.572 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 29/34 2024-03-11T21:59:41.613 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 29/34 2024-03-11T21:59:41.657 INFO:teuthology.orchestra.run.smithi045.stdout:(31/32): librbd1-19.0.0-1607.g94a53970.el9.x86_ 9.2 MB/s | 3.1 MB 00:00 2024-03-11T21:59:42.170 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 29/34 2024-03-11T21:59:42.208 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 30/34 2024-03-11T21:59:42.307 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 30/34 2024-03-11T21:59:42.307 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T21:59:42.307 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T21:59:42.307 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:59:42.335 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 31/34 2024-03-11T21:59:47.448 INFO:teuthology.orchestra.run.smithi045.stdout:(32/32): ceph-radosgw-19.0.0-1607.g94a53970.el9 1.0 MB/s | 9.8 MB 00:09 2024-03-11T21:59:47.452 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:59:47.452 INFO:teuthology.orchestra.run.smithi045.stdout:Total 5.9 MB/s | 58 MB 00:09 2024-03-11T21:59:47.487 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T21:59:47.529 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T21:59:47.529 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T21:59:48.020 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T21:59:48.020 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T21:59:49.454 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T21:59:49.541 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-11T21:59:49.570 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-11T21:59:49.599 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-11T21:59:49.626 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T21:59:49.642 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T21:59:49.813 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T21:59:49.866 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T21:59:50.211 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-11T21:59:50.272 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 7/34 2024-03-11T21:59:50.367 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 7/34 2024-03-11T21:59:50.463 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 8/34 2024-03-11T21:59:50.490 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 9/34 2024-03-11T21:59:50.561 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 9/34 2024-03-11T21:59:50.590 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-11T21:59:50.634 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9. 11/34 2024-03-11T21:59:50.712 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 12/34 2024-03-11T21:59:50.734 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_6 13/34 2024-03-11T21:59:50.797 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_6 13/34 2024-03-11T21:59:50.824 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-11T21:59:50.928 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-11T21:59:50.967 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-11T21:59:51.008 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-11T21:59:51.041 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-11T21:59:51.070 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-11T21:59:51.102 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-11T21:59:51.428 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-11T21:59:51.465 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 22/34 2024-03-11T21:59:51.557 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 22/34 2024-03-11T21:59:51.587 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 23/34 2024-03-11T21:59:51.634 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-11T21:59:51.793 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 31/34 2024-03-11T21:59:51.793 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /sys 2024-03-11T21:59:51.793 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /proc 2024-03-11T21:59:51.794 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /mnt 2024-03-11T21:59:51.794 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /var/tmp 2024-03-11T21:59:51.794 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /home 2024-03-11T21:59:51.794 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /root 2024-03-11T21:59:51.794 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /tmp 2024-03-11T21:59:51.794 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:59:52.112 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-11T21:59:52.143 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 26/34 2024-03-11T21:59:52.214 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 26/34 2024-03-11T21:59:52.294 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 27/34 2024-03-11T21:59:52.328 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x8 28/34 2024-03-11T21:59:52.700 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 32/34 2024-03-11T21:59:52.731 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 32/34 2024-03-11T21:59:52.731 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:59:52.731 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T21:59:52.731 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T21:59:52.731 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T21:59:52.731 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:59:52.733 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T21:59:52.820 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T21:59:52.820 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T21:59:54.049 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T21:59:54.049 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 1/34 2024-03-11T21:59:54.049 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2/34 2024-03-11T21:59:54.050 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 3/34 2024-03-11T21:59:54.050 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 4/34 2024-03-11T21:59:54.050 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 5/34 2024-03-11T21:59:54.050 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_6 6/34 2024-03-11T21:59:54.050 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 7/34 2024-03-11T21:59:54.050 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9. 8/34 2024-03-11T21:59:54.050 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x8 9/34 2024-03-11T21:59:54.050 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 10/34 2024-03-11T21:59:54.050 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 11/34 2024-03-11T21:59:54.050 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 12/34 2024-03-11T21:59:54.050 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 13/34 2024-03-11T21:59:54.051 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-11T21:59:54.051 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-11T21:59:54.051 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-11T21:59:54.051 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-11T21:59:54.051 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-11T21:59:54.051 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-11T21:59:54.051 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-11T21:59:54.051 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-11T21:59:54.051 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-11T21:59:54.051 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-11T21:59:54.052 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-11T21:59:54.052 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-11T21:59:54.052 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-11T21:59:54.052 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-11T21:59:54.052 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-11T21:59:54.052 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-11T21:59:54.052 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-11T21:59:54.052 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 31/34 2024-03-11T21:59:54.052 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-11T21:59:54.052 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 33/34 2024-03-11T21:59:54.589 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 29/34 2024-03-11T21:59:54.633 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 29/34 2024-03-11T21:59:55.124 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T21:59:55.124 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:59:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:Upgraded: 2024-03-11T21:59:55.124 INFO:teuthology.orchestra.run.smithi144.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.124 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.124 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T21:59:55.124 INFO:teuthology.orchestra.run.smithi144.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T21:59:55.124 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.124 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.124 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.124 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.125 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.126 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T21:59:55.126 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.126 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.126 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T21:59:55.126 INFO:teuthology.orchestra.run.smithi144.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T21:59:55.126 INFO:teuthology.orchestra.run.smithi144.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T21:59:55.126 INFO:teuthology.orchestra.run.smithi144.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T21:59:55.126 INFO:teuthology.orchestra.run.smithi144.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T21:59:55.126 INFO:teuthology.orchestra.run.smithi144.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T21:59:55.127 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:59:55.127 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T21:59:55.237 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 29/34 2024-03-11T21:59:55.270 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 30/34 2024-03-11T21:59:55.371 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 30/34 2024-03-11T21:59:55.371 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T21:59:55.371 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T21:59:55.372 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T21:59:55.412 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 31/34 2024-03-11T21:59:55.485 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-test 2024-03-11T21:59:56.052 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:31 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T21:59:56.201 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T21:59:56.201 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T21:59:56.201 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-03-11T21:59:56.201 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T21:59:56.201 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T21:59:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-11T21:59:56.202 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-03-11T21:59:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T21:59:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-11T21:59:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T21:59:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T21:59:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T21:59:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:59:56.202 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T21:59:56.202 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T21:59:56.202 INFO:teuthology.orchestra.run.smithi144.stdout:Install 6 Packages 2024-03-11T21:59:56.202 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T21:59:56.203 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 48 M 2024-03-11T21:59:56.203 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 205 M 2024-03-11T21:59:56.203 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T21:59:56.408 INFO:teuthology.orchestra.run.smithi144.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-11T21:59:56.466 INFO:teuthology.orchestra.run.smithi144.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-11T21:59:56.508 INFO:teuthology.orchestra.run.smithi144.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-11T21:59:56.941 INFO:teuthology.orchestra.run.smithi144.stdout:(4/6): libcephsqlite-19.0.0-1607.g94a53970.el9. 236 kB/s | 163 kB 00:00 2024-03-11T21:59:57.250 INFO:teuthology.orchestra.run.smithi144.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 252 kB/s | 187 kB 00:00 2024-03-11T22:00:05.504 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 31/34 2024-03-11T22:00:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2024-03-11T22:00:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2024-03-11T22:00:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2024-03-11T22:00:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2024-03-11T22:00:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2024-03-11T22:00:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /root 2024-03-11T22:00:05.505 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2024-03-11T22:00:05.505 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:06.433 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 32/34 2024-03-11T22:00:06.457 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 32/34 2024-03-11T22:00:06.457 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:00:06.457 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T22:00:06.457 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T22:00:06.457 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T22:00:06.458 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:06.459 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T22:00:06.543 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T22:00:06.544 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T22:00:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T22:00:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 1/34 2024-03-11T22:00:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2/34 2024-03-11T22:00:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 3/34 2024-03-11T22:00:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 4/34 2024-03-11T22:00:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 5/34 2024-03-11T22:00:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_6 6/34 2024-03-11T22:00:07.712 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 7/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9. 8/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x8 9/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 10/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 11/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 12/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 13/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-11T22:00:07.713 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-11T22:00:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-11T22:00:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-11T22:00:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-11T22:00:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-11T22:00:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-11T22:00:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-11T22:00:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-11T22:00:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:19.0.0-1607.g94a53970.el9.x86_64 31/34 2024-03-11T22:00:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-11T22:00:07.714 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 33/34 2024-03-11T22:00:08.645 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T22:00:08.645 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:08.646 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-03-11T22:00:08.646 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.646 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.646 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:00:08.646 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T22:00:08.646 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.647 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.647 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.647 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.647 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T22:00:08.647 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T22:00:08.647 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T22:00:08.647 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.647 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T22:00:08.647 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T22:00:08.647 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T22:00:08.648 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.648 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T22:00:08.648 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.648 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T22:00:08.648 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T22:00:08.648 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T22:00:08.648 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.648 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.648 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.648 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T22:00:08.648 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.648 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.648 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:08.649 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T22:00:08.649 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T22:00:08.649 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T22:00:08.649 INFO:teuthology.orchestra.run.smithi045.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T22:00:08.649 INFO:teuthology.orchestra.run.smithi045.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T22:00:08.649 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:08.649 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:00:08.974 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-test 2024-03-11T22:00:09.537 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:35 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:00:09.699 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:00:09.700 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:00:09.700 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-11T22:00:09.700 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:00:09.700 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:00:09.700 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:19.0.0-1607.g94a53970.el9 ceph 47 M 2024-03-11T22:00:09.700 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-11T22:00:09.700 INFO:teuthology.orchestra.run.smithi045.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T22:00:09.701 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:19.0.0-1607.g94a53970.el9 ceph 163 k 2024-03-11T22:00:09.701 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T22:00:09.701 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T22:00:09.701 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T22:00:09.701 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:09.701 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:00:09.701 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:00:09.701 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2024-03-11T22:00:09.701 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:09.702 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 48 M 2024-03-11T22:00:09.702 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 205 M 2024-03-11T22:00:09.702 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:00:10.190 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 813 kB/s | 217 kB 00:00 2024-03-11T22:00:10.248 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-11T22:00:10.282 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-11T22:00:10.482 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): libcephsqlite-19.0.0-1607.g94a53970.el9. 292 kB/s | 163 kB 00:00 2024-03-11T22:00:10.507 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 831 kB/s | 187 kB 00:00 2024-03-11T22:00:18.732 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): ceph-test-19.0.0-1607.g94a53970.el9.x86_ 5.4 MB/s | 47 MB 00:08 2024-03-11T22:00:18.735 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:00:18.735 INFO:teuthology.orchestra.run.smithi045.stdout:Total 5.3 MB/s | 48 MB 00:09 2024-03-11T22:00:18.749 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:00:18.773 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:00:18.773 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:00:19.174 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:00:19.175 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:00:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:00:19.858 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-11T22:00:19.894 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-11T22:00:19.935 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T22:00:19.984 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-11T22:00:20.010 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 5/6 2024-03-11T22:00:24.753 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 5/6 2024-03-11T22:00:24.828 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 6/6 2024-03-11T22:00:26.538 INFO:teuthology.orchestra.run.smithi144.stdout:(6/6): ceph-test-19.0.0-1607.g94a53970.el9.x86_ 1.6 MB/s | 47 MB 00:30 2024-03-11T22:00:26.542 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:00:26.542 INFO:teuthology.orchestra.run.smithi144.stdout:Total 1.6 MB/s | 48 MB 00:30 2024-03-11T22:00:26.557 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:00:26.578 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:00:26.579 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:00:26.961 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:00:26.962 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:00:27.190 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 6/6 2024-03-11T22:00:27.191 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 1/6 2024-03-11T22:00:27.191 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2/6 2024-03-11T22:00:27.191 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T22:00:27.191 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-11T22:00:27.191 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-11T22:00:27.558 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:00:27.639 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-11T22:00:27.663 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-11T22:00:27.663 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:27.663 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:00:27.663 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:27.663 INFO:teuthology.orchestra.run.smithi045.stdout: jq-1.6-16.el9.x86_64 2024-03-11T22:00:27.663 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:27.663 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T22:00:27.663 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T22:00:27.663 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T22:00:27.663 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:27.663 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:00:27.680 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-11T22:00:27.720 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T22:00:27.773 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-11T22:00:27.800 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 5/6 2024-03-11T22:00:27.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph 2024-03-11T22:00:28.450 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:54 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:00:28.608 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:00:28.611 INFO:teuthology.orchestra.run.smithi045.stdout:========================================================================================= 2024-03-11T22:00:28.611 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-11T22:00:28.611 INFO:teuthology.orchestra.run.smithi045.stdout:========================================================================================= 2024-03-11T22:00:28.611 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:00:28.611 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-11T22:00:28.611 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T22:00:28.612 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T22:00:28.613 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T22:00:28.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T22:00:28.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T22:00:28.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T22:00:28.614 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T22:00:28.614 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:28.614 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:00:28.614 INFO:teuthology.orchestra.run.smithi045.stdout:========================================================================================= 2024-03-11T22:00:28.614 INFO:teuthology.orchestra.run.smithi045.stdout:Install 36 Packages 2024-03-11T22:00:28.614 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:28.615 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 30 M 2024-03-11T22:00:28.615 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 107 M 2024-03-11T22:00:28.615 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:00:29.204 INFO:teuthology.orchestra.run.smithi045.stdout:(1/36): ceph-19.0.0-1607.g94a53970.el9.x86_64.r 35 kB/s | 6.6 kB 00:00 2024-03-11T22:00:29.470 INFO:teuthology.orchestra.run.smithi045.stdout:(2/36): ceph-mds-19.0.0-1607.g94a53970.el9.x86_ 4.7 MB/s | 2.1 MB 00:00 2024-03-11T22:00:29.804 INFO:teuthology.orchestra.run.smithi045.stdout:(3/36): ceph-mon-19.0.0-1607.g94a53970.el9.x86_ 7.8 MB/s | 4.7 MB 00:00 2024-03-11T22:00:29.862 INFO:teuthology.orchestra.run.smithi045.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1607.g94a5 4.2 MB/s | 247 kB 00:00 2024-03-11T22:00:29.938 INFO:teuthology.orchestra.run.smithi045.stdout:(5/36): ceph-mgr-19.0.0-1607.g94a53970.el9.x86_ 1.6 MB/s | 1.5 MB 00:00 2024-03-11T22:00:30.021 INFO:teuthology.orchestra.run.smithi045.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-11T22:00:30.046 INFO:teuthology.orchestra.run.smithi045.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-03-11T22:00:30.072 INFO:teuthology.orchestra.run.smithi045.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 633 kB/s | 32 kB 00:00 2024-03-11T22:00:30.097 INFO:teuthology.orchestra.run.smithi045.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 939 kB/s | 47 kB 00:00 2024-03-11T22:00:30.122 INFO:teuthology.orchestra.run.smithi045.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-03-11T22:00:30.147 INFO:teuthology.orchestra.run.smithi045.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-11T22:00:30.206 INFO:teuthology.orchestra.run.smithi045.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-11T22:00:30.231 INFO:teuthology.orchestra.run.smithi045.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-11T22:00:30.257 INFO:teuthology.orchestra.run.smithi045.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-11T22:00:30.282 INFO:teuthology.orchestra.run.smithi045.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-11T22:00:30.533 INFO:teuthology.orchestra.run.smithi045.stdout:(16/36): ceph-osd-19.0.0-1607.g94a53970.el9.x86 16 MB/s | 17 MB 00:01 2024-03-11T22:00:30.550 INFO:teuthology.orchestra.run.smithi045.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 83 kB/s | 22 kB 00:00 2024-03-11T22:00:30.642 INFO:teuthology.orchestra.run.smithi045.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 325 kB/s | 125 kB 00:00 2024-03-11T22:00:31.083 INFO:teuthology.orchestra.run.smithi045.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 81 kB/s | 43 kB 00:00 2024-03-11T22:00:31.217 INFO:teuthology.orchestra.run.smithi045.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 221 kB/s | 151 kB 00:00 2024-03-11T22:00:31.242 INFO:teuthology.orchestra.run.smithi045.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 286 kB/s | 172 kB 00:00 2024-03-11T22:00:31.342 INFO:teuthology.orchestra.run.smithi045.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2024-03-11T22:00:31.360 INFO:teuthology.orchestra.run.smithi045.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-03-11T22:00:31.435 INFO:teuthology.orchestra.run.smithi045.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-03-11T22:00:31.460 INFO:teuthology.orchestra.run.smithi045.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 197 kB/s | 23 kB 00:00 2024-03-11T22:00:31.485 INFO:teuthology.orchestra.run.smithi045.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2024-03-11T22:00:31.553 INFO:teuthology.orchestra.run.smithi045.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 167 kB/s | 20 kB 00:00 2024-03-11T22:00:31.594 INFO:teuthology.orchestra.run.smithi045.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 345 kB/s | 46 kB 00:00 2024-03-11T22:00:31.636 INFO:teuthology.orchestra.run.smithi045.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 522 kB/s | 79 kB 00:00 2024-03-11T22:00:31.745 INFO:teuthology.orchestra.run.smithi045.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-11T22:00:31.770 INFO:teuthology.orchestra.run.smithi045.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 93 kB/s | 16 kB 00:00 2024-03-11T22:00:31.796 INFO:teuthology.orchestra.run.smithi045.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 566 kB/s | 90 kB 00:00 2024-03-11T22:00:31.871 INFO:teuthology.orchestra.run.smithi045.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 285 kB/s | 36 kB 00:00 2024-03-11T22:00:31.996 INFO:teuthology.orchestra.run.smithi045.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 160 kB/s | 20 kB 00:00 2024-03-11T22:00:32.021 INFO:teuthology.orchestra.run.smithi045.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 916 kB/s | 230 kB 00:00 2024-03-11T22:00:32.047 INFO:teuthology.orchestra.run.smithi045.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.7 MB/s | 427 kB 00:00 2024-03-11T22:00:32.049 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:00:32.050 INFO:teuthology.orchestra.run.smithi045.stdout:Total 8.6 MB/s | 30 MB 00:03 2024-03-11T22:00:32.075 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:00:32.120 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:00:32.120 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:00:32.401 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 5/6 2024-03-11T22:00:32.473 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 6/6 2024-03-11T22:00:32.546 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:00:32.546 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:00:33.902 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:00:33.966 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-11T22:00:33.999 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-11T22:00:34.035 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-11T22:00:34.110 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-11T22:00:34.159 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-11T22:00:34.193 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-11T22:00:34.290 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-11T22:00:34.374 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-11T22:00:34.421 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-11T22:00:34.496 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-11T22:00:34.548 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-11T22:00:34.585 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-11T22:00:34.643 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-11T22:00:34.707 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-11T22:00:34.834 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 6/6 2024-03-11T22:00:34.834 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 1/6 2024-03-11T22:00:34.834 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2/6 2024-03-11T22:00:34.834 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T22:00:34.834 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-11T22:00:34.835 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-11T22:00:34.902 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-11T22:00:34.947 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-11T22:00:34.986 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-11T22:00:35.021 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-11T22:00:35.051 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-11T22:00:35.132 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-11T22:00:35.173 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T22:00:35.195 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-11T22:00:35.236 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-11T22:00:35.247 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-11T22:00:35.247 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:35.247 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:00:35.247 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:35.247 INFO:teuthology.orchestra.run.smithi144.stdout: jq-1.6-16.el9.x86_64 2024-03-11T22:00:35.248 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:35.248 INFO:teuthology.orchestra.run.smithi144.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T22:00:35.248 INFO:teuthology.orchestra.run.smithi144.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T22:00:35.248 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T22:00:35.248 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:35.248 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:00:35.279 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-11T22:00:35.313 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-11T22:00:35.406 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-11T22:00:35.444 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-11T22:00:35.467 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph 2024-03-11T22:00:35.520 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-11T22:00:35.669 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-11T22:00:35.801 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T22:00:35.969 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9. 31/36 2024-03-11T22:00:35.999 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 32/36 2024-03-11T22:00:36.024 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 32/36 2024-03-11T22:00:36.025 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:00:36.025 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T22:00:36.025 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T22:00:36.025 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T22:00:36.025 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:36.026 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:11 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:00:36.191 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:00:36.194 INFO:teuthology.orchestra.run.smithi144.stdout:========================================================================================= 2024-03-11T22:00:36.194 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-03-11T22:00:36.194 INFO:teuthology.orchestra.run.smithi144.stdout:========================================================================================= 2024-03-11T22:00:36.194 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:00:36.194 INFO:teuthology.orchestra.run.smithi144.stdout: ceph x86_64 2:19.0.0-1607.g94a53970.el9 ceph 6.6 k 2024-03-11T22:00:36.194 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-03-11T22:00:36.194 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds x86_64 2:19.0.0-1607.g94a53970.el9 ceph 2.1 M 2024-03-11T22:00:36.194 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr x86_64 2:19.0.0-1607.g94a53970.el9 ceph 1.5 M 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 247 k 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon x86_64 2:19.0.0-1607.g94a53970.el9 ceph 4.7 M 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 17 M 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T22:00:36.195 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T22:00:36.196 INFO:teuthology.orchestra.run.smithi144.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T22:00:36.197 INFO:teuthology.orchestra.run.smithi144.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T22:00:36.197 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T22:00:36.197 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T22:00:36.197 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T22:00:36.197 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:36.197 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:00:36.197 INFO:teuthology.orchestra.run.smithi144.stdout:========================================================================================= 2024-03-11T22:00:36.197 INFO:teuthology.orchestra.run.smithi144.stdout:Install 36 Packages 2024-03-11T22:00:36.197 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:36.198 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 30 M 2024-03-11T22:00:36.198 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 107 M 2024-03-11T22:00:36.198 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:00:37.116 INFO:teuthology.orchestra.run.smithi144.stdout:(1/36): ceph-19.0.0-1607.g94a53970.el9.x86_64.r 36 kB/s | 6.6 kB 00:00 2024-03-11T22:00:37.567 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 33/36 2024-03-11T22:00:37.593 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 33/36 2024-03-11T22:00:37.593 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:00:37.593 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T22:00:37.593 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T22:00:37.593 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T22:00:37.593 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:37.791 INFO:teuthology.orchestra.run.smithi144.stdout:(2/36): ceph-mon-19.0.0-1607.g94a53970.el9.x86_ 6.9 MB/s | 4.7 MB 00:00 2024-03-11T22:00:38.059 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 34/36 2024-03-11T22:00:38.082 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 34/36 2024-03-11T22:00:38.082 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:00:38.082 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T22:00:38.082 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T22:00:38.083 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T22:00:38.083 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:38.225 INFO:teuthology.orchestra.run.smithi144.stdout:(3/36): ceph-mgr-19.0.0-1607.g94a53970.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2024-03-11T22:00:38.310 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 35/36 2024-03-11T22:00:38.333 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 35/36 2024-03-11T22:00:38.333 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:00:38.333 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T22:00:38.333 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T22:00:38.333 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T22:00:38.333 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:38.400 INFO:teuthology.orchestra.run.smithi144.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1607.g94a5 1.4 MB/s | 247 kB 00:00 2024-03-11T22:00:38.402 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 36/36 2024-03-11T22:00:38.442 INFO:teuthology.orchestra.run.smithi144.stdout:(5/36): ceph-mds-19.0.0-1607.g94a53970.el9.x86_ 1.4 MB/s | 2.1 MB 00:01 2024-03-11T22:00:38.709 INFO:teuthology.orchestra.run.smithi144.stdout:(6/36): ceph-osd-19.0.0-1607.g94a53970.el9.x86_ 18 MB/s | 17 MB 00:00 2024-03-11T22:00:38.734 INFO:teuthology.orchestra.run.smithi144.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 560 kB/s | 187 kB 00:00 2024-03-11T22:00:38.759 INFO:teuthology.orchestra.run.smithi144.stdout:(8/36): python3-mako-1.1.4-6.el9.noarch.rpm 455 kB/s | 144 kB 00:00 2024-03-11T22:00:38.776 INFO:teuthology.orchestra.run.smithi144.stdout:(9/36): python3-markupsafe-1.1.1-12.el9.x86_64. 475 kB/s | 32 kB 00:00 2024-03-11T22:00:38.793 INFO:teuthology.orchestra.run.smithi144.stdout:(10/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-11T22:00:38.818 INFO:teuthology.orchestra.run.smithi144.stdout:(11/36): python3-toml-0.10.2-6.el9.noarch.rpm 660 kB/s | 39 kB 00:00 2024-03-11T22:00:38.843 INFO:teuthology.orchestra.run.smithi144.stdout:(12/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-11T22:00:38.877 INFO:teuthology.orchestra.run.smithi144.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-11T22:00:38.902 INFO:teuthology.orchestra.run.smithi144.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-11T22:00:38.919 INFO:teuthology.orchestra.run.smithi144.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 1.6 MB/s | 124 kB 00:00 2024-03-11T22:00:38.944 INFO:teuthology.orchestra.run.smithi144.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-11T22:00:39.020 INFO:teuthology.orchestra.run.smithi144.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-11T22:00:39.095 INFO:teuthology.orchestra.run.smithi144.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 650 kB/s | 125 kB 00:00 2024-03-11T22:00:39.304 INFO:teuthology.orchestra.run.smithi144.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 421 kB/s | 151 kB 00:00 2024-03-11T22:00:39.345 INFO:teuthology.orchestra.run.smithi144.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 133 kB/s | 43 kB 00:00 2024-03-11T22:00:39.413 INFO:teuthology.orchestra.run.smithi144.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-11T22:00:39.479 INFO:teuthology.orchestra.run.smithi144.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 447 kB/s | 172 kB 00:00 2024-03-11T22:00:39.505 INFO:teuthology.orchestra.run.smithi144.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-11T22:00:39.555 INFO:teuthology.orchestra.run.smithi144.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-03-11T22:00:39.580 INFO:teuthology.orchestra.run.smithi144.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-03-11T22:00:39.606 INFO:teuthology.orchestra.run.smithi144.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2024-03-11T22:00:39.631 INFO:teuthology.orchestra.run.smithi144.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-03-11T22:00:39.656 INFO:teuthology.orchestra.run.smithi144.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-03-11T22:00:39.714 INFO:teuthology.orchestra.run.smithi144.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 725 kB/s | 79 kB 00:00 2024-03-11T22:00:39.740 INFO:teuthology.orchestra.run.smithi144.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-11T22:00:39.807 INFO:teuthology.orchestra.run.smithi144.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 537 kB/s | 36 kB 00:00 2024-03-11T22:00:39.841 INFO:teuthology.orchestra.run.smithi144.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-11T22:00:39.866 INFO:teuthology.orchestra.run.smithi144.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 595 kB/s | 90 kB 00:00 2024-03-11T22:00:39.916 INFO:teuthology.orchestra.run.smithi144.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 401 kB/s | 20 kB 00:00 2024-03-11T22:00:39.975 INFO:teuthology.orchestra.run.smithi144.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-03-11T22:00:40.056 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 36/36 2024-03-11T22:00:40.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/36 2024-03-11T22:00:40.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2/36 2024-03-11T22:00:40.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 3/36 2024-03-11T22:00:40.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 4/36 2024-03-11T22:00:40.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 5/36 2024-03-11T22:00:40.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9. 6/36 2024-03-11T22:00:40.056 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-11T22:00:40.057 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T22:00:40.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-11T22:00:40.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-11T22:00:40.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-11T22:00:40.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-11T22:00:40.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-11T22:00:40.058 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-11T22:00:40.059 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-11T22:00:40.059 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-11T22:00:40.059 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T22:00:40.059 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-11T22:00:40.059 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-11T22:00:40.059 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-11T22:00:40.059 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-11T22:00:40.059 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-11T22:00:40.116 INFO:teuthology.orchestra.run.smithi144.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.7 MB/s | 427 kB 00:00 2024-03-11T22:00:40.124 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:00:40.124 INFO:teuthology.orchestra.run.smithi144.stdout:Total 7.6 MB/s | 30 MB 00:03 2024-03-11T22:00:40.148 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:00:40.192 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:00:40.192 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:00:40.619 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:00:40.620 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:00:40.889 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-11T22:00:40.890 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:40.890 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:00:40.890 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:40.890 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:40.890 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:40.890 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:00:40.890 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:40.890 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:40.890 INFO:teuthology.orchestra.run.smithi045.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T22:00:40.890 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T22:00:40.890 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T22:00:40.890 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T22:00:40.891 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T22:00:40.892 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T22:00:40.892 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T22:00:40.892 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T22:00:40.892 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T22:00:40.892 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T22:00:40.892 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T22:00:40.892 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T22:00:40.892 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T22:00:40.892 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T22:00:40.892 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T22:00:40.892 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:40.892 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:00:41.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-base 2024-03-11T22:00:41.845 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:07 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:00:41.946 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:00:41.948 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:00:42.002 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:00:42.003 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-11T22:00:42.003 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:00:42.019 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-11T22:00:42.067 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-11T22:00:42.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install cephadm 2024-03-11T22:00:42.097 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-11T22:00:42.167 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-11T22:00:42.212 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-11T22:00:42.245 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-11T22:00:42.336 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-11T22:00:42.414 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-11T22:00:42.464 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-11T22:00:42.535 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-11T22:00:42.590 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-11T22:00:42.635 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:08 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:00:42.636 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-11T22:00:42.689 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-11T22:00:42.749 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-11T22:00:42.791 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:00:42.792 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:00:42.792 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-11T22:00:42.792 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:00:42.792 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:00:42.792 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-11T22:00:42.792 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:42.792 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:00:42.793 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:00:42.793 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-11T22:00:42.793 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:42.793 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 748 k 2024-03-11T22:00:42.793 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 754 k 2024-03-11T22:00:42.793 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:00:42.939 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-11T22:00:42.991 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-11T22:00:43.021 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-11T22:00:43.055 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-11T22:00:43.086 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-11T22:00:43.134 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm-19.0.0-1607.g94a53970.el9.noarch.rpm 2.1 MB/s | 748 kB 00:00 2024-03-11T22:00:43.135 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:00:43.136 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.1 MB/s | 748 kB 00:00 2024-03-11T22:00:43.136 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:00:43.141 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:00:43.141 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:00:43.147 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:00:43.147 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:00:43.161 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-11T22:00:43.196 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T22:00:43.223 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:00:43.229 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-11T22:00:43.270 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-11T22:00:43.305 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-11T22:00:43.339 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-11T22:00:43.428 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-11T22:00:43.479 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-11T22:00:43.550 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-11T22:00:43.686 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-11T22:00:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 1/1 2024-03-11T22:00:43.792 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 1/1 2024-03-11T22:00:43.804 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T22:00:43.970 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9. 31/36 2024-03-11T22:00:44.000 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 32/36 2024-03-11T22:00:44.024 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 32/36 2024-03-11T22:00:44.025 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:00:44.025 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T22:00:44.025 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T22:00:44.025 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T22:00:44.025 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:44.181 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 1/1 2024-03-11T22:00:44.515 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 1/1 2024-03-11T22:00:44.515 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:44.515 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:00:44.516 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:00:44.516 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:44.516 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:00:44.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-immutable-object-cache 2024-03-11T22:00:45.286 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:11 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:00:45.440 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:00:45.441 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:00:45.441 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-03-11T22:00:45.441 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:00:45.441 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:00:45.441 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-11T22:00:45.441 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:45.441 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:00:45.441 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:00:45.441 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-11T22:00:45.442 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:45.442 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 145 k 2024-03-11T22:00:45.442 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 436 k 2024-03-11T22:00:45.442 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:00:45.459 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 33/36 2024-03-11T22:00:45.486 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 33/36 2024-03-11T22:00:45.486 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:00:45.486 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T22:00:45.486 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T22:00:45.486 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T22:00:45.486 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:45.801 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-immutable-object-cache-19.0.0-1607.g94a539 406 kB/s | 145 kB 00:00 2024-03-11T22:00:45.801 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:00:45.801 INFO:teuthology.orchestra.run.smithi045.stdout:Total 404 kB/s | 145 kB 00:00 2024-03-11T22:00:45.802 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:00:45.809 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:00:45.810 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:00:45.864 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:00:45.864 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:00:45.935 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 34/36 2024-03-11T22:00:45.959 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 34/36 2024-03-11T22:00:45.959 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:00:45.959 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T22:00:45.959 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T22:00:45.959 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T22:00:45.959 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:46.011 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:00:46.068 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53970. 1/1 2024-03-11T22:00:46.092 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970. 1/1 2024-03-11T22:00:46.092 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:00:46.092 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T22:00:46.092 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:46.169 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 35/36 2024-03-11T22:00:46.192 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 35/36 2024-03-11T22:00:46.192 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:00:46.192 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T22:00:46.193 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T22:00:46.193 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T22:00:46.193 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:46.258 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 36/36 2024-03-11T22:00:47.111 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53970. 1/1 2024-03-11T22:00:47.111 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:47.111 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:00:47.111 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:47.111 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:47.111 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:00:47.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr 2024-03-11T22:00:47.914 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:13 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:00:48.013 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:00:48.045 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 36/36 2024-03-11T22:00:48.045 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-2:19.0.0-1607.g94a53970.el9.x86_64 1/36 2024-03-11T22:00:48.045 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2/36 2024-03-11T22:00:48.045 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 3/36 2024-03-11T22:00:48.045 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 4/36 2024-03-11T22:00:48.045 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 5/36 2024-03-11T22:00:48.045 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9. 6/36 2024-03-11T22:00:48.045 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-11T22:00:48.045 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-11T22:00:48.045 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-11T22:00:48.045 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-11T22:00:48.046 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-11T22:00:48.047 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-11T22:00:48.047 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-11T22:00:48.047 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-11T22:00:48.047 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-11T22:00:48.048 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-11T22:00:48.048 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T22:00:48.048 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-11T22:00:48.048 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-11T22:00:48.048 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-11T22:00:48.048 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-11T22:00:48.048 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-11T22:00:48.069 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:00:48.071 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-11T22:00:48.071 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:00:48.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-dashboard 2024-03-11T22:00:48.714 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:14 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:00:48.871 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:00:48.872 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================== 2024-03-11T22:00:48.872 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-11T22:00:48.872 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================== 2024-03-11T22:00:48.872 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:00:48.872 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-11T22:00:48.872 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-11T22:00:48.873 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T22:00:48.873 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-11T22:00:48.873 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-11T22:00:48.873 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T22:00:48.873 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T22:00:48.873 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T22:00:48.873 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T22:00:48.873 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T22:00:48.873 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T22:00:48.873 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T22:00:48.873 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T22:00:48.873 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:48.874 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:00:48.874 INFO:teuthology.orchestra.run.smithi045.stdout:====================================================================================== 2024-03-11T22:00:48.874 INFO:teuthology.orchestra.run.smithi045.stdout:Install 12 Packages 2024-03-11T22:00:48.874 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:48.875 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 6.9 M 2024-03-11T22:00:48.875 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 26 M 2024-03-11T22:00:48.875 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:00:49.058 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-11T22:00:49.058 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:49.058 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:00:49.058 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:49.058 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:49.058 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:49.058 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:00:49.058 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:49.058 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:49.059 INFO:teuthology.orchestra.run.smithi144.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T22:00:49.059 INFO:teuthology.orchestra.run.smithi144.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T22:00:49.059 INFO:teuthology.orchestra.run.smithi144.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T22:00:49.059 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T22:00:49.059 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T22:00:49.059 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T22:00:49.059 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T22:00:49.059 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T22:00:49.059 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T22:00:49.059 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T22:00:49.059 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T22:00:49.059 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T22:00:49.059 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T22:00:49.060 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T22:00:49.061 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T22:00:49.061 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:49.061 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:00:49.412 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-base 2024-03-11T22:00:49.740 INFO:teuthology.orchestra.run.smithi045.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1607.g94 166 kB/s | 31 kB 00:00 2024-03-11T22:00:49.756 INFO:teuthology.orchestra.run.smithi045.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1607.g94a 74 kB/s | 15 kB 00:00 2024-03-11T22:00:49.924 INFO:teuthology.orchestra.run.smithi045.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 238 kB 00:00 2024-03-11T22:00:49.957 INFO:teuthology.orchestra.run.smithi045.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-11T22:00:49.987 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:24 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:00:50.085 INFO:teuthology.orchestra.run.smithi144.stdout:Package ceph-base-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:00:50.099 INFO:teuthology.orchestra.run.smithi045.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.8 MB/s | 548 kB 00:00 2024-03-11T22:00:50.137 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:00:50.138 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-03-11T22:00:50.138 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:00:50.141 INFO:teuthology.orchestra.run.smithi045.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 464 kB/s | 19 kB 00:00 2024-03-11T22:00:50.204 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install cephadm 2024-03-11T22:00:50.258 INFO:teuthology.orchestra.run.smithi045.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 17 MB/s | 2.0 MB 00:00 2024-03-11T22:00:50.308 INFO:teuthology.orchestra.run.smithi045.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-03-11T22:00:50.359 INFO:teuthology.orchestra.run.smithi045.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 614 kB/s | 31 kB 00:00 2024-03-11T22:00:50.417 INFO:teuthology.orchestra.run.smithi045.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-03-11T22:00:50.459 INFO:teuthology.orchestra.run.smithi045.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.6 MB/s | 863 kB 00:00 2024-03-11T22:00:50.685 INFO:teuthology.orchestra.run.smithi045.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1607.g94a539 1.6 MB/s | 1.8 MB 00:01 2024-03-11T22:00:50.688 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:00:50.689 INFO:teuthology.orchestra.run.smithi045.stdout:Total 3.8 MB/s | 6.9 MB 00:01 2024-03-11T22:00:50.707 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:00:50.725 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:00:50.725 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:00:50.772 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:25 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:00:50.906 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:00:50.906 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:00:50.919 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:00:50.920 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:00:50.920 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-03-11T22:00:50.920 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:00:50.920 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:00:50.920 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 748 k 2024-03-11T22:00:50.920 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:50.920 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:00:50.920 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:00:50.920 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-03-11T22:00:50.920 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:50.920 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 748 k 2024-03-11T22:00:50.921 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 754 k 2024-03-11T22:00:50.921 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:00:51.279 INFO:teuthology.orchestra.run.smithi144.stdout:cephadm-19.0.0-1607.g94a53970.el9.noarch.rpm 2.0 MB/s | 748 kB 00:00 2024-03-11T22:00:51.280 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:00:51.280 INFO:teuthology.orchestra.run.smithi144.stdout:Total 2.0 MB/s | 748 kB 00:00 2024-03-11T22:00:51.280 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:00:51.285 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:00:51.285 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:00:51.291 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:00:51.291 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:00:51.369 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:00:51.420 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:00:51.494 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-11T22:00:51.538 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-11T22:00:51.585 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-11T22:00:51.648 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-11T22:00:51.722 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 1/1 2024-03-11T22:00:51.761 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-11T22:00:51.819 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 1/1 2024-03-11T22:00:51.903 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-11T22:00:51.996 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-11T22:00:52.045 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-11T22:00:52.081 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-11T22:00:52.113 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9 10/12 2024-03-11T22:00:52.269 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 1/1 2024-03-11T22:00:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el 11/12 2024-03-11T22:00:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noa 12/12 2024-03-11T22:00:52.604 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephadm-2:19.0.0-1607.g94a53970.el9.noarch 1/1 2024-03-11T22:00:52.605 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:52.605 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:00:52.605 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:00:52.605 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:52.605 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:00:52.796 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-immutable-object-cache 2024-03-11T22:00:53.229 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noa 12/12 2024-03-11T22:00:53.230 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el 1/12 2024-03-11T22:00:53.230 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noa 2/12 2024-03-11T22:00:53.230 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9 3/12 2024-03-11T22:00:53.230 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-11T22:00:53.230 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-11T22:00:53.230 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-11T22:00:53.230 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-11T22:00:53.230 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-11T22:00:53.230 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-11T22:00:53.230 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-11T22:00:53.230 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-11T22:00:53.368 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:28 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:00:53.521 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:00:53.521 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:00:53.522 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-03-11T22:00:53.522 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:00:53.522 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:00:53.522 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1607.g94a53970.el9 ceph 145 k 2024-03-11T22:00:53.522 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:53.522 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:00:53.522 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:00:53.522 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-03-11T22:00:53.522 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:53.522 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 145 k 2024-03-11T22:00:53.522 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 436 k 2024-03-11T22:00:53.522 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:00:53.594 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-11T22:00:53.594 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:53.594 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:00:53.594 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T22:00:53.594 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:00:53.594 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:00:53.594 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:00:53.595 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T22:00:53.595 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T22:00:53.595 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T22:00:53.595 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T22:00:53.595 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T22:00:53.595 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T22:00:53.595 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T22:00:53.595 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T22:00:53.595 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:53.595 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:00:53.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-11T22:00:54.111 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-immutable-object-cache-19.0.0-1607.g94a539 247 kB/s | 145 kB 00:00 2024-03-11T22:00:54.112 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:00:54.112 INFO:teuthology.orchestra.run.smithi144.stdout:Total 246 kB/s | 145 kB 00:00 2024-03-11T22:00:54.112 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:00:54.120 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:00:54.120 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:00:54.172 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:00:54.173 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:00:54.317 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:00:54.379 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1607.g94a53970. 1/1 2024-03-11T22:00:54.397 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:20 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:00:54.404 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970. 1/1 2024-03-11T22:00:54.404 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:00:54.404 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T22:00:54.404 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:54.555 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:00:54.556 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-11T22:00:54.556 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-11T22:00:54.556 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-11T22:00:54.556 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:00:54.556 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-11T22:00:54.556 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-11T22:00:54.556 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T22:00:54.556 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T22:00:54.556 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T22:00:54.556 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T22:00:54.556 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T22:00:54.557 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T22:00:54.557 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T22:00:54.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T22:00:54.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T22:00:54.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T22:00:54.557 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T22:00:54.557 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:54.557 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:00:54.557 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-03-11T22:00:54.557 INFO:teuthology.orchestra.run.smithi045.stdout:Install 12 Packages 2024-03-11T22:00:54.557 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:00:54.558 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 37 M 2024-03-11T22:00:54.558 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 209 M 2024-03-11T22:00:54.558 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:00:54.793 INFO:teuthology.orchestra.run.smithi045.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 208 kB/s | 32 kB 00:00 2024-03-11T22:00:54.834 INFO:teuthology.orchestra.run.smithi045.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 412 kB/s | 17 kB 00:00 2024-03-11T22:00:54.876 INFO:teuthology.orchestra.run.smithi045.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 826 kB/s | 34 kB 00:00 2024-03-11T22:00:55.052 INFO:teuthology.orchestra.run.smithi045.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-11T22:00:55.152 INFO:teuthology.orchestra.run.smithi045.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 48 MB/s | 4.8 MB 00:00 2024-03-11T22:00:55.194 INFO:teuthology.orchestra.run.smithi045.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-11T22:00:55.269 INFO:teuthology.orchestra.run.smithi045.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.8 MB/s | 3.0 MB 00:00 2024-03-11T22:00:55.442 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1607.g94a53970. 1/1 2024-03-11T22:00:55.442 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:55.442 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:00:55.442 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:00:55.442 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:55.442 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:00:55.503 INFO:teuthology.orchestra.run.smithi045.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 51 MB/s | 16 MB 00:00 2024-03-11T22:00:55.595 INFO:teuthology.orchestra.run.smithi045.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-03-11T22:00:55.628 INFO:teuthology.orchestra.run.smithi045.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 195 kB 00:00 2024-03-11T22:00:55.666 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr 2024-03-11T22:00:55.746 INFO:teuthology.orchestra.run.smithi045.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-11T22:00:56.242 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:31 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:00:56.338 INFO:teuthology.orchestra.run.smithi144.stdout:Package ceph-mgr-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:00:56.389 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:00:56.390 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-03-11T22:00:56.390 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:00:56.465 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr-dashboard 2024-03-11T22:00:57.020 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:32 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:00:57.104 INFO:teuthology.orchestra.run.smithi045.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.0 MB/s | 7.4 MB 00:02 2024-03-11T22:00:57.106 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:00:57.107 INFO:teuthology.orchestra.run.smithi045.stdout:Total 15 MB/s | 37 MB 00:02 2024-03-11T22:00:57.150 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:00:57.167 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:00:57.169 INFO:teuthology.orchestra.run.smithi144.stdout:====================================================================================== 2024-03-11T22:00:57.169 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-03-11T22:00:57.169 INFO:teuthology.orchestra.run.smithi144.stdout:====================================================================================== 2024-03-11T22:00:57.169 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:00:57.169 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 1.8 M 2024-03-11T22:00:57.169 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-03-11T22:00:57.169 INFO:teuthology.orchestra.run.smithi144.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T22:00:57.169 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 31 k 2024-03-11T22:00:57.169 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 15 k 2024-03-11T22:00:57.169 INFO:teuthology.orchestra.run.smithi144.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T22:00:57.170 INFO:teuthology.orchestra.run.smithi144.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T22:00:57.170 INFO:teuthology.orchestra.run.smithi144.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T22:00:57.170 INFO:teuthology.orchestra.run.smithi144.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T22:00:57.170 INFO:teuthology.orchestra.run.smithi144.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T22:00:57.170 INFO:teuthology.orchestra.run.smithi144.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T22:00:57.170 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T22:00:57.170 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T22:00:57.170 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:57.170 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:00:57.170 INFO:teuthology.orchestra.run.smithi144.stdout:====================================================================================== 2024-03-11T22:00:57.170 INFO:teuthology.orchestra.run.smithi144.stdout:Install 12 Packages 2024-03-11T22:00:57.170 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:00:57.171 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 6.9 M 2024-03-11T22:00:57.171 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 26 M 2024-03-11T22:00:57.171 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:00:57.176 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:00:57.177 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:00:57.555 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:00:57.556 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:00:57.890 INFO:teuthology.orchestra.run.smithi144.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1607.g94 102 kB/s | 31 kB 00:00 2024-03-11T22:00:57.940 INFO:teuthology.orchestra.run.smithi144.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1607.g94a 43 kB/s | 15 kB 00:00 2024-03-11T22:00:58.099 INFO:teuthology.orchestra.run.smithi144.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-11T22:00:58.149 INFO:teuthology.orchestra.run.smithi144.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.9 MB/s | 1.0 MB 00:00 2024-03-11T22:00:58.292 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:00:58.369 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-11T22:00:58.400 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-11T22:00:58.458 INFO:teuthology.orchestra.run.smithi144.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1607.g94a5397 2.0 MB/s | 1.8 MB 00:00 2024-03-11T22:00:58.497 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-11T22:00:58.529 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-11T22:00:58.592 INFO:teuthology.orchestra.run.smithi144.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.7 MB/s | 863 kB 00:00 2024-03-11T22:00:58.712 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T22:00:58.717 INFO:teuthology.orchestra.run.smithi144.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-11T22:00:58.744 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T22:00:58.776 INFO:teuthology.orchestra.run.smithi144.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 875 kB/s | 548 kB 00:00 2024-03-11T22:00:58.827 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-11T22:00:58.893 INFO:teuthology.orchestra.run.smithi144.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 263 kB/s | 31 kB 00:00 2024-03-11T22:00:58.993 INFO:teuthology.orchestra.run.smithi144.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 524 kB/s | 144 kB 00:00 2024-03-11T22:00:59.018 INFO:teuthology.orchestra.run.smithi144.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-11T22:00:59.525 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-11T22:00:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-11T22:01:00.135 INFO:teuthology.orchestra.run.smithi144.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 1.3 MB/s | 2.0 MB 00:01 2024-03-11T22:01:00.138 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:00.138 INFO:teuthology.orchestra.run.smithi144.stdout:Total 2.3 MB/s | 6.9 MB 00:02 2024-03-11T22:01:00.158 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:01:00.176 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:01:00.176 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:01:00.363 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:01:00.363 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:01:00.884 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:01:00.960 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-11T22:01:00.977 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-11T22:01:01.012 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-11T22:01:01.058 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-11T22:01:01.089 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-11T22:01:01.182 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-11T22:01:01.305 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-11T22:01:01.391 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-11T22:01:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-11T22:01:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-11T22:01:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9 10/12 2024-03-11T22:01:01.744 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el 11/12 2024-03-11T22:01:01.771 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noa 12/12 2024-03-11T22:01:02.040 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-11T22:01:02.062 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53 12/12 2024-03-11T22:01:02.570 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noa 12/12 2024-03-11T22:01:02.570 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el 1/12 2024-03-11T22:01:02.570 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noa 2/12 2024-03-11T22:01:02.570 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9 3/12 2024-03-11T22:01:02.570 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-11T22:01:02.570 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-11T22:01:02.570 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-11T22:01:02.571 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-11T22:01:02.571 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-11T22:01:02.571 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-11T22:01:02.571 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-11T22:01:02.571 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-11T22:01:02.942 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-11T22:01:02.942 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T22:01:02.943 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:02.944 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:03.193 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-11T22:01:03.752 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:38 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:03.902 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================================ 2024-03-11T22:01:03.903 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-03-11T22:01:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================================ 2024-03-11T22:01:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:01:03.903 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 7.4 M 2024-03-11T22:01:03.903 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-03-11T22:01:03.903 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T22:01:03.903 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T22:01:03.903 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T22:01:03.903 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T22:01:03.903 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T22:01:03.903 INFO:teuthology.orchestra.run.smithi144.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T22:01:03.904 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T22:01:03.904 INFO:teuthology.orchestra.run.smithi144.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T22:01:03.904 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T22:01:03.904 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T22:01:03.904 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T22:01:03.904 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:01:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================================ 2024-03-11T22:01:03.904 INFO:teuthology.orchestra.run.smithi144.stdout:Install 12 Packages 2024-03-11T22:01:03.904 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 37 M 2024-03-11T22:01:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 209 M 2024-03-11T22:01:03.905 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:01:04.575 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53 12/12 2024-03-11T22:01:04.575 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53 1/12 2024-03-11T22:01:04.575 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-11T22:01:04.575 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-11T22:01:04.575 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-11T22:01:04.575 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T22:01:04.576 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T22:01:04.576 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-11T22:01:04.576 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-11T22:01:04.576 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-11T22:01:04.576 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-11T22:01:04.576 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-11T22:01:04.742 INFO:teuthology.orchestra.run.smithi144.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 182 kB/s | 32 kB 00:00 2024-03-11T22:01:04.784 INFO:teuthology.orchestra.run.smithi144.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 415 kB/s | 17 kB 00:00 2024-03-11T22:01:04.826 INFO:teuthology.orchestra.run.smithi144.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 836 kB/s | 34 kB 00:00 2024-03-11T22:01:04.876 INFO:teuthology.orchestra.run.smithi144.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.7 MB/s | 3.0 MB 00:00 2024-03-11T22:01:05.005 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-11T22:01:05.005 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:05.005 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:01:05.005 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:01:05.005 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T22:01:05.005 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T22:01:05.005 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T22:01:05.005 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T22:01:05.005 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T22:01:05.005 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T22:01:05.006 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T22:01:05.006 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T22:01:05.006 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T22:01:05.006 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T22:01:05.006 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T22:01:05.006 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:05.006 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:05.035 INFO:teuthology.orchestra.run.smithi144.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-11T22:01:05.101 INFO:teuthology.orchestra.run.smithi144.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-11T22:01:05.127 INFO:teuthology.orchestra.run.smithi144.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-11T22:01:05.344 INFO:teuthology.orchestra.run.smithi144.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 65 MB/s | 16 MB 00:00 2024-03-11T22:01:05.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-rook 2024-03-11T22:01:05.486 INFO:teuthology.orchestra.run.smithi144.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-16 8.1 MB/s | 7.4 MB 00:00 2024-03-11T22:01:05.678 INFO:teuthology.orchestra.run.smithi144.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 586 kB/s | 195 kB 00:00 2024-03-11T22:01:05.753 INFO:teuthology.orchestra.run.smithi144.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 920 kB/s | 245 kB 00:00 2024-03-11T22:01:05.786 INFO:teuthology.orchestra.run.smithi144.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-03-11T22:01:05.788 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:05.788 INFO:teuthology.orchestra.run.smithi144.stdout:Total 20 MB/s | 37 MB 00:01 2024-03-11T22:01:05.828 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:01:05.851 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:01:05.851 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:01:05.994 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:32 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:06.152 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:06.153 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-11T22:01:06.153 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-11T22:01:06.153 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-11T22:01:06.153 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:01:06.154 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-11T22:01:06.154 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-11T22:01:06.154 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T22:01:06.154 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T22:01:06.154 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T22:01:06.154 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T22:01:06.154 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T22:01:06.154 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T22:01:06.154 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T22:01:06.154 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T22:01:06.154 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T22:01:06.154 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T22:01:06.155 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T22:01:06.155 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T22:01:06.155 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:06.155 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:01:06.155 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-11T22:01:06.155 INFO:teuthology.orchestra.run.smithi045.stdout:Install 13 Packages 2024-03-11T22:01:06.155 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:06.155 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 2.0 M 2024-03-11T22:01:06.156 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 26 M 2024-03-11T22:01:06.156 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:01:06.222 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:01:06.222 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:01:06.700 INFO:teuthology.orchestra.run.smithi045.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 162 kB/s | 24 kB 00:00 2024-03-11T22:01:06.725 INFO:teuthology.orchestra.run.smithi045.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 102 kB/s | 18 kB 00:00 2024-03-11T22:01:06.784 INFO:teuthology.orchestra.run.smithi045.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-11T22:01:06.809 INFO:teuthology.orchestra.run.smithi045.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-11T22:01:06.834 INFO:teuthology.orchestra.run.smithi045.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-11T22:01:06.852 INFO:teuthology.orchestra.run.smithi045.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-11T22:01:06.868 INFO:teuthology.orchestra.run.smithi045.stdout:(7/13): ceph-mgr-rook-19.0.0-1607.g94a53970.el9 154 kB/s | 49 kB 00:00 2024-03-11T22:01:06.932 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:01:07.009 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-11T22:01:07.040 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-11T22:01:07.135 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-11T22:01:07.177 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-11T22:01:07.261 INFO:teuthology.orchestra.run.smithi045.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 37 kB/s | 15 kB 00:00 2024-03-11T22:01:07.277 INFO:teuthology.orchestra.run.smithi045.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 73 kB/s | 32 kB 00:00 2024-03-11T22:01:07.354 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T22:01:07.384 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T22:01:07.461 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-11T22:01:07.469 INFO:teuthology.orchestra.run.smithi045.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-03-11T22:01:07.528 INFO:teuthology.orchestra.run.smithi045.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 307 kB/s | 203 kB 00:00 2024-03-11T22:01:07.628 INFO:teuthology.orchestra.run.smithi045.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 565 kB/s | 90 kB 00:00 2024-03-11T22:01:07.862 INFO:teuthology.orchestra.run.smithi045.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-03-11T22:01:07.866 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:07.867 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-03-11T22:01:07.867 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:01:07.883 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:01:07.883 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:01:07.993 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:01:07.995 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:01:08.128 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-11T22:01:08.238 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-11T22:01:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:01:08.703 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-11T22:01:08.756 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-11T22:01:08.812 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-11T22:01:08.848 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-11T22:01:08.890 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-11T22:01:08.987 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-11T22:01:09.106 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-11T22:01:09.159 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-11T22:01:09.485 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-11T22:01:09.921 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-11T22:01:09.972 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-11T22:01:10.006 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-11T22:01:10.076 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-11T22:01:10.100 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 13/13 2024-03-11T22:01:10.510 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-11T22:01:10.542 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53 12/12 2024-03-11T22:01:10.911 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 13/13 2024-03-11T22:01:10.911 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 1/13 2024-03-11T22:01:10.911 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-11T22:01:10.911 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-11T22:01:10.911 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-11T22:01:10.911 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-11T22:01:10.911 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-11T22:01:10.911 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-11T22:01:10.911 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-11T22:01:10.911 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-11T22:01:10.912 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-11T22:01:10.912 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-11T22:01:10.912 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-11T22:01:11.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-11T22:01:11.300 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:11.300 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:01:11.300 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:11.301 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:11.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-cephadm 2024-03-11T22:01:12.121 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:38 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:12.277 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:12.278 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-11T22:01:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-11T22:01:12.278 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-11T22:01:12.278 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:01:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-11T22:01:12.278 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-11T22:01:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T22:01:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T22:01:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T22:01:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T22:01:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T22:01:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:12.279 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:01:12.279 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2024-03-11T22:01:12.279 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2024-03-11T22:01:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:12.280 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 6.8 M 2024-03-11T22:01:12.280 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 33 M 2024-03-11T22:01:12.280 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:01:12.882 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-11T22:01:12.965 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1607.g94a53970.e 541 kB/s | 138 kB 00:00 2024-03-11T22:01:12.983 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53 12/12 2024-03-11T22:01:12.983 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53 1/12 2024-03-11T22:01:12.983 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-11T22:01:12.984 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-11T22:01:12.984 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-11T22:01:12.984 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T22:01:12.984 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T22:01:12.984 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-11T22:01:12.984 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-11T22:01:12.984 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-11T22:01:12.984 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-11T22:01:12.984 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-11T22:01:13.074 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-03-11T22:01:13.224 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 225 kB/s | 58 kB 00:00 2024-03-11T22:01:13.250 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 291 kB/s | 51 kB 00:00 2024-03-11T22:01:13.274 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2024-03-11T22:01:13.276 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:13.276 INFO:teuthology.orchestra.run.smithi045.stdout:Total 6.8 MB/s | 6.8 MB 00:00 2024-03-11T22:01:13.289 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:01:13.300 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:01:13.300 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:01:13.394 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:01:13.395 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:01:13.424 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-11T22:01:13.424 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:13.424 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:01:13.424 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:01:13.424 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T22:01:13.424 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T22:01:13.424 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T22:01:13.424 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T22:01:13.424 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T22:01:13.424 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T22:01:13.424 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T22:01:13.425 INFO:teuthology.orchestra.run.smithi144.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T22:01:13.425 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T22:01:13.425 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T22:01:13.425 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T22:01:13.425 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:13.425 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:13.709 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:01:13.793 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr-rook 2024-03-11T22:01:13.875 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-11T22:01:13.920 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-11T22:01:14.362 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:49 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:14.479 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-11T22:01:14.514 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:14.515 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================== 2024-03-11T22:01:14.515 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-03-11T22:01:14.515 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================== 2024-03-11T22:01:14.515 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:01:14.515 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 49 k 2024-03-11T22:01:14.516 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-03-11T22:01:14.516 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T22:01:14.516 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T22:01:14.516 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T22:01:14.516 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T22:01:14.516 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T22:01:14.516 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T22:01:14.516 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T22:01:14.516 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T22:01:14.516 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T22:01:14.516 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T22:01:14.516 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T22:01:14.517 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T22:01:14.517 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:14.517 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:01:14.517 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================== 2024-03-11T22:01:14.517 INFO:teuthology.orchestra.run.smithi144.stdout:Install 13 Packages 2024-03-11T22:01:14.517 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:14.518 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 2.0 M 2024-03-11T22:01:14.518 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 26 M 2024-03-11T22:01:14.518 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:01:14.565 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-11T22:01:14.643 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-11T22:01:14.673 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 6/6 2024-03-11T22:01:15.009 INFO:teuthology.orchestra.run.smithi144.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 135 kB/s | 18 kB 00:00 2024-03-11T22:01:15.034 INFO:teuthology.orchestra.run.smithi144.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 154 kB/s | 24 kB 00:00 2024-03-11T22:01:15.093 INFO:teuthology.orchestra.run.smithi144.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-11T22:01:15.118 INFO:teuthology.orchestra.run.smithi144.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-11T22:01:15.144 INFO:teuthology.orchestra.run.smithi144.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-11T22:01:15.169 INFO:teuthology.orchestra.run.smithi144.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-11T22:01:15.194 INFO:teuthology.orchestra.run.smithi144.stdout:(7/13): ceph-mgr-rook-19.0.0-1607.g94a53970.el9 155 kB/s | 49 kB 00:00 2024-03-11T22:01:15.471 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 6/6 2024-03-11T22:01:15.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 1/6 2024-03-11T22:01:15.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-11T22:01:15.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-11T22:01:15.471 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-11T22:01:15.472 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-11T22:01:15.561 INFO:teuthology.orchestra.run.smithi144.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 38 kB/s | 15 kB 00:00 2024-03-11T22:01:15.587 INFO:teuthology.orchestra.run.smithi144.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 73 kB/s | 32 kB 00:00 2024-03-11T22:01:15.792 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-11T22:01:15.792 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:15.792 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:01:15.792 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:01:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T22:01:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T22:01:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T22:01:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T22:01:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T22:01:15.793 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:15.793 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:15.795 INFO:teuthology.orchestra.run.smithi144.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 284 kB/s | 59 kB 00:00 2024-03-11T22:01:15.837 INFO:teuthology.orchestra.run.smithi144.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 315 kB/s | 203 kB 00:00 2024-03-11T22:01:15.962 INFO:teuthology.orchestra.run.smithi144.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 537 kB/s | 90 kB 00:00 2024-03-11T22:01:16.026 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-fuse 2024-03-11T22:01:16.129 INFO:teuthology.orchestra.run.smithi144.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-03-11T22:01:16.133 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:16.133 INFO:teuthology.orchestra.run.smithi144.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2024-03-11T22:01:16.134 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:01:16.147 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:01:16.147 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:01:16.247 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:01:16.247 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:01:16.597 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:42 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:16.754 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:16.755 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-11T22:01:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:01:16.755 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-11T22:01:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-11T22:01:16.755 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T22:01:16.755 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:01:16.755 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:Install 2 Packages 2024-03-11T22:01:16.756 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 964 k 2024-03-11T22:01:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 2.8 M 2024-03-11T22:01:16.756 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:01:16.774 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:01:16.902 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 540 kB/s | 79 kB 00:00 2024-03-11T22:01:16.934 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-11T22:01:16.988 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-11T22:01:17.044 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-11T22:01:17.081 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-11T22:01:17.123 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-11T22:01:17.216 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-11T22:01:17.325 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-11T22:01:17.374 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-11T22:01:17.452 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): ceph-fuse-19.0.0-1607.g94a53970.el9.x86_ 1.2 MB/s | 886 kB 00:00 2024-03-11T22:01:17.454 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:17.454 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.3 MB/s | 964 kB 00:00 2024-03-11T22:01:17.454 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:01:17.477 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:01:17.477 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:01:17.558 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:01:17.558 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:01:17.747 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:01:17.891 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-11T22:01:17.963 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2/2 2024-03-11T22:01:18.090 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-11T22:01:18.138 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-11T22:01:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-11T22:01:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-11T22:01:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 13/13 2024-03-11T22:01:18.669 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2/2 2024-03-11T22:01:18.669 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 1/2 2024-03-11T22:01:18.998 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-11T22:01:18.999 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:18.999 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:01:18.999 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-11T22:01:18.999 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:18.999 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:19.022 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 13/13 2024-03-11T22:01:19.022 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 1/13 2024-03-11T22:01:19.022 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-11T22:01:19.022 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-11T22:01:19.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-11T22:01:19.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-11T22:01:19.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-11T22:01:19.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-11T22:01:19.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-11T22:01:19.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-11T22:01:19.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-11T22:01:19.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-11T22:01:19.024 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-11T22:01:19.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-volume 2024-03-11T22:01:19.390 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-11T22:01:19.390 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:19.390 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:01:19.390 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:01:19.390 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T22:01:19.391 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T22:01:19.391 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T22:01:19.391 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T22:01:19.391 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T22:01:19.391 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T22:01:19.391 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T22:01:19.391 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T22:01:19.391 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T22:01:19.391 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T22:01:19.391 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T22:01:19.391 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T22:01:19.391 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:19.392 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:19.631 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr-cephadm 2024-03-11T22:01:19.889 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:45 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:20.046 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:20.047 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:20.047 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-03-11T22:01:20.047 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:20.047 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:01:20.047 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-11T22:01:20.047 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-03-11T22:01:20.047 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T22:01:20.047 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:20.047 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:01:20.048 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:20.048 INFO:teuthology.orchestra.run.smithi045.stdout:Install 2 Packages 2024-03-11T22:01:20.048 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:20.048 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 332 k 2024-03-11T22:01:20.048 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 1.5 M 2024-03-11T22:01:20.048 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:01:20.200 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 453 kB/s | 69 kB 00:00 2024-03-11T22:01:20.201 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:55 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:20.350 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:20.352 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================== 2024-03-11T22:01:20.352 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-03-11T22:01:20.352 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================== 2024-03-11T22:01:20.352 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:01:20.352 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 138 k 2024-03-11T22:01:20.352 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-03-11T22:01:20.352 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T22:01:20.353 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T22:01:20.353 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T22:01:20.353 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T22:01:20.353 INFO:teuthology.orchestra.run.smithi144.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T22:01:20.353 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:20.353 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:01:20.353 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================== 2024-03-11T22:01:20.353 INFO:teuthology.orchestra.run.smithi144.stdout:Install 6 Packages 2024-03-11T22:01:20.353 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:20.353 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 6.8 M 2024-03-11T22:01:20.353 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 33 M 2024-03-11T22:01:20.354 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:01:20.459 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): ceph-volume-19.0.0-1607.g94a53970.el9.no 643 kB/s | 264 kB 00:00 2024-03-11T22:01:20.460 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:20.460 INFO:teuthology.orchestra.run.smithi045.stdout:Total 806 kB/s | 332 kB 00:00 2024-03-11T22:01:20.460 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:01:20.470 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:01:20.471 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:01:20.515 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:01:20.515 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:01:20.667 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:01:20.849 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-11T22:01:20.878 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2/2 2024-03-11T22:01:20.894 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2/2 2024-03-11T22:01:20.894 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:01:20.894 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T22:01:20.894 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:21.208 INFO:teuthology.orchestra.run.smithi144.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-11T22:01:21.233 INFO:teuthology.orchestra.run.smithi144.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1607.g94a53970.e 251 kB/s | 138 kB 00:00 2024-03-11T22:01:21.342 INFO:teuthology.orchestra.run.smithi144.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 377 kB/s | 248 kB 00:00 2024-03-11T22:01:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 1/2 2024-03-11T22:01:21.817 INFO:teuthology.orchestra.run.smithi144.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 99 kB/s | 58 kB 00:00 2024-03-11T22:01:21.868 INFO:teuthology.orchestra.run.smithi144.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 107 kB/s | 51 kB 00:00 2024-03-11T22:01:21.981 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-11T22:01:21.981 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:21.982 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:01:21.982 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:01:21.982 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T22:01:21.982 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:21.982 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:22.034 INFO:teuthology.orchestra.run.smithi144.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 663 kB/s | 548 kB 00:00 2024-03-11T22:01:22.039 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:22.039 INFO:teuthology.orchestra.run.smithi144.stdout:Total 4.0 MB/s | 6.8 MB 00:01 2024-03-11T22:01:22.052 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:01:22.063 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:01:22.063 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:01:22.157 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:01:22.158 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:01:22.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados-devel 2024-03-11T22:01:22.466 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:01:22.633 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-11T22:01:22.686 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-11T22:01:22.883 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:48 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:23.039 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:23.040 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:23.040 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-03-11T22:01:23.040 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:23.040 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:01:23.040 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-11T22:01:23.041 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:23.041 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:01:23.041 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:23.041 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-11T22:01:23.041 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:23.041 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 127 k 2024-03-11T22:01:23.041 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 456 k 2024-03-11T22:01:23.041 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:01:23.207 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-11T22:01:23.288 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-11T22:01:23.349 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-11T22:01:23.365 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 6/6 2024-03-11T22:01:23.424 INFO:teuthology.orchestra.run.smithi045.stdout:librados-devel-19.0.0-1607.g94a53970.el9.x86_64 331 kB/s | 127 kB 00:00 2024-03-11T22:01:23.424 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:23.424 INFO:teuthology.orchestra.run.smithi045.stdout:Total 330 kB/s | 127 kB 00:00 2024-03-11T22:01:23.424 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:01:23.430 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:01:23.430 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:01:23.488 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:01:23.489 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:01:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:01:23.784 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:24.173 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 6/6 2024-03-11T22:01:24.173 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 1/6 2024-03-11T22:01:24.173 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-11T22:01:24.173 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-11T22:01:24.173 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-11T22:01:24.174 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-11T22:01:24.274 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:24.487 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-11T22:01:24.487 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:24.487 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:01:24.488 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:01:24.488 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T22:01:24.488 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T22:01:24.488 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T22:01:24.488 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T22:01:24.488 INFO:teuthology.orchestra.run.smithi144.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T22:01:24.488 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:24.488 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:24.574 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:24.574 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:24.574 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:01:24.574 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:01:24.574 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:24.574 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:24.768 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-fuse 2024-03-11T22:01:24.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs2 2024-03-11T22:01:25.368 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:00 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:25.417 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:51 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:25.517 INFO:teuthology.orchestra.run.smithi045.stdout:Package libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:25.526 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:25.526 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:25.526 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-03-11T22:01:25.526 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:25.526 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:01:25.526 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 886 k 2024-03-11T22:01:25.526 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-03-11T22:01:25.527 INFO:teuthology.orchestra.run.smithi144.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T22:01:25.527 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:25.527 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:01:25.527 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:25.527 INFO:teuthology.orchestra.run.smithi144.stdout:Install 2 Packages 2024-03-11T22:01:25.527 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:25.527 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 964 k 2024-03-11T22:01:25.528 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 2.8 M 2024-03-11T22:01:25.528 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:01:25.572 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:25.573 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-11T22:01:25.573 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:25.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs-devel 2024-03-11T22:01:25.670 INFO:teuthology.orchestra.run.smithi144.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 554 kB/s | 79 kB 00:00 2024-03-11T22:01:25.962 INFO:teuthology.orchestra.run.smithi144.stdout:(2/2): ceph-fuse-19.0.0-1607.g94a53970.el9.x86_ 2.0 MB/s | 886 kB 00:00 2024-03-11T22:01:25.964 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:25.964 INFO:teuthology.orchestra.run.smithi144.stdout:Total 2.2 MB/s | 964 kB 00:00 2024-03-11T22:01:25.964 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:01:25.984 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:01:25.984 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:01:26.060 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:01:26.060 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:01:26.203 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:52 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:26.239 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:01:26.357 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:26.358 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:26.358 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-03-11T22:01:26.358 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:26.358 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:01:26.358 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-11T22:01:26.358 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:26.358 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:01:26.358 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:26.358 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-11T22:01:26.359 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:26.359 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 31 k 2024-03-11T22:01:26.359 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 140 k 2024-03-11T22:01:26.359 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:01:26.371 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-11T22:01:26.437 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2/2 2024-03-11T22:01:26.619 INFO:teuthology.orchestra.run.smithi045.stdout:libcephfs-devel-19.0.0-1607.g94a53970.el9.x86_6 119 kB/s | 31 kB 00:00 2024-03-11T22:01:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:Total 119 kB/s | 31 kB 00:00 2024-03-11T22:01:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:01:26.624 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:01:26.624 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:01:26.652 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:01:26.652 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:01:26.742 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:01:26.855 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:27.170 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2/2 2024-03-11T22:01:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 1/2 2024-03-11T22:01:27.245 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:27.497 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-11T22:01:27.497 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:27.497 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:01:27.497 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-11T22:01:27.497 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:27.497 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:27.538 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:27.538 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:27.538 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:01:27.538 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:01:27.538 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:27.538 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:27.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados2 2024-03-11T22:01:27.777 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-volume 2024-03-11T22:01:28.273 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:54 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:28.358 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:03 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:28.373 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:28.427 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:28.428 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-11T22:01:28.428 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:28.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librbd1 2024-03-11T22:01:28.511 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:28.511 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:28.511 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-03-11T22:01:28.511 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:28.511 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:01:28.512 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume noarch 2:19.0.0-1607.g94a53970.el9 ceph-noarch 264 k 2024-03-11T22:01:28.512 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-03-11T22:01:28.512 INFO:teuthology.orchestra.run.smithi144.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T22:01:28.512 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:28.512 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:01:28.512 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:28.512 INFO:teuthology.orchestra.run.smithi144.stdout:Install 2 Packages 2024-03-11T22:01:28.512 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:28.513 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 332 k 2024-03-11T22:01:28.513 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 1.5 M 2024-03-11T22:01:28.513 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:01:28.642 INFO:teuthology.orchestra.run.smithi144.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 534 kB/s | 69 kB 00:00 2024-03-11T22:01:28.775 INFO:teuthology.orchestra.run.smithi144.stdout:(2/2): ceph-volume-19.0.0-1607.g94a53970.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-11T22:01:28.776 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:28.777 INFO:teuthology.orchestra.run.smithi144.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-11T22:01:28.777 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:01:28.785 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:01:28.786 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:01:28.830 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:01:28.830 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:01:28.991 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:01:29.069 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:55 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:29.171 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:29.181 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-11T22:01:29.226 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:29.227 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-11T22:01:29.227 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:29.244 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2/2 2024-03-11T22:01:29.260 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2/2 2024-03-11T22:01:29.260 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:01:29.260 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T22:01:29.261 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:29.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rados 2024-03-11T22:01:29.871 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:55 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:29.970 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:30.015 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 1/2 2024-03-11T22:01:30.025 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:30.026 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-11T22:01:30.027 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:30.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rgw 2024-03-11T22:01:30.361 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-11T22:01:30.362 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:30.362 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:01:30.362 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume-2:19.0.0-1607.g94a53970.el9.noarch 2024-03-11T22:01:30.362 INFO:teuthology.orchestra.run.smithi144.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T22:01:30.362 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:30.362 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:30.664 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:56 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:30.699 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install librados-devel 2024-03-11T22:01:30.764 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:30.818 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:30.820 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-11T22:01:30.820 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:30.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-cephfs 2024-03-11T22:01:31.273 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:06 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:31.423 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:31.476 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:57 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:31.477 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:31.477 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-03-11T22:01:31.478 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:31.478 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:01:31.478 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 127 k 2024-03-11T22:01:31.478 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:31.479 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:01:31.479 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:31.479 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-03-11T22:01:31.479 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:31.479 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 127 k 2024-03-11T22:01:31.480 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 456 k 2024-03-11T22:01:31.480 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:01:31.573 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:31.628 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:31.630 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-11T22:01:31.630 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:31.640 INFO:teuthology.orchestra.run.smithi144.stdout:librados-devel-19.0.0-1607.g94a53970.el9.x86_64 588 kB/s | 127 kB 00:00 2024-03-11T22:01:31.642 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:31.642 INFO:teuthology.orchestra.run.smithi144.stdout:Total 583 kB/s | 127 kB 00:00 2024-03-11T22:01:31.642 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:01:31.647 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:01:31.647 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:01:31.703 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:01:31.703 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:01:31.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rbd 2024-03-11T22:01:31.893 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:01:31.996 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:32.283 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:58 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:32.384 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:32.440 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:32.441 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-03-11T22:01:32.441 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:32.466 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:32.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-fuse 2024-03-11T22:01:32.752 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:32.752 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:32.753 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:01:32.753 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:01:32.753 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:32.753 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:33.015 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install libcephfs2 2024-03-11T22:01:33.089 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:59 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:33.245 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:33.246 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:33.246 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-03-11T22:01:33.246 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:33.246 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:01:33.246 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-11T22:01:33.247 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:33.247 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:01:33.247 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:33.247 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-11T22:01:33.247 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:33.247 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 85 k 2024-03-11T22:01:33.247 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 231 k 2024-03-11T22:01:33.247 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:01:33.592 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:08 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:33.649 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-fuse-19.0.0-1607.g94a53970.el9.x86_64.rpm 212 kB/s | 85 kB 00:00 2024-03-11T22:01:33.650 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:33.650 INFO:teuthology.orchestra.run.smithi045.stdout:Total 211 kB/s | 85 kB 00:00 2024-03-11T22:01:33.650 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:01:33.658 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:01:33.658 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:01:33.693 INFO:teuthology.orchestra.run.smithi144.stdout:Package libcephfs2-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:33.716 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:01:33.716 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:01:33.746 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:33.747 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-03-11T22:01:33.747 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:33.799 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install libcephfs-devel 2024-03-11T22:01:33.866 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:01:33.958 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:34.353 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:34.367 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:09 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:34.517 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:34.517 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:34.517 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-03-11T22:01:34.518 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:34.518 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:01:34.518 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel x86_64 2:19.0.0-1607.g94a53970.el9 ceph 31 k 2024-03-11T22:01:34.518 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:34.518 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:01:34.518 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:34.518 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-03-11T22:01:34.518 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:34.518 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 31 k 2024-03-11T22:01:34.518 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 140 k 2024-03-11T22:01:34.518 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:01:34.662 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:34.662 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:34.662 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:01:34.662 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:01:34.662 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:34.662 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:34.760 INFO:teuthology.orchestra.run.smithi144.stdout:libcephfs-devel-19.0.0-1607.g94a53970.el9.x86_6 129 kB/s | 31 kB 00:00 2024-03-11T22:01:34.761 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:34.761 INFO:teuthology.orchestra.run.smithi144.stdout:Total 128 kB/s | 31 kB 00:00 2024-03-11T22:01:34.761 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:01:34.764 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:01:34.765 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:01:34.792 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:01:34.792 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:01:34.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-mirror 2024-03-11T22:01:34.894 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:01:35.001 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:35.407 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:35.464 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:01 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:35.619 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:35.620 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:35.620 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-03-11T22:01:35.620 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:35.620 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:01:35.620 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T22:01:35.620 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:35.620 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:01:35.620 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:35.620 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-11T22:01:35.620 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:35.620 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 3.1 M 2024-03-11T22:01:35.621 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 13 M 2024-03-11T22:01:35.621 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:01:35.703 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:35.703 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:35.703 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:01:35.703 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:01:35.703 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:35.703 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:35.861 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install librados2 2024-03-11T22:01:36.177 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-mirror-19.0.0-1607.g94a53970.el9.x86_64.rpm 5.6 MB/s | 3.1 MB 00:00 2024-03-11T22:01:36.178 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:36.178 INFO:teuthology.orchestra.run.smithi045.stdout:Total 5.5 MB/s | 3.1 MB 00:00 2024-03-11T22:01:36.179 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:01:36.189 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:01:36.189 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:01:36.266 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:01:36.266 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:01:36.431 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:11 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:36.527 INFO:teuthology.orchestra.run.smithi144.stdout:Package librados2-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:36.578 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:36.579 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-03-11T22:01:36.579 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:36.640 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install librbd1 2024-03-11T22:01:36.702 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:01:36.761 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:36.785 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:36.785 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:01:36.785 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T22:01:36.785 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T22:01:36.785 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T22:01:36.785 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:37.202 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:12 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:37.300 INFO:teuthology.orchestra.run.smithi144.stdout:Package librbd1-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:37.352 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:37.353 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-03-11T22:01:37.353 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:37.415 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-rados 2024-03-11T22:01:37.961 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:37.962 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:37.962 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:01:37.962 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:01:37.962 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:37.962 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:37.982 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:12 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:38.079 INFO:teuthology.orchestra.run.smithi144.stdout:Package python3-rados-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:38.130 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:38.131 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-03-11T22:01:38.131 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:38.193 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-rgw 2024-03-11T22:01:38.214 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-nbd 2024-03-11T22:01:38.749 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:13 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:38.787 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:04 ago on Mon 11 Mar 2024 09:59:34 PM UTC. 2024-03-11T22:01:38.846 INFO:teuthology.orchestra.run.smithi144.stdout:Package python3-rgw-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:38.897 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:38.898 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-03-11T22:01:38.898 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:38.942 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-03-11T22:01:38.942 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:38.943 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-03-11T22:01:38.943 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:38.943 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-03-11T22:01:38.943 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-11T22:01:38.943 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:38.943 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-03-11T22:01:38.943 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-03-11T22:01:38.943 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-03-11T22:01:38.943 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:38.943 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 171 k 2024-03-11T22:01:38.943 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 494 k 2024-03-11T22:01:38.943 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-03-11T22:01:38.956 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-cephfs 2024-03-11T22:01:39.517 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:14 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:39.615 INFO:teuthology.orchestra.run.smithi144.stdout:Package python3-cephfs-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:39.666 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:39.667 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-03-11T22:01:39.667 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:39.723 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-rbd 2024-03-11T22:01:39.749 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-nbd-19.0.0-1607.g94a53970.el9.x86_64.rpm 212 kB/s | 171 kB 00:00 2024-03-11T22:01:39.749 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:39.750 INFO:teuthology.orchestra.run.smithi045.stdout:Total 212 kB/s | 171 kB 00:00 2024-03-11T22:01:39.750 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-03-11T22:01:39.759 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-03-11T22:01:39.759 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-03-11T22:01:39.820 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-03-11T22:01:39.820 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-03-11T22:01:39.972 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-03-11T22:01:40.083 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:40.288 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:15 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:40.385 INFO:teuthology.orchestra.run.smithi144.stdout:Package python3-rbd-2:19.0.0-1607.g94a53970.el9.x86_64 is already installed. 2024-03-11T22:01:40.437 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:40.438 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-03-11T22:01:40.438 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:40.507 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install rbd-fuse 2024-03-11T22:01:40.535 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:40.830 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:40.830 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:40.830 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-03-11T22:01:40.830 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:01:40.830 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:01:40.830 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-03-11T22:01:41.040 DEBUG:teuthology.parallel:result is None 2024-03-11T22:01:41.070 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:16 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:41.218 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:41.219 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:41.219 INFO:teuthology.orchestra.run.smithi144.stdout: Package Architecture Version Repository Size 2024-03-11T22:01:41.219 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:41.219 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:01:41.219 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse x86_64 2:19.0.0-1607.g94a53970.el9 ceph 85 k 2024-03-11T22:01:41.219 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:41.219 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:01:41.220 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:41.220 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-03-11T22:01:41.220 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:41.220 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 85 k 2024-03-11T22:01:41.220 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 231 k 2024-03-11T22:01:41.220 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:01:41.671 INFO:teuthology.orchestra.run.smithi144.stdout:rbd-fuse-19.0.0-1607.g94a53970.el9.x86_64.rpm 189 kB/s | 85 kB 00:00 2024-03-11T22:01:41.672 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:41.672 INFO:teuthology.orchestra.run.smithi144.stdout:Total 188 kB/s | 85 kB 00:00 2024-03-11T22:01:41.672 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:01:41.680 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:01:41.680 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:01:41.737 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:01:41.737 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:01:41.880 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:01:41.978 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:42.367 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:42.651 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:42.652 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:42.652 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:01:42.652 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:01:42.652 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:42.652 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:42.849 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install rbd-mirror 2024-03-11T22:01:43.426 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:18 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:43.579 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:43.580 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:43.580 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-03-11T22:01:43.580 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:43.580 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:01:43.580 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror x86_64 2:19.0.0-1607.g94a53970.el9 ceph 3.1 M 2024-03-11T22:01:43.580 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:43.580 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:01:43.580 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:43.580 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-03-11T22:01:43.580 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:43.580 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 3.1 M 2024-03-11T22:01:43.581 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 13 M 2024-03-11T22:01:43.581 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:01:46.316 INFO:teuthology.orchestra.run.smithi144.stdout:rbd-mirror-19.0.0-1607.g94a53970.el9.x86_64.rpm 1.1 MB/s | 3.1 MB 00:02 2024-03-11T22:01:46.317 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:46.317 INFO:teuthology.orchestra.run.smithi144.stdout:Total 1.1 MB/s | 3.1 MB 00:02 2024-03-11T22:01:46.318 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:01:46.328 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:01:46.328 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:01:46.404 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:01:46.405 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:01:46.832 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:01:46.892 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:46.916 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:46.916 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:01:46.916 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T22:01:46.916 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T22:01:46.916 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T22:01:46.916 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:48.103 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:48.103 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:48.103 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:01:48.103 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:01:48.103 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:48.103 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:48.370 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install rbd-nbd 2024-03-11T22:01:48.944 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:23 ago on Mon 11 Mar 2024 09:59:25 PM UTC. 2024-03-11T22:01:49.093 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-03-11T22:01:49.094 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:49.094 INFO:teuthology.orchestra.run.smithi144.stdout: Package Architecture Version Repository Size 2024-03-11T22:01:49.094 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:49.094 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-03-11T22:01:49.094 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd x86_64 2:19.0.0-1607.g94a53970.el9 ceph 171 k 2024-03-11T22:01:49.094 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:49.094 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-03-11T22:01:49.094 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-03-11T22:01:49.094 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-03-11T22:01:49.094 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:49.095 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 171 k 2024-03-11T22:01:49.095 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 494 k 2024-03-11T22:01:49.095 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-03-11T22:01:49.478 INFO:teuthology.orchestra.run.smithi144.stdout:rbd-nbd-19.0.0-1607.g94a53970.el9.x86_64.rpm 446 kB/s | 171 kB 00:00 2024-03-11T22:01:49.478 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:01:49.478 INFO:teuthology.orchestra.run.smithi144.stdout:Total 445 kB/s | 171 kB 00:00 2024-03-11T22:01:49.479 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-03-11T22:01:49.486 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-03-11T22:01:49.486 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-03-11T22:01:49.541 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-03-11T22:01:49.541 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-03-11T22:01:49.689 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-03-11T22:01:49.785 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:50.246 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:50.567 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 1/1 2024-03-11T22:01:50.567 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:50.567 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-03-11T22:01:50.567 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd-2:19.0.0-1607.g94a53970.el9.x86_64 2024-03-11T22:01:50.567 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:01:50.567 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-03-11T22:01:50.756 DEBUG:teuthology.parallel:result is None 2024-03-11T22:01:50.756 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:01:50.977 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T22:01:51.002 INFO:teuthology.orchestra.run.smithi045.stdout:19.0.0-1607.g94a53970.el9 2024-03-11T22:01:51.003 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-11T22:01:51.003 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-11T22:01:51.005 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:01:51.226 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T22:01:51.247 INFO:teuthology.orchestra.run.smithi144.stdout:19.0.0-1607.g94a53970.el9 2024-03-11T22:01:51.247 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1607.g94a53970.el9 2024-03-11T22:01:51.248 INFO:teuthology.task.install:The correct ceph version 19.0.0-1607.g94a53970 is installed. 2024-03-11T22:01:51.249 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-11T22:01:51.249 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:01:51.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T22:01:51.282 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-03-11T22:01:51.282 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T22:01:51.316 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-11T22:01:51.316 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:01:51.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T22:01:51.351 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T22:01:51.421 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-03-11T22:01:51.421 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T22:01:51.448 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T22:01:51.515 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-11T22:01:51.515 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:01:51.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T22:01:51.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T22:01:51.612 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-03-11T22:01:51.612 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T22:01:51.640 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T22:01:51.706 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-11T22:01:51.707 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:01:51.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T22:01:51.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T22:01:51.803 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-03-11T22:01:51.803 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T22:01:51.830 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T22:01:51.897 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T22:01:52.005 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'cephadm_mode': 'root'} 2024-03-11T22:01:52.005 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:01:52.006 INFO:tasks.cephadm:Cluster fsid is f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:01:52.006 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T22:01:52.006 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.45', 'mon.c': '[v2:172.21.15.45:3301,v1:172.21.15.45:6790]', 'mon.b': '172.21.15.144'} 2024-03-11T22:01:52.006 INFO:tasks.cephadm:First mon is mon.a on smithi045 2024-03-11T22:01:52.006 INFO:tasks.cephadm:First mgr is y 2024-03-11T22:01:52.006 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T22:01:52.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo hostname $(hostname -s) 2024-03-11T22:01:52.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo hostname $(hostname -s) 2024-03-11T22:01:52.063 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T22:01:52.063 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:01:52.203 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77431/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:08:19.449750', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77429/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:06:59.696700', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-11T22:01:52.346 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 94a539707733325322964c6a187c6a1b41be1daf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:01:52.347 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm 2024-03-11T22:01:52.348 DEBUG:teuthology.orchestra.run.smithi045:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:01:52.928 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 22:01 /home/ubuntu/cephtest/cephadm 2024-03-11T22:01:52.929 DEBUG:teuthology.orchestra.run.smithi144:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:01:53.369 INFO:teuthology.orchestra.run.smithi144.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 22:01 /home/ubuntu/cephtest/cephadm 2024-03-11T22:01:53.370 DEBUG:teuthology.orchestra.run.smithi045:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:01:53.390 DEBUG:teuthology.orchestra.run.smithi144:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:01:53.425 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on all hosts... 2024-03-11T22:01:53.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T22:01:53.432 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T22:01:54.251 INFO:teuthology.orchestra.run.smithi045.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:01:54.423 INFO:teuthology.orchestra.run.smithi144.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:02:41.135 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:02:41.136 INFO:teuthology.orchestra.run.smithi144.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T22:02:41.136 INFO:teuthology.orchestra.run.smithi144.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T22:02:41.136 INFO:teuthology.orchestra.run.smithi144.stdout: "repo_digests": [ 2024-03-11T22:02:41.136 INFO:teuthology.orchestra.run.smithi144.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T22:02:41.136 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:02:41.136 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:02:41.321 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:02:41.321 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T22:02:41.321 INFO:teuthology.orchestra.run.smithi045.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T22:02:41.321 INFO:teuthology.orchestra.run.smithi045.stdout: "repo_digests": [ 2024-03-11T22:02:41.321 INFO:teuthology.orchestra.run.smithi045.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T22:02:41.321 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:02:41.321 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:02:41.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph 2024-03-11T22:02:41.389 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /etc/ceph 2024-03-11T22:02:41.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /etc/ceph 2024-03-11T22:02:41.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 777 /etc/ceph 2024-03-11T22:02:41.487 INFO:tasks.cephadm:Writing seed config... 2024-03-11T22:02:41.488 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T22:02:41.488 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T22:02:41.488 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-11T22:02:41.488 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-11T22:02:41.488 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-11T22:02:41.489 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T22:02:41.490 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-11T22:02:41.490 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:02:41.490 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T22:02:41.510 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = f75aafa4-dff2-11ee-95c8-87774f69a715 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-11T22:02:41.511 DEBUG:teuthology.orchestra.run.smithi045:mon.a> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T22:02:41.553 DEBUG:teuthology.orchestra.run.smithi045:mgr.y> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T22:02:41.595 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T22:02:41.596 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf -v bootstrap --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.45 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:02:41.792 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:02:41.792 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf', '-v', 'bootstrap', '--fsid', 'f75aafa4-dff2-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.45', '--skip-admin-label'] 2024-03-11T22:02:41.817 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:02:41.817 INFO:teuthology.orchestra.run.smithi045.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T22:02:41.818 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying podman|docker is present... 2024-03-11T22:02:41.839 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:02:41.839 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying lvm2 is present... 2024-03-11T22:02:41.839 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying time synchronization is in place... 2024-03-11T22:02:41.847 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T22:02:41.847 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T22:02:41.852 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T22:02:41.853 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-11T22:02:41.859 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout enabled 2024-03-11T22:02:41.865 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout active 2024-03-11T22:02:41.865 INFO:teuthology.orchestra.run.smithi045.stdout:Unit chronyd.service is enabled and running 2024-03-11T22:02:41.865 INFO:teuthology.orchestra.run.smithi045.stdout:Repeating the final host check... 2024-03-11T22:02:41.889 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:02:41.889 INFO:teuthology.orchestra.run.smithi045.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T22:02:41.889 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl is present 2024-03-11T22:02:41.889 INFO:teuthology.orchestra.run.smithi045.stdout:lvcreate is present 2024-03-11T22:02:41.895 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T22:02:41.895 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T22:02:41.901 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T22:02:41.901 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-11T22:02:41.907 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout enabled 2024-03-11T22:02:41.913 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout active 2024-03-11T22:02:41.913 INFO:teuthology.orchestra.run.smithi045.stdout:Unit chronyd.service is enabled and running 2024-03-11T22:02:41.913 INFO:teuthology.orchestra.run.smithi045.stdout:Host looks OK 2024-03-11T22:02:41.913 INFO:teuthology.orchestra.run.smithi045.stdout:Cluster fsid: f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:02:41.913 INFO:teuthology.orchestra.run.smithi045.stdout:Acquiring lock 140484372959184 on /run/cephadm/f75aafa4-dff2-11ee-95c8-87774f69a715.lock 2024-03-11T22:02:41.913 INFO:teuthology.orchestra.run.smithi045.stdout:Lock 140484372959184 acquired on /run/cephadm/f75aafa4-dff2-11ee-95c8-87774f69a715.lock 2024-03-11T22:02:41.914 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying IP 172.21.15.45 port 3300 ... 2024-03-11T22:02:41.914 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying IP 172.21.15.45 port 6789 ... 2024-03-11T22:02:41.914 INFO:teuthology.orchestra.run.smithi045.stdout:Base mon IP(s) is [172.21.15.45:3300, 172.21.15.45:6789], mon addrv is [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2024-03-11T22:02:41.917 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.45 metric 100 2024-03-11T22:02:41.917 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.45 metric 100 2024-03-11T22:02:41.920 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T22:02:41.920 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-11T22:02:41.920 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-11T22:02:41.922 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T22:02:41.922 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T22:02:41.922 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T22:02:41.922 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-11T22:02:41.922 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2024-03-11T22:02:41.922 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T22:02:41.923 INFO:teuthology.orchestra.run.smithi045.stdout:Mon IP `172.21.15.45` is in CIDR network `172.21.0.0/20` 2024-03-11T22:02:41.923 INFO:teuthology.orchestra.run.smithi045.stdout:Mon IP `172.21.15.45` is in CIDR network `172.21.0.0/20` 2024-03-11T22:02:41.923 INFO:teuthology.orchestra.run.smithi045.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T22:02:41.924 INFO:teuthology.orchestra.run.smithi045.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T22:02:41.924 INFO:teuthology.orchestra.run.smithi045.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:02:42.440 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T22:02:42.440 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:02:42.440 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T22:02:42.440 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying blob sha256:0b3c7bee4e66dab4b5376aaf0baf8ba5fc1c3e66c27dc601157782d617bd9dc6 2024-03-11T22:02:42.440 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T22:02:42.440 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying config sha256:a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T22:02:42.440 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T22:02:43.495 INFO:teuthology.orchestra.run.smithi045.stdout:ceph: stdout ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T22:02:43.495 INFO:teuthology.orchestra.run.smithi045.stdout:Ceph version: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T22:02:43.495 INFO:teuthology.orchestra.run.smithi045.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T22:02:44.500 INFO:teuthology.orchestra.run.smithi045.stdout:stat: stdout 167 167 2024-03-11T22:02:44.501 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial keys... 2024-03-11T22:02:45.564 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQCEf+9lcfdbOhAAAeCp+JsaHnz7jNLEEV6spQ== 2024-03-11T22:02:46.661 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQCFf+9l2ykdOhAA76elEWvfRVdMpxzzNuHgIg== 2024-03-11T22:02:47.642 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQCHf+9l0zI9AxAA6u59R5Xmysb2okWHDMeeCw== 2024-03-11T22:02:47.643 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial monmap... 2024-03-11T22:02:48.531 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T22:02:48.531 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T22:02:48.531 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:02:48.531 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T22:02:48.531 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool for a [v2:172.21.15.45:3300,v1:172.21.15.45:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T22:02:48.531 INFO:teuthology.orchestra.run.smithi045.stdout:setting min_mon_release = quincy 2024-03-11T22:02:48.531 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: set fsid to f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:02:48.531 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T22:02:48.531 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:02:48.531 INFO:teuthology.orchestra.run.smithi045.stdout:Creating mon... 2024-03-11T22:02:49.629 INFO:teuthology.orchestra.run.smithi045.stdout:create mon.a on 2024-03-11T22:02:49.839 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-11T22:02:49.977 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T22:02:50.145 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-f75aafa4-dff2-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-f75aafa4-dff2-11ee-95c8-87774f69a715.target. 2024-03-11T22:02:50.146 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-f75aafa4-dff2-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-f75aafa4-dff2-11ee-95c8-87774f69a715.target. 2024-03-11T22:02:50.375 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.a 2024-03-11T22:02:50.376 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T22:02:50.538 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f75aafa4-dff2-11ee-95c8-87774f69a715.target.wants/ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@.service. 2024-03-11T22:02:51.078 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:02:51.079 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-03-11T22:02:51.085 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:02:51.085 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-11T22:02:51.085 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-03-11T22:02:51.085 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T22:02:51.086 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mon to start... 2024-03-11T22:02:51.086 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mon... 2024-03-11T22:02:52.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:52 smithi045 ceph-mon[29691]: from='client.? 172.21.15.45:0/2011018254' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T22:02:52.230 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T22:02:52.232 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout id: f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:02:52.232 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T22:02:52.232 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-11T22:02:52.232 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout services: 2024-03-11T22:02:52.232 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.573904s) 2024-03-11T22:02:52.232 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T22:02:52.232 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T22:02:52.232 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-11T22:02:52.232 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout data: 2024-03-11T22:02:52.232 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T22:02:52.233 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T22:02:52.233 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T22:02:52.233 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T22:02:52.233 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-11T22:02:52.233 INFO:teuthology.orchestra.run.smithi045.stdout:mon is available 2024-03-11T22:02:52.233 INFO:teuthology.orchestra.run.smithi045.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T22:02:53.225 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:53 smithi045 ceph-mon[29691]: from='client.? 172.21.15.45:0/2113172311' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T22:02:53.225 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:53 smithi045 ceph-mon[29691]: from='client.? 172.21.15.45:0/2113172311' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T22:02:53.229 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-11T22:02:53.229 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T22:02:53.229 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout fsid = f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:02:53.229 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2024-03-11T22:02:53.229 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T22:02:53.229 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T22:02:53.229 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T22:02:53.229 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T22:02:53.230 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-11T22:02:53.230 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T22:02:53.230 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T22:02:53.230 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-11T22:02:53.230 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T22:02:53.230 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T22:02:53.230 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T22:02:53.230 INFO:teuthology.orchestra.run.smithi045.stdout:Generating new minimal ceph.conf... 2024-03-11T22:02:54.326 INFO:teuthology.orchestra.run.smithi045.stdout:Restarting the monitor... 2024-03-11T22:02:54.344 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:54 smithi045 ceph-mon[29691]: from='client.? 172.21.15.45:0/3094650956' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:02:54.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:54 smithi045 systemd[1]: Stopping Ceph mon.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:02:54.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:54 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a[29668]: 2024-03-11T22:02:54.584+0000 7fefd12a6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:02:54.630 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:54 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a[29668]: 2024-03-11T22:02:54.584+0000 7fefd12a6700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T22:02:54.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:54 smithi045 podman[30023]: 2024-03-11 22:02:54.636399698 +0000 UTC m=+0.138679286 container died e1f52240dee3cb76e8d7240c8a98af983b082e436621d80213998f125cf95f7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-11T22:02:55.234 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:54 smithi045 podman[30023]: 2024-03-11 22:02:54.998596099 +0000 UTC m=+0.500875686 container cleanup e1f52240dee3cb76e8d7240c8a98af983b082e436621d80213998f125cf95f7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.license=GPLv2, ceph=True) 2024-03-11T22:02:55.235 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:54 smithi045 bash[30023]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a 2024-03-11T22:02:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 podman[30042]: 2024-03-11 22:02:55.24109696 +0000 UTC m=+0.607501892 container remove e1f52240dee3cb76e8d7240c8a98af983b082e436621d80213998f125cf95f7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-11T22:02:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T22:02:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 systemd[1]: Stopped Ceph mon.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:02:55.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 systemd[1]: Starting Ceph mon.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:02:55.921 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 podman[30140]: 2024-03-11 22:02:55.476569836 +0000 UTC m=+0.018106247 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:02:55.922 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 podman[30140]: 2024-03-11 22:02:55.700384808 +0000 UTC m=+0.241921198 container create 8386a2a2886424ad728da9f8f0c981d7289bad54deb8f6d04e9573911b9361cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-11T22:02:55.922 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 podman[30140]: 2024-03-11 22:02:55.928086777 +0000 UTC m=+0.469623177 container init 8386a2a2886424ad728da9f8f0c981d7289bad54deb8f6d04e9573911b9361cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.build-date=20240307) 2024-03-11T22:02:56.084 INFO:teuthology.orchestra.run.smithi045.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T22:02:56.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 podman[30140]: 2024-03-11 22:02:55.933643112 +0000 UTC m=+0.475179502 container start 8386a2a2886424ad728da9f8f0c981d7289bad54deb8f6d04e9573911b9361cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-11T22:02:56.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:02:56.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T22:02:56.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: pidfile_write: ignore empty --pid-file 2024-03-11T22:02:56.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: load: jerasure load: lrc 2024-03-11T22:02:56.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: RocksDB version: 7.9.2 2024-03-11T22:02:56.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Git sha 0 2024-03-11T22:02:56.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T22:02:56.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: DB SUMMARY 2024-03-11T22:02:56.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: DB Session ID: 1AQGHGCL4JSPX8KQ62DF 2024-03-11T22:02:56.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:02:56.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:02:56.199 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T22:02:56.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T22:02:56.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 90819 ; 2024-03-11T22:02:56.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:02:56.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:02:56.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:02:56.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T22:02:56.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:02:56.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T22:02:56.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.env: 0x557fae135ae0 2024-03-11T22:02:56.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T22:02:56.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.info_log: 0x557faf07f8c0 2024-03-11T22:02:56.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:02:56.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.statistics: (nil) 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.use_fsync: 0 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.db_log_dir: 2024-03-11T22:02:56.201 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.wal_dir: 2024-03-11T22:02:56.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:02:56.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:02:56.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:02:56.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:02:56.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:02:56.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:02:56.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:02:56.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:02:56.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.write_buffer_manager: 0x557faf053400 2024-03-11T22:02:56.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:02:56.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:02:56.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:02:56.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:02:56.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:02:56.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:02:56.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:02:56.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:02:56.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.unordered_write: 0 2024-03-11T22:02:56.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:02:56.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:02:56.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:02:56.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:02:56.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.row_cache: None 2024-03-11T22:02:56.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.wal_filter: None 2024-03-11T22:02:56.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:02:56.203 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:02:56.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:02:56.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:02:56.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.wal_compression: 0 2024-03-11T22:02:56.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:02:56.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:02:56.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:02:56.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:02:56.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:02:56.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:02:56.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:02:56.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:02:56.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:02:56.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_open_files: -1 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:02:56.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Compression algorithms supported: 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: kZSTD supported: 0 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T22:02:56.207 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.merge_operator: 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compaction_filter: None 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557faf07f9c0) 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout: index_type: 0 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout: data_block_index_type: 0 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout: index_shortening: 1 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:02:56.208 INFO:journalctl@ceph.mon.a.smithi045.stdout: checksum: 4 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: no_block_cache: 0 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_cache: 0x557faf083090 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_cache_options: 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: capacity : 536870912 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: num_shard_bits : 4 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: strict_capacity_limit : 0 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_cache_compressed: (nil) 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: persistent_cache: (nil) 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_size: 4096 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_size_deviation: 10 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_restart_interval: 16 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: index_block_restart_interval: 1 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: metadata_block_size: 4096 2024-03-11T22:02:56.209 INFO:journalctl@ceph.mon.a.smithi045.stdout: partition_filters: 0 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout: use_delta_encoding: 1 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout: filter_policy: bloomfilter 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout: whole_key_filtering: 1 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout: verify_compression: 0 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout: format_version: 5 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout: enable_index_compression: 1 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout: block_align: 0 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout: max_auto_readahead_size: 262144 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout: prepopulate_block_cache: 0 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout: initial_auto_readahead_size: 8192 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:02:56.210 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compression: NoCompression 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.num_levels: 7 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:02:56.211 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:02:56.212 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T22:02:56.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:02:56.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.ttl: 2592000 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:02:56.215 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f9727d65-5b20-4873-8284-4ff7238475f3 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194575964276, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194575965267, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 86814, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 278, "table_properties": {"data_size": 84896, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11580, "raw_average_key_size": 48, "raw_value_size": 78517, "raw_average_value_size": 331, "num_data_blocks": 11, "num_entries": 237, "num_filter_entries": 237, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710194575, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f9727d65-5b20-4873-8284-4ff7238475f3", "db_session_id": "1AQGHGCL4JSPX8KQ62DF", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194575965355, "job": 1, "event": "recovery_finished"} 2024-03-11T22:02:56.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557faf174000 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: DB pointer 0x557faf15e000 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout: ** DB Stats ** 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout: ** Compaction Stats [default] ** 2024-03-11T22:02:56.217 INFO:journalctl@ceph.mon.a.smithi045.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: L0 2/0 86.61 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: Sum 2/0 86.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 123.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: ** Compaction Stats [default] ** 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 123.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:02:56.218 INFO:journalctl@ceph.mon.a.smithi045.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout: Cumulative compaction: 0.00 GB write, 26.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout: Interval compaction: 0.00 GB write, 26.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout: Block cache BinnedLRUCache@0x557faf083090#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: starting mon.a rank 0 at public addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] at bind addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: mon.a@-1(???) e1 preinit fsid f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: mon.a@-1(???).mds e1 new map 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: mon.a@-1(???).mds e1 print_map 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout: e1 2024-03-11T22:02:56.219 INFO:journalctl@ceph.mon.a.smithi045.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout: legacy client fscid: -1 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout: 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout: No filesystems configured 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:55 smithi045 ceph-mon[30178]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:56 smithi045 ceph-mon[30178]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:56 smithi045 ceph-mon[30178]: monmap e1: 1 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:56 smithi045 ceph-mon[30178]: fsmap 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:56 smithi045 ceph-mon[30178]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:56 smithi045 ceph-mon[30178]: mgrmap e1: no daemons active 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:56 smithi045 bash[30140]: 8386a2a2886424ad728da9f8f0c981d7289bad54deb8f6d04e9573911b9361cf 2024-03-11T22:02:56.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:56 smithi045 systemd[1]: Started Ceph mon.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:02:57.164 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T22:02:57.186 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:02:57.186 INFO:teuthology.orchestra.run.smithi045.stdout:Creating mgr... 2024-03-11T22:02:57.186 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T22:02:57.187 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T22:02:57.436 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mgr.y 2024-03-11T22:02:57.436 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to reset failed state of unit ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-11T22:02:57.579 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-f75aafa4-dff2-11ee-95c8-87774f69a715.target.wants/ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@.service. 2024-03-11T22:02:57.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:57 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1539307849' entity='client.admin' 2024-03-11T22:02:57.914 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:02:57 smithi045 podman[30421]: 2024-03-11 22:02:57.718766331 +0000 UTC m=+0.017487898 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:02:58.120 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:02:58.120 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-03-11T22:02:58.127 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:02:58.127 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-11T22:02:58.127 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-03-11T22:02:58.128 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T22:02:58.137 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:02:58.137 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-03-11T22:02:58.144 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:02:58.144 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-11T22:02:58.145 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-03-11T22:02:58.145 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T22:02:58.145 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr to start... 2024-03-11T22:02:58.145 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr... 2024-03-11T22:02:58.267 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:02:58 smithi045 podman[30421]: 2024-03-11 22:02:58.013959039 +0000 UTC m=+0.312680613 container init facc990c7c1b706faf5fd75546eda5969aa3204c32b4e9447d7e1f6d6a60668e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:02:58.267 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:02:58 smithi045 podman[30421]: 2024-03-11 22:02:58.01902197 +0000 UTC m=+0.317743526 container start facc990c7c1b706faf5fd75546eda5969aa3204c32b4e9447d7e1f6d6a60668e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-11T22:02:58.267 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:02:58 smithi045 bash[30421]: facc990c7c1b706faf5fd75546eda5969aa3204c32b4e9447d7e1f6d6a60668e 2024-03-11T22:02:58.267 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:02:58 smithi045 systemd[1]: Started Ceph mgr.y for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:02:59.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:02:58 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1290429609' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:02:59.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:02:58 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:02:58.583+0000 7f5515eac200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:02:59.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:02:58 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:02:58.681+0000 7f5515eac200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:02:59.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-11T22:02:59.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:02:59.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "f75aafa4-dff2-11ee-95c8-87774f69a715", 2024-03-11T22:02:59.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:02:59.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:02:59.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:02:59.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:02:59.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:02:59.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:02:59.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:02:59.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:02:59.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:02:59.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:02:59.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:02:59.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:02:51.076476+0000", 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:02:59.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:02:59.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:02:59.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:02:59.343 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (1/15)... 2024-03-11T22:02:59.581 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:02:59 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:02:59.349+0000 7f5515eac200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:02:59.581 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:02:59 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:02:59.444+0000 7f5515eac200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:03:00.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:02:59 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:02:59.586+0000 7f5515eac200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:03:00.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:02:59 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:02:59.683+0000 7f5515eac200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:03:00.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:02:59 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:02:59.767+0000 7f5515eac200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:03:00.322 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:00 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:00.034+0000 7f5515eac200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:03:00.322 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:00 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:00.145+0000 7f5515eac200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:03:00.322 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:00 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:00.232+0000 7f5515eac200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:03:00.322 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:00 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:00.327+0000 7f5515eac200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:03:00.774 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:00 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:00.505+0000 7f5515eac200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:03:01.750 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:01 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:01.521+0000 7f5515eac200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:03:01.750 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:01 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:01.657+0000 7f5515eac200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:03:02.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:01 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/975314542' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:03:02.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:02.000+0000 7f5515eac200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:03:02.387 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:02.092+0000 7f5515eac200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:03:02.387 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:02.290+0000 7f5515eac200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:03:02.392 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-11T22:03:02.392 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:03:02.392 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "f75aafa4-dff2-11ee-95c8-87774f69a715", 2024-03-11T22:03:02.392 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:03:02.392 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:03:02.392 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:02.393 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:03:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:03:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:03:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:03:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:03:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:03:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:03:02.394 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:03:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:03:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:03:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:03:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:03:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:03:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:03:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:03:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:03:02.398 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:02.399 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:03:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:03:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:02:51.076476+0000", 2024-03-11T22:03:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:03:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:03:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:03:02.400 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (2/15)... 2024-03-11T22:03:02.760 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:02.477+0000 7f5515eac200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:03:03.020 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:02.765+0000 7f5515eac200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:03:03.021 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:03 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:03.026+0000 7f5515eac200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:03:03.275 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:03 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:03.267+0000 7f5515eac200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:03:04.513 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:04 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:04.086+0000 7f5515eac200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:03:04.771 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:04 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:04.675+0000 7f5515eac200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:03:04.771 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:04 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:04.777+0000 7f5515eac200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:03:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:04 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2859746698' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:03:05.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:05.008+0000 7f5515eac200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:03:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-11T22:03:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:03:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "f75aafa4-dff2-11ee-95c8-87774f69a715", 2024-03-11T22:03:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:03:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:03:05.510 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:03:05.511 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:03:05.512 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:03:05.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:03:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:03:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:03:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:03:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:03:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:03:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:02:51.076476+0000", 2024-03-11T22:03:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:03:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:03:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:03:05.514 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (3/15)... 2024-03-11T22:03:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-mon[30178]: Activating manager daemon y 2024-03-11T22:03:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-mon[30178]: mgrmap e2: y(active, starting, since 0.00292007s) 2024-03-11T22:03:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-mon[30178]: from='mgr.14100 172.21.15.45:0/1421030065' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:03:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-mon[30178]: from='mgr.14100 172.21.15.45:0/1421030065' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:03:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-mon[30178]: from='mgr.14100 172.21.15.45:0/1421030065' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:03:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-mon[30178]: from='mgr.14100 172.21.15.45:0/1421030065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:03:06.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-mon[30178]: from='mgr.14100 172.21.15.45:0/1421030065' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:03:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-mon[30178]: Manager daemon y is now available 2024-03-11T22:03:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-mon[30178]: from='mgr.14100 172.21.15.45:0/1421030065' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:03:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-mon[30178]: from='mgr.14100 172.21.15.45:0/1421030065' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:03:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-mon[30178]: from='mgr.14100 172.21.15.45:0/1421030065' entity='mgr.y' 2024-03-11T22:03:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-mon[30178]: from='mgr.14100 172.21.15.45:0/1421030065' entity='mgr.y' 2024-03-11T22:03:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:05 smithi045 ceph-mon[30178]: from='mgr.14100 172.21.15.45:0/1421030065' entity='mgr.y' 2024-03-11T22:03:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:07 smithi045 ceph-mon[30178]: mgrmap e3: y(active, since 1.0072s) 2024-03-11T22:03:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:09 smithi045 ceph-mon[30178]: mgrmap e4: y(active, since 3s) 2024-03-11T22:03:11.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:11 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1929086948' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:03:16.067 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-11T22:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "f75aafa4-dff2-11ee-95c8-87774f69a715", 2024-03-11T22:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:03:16.068 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:03:16.069 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:03:16.070 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:16.070 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:03:16.070 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:03:16.070 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:03:16.070 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:03:16.070 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:03:16.070 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:03:16.070 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:03:16.070 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:03:16.070 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:03:16.070 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:16.070 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:03:16.070 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:03:16.071 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:03:16.071 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:03:16.071 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:16.071 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:03:16.071 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:03:16.071 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:03:16.071 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:03:16.071 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:03:16.072 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:03:16.072 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:03:16.072 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:03:16.072 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:03:16.072 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:16.072 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:03:16.072 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:03:16.072 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:02:51.076476+0000", 2024-03-11T22:03:16.072 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:03:16.072 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:03:16.072 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:03:16.073 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:03:16.073 INFO:teuthology.orchestra.run.smithi045.stdout:mgr is available 2024-03-11T22:03:17.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:16 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2836215113' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T22:03:17.250 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-11T22:03:17.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T22:03:17.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout fsid = f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:03:17.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T22:03:17.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T22:03:17.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T22:03:17.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T22:03:17.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-11T22:03:17.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T22:03:17.252 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T22:03:17.253 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-03-11T22:03:17.253 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T22:03:17.253 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T22:03:17.253 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T22:03:17.253 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling cephadm module... 2024-03-11T22:03:18.363 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:18 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4273109009' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T22:03:18.363 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:18 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ignoring --setuser ceph since I am not root 2024-03-11T22:03:18.363 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:18 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ignoring --setgroup ceph since I am not root 2024-03-11T22:03:18.957 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:18 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:18.641+0000 7f1fe8807200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:03:18.957 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:18 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:18.738+0000 7f1fe8807200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:03:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:19 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4273109009' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T22:03:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:19 smithi045 ceph-mon[30178]: mgrmap e5: y(active, since 13s) 2024-03-11T22:03:19.643 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:19.415+0000 7f1fe8807200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:03:19.643 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:19.511+0000 7f1fe8807200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:03:19.937 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:19.656+0000 7f1fe8807200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:03:19.937 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:19.754+0000 7f1fe8807200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:03:19.937 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:19.840+0000 7f1fe8807200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:03:19.939 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:03:19.939 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T22:03:19.940 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:03:19.940 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T22:03:19.940 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T22:03:19.940 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:03:19.940 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for the mgr to restart... 2024-03-11T22:03:19.940 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr epoch 5... 2024-03-11T22:03:20.190 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:20 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1277292253' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T22:03:20.191 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:20 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:20.115+0000 7f1fe8807200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:03:20.525 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:20 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:20.236+0000 7f1fe8807200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:03:20.525 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:20 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:20.324+0000 7f1fe8807200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:03:20.525 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:20 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:20.424+0000 7f1fe8807200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:03:21.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:20 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:20.603+0000 7f1fe8807200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:03:21.895 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:21 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:21.590+0000 7f1fe8807200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:03:21.895 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:21 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:21.724+0000 7f1fe8807200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:03:22.158 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:22.071+0000 7f1fe8807200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:03:22.158 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:22.163+0000 7f1fe8807200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:03:22.466 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:22.363+0000 7f1fe8807200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:03:22.775 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:22.557+0000 7f1fe8807200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:03:23.102 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:22.847+0000 7f1fe8807200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:03:23.102 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:23 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:23.107+0000 7f1fe8807200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:03:23.525 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:23 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:23.349+0000 7f1fe8807200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:03:24.525 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:24.171+0000 7f1fe8807200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:03:25.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:24.751+0000 7f1fe8807200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:03:25.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:24.851+0000 7f1fe8807200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:03:25.525 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:25.074+0000 7f1fe8807200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:03:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: Active manager daemon y restarted 2024-03-11T22:03:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: Activating manager daemon y 2024-03-11T22:03:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T22:03:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: mgrmap e6: y(active, starting, since 0.00337228s) 2024-03-11T22:03:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:03:25.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:03:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:03:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:03:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:03:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: Manager daemon y is now available 2024-03-11T22:03:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' 2024-03-11T22:03:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' 2024-03-11T22:03:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:03:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:03:25.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:25 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:03:26.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:26 smithi045 ceph-mon[30178]: Found migration_current of "None". Setting to last migration. 2024-03-11T22:03:26.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:26 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:03:26.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:26 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:03:26.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:26 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' 2024-03-11T22:03:26.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:26 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' 2024-03-11T22:03:26.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:26 smithi045 ceph-mon[30178]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T22:03:26.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:26 smithi045 ceph-mon[30178]: mgrmap e7: y(active, since 1.00514s) 2024-03-11T22:03:26.468 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:26 smithi045 ceph-mon[30178]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T22:03:26.721 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:03:26.721 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T22:03:26.722 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T22:03:26.722 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:03:26.722 INFO:teuthology.orchestra.run.smithi045.stdout:mgr epoch 5 is available 2024-03-11T22:03:26.722 INFO:teuthology.orchestra.run.smithi045.stdout:Setting orchestrator backend to cephadm... 2024-03-11T22:03:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:28 smithi045 ceph-mon[30178]: mgrmap e8: y(active, since 2s) 2024-03-11T22:03:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:28 smithi045 ceph-mon[30178]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:03:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:28 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' 2024-03-11T22:03:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:28 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:03:29.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:29 smithi045 ceph-mon[30178]: [11/Mar/2024:22:03:28] ENGINE Bus STARTING 2024-03-11T22:03:29.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:29 smithi045 ceph-mon[30178]: [11/Mar/2024:22:03:28] ENGINE Serving on https://172.21.15.45:7150 2024-03-11T22:03:29.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:29 smithi045 ceph-mon[30178]: [11/Mar/2024:22:03:28] ENGINE Serving on http://172.21.15.45:8765 2024-03-11T22:03:29.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:29 smithi045 ceph-mon[30178]: [11/Mar/2024:22:03:28] ENGINE Bus STARTED 2024-03-11T22:03:29.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:29 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:03:29.164 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:29 smithi045 ceph-mon[30178]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:03:29.167 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T22:03:29.167 INFO:teuthology.orchestra.run.smithi045.stdout:Generating ssh key... 2024-03-11T22:03:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-mon[30178]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:03:30.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-mon[30178]: Generating ssh key... 2024-03-11T22:03:30.962 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: Generating public/private rsa key pair. 2024-03-11T22:03:30.962 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: Your identification has been saved in /tmp/tmpjtp1y2i9/key. 2024-03-11T22:03:30.962 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: Your public key has been saved in /tmp/tmpjtp1y2i9/key.pub. 2024-03-11T22:03:30.962 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: The key fingerprint is: 2024-03-11T22:03:30.962 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: SHA256:pj+6LlTt0hYVX5sp9q/DRQC7mSA2ZV1C6LL243lDZWQ ceph-f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:03:30.962 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: The key's randomart image is: 2024-03-11T22:03:30.962 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: +---[RSA 3072]----+ 2024-03-11T22:03:30.962 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: | +=+oo | 2024-03-11T22:03:30.962 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: | oo..+E+ | 2024-03-11T22:03:30.962 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: | .+o. =o+. | 2024-03-11T22:03:30.963 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: | ..+o.o *o .| 2024-03-11T22:03:30.963 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: | . oS+ +o.. | 2024-03-11T22:03:30.963 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: | . .o* . ..| 2024-03-11T22:03:30.963 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: | . .+ . . . ..| 2024-03-11T22:03:30.963 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: | . .. o.o o. | 2024-03-11T22:03:30.963 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: | o+o.ooo ... | 2024-03-11T22:03:30.963 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:30 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: +----[SHA256]-----+ 2024-03-11T22:03:31.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:31 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' 2024-03-11T22:03:31.862 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:31 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' 2024-03-11T22:03:32.463 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7jXcGfm5m5P9Fv7u+a8+jP0GGYfN1hLt3snm+JX+tCWRuAP5fMI9jkN+t34bh3M/Y+chBa679wkRNdg2jgdqBbZepwT+rdgt+KjB1S+vVsRxX3cB4u4h6lIPbKxFhkRxVlb1S0E5kC+blrT6X+7NBjlyIOc5+Ows92KWXDsX38snu5XlX/eUF9KeBcUxTefwCXi7GXuPBBOKZExlY7NrvifooSduJPAhb8HPEL94lSaDh3eraDzQa9Bl2zZEieA05roLXUEgTgfSP5WHVBHjhxexxDRIs6YTfDXmB4F8KlVRRlhxkKZ1Zc/mGI+cYZ07V+6SQaJhQnO6KISUBdmZgkNfd2nUYKtbDKhDF6ZVtB+pwprwEMxWdU/kiCXk5UPKkg9asFb7w6a8uudoiNaPgCyVm+6XCKjOwy5HOmT0V/A24HrApsGA3cwKmEEoOLBdkNP6Tvx8kE5Z5s5fkLe6FOk3RL2f8AfRep4Uty0Mgqvd02hVwLOXoh841hFj5FK0= ceph-f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:03:32.464 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:03:32.464 INFO:teuthology.orchestra.run.smithi045.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T22:03:32.464 INFO:teuthology.orchestra.run.smithi045.stdout:Adding host smithi045... 2024-03-11T22:03:32.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:32 smithi045 ceph-mon[30178]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:03:33.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:33 smithi045 ceph-mon[30178]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi045", "addr": "172.21.15.45", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:03:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:34 smithi045 ceph-mon[30178]: Deploying cephadm binary to smithi045 2024-03-11T22:03:38.204 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Added host 'smithi045' with addr '172.21.15.45' 2024-03-11T22:03:38.204 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying unmanaged mon service... 2024-03-11T22:03:38.909 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:38 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' 2024-03-11T22:03:38.909 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:38 smithi045 ceph-mon[30178]: Added host smithi045 2024-03-11T22:03:38.909 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:38 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:03:39.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T22:03:39.547 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying unmanaged mgr service... 2024-03-11T22:03:40.123 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:39 smithi045 ceph-mon[30178]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:03:40.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:39 smithi045 ceph-mon[30178]: Saving service mon spec with placement count:5 2024-03-11T22:03:40.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:39 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' 2024-03-11T22:03:41.561 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T22:03:41.684 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:41 smithi045 ceph-mon[30178]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:03:41.684 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:41 smithi045 ceph-mon[30178]: Saving service mgr spec with placement count:2 2024-03-11T22:03:41.684 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:41 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' 2024-03-11T22:03:41.684 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:41 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' 2024-03-11T22:03:43.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:42 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' 2024-03-11T22:03:43.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:42 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3751557065' entity='client.admin' 2024-03-11T22:03:44.174 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling the dashboard module... 2024-03-11T22:03:44.624 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:44 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/460395157' entity='client.admin' 2024-03-11T22:03:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:45 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1808159418' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T22:03:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:45 smithi045 ceph-mon[30178]: from='mgr.14120 172.21.15.45:0/1828667201' entity='mgr.y' 2024-03-11T22:03:45.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:45 smithi045 ceph-mon[30178]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:03:45.775 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:45 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ignoring --setuser ceph since I am not root 2024-03-11T22:03:45.775 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:45 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ignoring --setgroup ceph since I am not root 2024-03-11T22:03:46.123 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:46 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:46.129+0000 7fb8311ef200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:03:46.390 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:46 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:46.233+0000 7fb8311ef200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:03:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:46 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1808159418' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T22:03:46.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:46 smithi045 ceph-mon[30178]: mgrmap e9: y(active, since 20s) 2024-03-11T22:03:47.227 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:46 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:46.904+0000 7fb8311ef200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:03:47.227 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:47 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:47.003+0000 7fb8311ef200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:03:47.227 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:47 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:47.148+0000 7fb8311ef200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:03:47.502 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:47 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:47.247+0000 7fb8311ef200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:03:47.502 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:47 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:47.331+0000 7fb8311ef200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:03:47.505 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:03:47.506 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T22:03:47.506 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:03:47.506 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T22:03:47.506 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T22:03:47.506 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:03:47.506 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for the mgr to restart... 2024-03-11T22:03:47.507 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr epoch 9... 2024-03-11T22:03:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1437117066' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T22:03:47.775 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:47 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:47.603+0000 7fb8311ef200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:03:47.775 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:47 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:47.715+0000 7fb8311ef200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:03:48.084 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:47 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:47.809+0000 7fb8311ef200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:03:48.084 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:47 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:47.907+0000 7fb8311ef200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:03:48.084 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:48 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:48.089+0000 7fb8311ef200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:03:49.525 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:49 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:49.069+0000 7fb8311ef200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:03:49.525 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:49 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:49.202+0000 7fb8311ef200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:03:49.827 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:49 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:49.541+0000 7fb8311ef200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:03:49.828 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:49 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:49.633+0000 7fb8311ef200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:03:49.828 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:49 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:49.833+0000 7fb8311ef200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:03:50.275 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:50 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:50.021+0000 7fb8311ef200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:03:50.566 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:50 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:50.311+0000 7fb8311ef200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:03:51.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:50 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:50.572+0000 7fb8311ef200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:03:51.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:50 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:50.813+0000 7fb8311ef200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:03:52.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:51 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:51.635+0000 7fb8311ef200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:03:52.525 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:52.216+0000 7fb8311ef200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:03:52.525 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:52.317+0000 7fb8311ef200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:03:53.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:03:52.543+0000 7fb8311ef200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:03:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-mon[30178]: Active manager daemon y restarted 2024-03-11T22:03:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-mon[30178]: Activating manager daemon y 2024-03-11T22:03:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-mon[30178]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T22:03:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-mon[30178]: mgrmap e10: y(active, starting, since 0.00301242s) 2024-03-11T22:03:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:03:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:03:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:03:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:03:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:03:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-mon[30178]: Manager daemon y is now available 2024-03-11T22:03:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:03:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:03:53.934 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:53 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:03:53.935 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:53 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:03:53.935 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:53 smithi045 ceph-mon[30178]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T22:03:53.935 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:53 smithi045 ceph-mon[30178]: mgrmap e11: y(active, since 1.0065s) 2024-03-11T22:03:53.935 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:53 smithi045 ceph-mon[30178]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T22:03:54.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:03:54.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T22:03:54.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T22:03:54.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:03:54.147 INFO:teuthology.orchestra.run.smithi045.stdout:mgr epoch 9 is available 2024-03-11T22:03:54.147 INFO:teuthology.orchestra.run.smithi045.stdout:Generating a dashboard self-signed certificate... 2024-03-11T22:03:54.904 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:54 smithi045 ceph-mon[30178]: [11/Mar/2024:22:03:53] ENGINE Bus STARTING 2024-03-11T22:03:54.904 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:54 smithi045 ceph-mon[30178]: [11/Mar/2024:22:03:53] ENGINE Serving on https://172.21.15.45:7150 2024-03-11T22:03:54.905 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:54 smithi045 ceph-mon[30178]: [11/Mar/2024:22:03:53] ENGINE Serving on http://172.21.15.45:8765 2024-03-11T22:03:54.905 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:54 smithi045 ceph-mon[30178]: [11/Mar/2024:22:03:53] ENGINE Bus STARTED 2024-03-11T22:03:55.479 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T22:03:55.479 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial admin user... 2024-03-11T22:03:55.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:55 smithi045 ceph-mon[30178]: mgrmap e12: y(active, since 2s) 2024-03-11T22:03:55.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:55 smithi045 ceph-mon[30178]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:03:55.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:03:55.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:03:55.733 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:03:57.059 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$1/kV2F0uXucjrqWBzK2tNOnpllu.QurL1uXtHPCKgo5A8QLF4p.oS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710194636, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T22:03:57.059 INFO:teuthology.orchestra.run.smithi045.stdout:Fetching dashboard port number... 2024-03-11T22:03:57.693 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:57 smithi045 ceph-mon[30178]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:03:57.694 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:57 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:03:57.694 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:57 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:03:58.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:03:58 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1308116248' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T22:03:58.383 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T22:03:58.393 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:03:58.393 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-03-11T22:03:58.401 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:03:58.401 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-03-11T22:03:58.401 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-03-11T22:03:58.401 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T22:03:58.403 INFO:teuthology.orchestra.run.smithi045.stdout:Ceph Dashboard is now available at: 2024-03-11T22:03:58.403 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:03:58.403 INFO:teuthology.orchestra.run.smithi045.stdout: URL: https://smithi045.front.sepia.ceph.com:8443/ 2024-03-11T22:03:58.403 INFO:teuthology.orchestra.run.smithi045.stdout: User: admin 2024-03-11T22:03:58.403 INFO:teuthology.orchestra.run.smithi045.stdout: Password: 4qc0gmqf64 2024-03-11T22:03:58.403 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:03:58.404 INFO:teuthology.orchestra.run.smithi045.stdout:Saving cluster configuration to /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config directory 2024-03-11T22:03:58.404 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling autotune for osd_memory_target 2024-03-11T22:04:00.482 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:00 smithi045 ceph-mon[30178]: mgrmap e13: y(active, since 6s) 2024-03-11T22:04:01.842 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T22:04:01.842 INFO:teuthology.orchestra.run.smithi045.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T22:04:01.842 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:04:01.842 INFO:teuthology.orchestra.run.smithi045.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:04:01.842 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:04:01.842 INFO:teuthology.orchestra.run.smithi045.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T22:04:01.842 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:04:01.842 INFO:teuthology.orchestra.run.smithi045.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T22:04:01.842 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:04:01.842 INFO:teuthology.orchestra.run.smithi045.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T22:04:01.842 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:04:01.842 INFO:teuthology.orchestra.run.smithi045.stdout: ceph telemetry on 2024-03-11T22:04:01.843 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:04:01.843 INFO:teuthology.orchestra.run.smithi045.stdout:For more information see: 2024-03-11T22:04:01.843 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:04:01.843 INFO:teuthology.orchestra.run.smithi045.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T22:04:01.843 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:04:01.843 INFO:teuthology.orchestra.run.smithi045.stdout:Bootstrap complete. 2024-03-11T22:04:01.887 INFO:tasks.cephadm:Fetching config... 2024-03-11T22:04:01.887 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:04:01.887 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T22:04:01.903 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T22:04:01.903 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:04:01.903 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T22:04:01.958 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T22:04:01.958 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:04:01.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T22:04:02.024 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T22:04:02.025 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:04:02.025 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T22:04:02.080 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T22:04:02.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7jXcGfm5m5P9Fv7u+a8+jP0GGYfN1hLt3snm+JX+tCWRuAP5fMI9jkN+t34bh3M/Y+chBa679wkRNdg2jgdqBbZepwT+rdgt+KjB1S+vVsRxX3cB4u4h6lIPbKxFhkRxVlb1S0E5kC+blrT6X+7NBjlyIOc5+Ows92KWXDsX38snu5XlX/eUF9KeBcUxTefwCXi7GXuPBBOKZExlY7NrvifooSduJPAhb8HPEL94lSaDh3eraDzQa9Bl2zZEieA05roLXUEgTgfSP5WHVBHjhxexxDRIs6YTfDXmB4F8KlVRRlhxkKZ1Zc/mGI+cYZ07V+6SQaJhQnO6KISUBdmZgkNfd2nUYKtbDKhDF6ZVtB+pwprwEMxWdU/kiCXk5UPKkg9asFb7w6a8uudoiNaPgCyVm+6XCKjOwy5HOmT0V/A24HrApsGA3cwKmEEoOLBdkNP6Tvx8kE5Z5s5fkLe6FOk3RL2f8AfRep4Uty0Mgqvd02hVwLOXoh841hFj5FK0= ceph-f75aafa4-dff2-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:04:02.164 INFO:teuthology.orchestra.run.smithi045.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7jXcGfm5m5P9Fv7u+a8+jP0GGYfN1hLt3snm+JX+tCWRuAP5fMI9jkN+t34bh3M/Y+chBa679wkRNdg2jgdqBbZepwT+rdgt+KjB1S+vVsRxX3cB4u4h6lIPbKxFhkRxVlb1S0E5kC+blrT6X+7NBjlyIOc5+Ows92KWXDsX38snu5XlX/eUF9KeBcUxTefwCXi7GXuPBBOKZExlY7NrvifooSduJPAhb8HPEL94lSaDh3eraDzQa9Bl2zZEieA05roLXUEgTgfSP5WHVBHjhxexxDRIs6YTfDXmB4F8KlVRRlhxkKZ1Zc/mGI+cYZ07V+6SQaJhQnO6KISUBdmZgkNfd2nUYKtbDKhDF6ZVtB+pwprwEMxWdU/kiCXk5UPKkg9asFb7w6a8uudoiNaPgCyVm+6XCKjOwy5HOmT0V/A24HrApsGA3cwKmEEoOLBdkNP6Tvx8kE5Z5s5fkLe6FOk3RL2f8AfRep4Uty0Mgqvd02hVwLOXoh841hFj5FK0= ceph-f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:04:02.179 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7jXcGfm5m5P9Fv7u+a8+jP0GGYfN1hLt3snm+JX+tCWRuAP5fMI9jkN+t34bh3M/Y+chBa679wkRNdg2jgdqBbZepwT+rdgt+KjB1S+vVsRxX3cB4u4h6lIPbKxFhkRxVlb1S0E5kC+blrT6X+7NBjlyIOc5+Ows92KWXDsX38snu5XlX/eUF9KeBcUxTefwCXi7GXuPBBOKZExlY7NrvifooSduJPAhb8HPEL94lSaDh3eraDzQa9Bl2zZEieA05roLXUEgTgfSP5WHVBHjhxexxDRIs6YTfDXmB4F8KlVRRlhxkKZ1Zc/mGI+cYZ07V+6SQaJhQnO6KISUBdmZgkNfd2nUYKtbDKhDF6ZVtB+pwprwEMxWdU/kiCXk5UPKkg9asFb7w6a8uudoiNaPgCyVm+6XCKjOwy5HOmT0V/A24HrApsGA3cwKmEEoOLBdkNP6Tvx8kE5Z5s5fkLe6FOk3RL2f8AfRep4Uty0Mgqvd02hVwLOXoh841hFj5FK0= ceph-f75aafa4-dff2-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:04:02.224 INFO:teuthology.orchestra.run.smithi144.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7jXcGfm5m5P9Fv7u+a8+jP0GGYfN1hLt3snm+JX+tCWRuAP5fMI9jkN+t34bh3M/Y+chBa679wkRNdg2jgdqBbZepwT+rdgt+KjB1S+vVsRxX3cB4u4h6lIPbKxFhkRxVlb1S0E5kC+blrT6X+7NBjlyIOc5+Ows92KWXDsX38snu5XlX/eUF9KeBcUxTefwCXi7GXuPBBOKZExlY7NrvifooSduJPAhb8HPEL94lSaDh3eraDzQa9Bl2zZEieA05roLXUEgTgfSP5WHVBHjhxexxDRIs6YTfDXmB4F8KlVRRlhxkKZ1Zc/mGI+cYZ07V+6SQaJhQnO6KISUBdmZgkNfd2nUYKtbDKhDF6ZVtB+pwprwEMxWdU/kiCXk5UPKkg9asFb7w6a8uudoiNaPgCyVm+6XCKjOwy5HOmT0V/A24HrApsGA3cwKmEEoOLBdkNP6Tvx8kE5Z5s5fkLe6FOk3RL2f8AfRep4Uty0Mgqvd02hVwLOXoh841hFj5FK0= ceph-f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:04:02.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T22:04:02.478 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:04:02.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:02 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1435246554' entity='client.admin' 2024-03-11T22:04:05.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:05 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:05.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:05 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:05.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:05 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:04:05.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:05 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:05.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:05 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:05.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:05 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:05.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:05 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1114715389' entity='client.admin' 2024-03-11T22:04:05.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:05 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:05.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:05 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:05.402 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T22:04:05.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T22:04:05.640 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:04:08.202 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi144 2024-03-11T22:04:08.202 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-03-11T22:04:08.202 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/etc/ceph/ceph.conf 2024-03-11T22:04:08.222 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-03-11T22:04:08.222 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:04:08.280 INFO:tasks.cephadm:Adding host smithi144 to orchestrator... 2024-03-11T22:04:08.280 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch host add smithi144 2024-03-11T22:04:08.521 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:04:08.749 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:08 smithi045 ceph-mon[30178]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:04:08.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:08 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:08.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:08 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:08.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:08 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:08.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:08 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:08.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:08 smithi045 ceph-mon[30178]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-11T22:04:08.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:08 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:08.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:08 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:08.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:08 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:09 smithi045 ceph-mon[30178]: Updating smithi045:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:04:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:09 smithi045 ceph-mon[30178]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:04:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:09 smithi045 ceph-mon[30178]: Updating smithi045:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:04:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:10 smithi045 ceph-mon[30178]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi144", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:04:12.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:11 smithi045 ceph-mon[30178]: Deploying cephadm binary to smithi144 2024-03-11T22:04:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:12 smithi045 ceph-mon[30178]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:14.526 INFO:teuthology.orchestra.run.smithi045.stdout:Added host 'smithi144' with addr '172.21.15.144' 2024-03-11T22:04:15.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T22:04:15.455 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:04:15.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:15 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:15.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:15 smithi045 ceph-mon[30178]: Added host smithi144 2024-03-11T22:04:15.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:15 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:15.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:15 smithi045 ceph-mon[30178]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:15.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:15 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:16 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:04:17.247 INFO:teuthology.orchestra.run.smithi045.stdout:[{"addr": "172.21.15.45", "hostname": "smithi045", "labels": [], "status": ""}, {"addr": "172.21.15.144", "hostname": "smithi144", "labels": [], "status": ""}] 2024-03-11T22:04:17.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:17 smithi045 ceph-mon[30178]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:17.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:17 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:17.841 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T22:04:17.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T22:04:18.084 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:04:18.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:18 smithi045 ceph-mon[30178]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:04:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:19 smithi045 ceph-mon[30178]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:20.539 INFO:teuthology.orchestra.run.smithi045.stderr:adjusted tunables profile to default 2024-03-11T22:04:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:20 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/212318803' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:04:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:20 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:20 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:20 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:20 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:20 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:04:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:20 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:20.865 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:20 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:21.224 INFO:tasks.cephadm:Adding mon.a on smithi045 2024-03-11T22:04:21.225 INFO:tasks.cephadm:Adding mon.c on smithi045 2024-03-11T22:04:21.225 INFO:tasks.cephadm:Adding mon.b on smithi144 2024-03-11T22:04:21.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi045:172.21.15.45=a;smithi045:[v2:172.21.15.45:3301,v1:172.21.15.45:6790]=c;smithi144:172.21.15.144=b' 2024-03-11T22:04:21.451 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:04:22.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:21 smithi045 ceph-mon[30178]: Updating smithi144:/etc/ceph/ceph.conf 2024-03-11T22:04:22.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:21 smithi045 ceph-mon[30178]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:04:22.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:21 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/212318803' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:04:22.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:21 smithi045 ceph-mon[30178]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:04:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:21 smithi045 ceph-mon[30178]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:22.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:22 smithi045 ceph-mon[30178]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:04:22.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:22 smithi045 ceph-mon[30178]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:04:23.106 INFO:teuthology.orchestra.run.smithi144.stdout:Scheduled mon update... 2024-03-11T22:04:23.691 DEBUG:teuthology.orchestra.run.smithi045:mon.c> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T22:04:23.693 DEBUG:teuthology.orchestra.run.smithi144:mon.b> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T22:04:23.695 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T22:04:23.696 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:04:24.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:23 smithi045 ceph-mon[30178]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:24.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:23 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:24.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:23 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:24.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:23 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:24.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:23 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:23 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:23 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:24.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:23 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:24.442 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:04:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:24 smithi045 ceph-mon[30178]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi045:172.21.15.45=a;smithi045:[v2:172.21.15.45:3301,v1:172.21.15.45:6790]=c;smithi144:172.21.15.144=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:04:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:24 smithi045 ceph-mon[30178]: Saving service mon spec with placement smithi045:172.21.15.45=a;smithi045:[v2:172.21.15.45:3301,v1:172.21.15.45:6790]=c;smithi144:172.21.15.144=b;count:3 2024-03-11T22:04:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:24 smithi045 ceph-mon[30178]: Deploying daemon mon.b on smithi144 2024-03-11T22:04:26.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:25 smithi045 ceph-mon[30178]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:26.690 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:04:26.690 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"f75aafa4-dff2-11ee-95c8-87774f69a715","modified":"2024-03-11T22:02:48.032741Z","created":"2024-03-11T22:02:48.032741Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:04:26.692 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2024-03-11T22:04:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:26 smithi045 ceph-mon[30178]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:28.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:27 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/1821123570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:04:28.556 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 podman[33946]: 2024-03-11 22:04:28.359934717 +0000 UTC m=+0.018113910 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:04:28.775 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T22:04:28.776 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:04:29.013 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:04:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:28 smithi045 ceph-mon[30178]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:29.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 podman[33946]: 2024-03-11 22:04:28.68239022 +0000 UTC m=+0.340569426 container init 061bce8bb349978ea4d86db352a1757b4456c8edc34aa72cdb0ce79e893b847d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-94a5397) 2024-03-11T22:04:29.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 podman[33946]: 2024-03-11 22:04:28.687479845 +0000 UTC m=+0.345659036 container start 061bce8bb349978ea4d86db352a1757b4456c8edc34aa72cdb0ce79e893b847d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS) 2024-03-11T22:04:29.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:04:29.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T22:04:29.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: pidfile_write: ignore empty --pid-file 2024-03-11T22:04:29.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: load: jerasure load: lrc 2024-03-11T22:04:29.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: RocksDB version: 7.9.2 2024-03-11T22:04:29.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Git sha 0 2024-03-11T22:04:29.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T22:04:29.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: DB SUMMARY 2024-03-11T22:04:29.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: DB Session ID: 831SG51Y42DK19YPSNZC 2024-03-11T22:04:29.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:04:29.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:04:29.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T22:04:29.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-11T22:04:29.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-03-11T22:04:29.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:04:29.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:04:29.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:04:29.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T22:04:29.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:04:29.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T22:04:29.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.env: 0x56245bf16ae0 2024-03-11T22:04:29.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T22:04:29.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.info_log: 0x56245e5a78c0 2024-03-11T22:04:29.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:04:29.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.statistics: (nil) 2024-03-11T22:04:29.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.use_fsync: 0 2024-03-11T22:04:29.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:04:29.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:04:29.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:04:29.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:04:29.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:04:29.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:04:29.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:04:29.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:04:29.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.db_log_dir: 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.wal_dir: 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.write_buffer_manager: 0x56245e57b400 2024-03-11T22:04:29.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:04:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:04:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:04:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:04:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:04:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:04:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:04:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:04:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.unordered_write: 0 2024-03-11T22:04:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:04:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:04:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:04:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:04:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.row_cache: None 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.wal_filter: None 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.wal_compression: 0 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:04:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:04:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_open_files: -1 2024-03-11T22:04:29.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:04:29.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:04:29.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:04:29.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:04:29.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:04:29.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Compression algorithms supported: 2024-03-11T22:04:29.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:04:29.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: kZSTD supported: 0 2024-03-11T22:04:29.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:04:29.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:04:29.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:04:29.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:04:29.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.merge_operator: 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compaction_filter: None 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56245e5a79c0) 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:04:29.099 INFO:journalctl@ceph.mon.b.smithi144.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: index_type: 0 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: data_block_index_type: 0 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: index_shortening: 1 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: checksum: 4 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: no_block_cache: 0 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: block_cache: 0x56245e5ab090 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: block_cache_options: 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: capacity : 536870912 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: num_shard_bits : 4 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: strict_capacity_limit : 0 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:04:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout: block_cache_compressed: (nil) 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: persistent_cache: (nil) 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: block_size: 4096 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: block_size_deviation: 10 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: block_restart_interval: 16 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: index_block_restart_interval: 1 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: metadata_block_size: 4096 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: partition_filters: 0 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: use_delta_encoding: 1 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: filter_policy: bloomfilter 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: whole_key_filtering: 1 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: verify_compression: 0 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: format_version: 5 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: enable_index_compression: 1 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: block_align: 0 2024-03-11T22:04:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout: max_auto_readahead_size: 262144 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout: prepopulate_block_cache: 0 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout: initial_auto_readahead_size: 8192 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compression: NoCompression 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.num_levels: 7 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:04:29.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:04:29.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:04:29.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:04:29.105 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:04:29.106 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.ttl: 2592000 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T22:04:29.107 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7dc69a10-9345-4d7f-b67c-5abe9444f892 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194668717319, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194668717864, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710194668, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7dc69a10-9345-4d7f-b67c-5abe9444f892", "db_session_id": "831SG51Y42DK19YPSNZC", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194668717963, "job": 1, "event": "recovery_finished"} 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56245e69c000 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: DB pointer 0x56245e686000 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout: ** DB Stats ** 2024-03-11T22:04:29.108 INFO:journalctl@ceph.mon.b.smithi144.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: ** Compaction Stats [default] ** 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: ** Compaction Stats [default] ** 2024-03-11T22:04:29.109 INFO:journalctl@ceph.mon.b.smithi144.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: Cumulative compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: Interval compaction: 0.00 GB write, 0.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:04:29.110 INFO:journalctl@ceph.mon.b.smithi144.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout: Block cache BinnedLRUCache@0x56245e5ab090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout: 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: using public_addr v2:172.21.15.144:0/0 -> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: starting mon.b rank -1 at public addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] at bind addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(???) e0 preinit fsid f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(synchronizing).mds e1 new map 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout: e1 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout: legacy client fscid: -1 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout: 2024-03-11T22:04:29.111 INFO:journalctl@ceph.mon.b.smithi144.stdout: No filesystems configured 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:29.112 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/212318803' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:04:29.113 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:29.113 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:29.113 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:29.113 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:29.113 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:04:29.113 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:29.113 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:29.113 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: Updating smithi144:/etc/ceph/ceph.conf 2024-03-11T22:04:29.113 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:04:29.113 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/212318803' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:04:29.113 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:04:29.113 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:29.113 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi045:172.21.15.45=a;smithi045:[v2:172.21.15.45:3301,v1:172.21.15.45:6790]=c;smithi144:172.21.15.144=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:04:29.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: Saving service mon spec with placement smithi045:172.21.15.45=a;smithi045:[v2:172.21.15.45:3301,v1:172.21.15.45:6790]=c;smithi144:172.21.15.144=b;count:3 2024-03-11T22:04:29.115 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: Deploying daemon mon.b on smithi144 2024-03-11T22:04:29.115 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:29.115 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:29.115 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/1821123570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:04:29.115 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:29.115 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 ceph-mon[33984]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T22:04:29.115 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 bash[33946]: 061bce8bb349978ea4d86db352a1757b4456c8edc34aa72cdb0ce79e893b847d 2024-03-11T22:04:29.115 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:28 smithi144 systemd[1]: Started Ceph mon.b for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:04:34.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:04:34.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: mon.a calling monitor election 2024-03-11T22:04:34.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:34.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: mon.b calling monitor election 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: monmap e2: 2 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: fsmap 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: mgrmap e13: y(active, since 41s) 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: overall HEALTH_OK 2024-03-11T22:04:34.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:34.092 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:34.092 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:34.092 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:34.092 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:34.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:04:34.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: mon.a calling monitor election 2024-03-11T22:04:34.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: mon.b calling monitor election 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: monmap e2: 2 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: fsmap 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: mgrmap e13: y(active, since 41s) 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: overall HEALTH_OK 2024-03-11T22:04:34.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:34.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:34.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:34.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:34.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:34.440 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:04:34.440 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":2,"fsid":"f75aafa4-dff2-11ee-95c8-87774f69a715","modified":"2024-03-11T22:04:28.740519Z","created":"2024-03-11T22:02:48.032741Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T22:04:34.442 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 2 2024-03-11T22:04:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:34 smithi045 ceph-mon[30178]: Deploying daemon mon.c on smithi045 2024-03-11T22:04:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:34 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/114641082' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:04:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:34 smithi045 ceph-mon[30178]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:34 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:35.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:34 smithi144 ceph-mon[33984]: Deploying daemon mon.c on smithi045 2024-03-11T22:04:35.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:34 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/114641082' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:04:35.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:34 smithi144 ceph-mon[33984]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:35.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:34 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:35.951 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T22:04:35.951 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:04:36.196 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:04:36.924 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:36 smithi045 ceph-mon[30178]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:36.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:36 smithi045 podman[39677]: 2024-03-11 22:04:36.768948291 +0000 UTC m=+0.018918280 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:04:36.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:36 smithi045 podman[39677]: 2024-03-11 22:04:36.930806781 +0000 UTC m=+0.180776769 container create 28174753370cb1c38ce141290547f95bb7beb15efe5d84106e8696f50808903f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-c, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:04:36.978 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:36 smithi144 ceph-mon[33984]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 podman[39677]: 2024-03-11 22:04:37.194920211 +0000 UTC m=+0.444890215 container init 28174753370cb1c38ce141290547f95bb7beb15efe5d84106e8696f50808903f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-11T22:04:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 podman[39677]: 2024-03-11 22:04:37.200142374 +0000 UTC m=+0.450112373 container start 28174753370cb1c38ce141290547f95bb7beb15efe5d84106e8696f50808903f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-11T22:04:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:04:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T22:04:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: pidfile_write: ignore empty --pid-file 2024-03-11T22:04:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: load: jerasure load: lrc 2024-03-11T22:04:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: RocksDB version: 7.9.2 2024-03-11T22:04:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Git sha 0 2024-03-11T22:04:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T22:04:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: DB SUMMARY 2024-03-11T22:04:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: DB Session ID: ZEW0572PI31IWUP8OT1K 2024-03-11T22:04:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.env: 0x55578e0f0ae0 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.info_log: 0x55578f7d98c0 2024-03-11T22:04:37.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.statistics: (nil) 2024-03-11T22:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.use_fsync: 0 2024-03-11T22:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:04:37.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.db_log_dir: 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.wal_dir: 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.write_buffer_manager: 0x55578f7ad400 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:04:37.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.unordered_write: 0 2024-03-11T22:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.row_cache: None 2024-03-11T22:04:37.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.wal_filter: None 2024-03-11T22:04:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:04:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:04:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:04:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:04:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.wal_compression: 0 2024-03-11T22:04:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:04:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:04:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:04:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:04:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:04:37.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:04:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:04:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:04:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:04:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:04:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:04:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T22:04:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:04:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:04:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:04:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:04:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:04:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:04:37.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:04:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:04:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:04:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:04:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:04:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_open_files: -1 2024-03-11T22:04:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:04:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:04:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:04:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:04:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:04:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Compression algorithms supported: 2024-03-11T22:04:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:04:37.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: kZSTD supported: 0 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.merge_operator: 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compaction_filter: None 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:04:37.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55578f7d99c0) 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout: index_type: 0 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout: data_block_index_type: 0 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout: index_shortening: 1 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout: checksum: 4 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout: no_block_cache: 0 2024-03-11T22:04:37.536 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_cache: 0x55578f7dd090 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_cache_options: 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: capacity : 536870912 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: num_shard_bits : 4 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: strict_capacity_limit : 0 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_cache_compressed: (nil) 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: persistent_cache: (nil) 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_size: 4096 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_size_deviation: 10 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_restart_interval: 16 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: index_block_restart_interval: 1 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: metadata_block_size: 4096 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: partition_filters: 0 2024-03-11T22:04:37.537 INFO:journalctl@ceph.mon.c.smithi045.stdout: use_delta_encoding: 1 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout: filter_policy: bloomfilter 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout: whole_key_filtering: 1 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout: verify_compression: 0 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout: format_version: 5 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout: enable_index_compression: 1 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout: block_align: 0 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout: max_auto_readahead_size: 262144 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout: prepopulate_block_cache: 0 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout: initial_auto_readahead_size: 8192 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:04:37.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compression: NoCompression 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.num_levels: 7 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:04:37.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:04:37.540 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:04:37.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:04:37.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.ttl: 2592000 2024-03-11T22:04:37.543 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T22:04:37.544 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 94344264-5b79-4421-88cc-93d4caaae56e 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194677229798, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194677230205, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710194677, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "94344264-5b79-4421-88cc-93d4caaae56e", "db_session_id": "ZEW0572PI31IWUP8OT1K", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710194677230288, "job": 1, "event": "recovery_finished"} 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55578f8ce000 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: DB pointer 0x55578f8b8000 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout: ** DB Stats ** 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:04:37.545 INFO:journalctl@ceph.mon.c.smithi045.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:04:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:04:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:04:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:04:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:04:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:04:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout: 2024-03-11T22:04:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout: ** Compaction Stats [default] ** 2024-03-11T22:04:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:04:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T22:04:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:04:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:04:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:04:37.546 INFO:journalctl@ceph.mon.c.smithi045.stdout: 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: ** Compaction Stats [default] ** 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:04:37.547 INFO:journalctl@ceph.mon.c.smithi045.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout: Block cache BinnedLRUCache@0x55578f7dd090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout: 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: using public_addrv [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: starting mon.c rank -1 at public addrs [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] at bind addrs [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(???) e0 preinit fsid f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(synchronizing).mds e1 new map 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout: e1 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T22:04:37.548 INFO:journalctl@ceph.mon.c.smithi045.stdout: legacy client fscid: -1 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout: 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout: No filesystems configured 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:37.549 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:37.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:04:37.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:37.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/212318803' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:04:37.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:37.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:37.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:37.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:37.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:04:37.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:37.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:37.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: Updating smithi144:/etc/ceph/ceph.conf 2024-03-11T22:04:37.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:04:37.550 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/212318803' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:37.551 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi045:172.21.15.45=a;smithi045:[v2:172.21.15.45:3301,v1:172.21.15.45:6790]=c;smithi144:172.21.15.144=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: Saving service mon spec with placement smithi045:172.21.15.45=a;smithi045:[v2:172.21.15.45:3301,v1:172.21.15.45:6790]=c;smithi144:172.21.15.144=b;count:3 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: Deploying daemon mon.b on smithi144 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/1821123570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.a calling monitor election 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:37.552 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.b calling monitor election 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: monmap e2: 2 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: fsmap 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mgrmap e13: y(active, since 41s) 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: overall HEALTH_OK 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:37.553 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: Deploying daemon mon.c on smithi045 2024-03-11T22:04:37.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/114641082' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:04:37.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:37.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:37.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:37.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 ceph-mon[39715]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T22:04:37.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 bash[39677]: 28174753370cb1c38ce141290547f95bb7beb15efe5d84106e8696f50808903f 2024-03-11T22:04:37.554 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:37 smithi045 systemd[1]: Started Ceph mon.c for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:04:42.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:04:42.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: mon.a calling monitor election 2024-03-11T22:04:42.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: mon.b calling monitor election 2024-03-11T22:04:42.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:42.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:42.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: mon.c calling monitor election 2024-03-11T22:04:42.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:42.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T22:04:42.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0],c=[v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:04:42.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: fsmap 2024-03-11T22:04:42.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:04:42.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: mgrmap e13: y(active, since 49s) 2024-03-11T22:04:42.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: overall HEALTH_OK 2024-03-11T22:04:42.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:42.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:04:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: mon.a calling monitor election 2024-03-11T22:04:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: mon.b calling monitor election 2024-03-11T22:04:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: mon.c calling monitor election 2024-03-11T22:04:42.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T22:04:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0],c=[v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:04:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: fsmap 2024-03-11T22:04:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:04:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: mgrmap e13: y(active, since 49s) 2024-03-11T22:04:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: overall HEALTH_OK 2024-03-11T22:04:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:42.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:42.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: mon.a calling monitor election 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: mon.b calling monitor election 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: mon.c calling monitor election 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T22:04:42.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: monmap e3: 3 mons at {a=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],b=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0],c=[v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:04:42.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: fsmap 2024-03-11T22:04:42.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:04:42.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: mgrmap e13: y(active, since 49s) 2024-03-11T22:04:42.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: overall HEALTH_OK 2024-03-11T22:04:42.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:42.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:43.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:43 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:43.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:43 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:43.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:43 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:43.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:43 smithi144 ceph-mon[33984]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:43.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:43 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:43.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:43 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:43 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:43 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:43 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:43 smithi045 ceph-mon[30178]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:43 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:43 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:43 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:43 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:43 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:43 smithi045 ceph-mon[39715]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:43 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:43 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:04:44.007 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-03-11T22:04:44.007 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":3,"fsid":"f75aafa4-dff2-11ee-95c8-87774f69a715","modified":"2024-03-11T22:04:37.250128Z","created":"2024-03-11T22:02:48.032741Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3301","nonce":0},{"type":"v1","addr":"172.21.15.45:6790","nonce":0}]},"addr":"172.21.15.45:6790/0","public_addr":"172.21.15.45:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-11T22:04:44.009 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 3 2024-03-11T22:04:44.631 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T22:04:44.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T22:04:44.881 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:04:45.004 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: Updating smithi144:/etc/ceph/ceph.conf 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/2577102942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: Updating smithi045:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T22:04:45.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:45.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:04:45.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:45.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: Reconfiguring daemon mon.a on smithi045 2024-03-11T22:04:45.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[30178]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:45.006 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:45.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:45.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-11T22:04:45.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: Updating smithi144:/etc/ceph/ceph.conf 2024-03-11T22:04:45.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/2577102942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:04:45.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:04:45.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: Updating smithi045:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: Reconfiguring daemon mon.a on smithi045 2024-03-11T22:04:45.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:44 smithi045 ceph-mon[39715]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:45.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:45.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:45.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-11T22:04:45.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: Updating smithi144:/etc/ceph/ceph.conf 2024-03-11T22:04:45.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/2577102942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:04:45.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:04:45.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: Updating smithi045:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:04:45.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:45.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T22:04:45.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:45.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:04:45.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:45.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: Reconfiguring daemon mon.a on smithi045 2024-03-11T22:04:45.091 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:44 smithi144 ceph-mon[33984]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:47.162 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:46 smithi045 ceph-mon[39715]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:47.163 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:46 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:47.163 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:46 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:47.163 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:46 smithi045 ceph-mon[30178]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:47.163 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:46 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:47.163 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:46 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:47.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:46 smithi144 ceph-mon[33984]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:47.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:46 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:47.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:46 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:47.640 INFO:teuthology.orchestra.run.smithi045.stdout:# minimal ceph.conf for f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:04:47.640 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2024-03-11T22:04:47.640 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:04:47.640 INFO:teuthology.orchestra.run.smithi045.stdout: mon_host = [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] [v2:172.21.15.45:3301/0,v1:172.21.15.45:6790/0] 2024-03-11T22:04:48.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:47 smithi045 ceph-mon[30178]: Reconfiguring mon.c (monmap changed)... 2024-03-11T22:04:48.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:48.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:04:48.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:48.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:47 smithi045 ceph-mon[30178]: Reconfiguring daemon mon.c on smithi045 2024-03-11T22:04:48.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1249325862' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:48.205 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:47 smithi045 ceph-mon[39715]: Reconfiguring mon.c (monmap changed)... 2024-03-11T22:04:48.205 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:48.205 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:04:48.205 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:48.205 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:47 smithi045 ceph-mon[39715]: Reconfiguring daemon mon.c on smithi045 2024-03-11T22:04:48.205 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:47 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1249325862' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:48.229 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T22:04:48.229 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:04:48.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:04:48.259 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:04:48.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:04:48.327 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-03-11T22:04:48.328 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:04:48.338 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:47 smithi144 ceph-mon[33984]: Reconfiguring mon.c (monmap changed)... 2024-03-11T22:04:48.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:48.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:04:48.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:48.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:47 smithi144 ceph-mon[33984]: Reconfiguring daemon mon.c on smithi045 2024-03-11T22:04:48.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:47 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1249325862' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:48.362 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-03-11T22:04:48.362 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:04:48.432 INFO:tasks.cephadm:Adding mgr.y on smithi045 2024-03-11T22:04:48.432 INFO:tasks.cephadm:Adding mgr.x on smithi144 2024-03-11T22:04:48.432 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch apply mgr '2;smithi045=y;smithi144=x' 2024-03-11T22:04:48.709 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:04:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:49 smithi045 ceph-mon[30178]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:49.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:49 smithi045 ceph-mon[39715]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:49.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:49 smithi144 ceph-mon[33984]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:50.166 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:50.166 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:50.166 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:50 smithi144 ceph-mon[33984]: Reconfiguring mon.b (monmap changed)... 2024-03-11T22:04:50.166 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:50.166 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:04:50.166 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:50.166 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:50 smithi144 ceph-mon[33984]: Reconfiguring daemon mon.b on smithi144 2024-03-11T22:04:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[30178]: Reconfiguring mon.b (monmap changed)... 2024-03-11T22:04:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:04:50.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[30178]: Reconfiguring daemon mon.b on smithi144 2024-03-11T22:04:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[39715]: Reconfiguring mon.b (monmap changed)... 2024-03-11T22:04:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:04:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:04:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:50 smithi045 ceph-mon[39715]: Reconfiguring daemon mon.b on smithi144 2024-03-11T22:04:50.747 INFO:teuthology.orchestra.run.smithi144.stdout:Scheduled mgr update... 2024-03-11T22:04:51.297 DEBUG:teuthology.orchestra.run.smithi144:mgr.x> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T22:04:51.299 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T22:04:51.299 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:04:51.299 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:04:51.318 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:04:51.318 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-03-11T22:04:51.321 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:51 smithi144 ceph-mon[33984]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:51.321 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:51 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:51.375 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:04:51.375 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:04:51.375 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-11T22:04:51.375 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:04:51.375 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:04:51.375 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-11 22:04:36.585642303 +0000 2024-03-11T22:04:51.375 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-11 21:58:34.334006385 +0000 2024-03-11T22:04:51.375 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-11 21:58:34.334006385 +0000 2024-03-11T22:04:51.375 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-11 21:58:34.334006385 +0000 2024-03-11T22:04:51.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:04:51.441 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:51 smithi045 ceph-mon[30178]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:51.441 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:51 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:51.442 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:51 smithi045 ceph-mon[39715]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:51.442 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:51 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:51.444 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-11T22:04:51.444 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-11T22:04:51.444 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000108517 s, 4.7 MB/s 2024-03-11T22:04:51.445 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:04:51.502 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-03-11T22:04:51.559 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:04:51.559 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:04:51.560 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-11T22:04:51.560 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:04:51.560 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:04:51.560 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-11 22:04:36.586642288 +0000 2024-03-11T22:04:51.560 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-11 21:58:34.667001400 +0000 2024-03-11T22:04:51.560 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-11 21:58:34.667001400 +0000 2024-03-11T22:04:51.560 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-11 21:58:34.667001400 +0000 2024-03-11T22:04:51.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:04:51.626 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-11T22:04:51.634 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-11T22:04:51.634 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000112152 s, 4.6 MB/s 2024-03-11T22:04:51.635 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:04:51.691 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-03-11T22:04:51.748 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:04:51.749 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:04:51.749 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-11T22:04:51.749 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:04:51.749 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:04:51.749 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-11 22:04:36.586642288 +0000 2024-03-11T22:04:51.749 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-11 21:58:35.004996340 +0000 2024-03-11T22:04:51.749 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-11 21:58:35.004996340 +0000 2024-03-11T22:04:51.749 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-11 21:58:35.004996340 +0000 2024-03-11T22:04:51.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:04:51.815 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-11T22:04:51.815 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-11T22:04:51.815 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000116886 s, 4.4 MB/s 2024-03-11T22:04:51.816 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:04:51.872 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-03-11T22:04:51.927 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:04:51.927 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:04:51.927 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T22:04:51.927 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:04:51.927 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:04:51.927 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-03-11 22:04:36.586642288 +0000 2024-03-11T22:04:51.927 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-03-11 21:58:35.345991236 +0000 2024-03-11T22:04:51.928 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-03-11 21:58:35.345991236 +0000 2024-03-11T22:04:51.928 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-03-11 21:58:35.345991236 +0000 2024-03-11T22:04:51.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:04:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-03-11T22:04:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-03-11T22:04:51.993 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000111414 s, 4.6 MB/s 2024-03-11T22:04:51.994 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:04:52.049 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-03-11T22:04:52.050 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:04:52.065 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:04:52.065 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_1 2024-03-11T22:04:52.120 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:04:52.120 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:04:52.120 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T22:04:52.121 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:04:52.121 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:04:52.121 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-03-11 22:04:28.165952289 +0000 2024-03-11T22:04:52.121 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-03-11 21:58:33.703100040 +0000 2024-03-11T22:04:52.121 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-03-11 21:58:33.703100040 +0000 2024-03-11T22:04:52.121 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-03-11 21:58:33.703100040 +0000 2024-03-11T22:04:52.121 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:04:52.188 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-03-11T22:04:52.188 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-03-11T22:04:52.188 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000191954 s, 2.7 MB/s 2024-03-11T22:04:52.189 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:04:52.245 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_2 2024-03-11T22:04:52.301 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:04:52.301 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:04:52.301 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-11T22:04:52.301 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:04:52.301 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:04:52.301 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-03-11 22:04:28.165952289 +0000 2024-03-11T22:04:52.301 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-03-11 21:58:34.053094205 +0000 2024-03-11T22:04:52.301 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-03-11 21:58:34.053094205 +0000 2024-03-11T22:04:52.302 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-03-11 21:58:34.053094205 +0000 2024-03-11T22:04:52.302 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:04:52.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:52 smithi144 ceph-mon[33984]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi045=y;smithi144=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:04:52.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:52 smithi144 ceph-mon[33984]: Saving service mgr spec with placement smithi045=y;smithi144=x;count:2 2024-03-11T22:04:52.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:52 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:52.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:52 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:52.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:52 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:52.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:52 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:52.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:52 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:52.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:52 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:52.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:52 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:04:52.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:52 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:04:52.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:52 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:04:52.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:52 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:52.364 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-03-11T22:04:52.364 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-03-11T22:04:52.365 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000212487 s, 2.4 MB/s 2024-03-11T22:04:52.366 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:04:52.421 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_3 2024-03-11T22:04:52.476 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:04:52.476 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:04:52.476 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-11T22:04:52.476 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:04:52.476 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:04:52.476 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-03-11 22:04:28.166952272 +0000 2024-03-11T22:04:52.477 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-03-11 21:58:34.393088536 +0000 2024-03-11T22:04:52.477 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-03-11 21:58:34.393088536 +0000 2024-03-11T22:04:52.477 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-03-11 21:58:34.393088536 +0000 2024-03-11T22:04:52.477 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:04:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[30178]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi045=y;smithi144=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:04:52.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[30178]: Saving service mgr spec with placement smithi045=y;smithi144=x;count:2 2024-03-11T22:04:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:04:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:04:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:04:52.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[39715]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi045=y;smithi144=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[39715]: Saving service mgr spec with placement smithi045=y;smithi144=x;count:2 2024-03-11T22:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:04:52.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:04:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:04:52.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:52 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:52.543 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-03-11T22:04:52.543 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-03-11T22:04:52.544 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000177377 s, 2.9 MB/s 2024-03-11T22:04:52.545 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:04:52.601 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_4 2024-03-11T22:04:52.655 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:04:52.656 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:04:52.656 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-11T22:04:52.656 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:04:52.656 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:04:52.656 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-03-11 22:04:28.166952272 +0000 2024-03-11T22:04:52.656 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-03-11 21:58:34.731082902 +0000 2024-03-11T22:04:52.656 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-03-11 21:58:34.731082902 +0000 2024-03-11T22:04:52.656 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-03-11 21:58:34.731082902 +0000 2024-03-11T22:04:52.656 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:04:52.722 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-03-11T22:04:52.722 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-03-11T22:04:52.722 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000189807 s, 2.7 MB/s 2024-03-11T22:04:52.723 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:04:52.779 INFO:tasks.cephadm:Deploying osd.0 on smithi045 with /dev/vg_nvme/lv_4... 2024-03-11T22:04:52.779 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T22:04:53.027 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:04:53.312 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:53 smithi144 ceph-mon[33984]: Deploying daemon mgr.x on smithi144 2024-03-11T22:04:53.312 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:53 smithi144 ceph-mon[33984]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:53.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:53 smithi045 ceph-mon[30178]: Deploying daemon mgr.x on smithi144 2024-03-11T22:04:53.328 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:53 smithi045 ceph-mon[30178]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:53.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:53 smithi045 ceph-mon[39715]: Deploying daemon mgr.x on smithi144 2024-03-11T22:04:53.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:53 smithi045 ceph-mon[39715]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:54.013 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:53 smithi144 podman[36051]: 2024-03-11 22:04:53.904997721 +0000 UTC m=+0.125045515 container create 06fde1e7966f9ce709c1a16c4b306c8f0260420d51e20dc0c9648863328f6db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-11T22:04:54.589 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:54 smithi144 podman[36051]: 2024-03-11 22:04:54.115490633 +0000 UTC m=+0.335538419 container init 06fde1e7966f9ce709c1a16c4b306c8f0260420d51e20dc0c9648863328f6db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T22:04:54.589 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:54 smithi144 podman[36051]: 2024-03-11 22:04:54.120540506 +0000 UTC m=+0.340588293 container start 06fde1e7966f9ce709c1a16c4b306c8f0260420d51e20dc0c9648863328f6db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-11T22:04:54.589 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:54 smithi144 ceph-mgr[36089]: -- 172.21.15.144:0/2912889582 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56486b84e1a0 con 0x56486b825000 2024-03-11T22:04:54.589 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:54 smithi144 bash[36051]: 06fde1e7966f9ce709c1a16c4b306c8f0260420d51e20dc0c9648863328f6db5 2024-03-11T22:04:54.589 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:54 smithi144 systemd[1]: Started Ceph mgr.x for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:04:55.023 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T22:04:55.023 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T22:04:55.023 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-03-11T22:04:55.023 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-03-11T22:04:55.023 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00830707 s, 1.3 GB/s 2024-03-11T22:04:55.023 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-03-11T22:04:55.089 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:54 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:54.678+0000 7f8ce9044200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:04:55.089 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:54 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:54.775+0000 7f8ce9044200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:04:55.467 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:55.467 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:55.467 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:55.467 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:55.467 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:55.467 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:55 smithi144 ceph-mon[33984]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:55.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:55.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:55.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:55.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:55.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:55.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:55 smithi045 ceph-mon[39715]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:55.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:55.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:55.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:55.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:55.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:55.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:55 smithi045 ceph-mon[30178]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:55.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_4 2024-03-11T22:04:55.806 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:55 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:55.472+0000 7f8ce9044200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:04:55.806 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:55 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:55.564+0000 7f8ce9044200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:04:55.806 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:55 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:55.714+0000 7f8ce9044200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:04:55.807 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:55 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:55.811+0000 7f8ce9044200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:04:55.901 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:04:56.089 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:55 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:55.894+0000 7f8ce9044200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:04:56.444 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:56.162+0000 7f8ce9044200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:04:56.444 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:56.270+0000 7f8ce9044200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:04:56.444 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:56.356+0000 7f8ce9044200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:04:56.444 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:56.449+0000 7f8ce9044200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:04:56.839 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:56.626+0000 7f8ce9044200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:04:56.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:56.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:56.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:56.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:56.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:56.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-mon[33984]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T22:04:56.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:04:56.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:04:56.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:56.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-mon[33984]: Reconfiguring daemon mgr.y on smithi045 2024-03-11T22:04:56.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:56 smithi144 ceph-mon[33984]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:56.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:56.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:56.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:56.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:56.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:56.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[30178]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T22:04:56.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:04:56.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:04:56.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:56.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[30178]: Reconfiguring daemon mgr.y on smithi045 2024-03-11T22:04:56.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[30178]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:56.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:56.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:56.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:56.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:56.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:56.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[39715]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T22:04:56.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:04:56.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:04:56.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:56.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[39715]: Reconfiguring daemon mgr.y on smithi045 2024-03-11T22:04:56.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:56 smithi045 ceph-mon[39715]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:57.839 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:57 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:57.584+0000 7f8ce9044200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:04:57.839 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:57 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:57.714+0000 7f8ce9044200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:04:58.327 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:58.048+0000 7f8ce9044200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:04:58.327 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:58.138+0000 7f8ce9044200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:04:58.327 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:58.332+0000 7f8ce9044200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:04:58.589 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:58.517+0000 7f8ce9044200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:04:58.733 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:58.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[30178]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:04:58.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:04:58.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:04:58.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:58.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:58.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:58.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:58.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[30178]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:58.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:58.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[39715]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:04:58.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:04:58.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:04:58.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:58.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:58.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:58.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:58.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:58 smithi045 ceph-mon[39715]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:59.054 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:58.797+0000 7f8ce9044200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:04:59.054 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:59 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:59.058+0000 7f8ce9044200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:04:59.054 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:59.054 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-mon[33984]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:04:59.054 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:04:59.054 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:04:59.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:59.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:59.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:59.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:04:59.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:58 smithi144 ceph-mon[33984]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:04:59.339 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:04:59 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:04:59.292+0000 7f8ce9044200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:04:59.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:59 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:59.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:59 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:59.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:04:59 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:04:59.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:59 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:04:59.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:59 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:04:59.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:04:59 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:00.083 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:59 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:00.083 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:59 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:05:00.083 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:04:59 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:00.339 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:05:00 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:05:00.088+0000 7f8ce9044200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:05:00.971 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:00 smithi144 ceph-mon[33984]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:00.971 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:05:00 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:05:00.664+0000 7f8ce9044200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:05:00.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:05:00 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:05:00.762+0000 7f8ce9044200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:05:00.972 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:05:00 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:05:00.976+0000 7f8ce9044200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:05:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:00 smithi045 ceph-mon[39715]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:00 smithi045 ceph-mon[30178]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:02.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1195484244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55a4e633-f2a6-4d27-b10c-de1c0f87cf32"}]: dispatch 2024-03-11T22:05:02.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1195484244' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55a4e633-f2a6-4d27-b10c-de1c0f87cf32"}]': finished 2024-03-11T22:05:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[39715]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T22:05:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[39715]: Standby manager daemon x started 2024-03-11T22:05:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[39715]: from='mgr.? 172.21.15.144:0/1175084058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:05:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[39715]: from='mgr.? 172.21.15.144:0/1175084058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:05:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[39715]: from='mgr.? 172.21.15.144:0/1175084058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:05:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[39715]: from='mgr.? 172.21.15.144:0/1175084058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:05:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2525098959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:05:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1195484244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55a4e633-f2a6-4d27-b10c-de1c0f87cf32"}]: dispatch 2024-03-11T22:05:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1195484244' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55a4e633-f2a6-4d27-b10c-de1c0f87cf32"}]': finished 2024-03-11T22:05:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[30178]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T22:05:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[30178]: Standby manager daemon x started 2024-03-11T22:05:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[30178]: from='mgr.? 172.21.15.144:0/1175084058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:05:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[30178]: from='mgr.? 172.21.15.144:0/1175084058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:05:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[30178]: from='mgr.? 172.21.15.144:0/1175084058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:05:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[30178]: from='mgr.? 172.21.15.144:0/1175084058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:05:02.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:01 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2525098959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:05:02.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:01 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1195484244' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55a4e633-f2a6-4d27-b10c-de1c0f87cf32"}]: dispatch 2024-03-11T22:05:02.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:01 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1195484244' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55a4e633-f2a6-4d27-b10c-de1c0f87cf32"}]': finished 2024-03-11T22:05:02.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:01 smithi144 ceph-mon[33984]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T22:05:02.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:01 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:02.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:01 smithi144 ceph-mon[33984]: Standby manager daemon x started 2024-03-11T22:05:02.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:01 smithi144 ceph-mon[33984]: from='mgr.? 172.21.15.144:0/1175084058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:05:02.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:01 smithi144 ceph-mon[33984]: from='mgr.? 172.21.15.144:0/1175084058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:05:02.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:01 smithi144 ceph-mon[33984]: from='mgr.? 172.21.15.144:0/1175084058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:05:02.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:01 smithi144 ceph-mon[33984]: from='mgr.? 172.21.15.144:0/1175084058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:05:02.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:01 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2525098959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:05:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:02 smithi045 ceph-mon[39715]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-11T22:05:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:02 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:05:03.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:02 smithi045 ceph-mon[39715]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:02 smithi045 ceph-mon[30178]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-11T22:05:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:02 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:05:03.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:02 smithi045 ceph-mon[30178]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:03.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:02 smithi144 ceph-mon[33984]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-11T22:05:03.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:02 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:05:03.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:02 smithi144 ceph-mon[33984]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:04.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:04 smithi045 ceph-mon[30178]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:04.914 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:04 smithi045 ceph-mon[39715]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:05.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:04 smithi144 ceph-mon[33984]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:06 smithi045 ceph-mon[30178]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:07.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:06 smithi045 ceph-mon[39715]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:07.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:06 smithi144 ceph-mon[33984]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:08.187 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:07 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:05:08.187 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:07 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:08.187 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:07 smithi045 ceph-mon[30178]: Deploying daemon osd.0 on smithi045 2024-03-11T22:05:08.188 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:07 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:05:08.188 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:07 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:08.188 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:07 smithi045 ceph-mon[39715]: Deploying daemon osd.0 on smithi045 2024-03-11T22:05:08.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:07 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:05:08.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:07 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:08.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:07 smithi144 ceph-mon[33984]: Deploying daemon osd.0 on smithi045 2024-03-11T22:05:09.185 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:08 smithi045 ceph-mon[30178]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:09.186 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:08 smithi045 ceph-mon[39715]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:09.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:08 smithi144 ceph-mon[33984]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:10 smithi045 ceph-mon[30178]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:10 smithi045 ceph-mon[39715]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:11.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:10 smithi144 ceph-mon[33984]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:13.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:12 smithi045 ceph-mon[39715]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:13.009 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:12 smithi045 ceph-mon[30178]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:13.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:12 smithi144 ceph-mon[33984]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:14.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:13 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:14.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:13 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:14.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:13 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:05:14.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:13 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:14.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:13 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:13 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:05:14.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:13 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:14.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:13 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:14.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:13 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:05:15.185 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:14 smithi045 ceph-mon[30178]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:15.186 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:14 smithi045 ceph-mon[39715]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:15.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:14 smithi144 ceph-mon[33984]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:16.998 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 0 on host 'smithi045' 2024-03-11T22:05:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:05:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[30178]: from='osd.0 [v2:172.21.15.45:6802/2216353820,v1:172.21.15.45:6803/2216353820]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:05:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[30178]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:05:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[30178]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:05:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:05:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[39715]: from='osd.0 [v2:172.21.15.45:6802/2216353820,v1:172.21.15.45:6803/2216353820]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:05:17.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[39715]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:05:17.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[39715]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:17.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:17 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:05:17.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:17 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:17 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:17 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:17.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:17 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:05:17.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:17 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:17 smithi144 ceph-mon[33984]: from='osd.0 [v2:172.21.15.45:6802/2216353820,v1:172.21.15.45:6803/2216353820]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:05:17.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:17 smithi144 ceph-mon[33984]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:05:17.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:17 smithi144 ceph-mon[33984]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:17.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:17 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:17 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:17.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:17 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:05:17.566 DEBUG:teuthology.orchestra.run.smithi045:osd.0> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T22:05:17.569 INFO:tasks.cephadm:Deploying osd.1 on smithi045 with /dev/vg_nvme/lv_3... 2024-03-11T22:05:17.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T22:05:18.273 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:05:18.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:18 smithi045 ceph-mon[39715]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:05:18.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:18 smithi045 ceph-mon[39715]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:05:18.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:18 smithi045 ceph-mon[39715]: from='osd.0 [v2:172.21.15.45:6802/2216353820,v1:172.21.15.45:6803/2216353820]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:05:18.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:18.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:18 smithi045 ceph-mon[39715]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:05:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:18 smithi045 ceph-mon[30178]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:05:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:18 smithi045 ceph-mon[30178]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:05:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:18 smithi045 ceph-mon[30178]: from='osd.0 [v2:172.21.15.45:6802/2216353820,v1:172.21.15.45:6803/2216353820]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:05:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:18 smithi045 ceph-mon[30178]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:05:18.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:18 smithi144 ceph-mon[33984]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:05:18.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:18 smithi144 ceph-mon[33984]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:05:18.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:18 smithi144 ceph-mon[33984]: from='osd.0 [v2:172.21.15.45:6802/2216353820,v1:172.21.15.45:6803/2216353820]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:05:18.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:18.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:18 smithi144 ceph-mon[33984]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:05:18.653 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:05:18 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0[44097]: 2024-03-11T22:05:18.329+0000 7f0695431700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T22:05:18.654 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:05:18 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0[44097]: 2024-03-11T22:05:18.331+0000 7f068ca1b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:05:19.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:19 smithi045 ceph-mon[30178]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-11T22:05:19.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:19 smithi045 ceph-mon[30178]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:05:19.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:19 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:19.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:19 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:19.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:19 smithi045 ceph-mon[30178]: from='osd.0 ' entity='osd.0' 2024-03-11T22:05:19.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:19 smithi045 ceph-mon[30178]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:19.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:19 smithi045 ceph-mon[39715]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-11T22:05:19.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:19 smithi045 ceph-mon[39715]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:05:19.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:19 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:19.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:19 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:19.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:19 smithi045 ceph-mon[39715]: from='osd.0 ' entity='osd.0' 2024-03-11T22:05:19.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:19 smithi045 ceph-mon[39715]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:19.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:19 smithi144 ceph-mon[33984]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-11T22:05:19.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:19 smithi144 ceph-mon[33984]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:05:19.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:19 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:19.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:19 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:19.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:19 smithi144 ceph-mon[33984]: from='osd.0 ' entity='osd.0' 2024-03-11T22:05:19.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:19 smithi144 ceph-mon[33984]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:05:20.091 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:20 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:05:20.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:20 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:05:20.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:20 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:20.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:20 smithi045 ceph-mon[30178]: osd.0 [v2:172.21.15.45:6802/2216353820,v1:172.21.15.45:6803/2216353820] boot 2024-03-11T22:05:20.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:20 smithi045 ceph-mon[30178]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T22:05:20.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:20 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:20.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:20 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:05:20.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:20 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:05:20.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:20 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:20.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:20 smithi045 ceph-mon[39715]: osd.0 [v2:172.21.15.45:6802/2216353820,v1:172.21.15.45:6803/2216353820] boot 2024-03-11T22:05:20.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:20 smithi045 ceph-mon[39715]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T22:05:20.092 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:20 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:20.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:20 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:05:20.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:20 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:05:20.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:20 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:20.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:20 smithi144 ceph-mon[33984]: osd.0 [v2:172.21.15.45:6802/2216353820,v1:172.21.15.45:6803/2216353820] boot 2024-03-11T22:05:20.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:20 smithi144 ceph-mon[33984]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T22:05:20.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:20 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:05:21.063 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T22:05:21.064 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T22:05:21.064 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-03-11T22:05:21.064 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-03-11T22:05:21.064 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113609 s, 923 MB/s 2024-03-11T22:05:21.064 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-03-11T22:05:21.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:21 smithi144 ceph-mon[33984]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:21.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:21 smithi045 ceph-mon[30178]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:21 smithi045 ceph-mon[39715]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:22.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[30178]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:05:22.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:22.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:22.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:05:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:05:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[39715]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:05:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:05:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:22.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:05:22.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:22 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:22.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:22 smithi144 ceph-mon[33984]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:05:22.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:22 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:22.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:22 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:22.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:22 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:05:22.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:22 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:22.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:22 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:22.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:22 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:05:22.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:22 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:22.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_3 2024-03-11T22:05:22.587 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:05:23.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:23 smithi144 ceph-mon[33984]: Detected new or changed devices on smithi045 2024-03-11T22:05:23.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:23 smithi144 ceph-mon[33984]: Adjusting osd_memory_target on smithi045 to 16069M 2024-03-11T22:05:23.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:23 smithi144 ceph-mon[33984]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:23.400 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:23 smithi045 ceph-mon[30178]: Detected new or changed devices on smithi045 2024-03-11T22:05:23.400 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:23 smithi045 ceph-mon[30178]: Adjusting osd_memory_target on smithi045 to 16069M 2024-03-11T22:05:23.400 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:23 smithi045 ceph-mon[30178]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:23.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:23 smithi045 ceph-mon[39715]: Detected new or changed devices on smithi045 2024-03-11T22:05:23.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:23 smithi045 ceph-mon[39715]: Adjusting osd_memory_target on smithi045 to 16069M 2024-03-11T22:05:23.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:23 smithi045 ceph-mon[39715]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:24 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:05:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:24 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:05:24.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:24 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:24.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:24 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:05:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:24 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:05:24.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:24 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:24.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:24 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:05:24.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:24 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:05:24.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:24 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:25 smithi045 ceph-mon[30178]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:05:25.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:25 smithi045 ceph-mon[30178]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:25 smithi045 ceph-mon[39715]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:05:25.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:25 smithi045 ceph-mon[39715]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:25.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:25 smithi144 ceph-mon[33984]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:05:25.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:25 smithi144 ceph-mon[33984]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:26 smithi045 ceph-mon[30178]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:26 smithi045 ceph-mon[39715]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:27.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:26 smithi144 ceph-mon[33984]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:27 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3463179869' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60274eeb-591a-43fe-a579-f13d13631e14"}]: dispatch 2024-03-11T22:05:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:27 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3463179869' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60274eeb-591a-43fe-a579-f13d13631e14"}]': finished 2024-03-11T22:05:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:27 smithi045 ceph-mon[30178]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T22:05:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:27 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:27 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/559889801' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:05:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:27 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3463179869' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60274eeb-591a-43fe-a579-f13d13631e14"}]: dispatch 2024-03-11T22:05:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:27 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3463179869' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60274eeb-591a-43fe-a579-f13d13631e14"}]': finished 2024-03-11T22:05:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:27 smithi045 ceph-mon[39715]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T22:05:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:27 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:28.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:27 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/559889801' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:05:28.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:27 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3463179869' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60274eeb-591a-43fe-a579-f13d13631e14"}]: dispatch 2024-03-11T22:05:28.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:27 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3463179869' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60274eeb-591a-43fe-a579-f13d13631e14"}]': finished 2024-03-11T22:05:28.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:27 smithi144 ceph-mon[33984]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T22:05:28.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:27 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:28.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:27 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/559889801' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:05:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:29 smithi045 ceph-mon[30178]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:29.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:29 smithi045 ceph-mon[39715]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:29.338 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:29 smithi144 ceph-mon[33984]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:30.750 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:30 smithi045 ceph-mon[30178]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:30.750 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:30 smithi045 ceph-mon[39715]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:31.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:30 smithi144 ceph-mon[33984]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:32.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:32 smithi045 ceph-mon[30178]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:32.683 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:32 smithi045 ceph-mon[39715]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:33.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:32 smithi144 ceph-mon[33984]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:34.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:05:34.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:34.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:33 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:05:34.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:33 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:34.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:05:34.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:34 smithi045 ceph-mon[30178]: Deploying daemon osd.1 on smithi045 2024-03-11T22:05:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:34 smithi045 ceph-mon[30178]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:34 smithi045 ceph-mon[39715]: Deploying daemon osd.1 on smithi045 2024-03-11T22:05:35.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:34 smithi045 ceph-mon[39715]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:35.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:34 smithi144 ceph-mon[33984]: Deploying daemon osd.1 on smithi045 2024-03-11T22:05:35.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:34 smithi144 ceph-mon[33984]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:36.706 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:36 smithi045 ceph-mon[39715]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:36.707 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:36 smithi045 ceph-mon[30178]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:37.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:36 smithi144 ceph-mon[33984]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:38.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:38 smithi045 ceph-mon[30178]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:38.915 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:38 smithi045 ceph-mon[39715]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:39.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:38 smithi144 ceph-mon[33984]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:40 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:40 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:40 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:05:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:40 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:40 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:40 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:05:40.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:40 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:40.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:40 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:40.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:40 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:05:41.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:41 smithi045 ceph-mon[30178]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:41 smithi045 ceph-mon[39715]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:41.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:41 smithi144 ceph-mon[33984]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:42.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:42 smithi045 ceph-mon[30178]: from='osd.1 [v2:172.21.15.45:6810/1241486091,v1:172.21.15.45:6811/1241486091]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:05:42.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:42 smithi045 ceph-mon[30178]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:05:42.615 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:42 smithi045 ceph-mon[39715]: from='osd.1 [v2:172.21.15.45:6810/1241486091,v1:172.21.15.45:6811/1241486091]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:05:42.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:42 smithi045 ceph-mon[39715]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:05:42.705 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 1 on host 'smithi045' 2024-03-11T22:05:42.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:42 smithi144 ceph-mon[33984]: from='osd.1 [v2:172.21.15.45:6810/1241486091,v1:172.21.15.45:6811/1241486091]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:05:42.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:42 smithi144 ceph-mon[33984]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:05:43.334 DEBUG:teuthology.orchestra.run.smithi045:osd.1> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T22:05:43.336 INFO:tasks.cephadm:Deploying osd.2 on smithi045 with /dev/vg_nvme/lv_2... 2024-03-11T22:05:43.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T22:05:43.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[39715]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:05:43.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[39715]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:05:43.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:43.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[39715]: from='osd.1 [v2:172.21.15.45:6810/1241486091,v1:172.21.15.45:6811/1241486091]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:05:43.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[39715]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:05:43.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[39715]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:43.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:43.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:43.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:43.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:43.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[30178]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:05:43.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[30178]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:05:43.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:43.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[30178]: from='osd.1 [v2:172.21.15.45:6810/1241486091,v1:172.21.15.45:6811/1241486091]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:05:43.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[30178]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:05:43.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[30178]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:43.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:43.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:43.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:43.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:43.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:43 smithi144 ceph-mon[33984]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:05:43.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:43 smithi144 ceph-mon[33984]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:05:43.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:43 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:43.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:43 smithi144 ceph-mon[33984]: from='osd.1 [v2:172.21.15.45:6810/1241486091,v1:172.21.15.45:6811/1241486091]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:05:43.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:43 smithi144 ceph-mon[33984]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:05:43.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:43 smithi144 ceph-mon[33984]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:43.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:43 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:43.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:43 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:43.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:43 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:43.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:43 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:43.865 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1[50343]: 2024-03-11T22:05:43.709+0000 7ff2d646c700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T22:05:43.865 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:05:43 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1[50343]: 2024-03-11T22:05:43.712+0000 7ff2ce257700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:05:44.106 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:05:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:44 smithi045 ceph-mon[30178]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-11T22:05:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:44 smithi045 ceph-mon[30178]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:05:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:44.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:44 smithi045 ceph-mon[30178]: from='osd.1 ' entity='osd.1' 2024-03-11T22:05:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:44 smithi045 ceph-mon[39715]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-11T22:05:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:44 smithi045 ceph-mon[39715]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:05:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:44.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:44 smithi045 ceph-mon[39715]: from='osd.1 ' entity='osd.1' 2024-03-11T22:05:44.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:44 smithi144 ceph-mon[33984]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-11T22:05:44.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:44 smithi144 ceph-mon[33984]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:05:44.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:44.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:44.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:44 smithi144 ceph-mon[33984]: from='osd.1 ' entity='osd.1' 2024-03-11T22:05:45.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:05:45.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:05:45.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:45.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[39715]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:45.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[39715]: osd.1 [v2:172.21.15.45:6810/1241486091,v1:172.21.15.45:6811/1241486091] boot 2024-03-11T22:05:45.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[39715]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T22:05:45.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:45.727 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:05:45.727 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:05:45.727 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:45.727 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[30178]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:45.727 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[30178]: osd.1 [v2:172.21.15.45:6810/1241486091,v1:172.21.15.45:6811/1241486091] boot 2024-03-11T22:05:45.727 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[30178]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T22:05:45.728 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:45 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:45.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:45 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:05:45.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:45 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:05:45.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:45 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:45.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:45 smithi144 ceph-mon[33984]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:05:45.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:45 smithi144 ceph-mon[33984]: osd.1 [v2:172.21.15.45:6810/1241486091,v1:172.21.15.45:6811/1241486091] boot 2024-03-11T22:05:45.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:45 smithi144 ceph-mon[33984]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T22:05:45.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:45 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:05:47.579 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T22:05:47.579 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T22:05:47.579 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-03-11T22:05:47.579 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-03-11T22:05:47.579 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00839353 s, 1.2 GB/s 2024-03-11T22:05:47.579 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-03-11T22:05:47.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:05:47.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:47.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:05:47.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:05:47.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:05:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:05:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:05:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:05:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:05:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:05:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:05:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:05:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:05:47.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:05:47.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:05:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:05:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:05:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:05:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:05:47.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:05:48.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_2 2024-03-11T22:05:48.423 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:05:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:48 smithi045 ceph-mon[30178]: Detected new or changed devices on smithi045 2024-03-11T22:05:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:48 smithi045 ceph-mon[30178]: Adjusting osd_memory_target on smithi045 to 8034M 2024-03-11T22:05:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:48 smithi045 ceph-mon[39715]: Detected new or changed devices on smithi045 2024-03-11T22:05:48.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:48 smithi045 ceph-mon[39715]: Adjusting osd_memory_target on smithi045 to 8034M 2024-03-11T22:05:48.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:48 smithi144 ceph-mon[33984]: Detected new or changed devices on smithi045 2024-03-11T22:05:48.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:48 smithi144 ceph-mon[33984]: Adjusting osd_memory_target on smithi045 to 8034M 2024-03-11T22:05:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:49 smithi045 ceph-mon[30178]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:49 smithi045 ceph-mon[39715]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:49.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:49 smithi144 ceph-mon[33984]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:05:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:05:50.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:05:50.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:05:50.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:50.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:05:50.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:05:50.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:05:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:51 smithi045 ceph-mon[30178]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:05:51.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:51 smithi045 ceph-mon[30178]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:51.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:51 smithi045 ceph-mon[39715]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:05:51.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:51 smithi045 ceph-mon[39715]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:51.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:51 smithi144 ceph-mon[33984]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:05:51.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:51 smithi144 ceph-mon[33984]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:52 smithi045 ceph-mon[30178]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:52 smithi045 ceph-mon[39715]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:53.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:52 smithi144 ceph-mon[33984]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:54.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:53 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/308506543' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b"}]: dispatch 2024-03-11T22:05:54.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:53 smithi045 ceph-mon[39715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b"}]: dispatch 2024-03-11T22:05:54.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:53 smithi045 ceph-mon[39715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b"}]': finished 2024-03-11T22:05:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:53 smithi045 ceph-mon[39715]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T22:05:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:53 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:05:54.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:53 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1092501068' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:05:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:53 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/308506543' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b"}]: dispatch 2024-03-11T22:05:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:53 smithi045 ceph-mon[30178]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b"}]: dispatch 2024-03-11T22:05:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:53 smithi045 ceph-mon[30178]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b"}]': finished 2024-03-11T22:05:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:53 smithi045 ceph-mon[30178]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T22:05:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:53 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:05:54.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:53 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1092501068' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:05:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:53 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/308506543' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b"}]: dispatch 2024-03-11T22:05:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:53 smithi144 ceph-mon[33984]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b"}]: dispatch 2024-03-11T22:05:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:53 smithi144 ceph-mon[33984]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b"}]': finished 2024-03-11T22:05:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:53 smithi144 ceph-mon[33984]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T22:05:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:53 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:05:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:53 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1092501068' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:05:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:54 smithi045 ceph-mon[30178]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:54 smithi045 ceph-mon[39715]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:55.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:54 smithi144 ceph-mon[33984]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:56.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:56 smithi045 ceph-mon[30178]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:56.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:56 smithi045 ceph-mon[39715]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:57.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:56 smithi144 ceph-mon[33984]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:58.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:58 smithi045 ceph-mon[30178]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:58 smithi045 ceph-mon[39715]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:05:59.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:58 smithi144 ceph-mon[33984]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:00.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:59 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:06:00.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:59 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:00.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:05:59 smithi045 ceph-mon[39715]: Deploying daemon osd.2 on smithi045 2024-03-11T22:06:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:59 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:06:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:59 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:00.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:05:59 smithi045 ceph-mon[30178]: Deploying daemon osd.2 on smithi045 2024-03-11T22:06:00.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:59 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:06:00.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:59 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:00.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:05:59 smithi144 ceph-mon[33984]: Deploying daemon osd.2 on smithi045 2024-03-11T22:06:00.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:00 smithi045 ceph-mon[30178]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:00.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:00 smithi045 ceph-mon[39715]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:01.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:00 smithi144 ceph-mon[33984]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:02.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:02 smithi045 ceph-mon[30178]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:02.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:02 smithi045 ceph-mon[39715]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:03.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:02 smithi144 ceph-mon[33984]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:04.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:04 smithi045 ceph-mon[30178]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:04.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:04 smithi045 ceph-mon[39715]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:05.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:04 smithi144 ceph-mon[33984]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:06.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:06.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:06.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:07.396 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:07 smithi045 ceph-mon[39715]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:07.397 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:07 smithi045 ceph-mon[30178]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:07.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:07 smithi144 ceph-mon[33984]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:09.063 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 2 on host 'smithi045' 2024-03-11T22:06:09.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[30178]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:06:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[30178]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:09.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[39715]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:06:09.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[39715]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:09.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:09.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:09.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:09 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:09.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:09 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:09 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:09 smithi144 ceph-mon[33984]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:06:09.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:09 smithi144 ceph-mon[33984]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:09.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:09 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:09.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:09 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:09.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:09 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:09 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:09 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:09.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:09 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:09.754 DEBUG:teuthology.orchestra.run.smithi045:osd.2> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T22:06:09.757 INFO:tasks.cephadm:Deploying osd.3 on smithi045 with /dev/vg_nvme/lv_1... 2024-03-11T22:06:09.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T22:06:10.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:10 smithi045 ceph-mon[39715]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:06:10.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:10 smithi045 ceph-mon[39715]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:06:10.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:10 smithi045 ceph-mon[39715]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:06:10.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:10 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:10.248 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:10 smithi045 ceph-mon[30178]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:06:10.248 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:10 smithi045 ceph-mon[30178]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:06:10.249 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:10 smithi045 ceph-mon[30178]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:06:10.249 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:10 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:10.525 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:06:10 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2[56488]: 2024-03-11T22:06:10.478+0000 7fbcac827700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T22:06:10.525 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:06:10 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2[56488]: 2024-03-11T22:06:10.481+0000 7fbca4612700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:06:10.587 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:06:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:10 smithi144 ceph-mon[33984]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:06:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:10 smithi144 ceph-mon[33984]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:06:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:10 smithi144 ceph-mon[33984]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:06:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:10 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:11.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[39715]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-11T22:06:11.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[39715]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:06:11.303 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:11.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:11.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[39715]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' 2024-03-11T22:06:11.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[39715]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:11.304 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:11.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[30178]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-11T22:06:11.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[30178]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:06:11.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:11.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:11.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[30178]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' 2024-03-11T22:06:11.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[30178]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:11.304 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:11 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:11 smithi144 ceph-mon[33984]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-11T22:06:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:11 smithi144 ceph-mon[33984]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:06:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:11 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:11 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:11 smithi144 ceph-mon[33984]: from='osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518]' entity='osd.2' 2024-03-11T22:06:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:11 smithi144 ceph-mon[33984]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:06:11.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:11 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:12 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:06:12.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:12 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:06:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:12 smithi045 ceph-mon[30178]: osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518] boot 2024-03-11T22:06:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:12 smithi045 ceph-mon[30178]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T22:06:12.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:12 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:12 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:06:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:12 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:06:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:12 smithi045 ceph-mon[39715]: osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518] boot 2024-03-11T22:06:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:12 smithi045 ceph-mon[39715]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T22:06:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:12 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:12.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:12 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:06:12.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:12 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:06:12.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:12 smithi144 ceph-mon[33984]: osd.2 [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518] boot 2024-03-11T22:06:12.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:12 smithi144 ceph-mon[33984]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T22:06:12.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:12 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:06:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:13 smithi045 ceph-mon[30178]: pgmap v78: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:13.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:13 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:06:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:13 smithi045 ceph-mon[39715]: pgmap v78: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:13.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:13 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:06:13.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:13 smithi144 ceph-mon[33984]: pgmap v78: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:13.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:13 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:06:14.362 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T22:06:14.362 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T22:06:14.362 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2024-03-11T22:06:14.362 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2024-03-11T22:06:14.362 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0084037 s, 1.2 GB/s 2024-03-11T22:06:14.362 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2024-03-11T22:06:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T22:06:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[30178]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:06:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:06:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[30178]: Detected new or changed devices on smithi045 2024-03-11T22:06:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[39715]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[39715]: Detected new or changed devices on smithi045 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:14 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:14.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:14 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T22:06:14.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:14 smithi144 ceph-mon[33984]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:06:14.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:14 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:06:14.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:14 smithi144 ceph-mon[33984]: Detected new or changed devices on smithi045 2024-03-11T22:06:14.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:14 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:14.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:14 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:14.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:14 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:14.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:14 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:14.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:14 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:14.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:14 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:14.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:14 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:14.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:14 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:14.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:14 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:14.992 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_1 2024-03-11T22:06:15.276 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:06:15.525 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:06:15 smithi045 sudo[60881]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:06:15.525 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:06:15 smithi045 sudo[60881]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:06:15.525 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:06:15 smithi045 sudo[60881]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:06:15.525 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:06:15 smithi045 sudo[60881]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:06:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:15 smithi045 ceph-mon[30178]: Adjusting osd_memory_target on smithi045 to 5356M 2024-03-11T22:06:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:15 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:06:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:15 smithi045 ceph-mon[30178]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:06:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:15 smithi045 ceph-mon[30178]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:15 smithi045 ceph-mon[39715]: Adjusting osd_memory_target on smithi045 to 5356M 2024-03-11T22:06:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:15 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:06:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:15 smithi045 ceph-mon[39715]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:06:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:15 smithi045 ceph-mon[39715]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:15.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:15 smithi144 ceph-mon[33984]: Adjusting osd_memory_target on smithi045 to 5356M 2024-03-11T22:06:15.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:15 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:06:15.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:15 smithi144 ceph-mon[33984]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:06:15.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:15 smithi144 ceph-mon[33984]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:15.826 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:06:15 smithi045 sudo[60904]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:06:15.826 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:06:15 smithi045 sudo[60904]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:06:15.826 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:06:15 smithi045 sudo[60904]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:06:15.826 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:06:15 smithi045 sudo[60904]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:06:16.204 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:06:15 smithi045 sudo[60925]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:06:16.204 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:06:15 smithi045 sudo[60925]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:06:16.205 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:06:15 smithi045 sudo[60925]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:06:16.205 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:06:16 smithi045 sudo[60925]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:06:16.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:16 smithi045 ceph-mon[39715]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:06:16.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:16 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:16.480 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:16 smithi045 ceph-mon[30178]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:06:16.480 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:16 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:16.480 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:06:16 smithi045 sudo[60936]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:06:16.480 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:06:16 smithi045 sudo[60936]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:06:16.480 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:06:16 smithi045 sudo[60936]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:06:16.481 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:06:16 smithi045 sudo[60936]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:06:16.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:16 smithi144 ceph-mon[33984]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:06:16.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:16 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:16.770 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:06:16 smithi045 sudo[60942]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:06:16.770 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:06:16 smithi045 sudo[60942]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:06:16.770 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:06:16 smithi045 sudo[60942]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:06:16.771 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:06:16 smithi045 sudo[60942]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:06:17.025 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:06:16 smithi045 sudo[60976]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:06:17.025 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:06:16 smithi045 sudo[60976]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:06:17.025 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:06:16 smithi045 sudo[60976]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:06:17.025 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:06:16 smithi045 sudo[60976]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:06:17.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:17 smithi045 sudo[61013]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:06:17.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:17 smithi045 sudo[61013]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:06:17.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:17 smithi045 sudo[61013]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:06:17.340 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:17 smithi045 sudo[61013]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:06:17.609 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:17 smithi144 ceph-mon[33984]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T22:06:17.609 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:17 smithi144 ceph-mon[33984]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 93 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:17.609 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:17 smithi144 ceph-mon[33984]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:06:17.609 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:17 smithi144 sudo[36809]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:06:17.609 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:17 smithi144 sudo[36809]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:06:17.609 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:17 smithi144 sudo[36809]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:06:17.609 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:17 smithi144 sudo[36809]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:06:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:17 smithi045 ceph-mon[39715]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T22:06:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:17 smithi045 ceph-mon[39715]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 93 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:17.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:17 smithi045 ceph-mon[39715]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:06:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:17 smithi045 sudo[61017]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:06:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:17 smithi045 sudo[61017]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:06:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:17 smithi045 sudo[61017]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:06:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:17 smithi045 sudo[61017]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:06:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:17 smithi045 ceph-mon[30178]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T22:06:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:17 smithi045 ceph-mon[30178]: pgmap v83: 1 pgs: 1 active+clean; 449 KiB data, 93 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:17.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:17 smithi045 ceph-mon[30178]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:06:18.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:17 smithi144 sudo[36812]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:06:18.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:17 smithi144 sudo[36812]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:06:18.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:17 smithi144 sudo[36812]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:06:18.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:17 smithi144 sudo[36812]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:06:18.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 sudo[61162]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:06:18.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 sudo[61162]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:06:18.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 sudo[61162]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:06:18.245 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 sudo[61162]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:06:18.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:06:18.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:06:18.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:06:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:06:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 sudo[61184]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:06:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 sudo[61184]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:06:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 sudo[61184]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:06:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 sudo[61184]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:06:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:06:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:06:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:06:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:06:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:06:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:06:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:06:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:06:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:06:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:06:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:06:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:06:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:06:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:06:18.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:18 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:06:18.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:06:18.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:06:18.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:06:18.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:18 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:06:19.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:19 smithi045 ceph-mon[30178]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:06:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:19 smithi045 ceph-mon[30178]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:06:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:19 smithi045 ceph-mon[30178]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:19 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:06:19.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:19 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:06:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:19 smithi045 ceph-mon[39715]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:06:19.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:19 smithi045 ceph-mon[39715]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:06:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:19 smithi045 ceph-mon[39715]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:19 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:06:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:19 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:06:19.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:19 smithi144 ceph-mon[33984]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:06:19.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:19 smithi144 ceph-mon[33984]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:06:19.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:19 smithi144 ceph-mon[33984]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:19.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:19 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:06:19.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:19 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:06:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:20 smithi045 ceph-mon[30178]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:06:20.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:20 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4027441604' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e51a65f3-96f6-478c-8d26-df1ace4a5058"}]: dispatch 2024-03-11T22:06:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:20 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4027441604' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e51a65f3-96f6-478c-8d26-df1ace4a5058"}]': finished 2024-03-11T22:06:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:20 smithi045 ceph-mon[30178]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T22:06:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:20 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:20 smithi045 ceph-mon[39715]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:06:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:20 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4027441604' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e51a65f3-96f6-478c-8d26-df1ace4a5058"}]: dispatch 2024-03-11T22:06:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:20 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4027441604' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e51a65f3-96f6-478c-8d26-df1ace4a5058"}]': finished 2024-03-11T22:06:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:20 smithi045 ceph-mon[39715]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T22:06:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:20 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:20.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:20 smithi144 ceph-mon[33984]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:06:20.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:20 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4027441604' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e51a65f3-96f6-478c-8d26-df1ace4a5058"}]: dispatch 2024-03-11T22:06:20.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:20 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4027441604' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e51a65f3-96f6-478c-8d26-df1ace4a5058"}]': finished 2024-03-11T22:06:20.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:20 smithi144 ceph-mon[33984]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T22:06:20.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:20 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:21 smithi045 ceph-mon[30178]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 326 KiB/s wr, 22 op/s 2024-03-11T22:06:21.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:21 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2038435037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:06:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:21 smithi045 ceph-mon[39715]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 326 KiB/s wr, 22 op/s 2024-03-11T22:06:21.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:21 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2038435037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:06:21.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:21 smithi144 ceph-mon[33984]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 326 KiB/s wr, 22 op/s 2024-03-11T22:06:21.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:21 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2038435037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:06:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:22 smithi045 ceph-mon[30178]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-11T22:06:23.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:22 smithi045 ceph-mon[39715]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-11T22:06:23.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:22 smithi144 ceph-mon[33984]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-03-11T22:06:24.754 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:24 smithi045 ceph-mon[30178]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-11T22:06:24.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:24 smithi045 ceph-mon[39715]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-11T22:06:25.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:24 smithi144 ceph-mon[33984]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-11T22:06:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:27 smithi045 ceph-mon[30178]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-11T22:06:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:27 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T22:06:27.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:27 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:27 smithi045 ceph-mon[39715]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-11T22:06:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:27 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T22:06:27.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:27 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:27.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:27 smithi144 ceph-mon[33984]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-11T22:06:27.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:27 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T22:06:27.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:27 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:28.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:28 smithi045 ceph-mon[30178]: Deploying daemon osd.3 on smithi045 2024-03-11T22:06:28.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:28 smithi045 ceph-mon[39715]: Deploying daemon osd.3 on smithi045 2024-03-11T22:06:28.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:28 smithi144 ceph-mon[33984]: Deploying daemon osd.3 on smithi045 2024-03-11T22:06:29.269 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:29 smithi045 ceph-mon[30178]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.9 KiB/s rd, 143 KiB/s wr, 9 op/s 2024-03-11T22:06:29.269 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:29 smithi045 ceph-mon[39715]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.9 KiB/s rd, 143 KiB/s wr, 9 op/s 2024-03-11T22:06:29.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:29 smithi144 ceph-mon[33984]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.9 KiB/s rd, 143 KiB/s wr, 9 op/s 2024-03-11T22:06:30.964 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:30 smithi045 ceph-mon[30178]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 126 KiB/s wr, 8 op/s 2024-03-11T22:06:30.964 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:30 smithi045 ceph-mon[39715]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 126 KiB/s wr, 8 op/s 2024-03-11T22:06:31.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:30 smithi144 ceph-mon[33984]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 126 KiB/s wr, 8 op/s 2024-03-11T22:06:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:33 smithi045 ceph-mon[30178]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:33 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:33 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:33 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:33.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:33 smithi045 ceph-mon[39715]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:33.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:33.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:33.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:33.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:33 smithi144 ceph-mon[33984]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:34.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:34 smithi045 ceph-mon[39715]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:34.843 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:34 smithi045 ceph-mon[30178]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:35.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:34 smithi144 ceph-mon[33984]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:36.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:35 smithi045 ceph-mon[30178]: from='osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T22:06:36.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:35 smithi045 ceph-mon[39715]: from='osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T22:06:36.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:35 smithi144 ceph-mon[33984]: from='osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T22:06:36.455 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 3 on host 'smithi045' 2024-03-11T22:06:36.679 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[39715]: from='osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T22:06:36.679 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[39715]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T22:06:36.679 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[39715]: from='osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:06:36.679 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[30178]: from='osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T22:06:36.679 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[30178]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T22:06:36.679 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[30178]: from='osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:06:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[30178]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:37.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:36 smithi045 ceph-mon[39715]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:37.057 DEBUG:teuthology.orchestra.run.smithi045:osd.3> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.3.service 2024-03-11T22:06:37.059 INFO:tasks.cephadm:Deploying osd.4 on smithi144 with /dev/vg_nvme/lv_4... 2024-03-11T22:06:37.059 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T22:06:37.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:36 smithi144 ceph-mon[33984]: from='osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T22:06:37.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:36 smithi144 ceph-mon[33984]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T22:06:37.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:36 smithi144 ceph-mon[33984]: from='osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2024-03-11T22:06:37.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:36 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:37.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:36 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:36 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:36 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:37.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:36 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:37.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:36 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:36 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:36 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:37.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:36 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:37.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:36 smithi144 ceph-mon[33984]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:37.302 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:06:37.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:37 smithi045 ceph-mon[39715]: from='osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-11T22:06:37.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:37 smithi045 ceph-mon[39715]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T22:06:37.998 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:37.998 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:37 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:37.998 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:37 smithi045 ceph-mon[30178]: from='osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-11T22:06:37.998 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:37 smithi045 ceph-mon[30178]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T22:06:37.998 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:37 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:37.998 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:37 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:38.021 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:37 smithi144 ceph-mon[33984]: from='osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-03-11T22:06:38.021 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:37 smithi144 ceph-mon[33984]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T22:06:38.021 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:37 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:38.021 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:37 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:38 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:06:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:38 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:06:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:38 smithi045 ceph-mon[30178]: osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830] boot 2024-03-11T22:06:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:38 smithi045 ceph-mon[30178]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T22:06:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:38 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:38 smithi045 ceph-mon[30178]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:38 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:06:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:38 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:06:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:38 smithi045 ceph-mon[39715]: osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830] boot 2024-03-11T22:06:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:38 smithi045 ceph-mon[39715]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T22:06:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:38 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:39.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:38 smithi045 ceph-mon[39715]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:39.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:38 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:06:39.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:38 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:06:39.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:38 smithi144 ceph-mon[33984]: osd.3 [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830] boot 2024-03-11T22:06:39.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:38 smithi144 ceph-mon[33984]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T22:06:39.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:38 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:06:39.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:38 smithi144 ceph-mon[33984]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:06:39.157 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T22:06:39.158 INFO:teuthology.orchestra.run.smithi144.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T22:06:39.158 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10+0 records in 2024-03-11T22:06:39.158 INFO:teuthology.orchestra.run.smithi144.stderr:10+0 records out 2024-03-11T22:06:39.158 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113948 s, 920 MB/s 2024-03-11T22:06:39.158 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping successful for: 2024-03-11T22:06:39.742 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:39 smithi144 ceph-mon[33984]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T22:06:39.768 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi144:vg_nvme/lv_4 2024-03-11T22:06:39.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:39 smithi045 ceph-mon[39715]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T22:06:39.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:39 smithi045 ceph-mon[30178]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T22:06:40.045 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:06:41.419 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:41 smithi144 ceph-mon[33984]: Detected new or changed devices on smithi045 2024-03-11T22:06:41.420 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:41.420 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:41.420 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:41.420 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:41.420 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:41.420 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:41.420 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:41 smithi144 ceph-mon[33984]: Adjusting osd_memory_target on smithi045 to 4017M 2024-03-11T22:06:41.420 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:41.420 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:41.420 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:41.420 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:41.421 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:41 smithi144 ceph-mon[33984]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[30178]: Detected new or changed devices on smithi045 2024-03-11T22:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[30178]: Adjusting osd_memory_target on smithi045 to 4017M 2024-03-11T22:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[30178]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[39715]: Detected new or changed devices on smithi045 2024-03-11T22:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[39715]: Adjusting osd_memory_target on smithi045 to 4017M 2024-03-11T22:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:41 smithi045 ceph-mon[39715]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:42.446 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:06:42.446 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:06:42.446 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:06:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:06:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:06:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:06:42.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:43.443 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:43 smithi045 ceph-mon[39715]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:06:43.443 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:43 smithi045 ceph-mon[39715]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:43.443 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:43 smithi045 ceph-mon[30178]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:06:43.443 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:43 smithi045 ceph-mon[30178]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:43.552 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:43 smithi144 ceph-mon[33984]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:06:43.552 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:43 smithi144 ceph-mon[33984]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:44.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:44 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/1002107503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32d0b71f-c696-4677-bdd3-b88a04c97604"}]: dispatch 2024-03-11T22:06:44.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:44 smithi144 ceph-mon[33984]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32d0b71f-c696-4677-bdd3-b88a04c97604"}]: dispatch 2024-03-11T22:06:44.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:44 smithi144 ceph-mon[33984]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32d0b71f-c696-4677-bdd3-b88a04c97604"}]': finished 2024-03-11T22:06:44.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:44 smithi144 ceph-mon[33984]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T22:06:44.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:44 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:06:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:44 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/1002107503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32d0b71f-c696-4677-bdd3-b88a04c97604"}]: dispatch 2024-03-11T22:06:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:44 smithi045 ceph-mon[30178]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32d0b71f-c696-4677-bdd3-b88a04c97604"}]: dispatch 2024-03-11T22:06:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:44 smithi045 ceph-mon[30178]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32d0b71f-c696-4677-bdd3-b88a04c97604"}]': finished 2024-03-11T22:06:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:44 smithi045 ceph-mon[30178]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T22:06:45.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:44 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:06:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:44 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/1002107503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32d0b71f-c696-4677-bdd3-b88a04c97604"}]: dispatch 2024-03-11T22:06:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:44 smithi045 ceph-mon[39715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32d0b71f-c696-4677-bdd3-b88a04c97604"}]: dispatch 2024-03-11T22:06:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:44 smithi045 ceph-mon[39715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32d0b71f-c696-4677-bdd3-b88a04c97604"}]': finished 2024-03-11T22:06:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:44 smithi045 ceph-mon[39715]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T22:06:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:44 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:06:45.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:45 smithi144 ceph-mon[33984]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:45.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:45 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/1179612819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:06:46.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:45 smithi045 ceph-mon[30178]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:46.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:45 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/1179612819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:06:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:45 smithi045 ceph-mon[39715]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:46.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:45 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/1179612819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:06:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:47 smithi045 ceph-mon[39715]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:47.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:47 smithi045 ceph-mon[30178]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:47.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:47 smithi144 ceph-mon[33984]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:48.767 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:48 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:06:48.767 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:48 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:06:48.767 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:48 smithi144 ceph-mon[33984]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T22:06:48.767 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:48 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:06:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:48 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:06:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:48 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:06:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:48 smithi045 ceph-mon[39715]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T22:06:48.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:48 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:06:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:48 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:06:48.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:48 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:06:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:48 smithi045 ceph-mon[30178]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T22:06:48.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:48 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:06:49.762 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:49 smithi144 ceph-mon[33984]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:49.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:49 smithi045 ceph-mon[30178]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:49 smithi045 ceph-mon[39715]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:50.861 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:50 smithi144 ceph-mon[33984]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:51.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:50 smithi045 ceph-mon[30178]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:51.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:50 smithi045 ceph-mon[39715]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:51.753 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:51 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T22:06:51.754 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:51 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:51.754 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:51 smithi144 ceph-mon[33984]: Deploying daemon osd.4 on smithi144 2024-03-11T22:06:52.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:51 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T22:06:52.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:51 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:52.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:51 smithi045 ceph-mon[30178]: Deploying daemon osd.4 on smithi144 2024-03-11T22:06:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:51 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T22:06:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:51 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:51 smithi045 ceph-mon[39715]: Deploying daemon osd.4 on smithi144 2024-03-11T22:06:52.800 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:52 smithi144 ceph-mon[33984]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:53.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:52 smithi045 ceph-mon[30178]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:52 smithi045 ceph-mon[39715]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:54.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:54 smithi144 ceph-mon[33984]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:55.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:54 smithi045 ceph-mon[30178]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:55.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:54 smithi045 ceph-mon[39715]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:57.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:57 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:57.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:57 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:57.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:57 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:57.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:57 smithi144 ceph-mon[33984]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:57 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:57 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:57 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:57.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:57 smithi045 ceph-mon[39715]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:57 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:57 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:57 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:06:57.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:57 smithi045 ceph-mon[30178]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:58.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:58.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:58.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:58.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:58.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:59.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:58 smithi045 ceph-mon[39715]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:06:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:06:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:06:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:58 smithi045 ceph-mon[30178]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:06:59.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:58 smithi144 ceph-mon[33984]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:07:00.042 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:59 smithi144 ceph-mon[33984]: from='osd.4 [v2:172.21.15.144:6800/912745159,v1:172.21.15.144:6801/912745159]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:07:00.042 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:06:59 smithi144 ceph-mon[33984]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:07:00.108 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 4 on host 'smithi144' 2024-03-11T22:07:00.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:59 smithi045 ceph-mon[30178]: from='osd.4 [v2:172.21.15.144:6800/912745159,v1:172.21.15.144:6801/912745159]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:07:00.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:06:59 smithi045 ceph-mon[30178]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:07:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:59 smithi045 ceph-mon[39715]: from='osd.4 [v2:172.21.15.144:6800/912745159,v1:172.21.15.144:6801/912745159]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:07:00.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:06:59 smithi045 ceph-mon[39715]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:07:00.713 DEBUG:teuthology.orchestra.run.smithi144:osd.4> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.4.service 2024-03-11T22:07:00.716 INFO:tasks.cephadm:Deploying osd.5 on smithi144 with /dev/vg_nvme/lv_3... 2024-03-11T22:07:00.716 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T22:07:01.008 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:00 smithi144 ceph-mon[33984]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T22:07:01.008 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:00 smithi144 ceph-mon[33984]: from='osd.4 [v2:172.21.15.144:6800/912745159,v1:172.21.15.144:6801/912745159]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:01.008 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:00 smithi144 ceph-mon[33984]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T22:07:01.008 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:00 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:01.008 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:00 smithi144 ceph-mon[33984]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:01.008 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:00 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:01.008 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:00 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:01.008 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:00 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:01.008 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:00 smithi144 ceph-mon[33984]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:07:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[39715]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T22:07:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[39715]: from='osd.4 [v2:172.21.15.144:6800/912745159,v1:172.21.15.144:6801/912745159]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[39715]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T22:07:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:01.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[39715]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[39715]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:07:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[30178]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T22:07:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[30178]: from='osd.4 [v2:172.21.15.144:6800/912745159,v1:172.21.15.144:6801/912745159]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[30178]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T22:07:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[30178]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:01.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:00 smithi045 ceph-mon[30178]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:07:01.466 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:07:01.588 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:07:01 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:07:01.296+0000 7fe8965c7700 -1 osd.4 0 waiting for initial osdmap 2024-03-11T22:07:01.588 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:07:01 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:07:01.302+0000 7fe88e3b2700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:07:02.109 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:01 smithi144 ceph-mon[33984]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-03-11T22:07:02.109 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:01 smithi144 ceph-mon[33984]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T22:07:02.110 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:01 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:02.110 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:01 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:02.110 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:01 smithi144 ceph-mon[33984]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T22:07:02.110 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:01 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:02.110 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:01 smithi144 ceph-mon[33984]: from='osd.4 ' entity='osd.4' 2024-03-11T22:07:02.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[30178]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-03-11T22:07:02.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[30178]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T22:07:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[30178]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T22:07:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:02.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[30178]: from='osd.4 ' entity='osd.4' 2024-03-11T22:07:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[39715]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-03-11T22:07:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[39715]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T22:07:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[39715]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T22:07:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:02.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:01 smithi045 ceph-mon[39715]: from='osd.4 ' entity='osd.4' 2024-03-11T22:07:03.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:02 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:07:03.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:02 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:07:03.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:02 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:03.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:02 smithi144 ceph-mon[33984]: osd.4 [v2:172.21.15.144:6800/912745159,v1:172.21.15.144:6801/912745159] boot 2024-03-11T22:07:03.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:02 smithi144 ceph-mon[33984]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T22:07:03.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:02 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:03.114 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:02 smithi144 ceph-mon[33984]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:07:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:07:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:07:03.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[39715]: osd.4 [v2:172.21.15.144:6800/912745159,v1:172.21.15.144:6801/912745159] boot 2024-03-11T22:07:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[39715]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T22:07:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[39715]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:07:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:07:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:07:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[30178]: osd.4 [v2:172.21.15.144:6800/912745159,v1:172.21.15.144:6801/912745159] boot 2024-03-11T22:07:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[30178]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T22:07:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:07:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:02 smithi045 ceph-mon[30178]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-11T22:07:04.152 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T22:07:04.152 INFO:teuthology.orchestra.run.smithi144.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T22:07:04.152 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10+0 records in 2024-03-11T22:07:04.152 INFO:teuthology.orchestra.run.smithi144.stderr:10+0 records out 2024-03-11T22:07:04.152 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126232 s, 831 MB/s 2024-03-11T22:07:04.152 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping successful for: 2024-03-11T22:07:04.389 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:04 smithi144 ceph-mon[33984]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T22:07:04.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:04 smithi045 ceph-mon[30178]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T22:07:04.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:04 smithi045 ceph-mon[39715]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T22:07:04.988 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi144:vg_nvme/lv_3 2024-03-11T22:07:05.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:05 smithi144 ceph-mon[33984]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T22:07:05.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:05 smithi144 ceph-mon[33984]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:05.419 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:07:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:05 smithi045 ceph-mon[39715]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T22:07:05.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:05 smithi045 ceph-mon[39715]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:05 smithi045 ceph-mon[30178]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T22:07:05.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:05 smithi045 ceph-mon[30178]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:06.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:06 smithi144 ceph-mon[33984]: osdmap e36: 5 total, 5 up, 5 in 2024-03-11T22:07:06.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:06 smithi144 ceph-mon[33984]: Detected new or changed devices on smithi144 2024-03-11T22:07:06.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:06.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:06.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:06.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:06 smithi144 ceph-mon[33984]: Adjusting osd_memory_target on smithi144 to 17088M 2024-03-11T22:07:06.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:06.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:06.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:06.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[30178]: osdmap e36: 5 total, 5 up, 5 in 2024-03-11T22:07:06.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[30178]: Detected new or changed devices on smithi144 2024-03-11T22:07:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[30178]: Adjusting osd_memory_target on smithi144 to 17088M 2024-03-11T22:07:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[39715]: osdmap e36: 5 total, 5 up, 5 in 2024-03-11T22:07:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[39715]: Detected new or changed devices on smithi144 2024-03-11T22:07:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[39715]: Adjusting osd_memory_target on smithi144 to 17088M 2024-03-11T22:07:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:07 smithi045 ceph-mon[30178]: osdmap e37: 5 total, 5 up, 5 in 2024-03-11T22:07:07.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:07 smithi045 ceph-mon[30178]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T22:07:07.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:07 smithi045 ceph-mon[39715]: osdmap e37: 5 total, 5 up, 5 in 2024-03-11T22:07:07.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:07 smithi045 ceph-mon[39715]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T22:07:07.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:07 smithi144 ceph-mon[33984]: osdmap e37: 5 total, 5 up, 5 in 2024-03-11T22:07:07.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:07 smithi144 ceph-mon[33984]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T22:07:08.483 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:08 smithi144 ceph-mon[33984]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:07:08.483 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:08 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:07:08.483 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:08 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:07:08.483 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:08 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:08 smithi045 ceph-mon[30178]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:07:08.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:08 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:07:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:08 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:07:08.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:08 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:08 smithi045 ceph-mon[39715]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:07:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:08 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:07:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:08 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:07:08.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:08 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:09.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:09 smithi144 ceph-mon[33984]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-11T22:07:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:09 smithi045 ceph-mon[30178]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-11T22:07:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:09 smithi045 ceph-mon[39715]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-11T22:07:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:10 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/1041647755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f46b91af-d67d-404c-aad1-75348e35d1c4"}]: dispatch 2024-03-11T22:07:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:10 smithi045 ceph-mon[30178]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f46b91af-d67d-404c-aad1-75348e35d1c4"}]: dispatch 2024-03-11T22:07:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:10 smithi045 ceph-mon[30178]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f46b91af-d67d-404c-aad1-75348e35d1c4"}]': finished 2024-03-11T22:07:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:10 smithi045 ceph-mon[30178]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T22:07:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:10 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:10 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/1041647755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f46b91af-d67d-404c-aad1-75348e35d1c4"}]: dispatch 2024-03-11T22:07:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:10 smithi045 ceph-mon[39715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f46b91af-d67d-404c-aad1-75348e35d1c4"}]: dispatch 2024-03-11T22:07:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:10 smithi045 ceph-mon[39715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f46b91af-d67d-404c-aad1-75348e35d1c4"}]': finished 2024-03-11T22:07:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:10 smithi045 ceph-mon[39715]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T22:07:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:10 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:10 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/1041647755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f46b91af-d67d-404c-aad1-75348e35d1c4"}]: dispatch 2024-03-11T22:07:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:10 smithi144 ceph-mon[33984]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f46b91af-d67d-404c-aad1-75348e35d1c4"}]: dispatch 2024-03-11T22:07:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:10 smithi144 ceph-mon[33984]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f46b91af-d67d-404c-aad1-75348e35d1c4"}]': finished 2024-03-11T22:07:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:10 smithi144 ceph-mon[33984]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T22:07:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:10 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:11 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/677786908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:07:11.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:11 smithi045 ceph-mon[30178]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T22:07:11.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:11 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/677786908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:07:11.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:11 smithi045 ceph-mon[39715]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T22:07:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:11 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/677786908' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:07:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:11 smithi144 ceph-mon[33984]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T22:07:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:12 smithi045 ceph-mon[30178]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-11T22:07:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:12 smithi045 ceph-mon[39715]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-11T22:07:13.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:12 smithi144 ceph-mon[33984]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-11T22:07:14.964 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:14 smithi144 ceph-mon[33984]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:15.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:14 smithi045 ceph-mon[30178]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:15.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:14 smithi045 ceph-mon[39715]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:16.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:16 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T22:07:16.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:16 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:16 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T22:07:16.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:16 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:16.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:16 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T22:07:16.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:16 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:17.536 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:17 smithi144 ceph-mon[33984]: Deploying daemon osd.5 on smithi144 2024-03-11T22:07:17.537 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:17 smithi144 ceph-mon[33984]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:17 smithi045 ceph-mon[30178]: Deploying daemon osd.5 on smithi144 2024-03-11T22:07:17.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:17 smithi045 ceph-mon[30178]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:17 smithi045 ceph-mon[39715]: Deploying daemon osd.5 on smithi144 2024-03-11T22:07:17.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:17 smithi045 ceph-mon[39715]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:18.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:18 smithi144 ceph-mon[33984]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:19.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:18 smithi045 ceph-mon[30178]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:19.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:18 smithi045 ceph-mon[39715]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:20.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:20 smithi144 ceph-mon[33984]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:21.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:20 smithi045 ceph-mon[30178]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:21.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:20 smithi045 ceph-mon[39715]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:23.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:22 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:22 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:22 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:23.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:22 smithi045 ceph-mon[30178]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:22 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:22 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:22 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:23.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:22 smithi045 ceph-mon[39715]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:23.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:22 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:23.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:22 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:23.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:22 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:23.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:22 smithi144 ceph-mon[33984]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:24.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:24 smithi144 ceph-mon[33984]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:25.004 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 5 on host 'smithi144' 2024-03-11T22:07:25.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:24 smithi045 ceph-mon[30178]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:25.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:24 smithi045 ceph-mon[39715]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:25.653 DEBUG:teuthology.orchestra.run.smithi144:osd.5> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.5.service 2024-03-11T22:07:25.655 INFO:tasks.cephadm:Deploying osd.6 on smithi144 with /dev/vg_nvme/lv_2... 2024-03-11T22:07:25.655 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T22:07:25.990 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:07:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:25 smithi045 ceph-mon[30178]: from='osd.5 [v2:172.21.15.144:6808/3162869174,v1:172.21.15.144:6809/3162869174]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:07:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:25 smithi045 ceph-mon[30178]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:07:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:25 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:26.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:25 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:25 smithi045 ceph-mon[39715]: from='osd.5 [v2:172.21.15.144:6808/3162869174,v1:172.21.15.144:6809/3162869174]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:07:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:25 smithi045 ceph-mon[39715]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:07:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:25 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:26.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:25 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:26.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:25 smithi144 ceph-mon[33984]: from='osd.5 [v2:172.21.15.144:6808/3162869174,v1:172.21.15.144:6809/3162869174]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:07:26.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:25 smithi144 ceph-mon[33984]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:07:26.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:25 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:26.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:25 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:26.973 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:26 smithi144 ceph-mon[33984]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T22:07:26.973 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:26 smithi144 ceph-mon[33984]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T22:07:26.973 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:26 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:26.973 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:26 smithi144 ceph-mon[33984]: from='osd.5 [v2:172.21.15.144:6808/3162869174,v1:172.21.15.144:6809/3162869174]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:26.973 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:26 smithi144 ceph-mon[33984]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:26.973 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:26 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:26.974 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:26 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:26.974 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:26 smithi144 ceph-mon[33984]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:26.974 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:07:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:07:26.700+0000 7f8128481700 -1 osd.5 0 waiting for initial osdmap 2024-03-11T22:07:26.974 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:07:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:07:26.707+0000 7f8124274700 -1 osd.5 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:07:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[30178]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T22:07:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[30178]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T22:07:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:27.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[30178]: from='osd.5 [v2:172.21.15.144:6808/3162869174,v1:172.21.15.144:6809/3162869174]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[30178]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:27.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[30178]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[39715]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T22:07:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[39715]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T22:07:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[39715]: from='osd.5 [v2:172.21.15.144:6808/3162869174,v1:172.21.15.144:6809/3162869174]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[39715]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:27.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:26 smithi045 ceph-mon[39715]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-03-11T22:07:27.823 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T22:07:27.823 INFO:teuthology.orchestra.run.smithi144.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T22:07:27.823 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10+0 records in 2024-03-11T22:07:27.823 INFO:teuthology.orchestra.run.smithi144.stderr:10+0 records out 2024-03-11T22:07:27.823 INFO:teuthology.orchestra.run.smithi144.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130222 s, 805 MB/s 2024-03-11T22:07:27.823 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping successful for: 2024-03-11T22:07:27.938 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:27 smithi144 ceph-mon[33984]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-03-11T22:07:27.938 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:27 smithi144 ceph-mon[33984]: osdmap e40: 6 total, 5 up, 6 in 2024-03-11T22:07:27.938 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:27 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:27.938 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:27 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:27.938 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:27 smithi144 ceph-mon[33984]: osd.5 [v2:172.21.15.144:6808/3162869174,v1:172.21.15.144:6809/3162869174] boot 2024-03-11T22:07:27.938 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:27 smithi144 ceph-mon[33984]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T22:07:27.938 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:27 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:28.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[39715]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-03-11T22:07:28.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[39715]: osdmap e40: 6 total, 5 up, 6 in 2024-03-11T22:07:28.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:28.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:28.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[39715]: osd.5 [v2:172.21.15.144:6808/3162869174,v1:172.21.15.144:6809/3162869174] boot 2024-03-11T22:07:28.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[39715]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T22:07:28.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[30178]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-03-11T22:07:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[30178]: osdmap e40: 6 total, 5 up, 6 in 2024-03-11T22:07:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[30178]: osd.5 [v2:172.21.15.144:6808/3162869174,v1:172.21.15.144:6809/3162869174] boot 2024-03-11T22:07:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[30178]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T22:07:28.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:27 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:07:28.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:28 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:07:28.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:28 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:07:28.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:28 smithi144 ceph-mon[33984]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:28.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:28 smithi144 ceph-mon[33984]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T22:07:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:28 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:07:29.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:28 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:07:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:28 smithi045 ceph-mon[30178]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:29.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:28 smithi045 ceph-mon[30178]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T22:07:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:28 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:07:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:28 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:07:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:28 smithi045 ceph-mon[39715]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:29.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:28 smithi045 ceph-mon[39715]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T22:07:29.661 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi144:vg_nvme/lv_2 2024-03-11T22:07:29.912 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:07:30.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:30 smithi144 ceph-mon[33984]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T22:07:30.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:30 smithi144 ceph-mon[33984]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:30 smithi045 ceph-mon[30178]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T22:07:31.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:30 smithi045 ceph-mon[30178]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:30 smithi045 ceph-mon[39715]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T22:07:31.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:30 smithi045 ceph-mon[39715]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:31.919 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T22:07:31.919 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: Detected new or changed devices on smithi144 2024-03-11T22:07:31.919 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:31.919 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:31.920 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:31.920 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:31.920 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: Adjusting osd_memory_target on smithi144 to 8544M 2024-03-11T22:07:31.920 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:31.920 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:31.920 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:31.920 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:31.920 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:31.920 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:31.920 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:31.920 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:31 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:32.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T22:07:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: Detected new or changed devices on smithi144 2024-03-11T22:07:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: Adjusting osd_memory_target on smithi144 to 8544M 2024-03-11T22:07:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:32.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:32.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T22:07:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: Detected new or changed devices on smithi144 2024-03-11T22:07:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: Adjusting osd_memory_target on smithi144 to 8544M 2024-03-11T22:07:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:32.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:32.028 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:31 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:32.962 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:32 smithi144 ceph-mon[33984]: osdmap e45: 6 total, 6 up, 6 in 2024-03-11T22:07:32.962 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:32 smithi144 ceph-mon[33984]: 1.0 scrub starts 2024-03-11T22:07:32.962 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:32 smithi144 ceph-mon[33984]: 1.0 scrub ok 2024-03-11T22:07:32.962 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:32 smithi144 ceph-mon[33984]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:07:32.962 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:32 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:07:32.962 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:32 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:07:32.962 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:32 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:32.962 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:32 smithi144 ceph-mon[33984]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:33.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[30178]: osdmap e45: 6 total, 6 up, 6 in 2024-03-11T22:07:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[30178]: 1.0 scrub starts 2024-03-11T22:07:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[30178]: 1.0 scrub ok 2024-03-11T22:07:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[30178]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:07:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:07:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:07:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:33.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[30178]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[39715]: osdmap e45: 6 total, 6 up, 6 in 2024-03-11T22:07:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[39715]: 1.0 scrub starts 2024-03-11T22:07:33.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[39715]: 1.0 scrub ok 2024-03-11T22:07:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[39715]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:07:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:07:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:07:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:33.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:32 smithi045 ceph-mon[39715]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:35.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:34 smithi045 ceph-mon[30178]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:35.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:34 smithi045 ceph-mon[39715]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:35.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:34 smithi144 ceph-mon[33984]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:35.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:35 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/2927425206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "970b128d-cd56-4dff-8345-3196b46c1257"}]: dispatch 2024-03-11T22:07:35.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:35 smithi144 ceph-mon[33984]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "970b128d-cd56-4dff-8345-3196b46c1257"}]: dispatch 2024-03-11T22:07:35.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:35 smithi144 ceph-mon[33984]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "970b128d-cd56-4dff-8345-3196b46c1257"}]': finished 2024-03-11T22:07:35.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:35 smithi144 ceph-mon[33984]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T22:07:35.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:35 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:35.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:35 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/2477697212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:07:36.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:35 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/2927425206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "970b128d-cd56-4dff-8345-3196b46c1257"}]: dispatch 2024-03-11T22:07:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:35 smithi045 ceph-mon[30178]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "970b128d-cd56-4dff-8345-3196b46c1257"}]: dispatch 2024-03-11T22:07:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:35 smithi045 ceph-mon[30178]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "970b128d-cd56-4dff-8345-3196b46c1257"}]': finished 2024-03-11T22:07:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:35 smithi045 ceph-mon[30178]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T22:07:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:35 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:36.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:35 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/2477697212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:07:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:35 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/2927425206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "970b128d-cd56-4dff-8345-3196b46c1257"}]: dispatch 2024-03-11T22:07:36.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:35 smithi045 ceph-mon[39715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "970b128d-cd56-4dff-8345-3196b46c1257"}]: dispatch 2024-03-11T22:07:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:35 smithi045 ceph-mon[39715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "970b128d-cd56-4dff-8345-3196b46c1257"}]': finished 2024-03-11T22:07:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:35 smithi045 ceph-mon[39715]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T22:07:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:35 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:36.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:35 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/2477697212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:07:37.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:36 smithi045 ceph-mon[39715]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:37.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:36 smithi045 ceph-mon[30178]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:37.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:36 smithi144 ceph-mon[33984]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:38.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:37 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:07:38.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:37 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:07:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:37 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:07:38.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:37 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:07:38.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:37 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:07:38.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:37 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:07:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:38 smithi045 ceph-mon[30178]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T22:07:39.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:38 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:39.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:38 smithi045 ceph-mon[30178]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:38 smithi045 ceph-mon[39715]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T22:07:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:38 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:39.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:38 smithi045 ceph-mon[39715]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:39.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:38 smithi144 ceph-mon[33984]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T22:07:39.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:38 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:39.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:38 smithi144 ceph-mon[33984]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:39.948 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:39 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:07:39.948 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:39 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:07:40.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:39 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:07:40.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:39 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:07:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:39 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:07:40.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:39 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:07:41.022 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:40 smithi144 ceph-mon[33984]: osdmap e48: 7 total, 6 up, 7 in 2024-03-11T22:07:41.022 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:40 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:41.022 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:40 smithi144 ceph-mon[33984]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:40 smithi045 ceph-mon[30178]: osdmap e48: 7 total, 6 up, 7 in 2024-03-11T22:07:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:40 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:41.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:40 smithi045 ceph-mon[30178]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:41.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:40 smithi045 ceph-mon[39715]: osdmap e48: 7 total, 6 up, 7 in 2024-03-11T22:07:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:40 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:41.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:40 smithi045 ceph-mon[39715]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:42.015 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T22:07:42.015 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:42.015 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:41 smithi144 ceph-mon[33984]: Deploying daemon osd.6 on smithi144 2024-03-11T22:07:42.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T22:07:42.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:42.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:41 smithi045 ceph-mon[30178]: Deploying daemon osd.6 on smithi144 2024-03-11T22:07:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T22:07:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:42.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:41 smithi045 ceph-mon[39715]: Deploying daemon osd.6 on smithi144 2024-03-11T22:07:42.979 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:42 smithi144 ceph-mon[33984]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:43.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:42 smithi045 ceph-mon[30178]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:43.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:42 smithi045 ceph-mon[39715]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:45.013 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:44 smithi144 ceph-mon[33984]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:44 smithi045 ceph-mon[30178]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:45.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:44 smithi045 ceph-mon[39715]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:47 smithi045 ceph-mon[39715]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:47.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:47.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:47 smithi045 ceph-mon[30178]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:47.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:47.281 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:47 smithi144 ceph-mon[33984]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:47.281 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:47.281 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:47.281 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:48.982 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:48 smithi144 ceph-mon[33984]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:49.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:48 smithi045 ceph-mon[30178]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:49.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:48 smithi045 ceph-mon[39715]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:50.726 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 6 on host 'smithi144' 2024-03-11T22:07:51.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:50 smithi144 ceph-mon[33984]: from='osd.6 [v2:172.21.15.144:6816/3439446288,v1:172.21.15.144:6817/3439446288]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:07:51.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:50 smithi144 ceph-mon[33984]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:07:51.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:51.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:51.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:50 smithi144 ceph-mon[33984]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:51.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:50 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[30178]: from='osd.6 [v2:172.21.15.144:6816/3439446288,v1:172.21.15.144:6817/3439446288]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:07:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[30178]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:07:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[30178]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[39715]: from='osd.6 [v2:172.21.15.144:6816/3439446288,v1:172.21.15.144:6817/3439446288]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:07:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[39715]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T22:07:51.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:51.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:51.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[39715]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:51.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:51.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:50 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:07:51.385 DEBUG:teuthology.orchestra.run.smithi144:osd.6> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.6.service 2024-03-11T22:07:51.388 INFO:tasks.cephadm:Deploying osd.7 on smithi144 with /dev/vg_nvme/lv_1... 2024-03-11T22:07:51.388 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T22:07:51.888 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:51 smithi144 ceph-mon[33984]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T22:07:51.888 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:51 smithi144 ceph-mon[33984]: osdmap e49: 7 total, 6 up, 7 in 2024-03-11T22:07:51.888 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:51 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:51.888 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:51 smithi144 ceph-mon[33984]: from='osd.6 [v2:172.21.15.144:6816/3439446288,v1:172.21.15.144:6817/3439446288]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:51.888 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:51 smithi144 ceph-mon[33984]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:51.888 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:07:51 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:07:51.825+0000 7f818634a700 -1 osd.6 0 waiting for initial osdmap 2024-03-11T22:07:51.888 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:07:51 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:07:51.832+0000 7f818213d700 -1 osd.6 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:07:52.187 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:07:52.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:51 smithi045 ceph-mon[30178]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T22:07:52.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:51 smithi045 ceph-mon[30178]: osdmap e49: 7 total, 6 up, 7 in 2024-03-11T22:07:52.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:51 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:51 smithi045 ceph-mon[30178]: from='osd.6 [v2:172.21.15.144:6816/3439446288,v1:172.21.15.144:6817/3439446288]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:52.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:51 smithi045 ceph-mon[30178]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:51 smithi045 ceph-mon[39715]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T22:07:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:51 smithi045 ceph-mon[39715]: osdmap e49: 7 total, 6 up, 7 in 2024-03-11T22:07:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:51 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:51 smithi045 ceph-mon[39715]: from='osd.6 [v2:172.21.15.144:6816/3439446288,v1:172.21.15.144:6817/3439446288]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:52.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:51 smithi045 ceph-mon[39715]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:07:53.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:52 smithi144 ceph-mon[33984]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-03-11T22:07:53.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:52 smithi144 ceph-mon[33984]: osdmap e50: 7 total, 6 up, 7 in 2024-03-11T22:07:53.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:52 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:53.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:52 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:53.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:52 smithi144 ceph-mon[33984]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:52 smithi045 ceph-mon[30178]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-03-11T22:07:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:52 smithi045 ceph-mon[30178]: osdmap e50: 7 total, 6 up, 7 in 2024-03-11T22:07:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:53.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:53.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:52 smithi045 ceph-mon[30178]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:52 smithi045 ceph-mon[39715]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-03-11T22:07:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:52 smithi045 ceph-mon[39715]: osdmap e50: 7 total, 6 up, 7 in 2024-03-11T22:07:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:52 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:52 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:53.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:52 smithi045 ceph-mon[39715]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-03-11T22:07:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:53 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:07:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:53 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:07:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:53 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:53 smithi144 ceph-mon[33984]: osd.6 [v2:172.21.15.144:6816/3439446288,v1:172.21.15.144:6817/3439446288] boot 2024-03-11T22:07:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:53 smithi144 ceph-mon[33984]: osdmap e51: 7 total, 7 up, 7 in 2024-03-11T22:07:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:53 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:54.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:53 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:07:54.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:53 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:07:54.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:53 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:54.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:53 smithi045 ceph-mon[30178]: osd.6 [v2:172.21.15.144:6816/3439446288,v1:172.21.15.144:6817/3439446288] boot 2024-03-11T22:07:54.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:53 smithi045 ceph-mon[30178]: osdmap e51: 7 total, 7 up, 7 in 2024-03-11T22:07:54.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:53 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:53 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:07:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:53 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:07:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:53 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:53 smithi045 ceph-mon[39715]: osd.6 [v2:172.21.15.144:6816/3439446288,v1:172.21.15.144:6817/3439446288] boot 2024-03-11T22:07:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:53 smithi045 ceph-mon[39715]: osdmap e51: 7 total, 7 up, 7 in 2024-03-11T22:07:54.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:53 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:07:55.003 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:54 smithi144 ceph-mon[33984]: osdmap e52: 7 total, 7 up, 7 in 2024-03-11T22:07:55.003 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:54 smithi144 ceph-mon[33984]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:07:55.003 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T22:07:55.003 INFO:teuthology.orchestra.run.smithi144.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T22:07:55.003 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10+0 records in 2024-03-11T22:07:55.003 INFO:teuthology.orchestra.run.smithi144.stderr:10+0 records out 2024-03-11T22:07:55.003 INFO:teuthology.orchestra.run.smithi144.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012609 s, 832 MB/s 2024-03-11T22:07:55.004 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping successful for: 2024-03-11T22:07:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:54 smithi045 ceph-mon[30178]: osdmap e52: 7 total, 7 up, 7 in 2024-03-11T22:07:55.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:54 smithi045 ceph-mon[30178]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:07:55.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:54 smithi045 ceph-mon[39715]: osdmap e52: 7 total, 7 up, 7 in 2024-03-11T22:07:55.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:54 smithi045 ceph-mon[39715]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:07:56.080 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:55 smithi144 ceph-mon[33984]: osdmap e53: 7 total, 7 up, 7 in 2024-03-11T22:07:56.080 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:55 smithi144 ceph-mon[33984]: Detected new or changed devices on smithi144 2024-03-11T22:07:56.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:56.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:56.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:56.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:56.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:56.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:55 smithi144 ceph-mon[33984]: Adjusting osd_memory_target on smithi144 to 5696M 2024-03-11T22:07:56.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:56.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:56.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:56.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:55 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:56.107 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi144:vg_nvme/lv_1 2024-03-11T22:07:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[30178]: osdmap e53: 7 total, 7 up, 7 in 2024-03-11T22:07:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[30178]: Detected new or changed devices on smithi144 2024-03-11T22:07:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[30178]: Adjusting osd_memory_target on smithi144 to 5696M 2024-03-11T22:07:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:56.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[39715]: osdmap e53: 7 total, 7 up, 7 in 2024-03-11T22:07:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[39715]: Detected new or changed devices on smithi144 2024-03-11T22:07:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:07:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[39715]: Adjusting osd_memory_target on smithi144 to 5696M 2024-03-11T22:07:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:07:56.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:55 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:07:56.343 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:07:57.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:56 smithi144 ceph-mon[33984]: osdmap e54: 7 total, 7 up, 7 in 2024-03-11T22:07:57.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:56 smithi144 ceph-mon[33984]: pgmap v166: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T22:07:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:56 smithi045 ceph-mon[30178]: osdmap e54: 7 total, 7 up, 7 in 2024-03-11T22:07:57.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:56 smithi045 ceph-mon[30178]: pgmap v166: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T22:07:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:56 smithi045 ceph-mon[39715]: osdmap e54: 7 total, 7 up, 7 in 2024-03-11T22:07:57.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:56 smithi045 ceph-mon[39715]: pgmap v166: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T22:07:58.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:57 smithi045 ceph-mon[30178]: osdmap e55: 7 total, 7 up, 7 in 2024-03-11T22:07:58.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:57 smithi045 ceph-mon[39715]: osdmap e55: 7 total, 7 up, 7 in 2024-03-11T22:07:58.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:57 smithi144 ceph-mon[33984]: osdmap e55: 7 total, 7 up, 7 in 2024-03-11T22:07:59.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:58 smithi144 ceph-mon[33984]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:07:59.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:07:59.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:07:59.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:58 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:59.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:07:58 smithi144 ceph-mon[33984]: pgmap v168: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-11T22:07:59.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:58 smithi045 ceph-mon[30178]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:07:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:07:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:07:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:58 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:07:58 smithi045 ceph-mon[30178]: pgmap v168: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-11T22:07:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:58 smithi045 ceph-mon[39715]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:07:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:07:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:07:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:58 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:07:59.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:07:58 smithi045 ceph-mon[39715]: pgmap v168: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-11T22:08:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:08:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:08:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[30178]: osdmap e56: 7 total, 7 up, 7 in 2024-03-11T22:08:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[30178]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-11T22:08:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/1111401732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40100aaf-830d-4137-b037-5cf167cba308"}]: dispatch 2024-03-11T22:08:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[30178]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40100aaf-830d-4137-b037-5cf167cba308"}]: dispatch 2024-03-11T22:08:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[30178]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40100aaf-830d-4137-b037-5cf167cba308"}]': finished 2024-03-11T22:08:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[30178]: osdmap e57: 8 total, 7 up, 8 in 2024-03-11T22:08:01.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:08:01.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:08:01.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[39715]: osdmap e56: 7 total, 7 up, 7 in 2024-03-11T22:08:01.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[39715]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-11T22:08:01.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/1111401732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40100aaf-830d-4137-b037-5cf167cba308"}]: dispatch 2024-03-11T22:08:01.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[39715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40100aaf-830d-4137-b037-5cf167cba308"}]: dispatch 2024-03-11T22:08:01.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[39715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40100aaf-830d-4137-b037-5cf167cba308"}]': finished 2024-03-11T22:08:01.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[39715]: osdmap e57: 8 total, 7 up, 8 in 2024-03-11T22:08:01.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:00 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:01.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:00 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:08:01.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:00 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:08:01.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:00 smithi144 ceph-mon[33984]: osdmap e56: 7 total, 7 up, 7 in 2024-03-11T22:08:01.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:00 smithi144 ceph-mon[33984]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-11T22:08:01.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:00 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/1111401732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40100aaf-830d-4137-b037-5cf167cba308"}]: dispatch 2024-03-11T22:08:01.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:00 smithi144 ceph-mon[33984]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "40100aaf-830d-4137-b037-5cf167cba308"}]: dispatch 2024-03-11T22:08:01.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:00 smithi144 ceph-mon[33984]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "40100aaf-830d-4137-b037-5cf167cba308"}]': finished 2024-03-11T22:08:01.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:00 smithi144 ceph-mon[33984]: osdmap e57: 8 total, 7 up, 8 in 2024-03-11T22:08:01.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:00 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:02.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:01 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/3248448331' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:08:02.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:01 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/3248448331' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:08:02.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:01 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/3248448331' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:08:03.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:02 smithi045 ceph-mon[30178]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:03.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:02 smithi045 ceph-mon[39715]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:03.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:02 smithi144 ceph-mon[33984]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:05.023 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:04 smithi144 ceph-mon[33984]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:05.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:04 smithi045 ceph-mon[30178]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:05.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:04 smithi045 ceph-mon[39715]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:07 smithi045 ceph-mon[30178]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:07.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:07 smithi045 ceph-mon[39715]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:07.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:07 smithi144 ceph-mon[33984]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:08.273 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:08 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:08:08.273 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:08 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:08:08.273 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:08 smithi144 ceph-mon[33984]: osdmap e58: 8 total, 7 up, 8 in 2024-03-11T22:08:08.273 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:08 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:08.274 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:08 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T22:08:08.274 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:08 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:08.274 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:08 smithi144 ceph-mon[33984]: Deploying daemon osd.7 on smithi144 2024-03-11T22:08:08.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:08:08.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:08:08.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[30178]: osdmap e58: 8 total, 7 up, 8 in 2024-03-11T22:08:08.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T22:08:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:08.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[30178]: Deploying daemon osd.7 on smithi144 2024-03-11T22:08:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:08:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:08:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[39715]: osdmap e58: 8 total, 7 up, 8 in 2024-03-11T22:08:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T22:08:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:08.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:08 smithi045 ceph-mon[39715]: Deploying daemon osd.7 on smithi144 2024-03-11T22:08:09.294 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:09 smithi144 ceph-mon[33984]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:09.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:09 smithi045 ceph-mon[30178]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:09 smithi045 ceph-mon[39715]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:11.011 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:10 smithi144 ceph-mon[33984]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:11.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:10 smithi045 ceph-mon[30178]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:11.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:10 smithi045 ceph-mon[39715]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:12.770 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:12 smithi144 ceph-mon[33984]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:13.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:12 smithi045 ceph-mon[30178]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:13.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:12 smithi045 ceph-mon[39715]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:14.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:13 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:14.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:13 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:14.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:13 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:14.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:13 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:14.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:13 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:14.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:13 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:13 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:13 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:14.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:13 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:14.946 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:14 smithi144 ceph-mon[33984]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:15.177 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:14 smithi045 ceph-mon[39715]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:15.177 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:14 smithi045 ceph-mon[30178]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:16.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:15 smithi144 ceph-mon[33984]: from='osd.7 [v2:172.21.15.144:6824/2371064429,v1:172.21.15.144:6825/2371064429]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:08:16.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:15 smithi144 ceph-mon[33984]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:08:16.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:15 smithi045 ceph-mon[30178]: from='osd.7 [v2:172.21.15.144:6824/2371064429,v1:172.21.15.144:6825/2371064429]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:08:16.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:15 smithi045 ceph-mon[30178]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:08:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:15 smithi045 ceph-mon[39715]: from='osd.7 [v2:172.21.15.144:6824/2371064429,v1:172.21.15.144:6825/2371064429]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:08:16.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:15 smithi045 ceph-mon[39715]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T22:08:16.522 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 7 on host 'smithi144' 2024-03-11T22:08:17.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:16 smithi144 ceph-mon[33984]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T22:08:17.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:16 smithi144 ceph-mon[33984]: osdmap e59: 8 total, 7 up, 8 in 2024-03-11T22:08:17.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:16 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:17.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:16 smithi144 ceph-mon[33984]: from='osd.7 [v2:172.21.15.144:6824/2371064429,v1:172.21.15.144:6825/2371064429]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:08:17.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:16 smithi144 ceph-mon[33984]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:08:17.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:16 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:17.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:16 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:17.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:16 smithi144 ceph-mon[33984]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:17.119 DEBUG:teuthology.orchestra.run.smithi144:osd.7> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.7.service 2024-03-11T22:08:17.121 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-11T22:08:17.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T22:08:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[39715]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T22:08:17.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[39715]: osdmap e59: 8 total, 7 up, 8 in 2024-03-11T22:08:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[39715]: from='osd.7 [v2:172.21.15.144:6824/2371064429,v1:172.21.15.144:6825/2371064429]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:08:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[39715]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:08:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[39715]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[30178]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T22:08:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[30178]: osdmap e59: 8 total, 7 up, 8 in 2024-03-11T22:08:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[30178]: from='osd.7 [v2:172.21.15.144:6824/2371064429,v1:172.21.15.144:6825/2371064429]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:08:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[30178]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-03-11T22:08:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:17.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:16 smithi045 ceph-mon[30178]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-11T22:08:17.378 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:08:18.170 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:17 smithi144 ceph-mon[33984]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-03-11T22:08:18.170 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:17 smithi144 ceph-mon[33984]: osdmap e60: 8 total, 7 up, 8 in 2024-03-11T22:08:18.171 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:17 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:18.171 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:17 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:18.171 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:17 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:18.171 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:17 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:18.240 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:17 smithi045 ceph-mon[39715]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-03-11T22:08:18.241 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:17 smithi045 ceph-mon[39715]: osdmap e60: 8 total, 7 up, 8 in 2024-03-11T22:08:18.241 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:17 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:18.241 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:17 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:18.241 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:17 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:18.241 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:17 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:18.241 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:17 smithi045 ceph-mon[30178]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-03-11T22:08:18.241 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:17 smithi045 ceph-mon[30178]: osdmap e60: 8 total, 7 up, 8 in 2024-03-11T22:08:18.241 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:17 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:18.241 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:17 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:18.241 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:17 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:18.241 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:17 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:19.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:08:19.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:08:19.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[39715]: osd.7 [v2:172.21.15.144:6824/2371064429,v1:172.21.15.144:6825/2371064429] boot 2024-03-11T22:08:19.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[39715]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T22:08:19.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:19.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[39715]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:08:19.095 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[39715]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T22:08:19.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:08:19.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:08:19.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[30178]: osd.7 [v2:172.21.15.144:6824/2371064429,v1:172.21.15.144:6825/2371064429] boot 2024-03-11T22:08:19.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[30178]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T22:08:19.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:19.095 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[30178]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:08:19.096 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:19 smithi045 ceph-mon[30178]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T22:08:19.261 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:19 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:08:19.261 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:19 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:08:19.262 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:19 smithi144 ceph-mon[33984]: osd.7 [v2:172.21.15.144:6824/2371064429,v1:172.21.15.144:6825/2371064429] boot 2024-03-11T22:08:19.262 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:19 smithi144 ceph-mon[33984]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T22:08:19.262 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:19 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:08:19.262 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:19 smithi144 ceph-mon[33984]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:08:19.262 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:19 smithi144 ceph-mon[33984]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T22:08:19.522 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:08:20.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:20 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3501326776' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:08:20.266 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:20 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3501326776' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:08:20.273 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:20 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3501326776' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:08:20.299 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":62,"num_osds":8,"num_up_osds":8,"osd_up_since":1710194897,"num_in_osds":8,"osd_in_since":1710194880,"num_remapped_pgs":1} 2024-03-11T22:08:20.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:08:20.549 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:08:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T22:08:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: pgmap v187: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T22:08:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: Detected new or changed devices on smithi144 2024-03-11T22:08:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:08:21.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:08:21.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:08:21.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:08:21.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:21.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:21.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:21.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:21.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:21.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T22:08:21.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: pgmap v187: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T22:08:21.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: Detected new or changed devices on smithi144 2024-03-11T22:08:21.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:08:21.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:08:21.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:08:21.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:08:21.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:21.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:21.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:21.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:21.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:21.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:21 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T22:08:21.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: pgmap v187: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T22:08:21.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: Detected new or changed devices on smithi144 2024-03-11T22:08:21.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:08:21.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:08:21.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:08:21.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:08:21.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:21.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:21.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:21.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:21.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:21.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:21.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:21 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:22 smithi045 ceph-mon[30178]: Adjusting osd_memory_target on smithi144 to 4272M 2024-03-11T22:08:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:22 smithi045 ceph-mon[30178]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T22:08:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:22 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:08:22.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:22 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:08:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:22 smithi045 ceph-mon[39715]: Adjusting osd_memory_target on smithi144 to 4272M 2024-03-11T22:08:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:22 smithi045 ceph-mon[39715]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T22:08:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:22 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:08:22.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:22 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:08:22.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:22 smithi144 ceph-mon[33984]: Adjusting osd_memory_target on smithi144 to 4272M 2024-03-11T22:08:22.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:22 smithi144 ceph-mon[33984]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T22:08:22.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:22 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:08:22.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:22 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:08:22.442 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:08:22.442 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":65,"fsid":"f75aafa4-dff2-11ee-95c8-87774f69a715","created":"2024-03-11T22:02:51.075940+0000","modified":"2024-03-11T22:08:22.008482+0000","last_up_change":"2024-03-11T22:08:17.992246+0000","last_in_change":"2024-03-11T22:08:00.853801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:06:12.903386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55a4e633-f2a6-4d27-b10c-de1c0f87cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6803","nonce":2216353820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6805","nonce":2216353820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6809","nonce":2216353820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6807","nonce":2216353820}]},"public_addr":"172.21.15.45:6803/2216353820","cluster_addr":"172.21.15.45:6805/2216353820","heartbeat_back_addr":"172.21.15.45:6809/2216353820","heartbeat_front_addr":"172.21.15.45:6807/2216353820","state":["exists","up"]},{"osd":1,"uuid":"60274eeb-591a-43fe-a579-f13d13631e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6811","nonce":1241486091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6813","nonce":1241486091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6817","nonce":1241486091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6815","nonce":1241486091}]},"public_addr":"172.21.15.45:6811/1241486091","cluster_addr":"172.21.15.45:6813/1241486091","heartbeat_back_addr":"172.21.15.45:6817/1241486091","heartbeat_front_addr":"172.21.15.45:6815/1241486091","state":["exists","up"]},{"osd":2,"uuid":"15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6819","nonce":4251983518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6821","nonce":4251983518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6825","nonce":4251983518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6823","nonce":4251983518}]},"public_addr":"172.21.15.45:6819/4251983518","cluster_addr":"172.21.15.45:6821/4251983518","heartbeat_back_addr":"172.21.15.45:6825/4251983518","heartbeat_front_addr":"172.21.15.45:6823/4251983518","state":["exists","up"]},{"osd":3,"uuid":"e51a65f3-96f6-478c-8d26-df1ace4a5058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6827","nonce":3178755830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6829","nonce":3178755830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6833","nonce":3178755830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6831","nonce":3178755830}]},"public_addr":"172.21.15.45:6827/3178755830","cluster_addr":"172.21.15.45:6829/3178755830","heartbeat_back_addr":"172.21.15.45:6833/3178755830","heartbeat_front_addr":"172.21.15.45:6831/3178755830","state":["exists","up"]},{"osd":4,"uuid":"32d0b71f-c696-4677-bdd3-b88a04c97604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6801","nonce":912745159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6803","nonce":912745159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6807","nonce":912745159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6805","nonce":912745159}]},"public_addr":"172.21.15.144:6801/912745159","cluster_addr":"172.21.15.144:6803/912745159","heartbeat_back_addr":"172.21.15.144:6807/912745159","heartbeat_front_addr":"172.21.15.144:6805/912745159","state":["exists","up"]},{"osd":5,"uuid":"f46b91af-d67d-404c-aad1-75348e35d1c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6809","nonce":3162869174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6811","nonce":3162869174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6815","nonce":3162869174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6813","nonce":3162869174}]},"public_addr":"172.21.15.144:6809/3162869174","cluster_addr":"172.21.15.144:6811/3162869174","heartbeat_back_addr":"172.21.15.144:6815/3162869174","heartbeat_front_addr":"172.21.15.144:6813/3162869174","state":["exists","up"]},{"osd":6,"uuid":"970b128d-cd56-4dff-8345-3196b46c1257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6817","nonce":3439446288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6819","nonce":3439446288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6823","nonce":3439446288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6821","nonce":3439446288}]},"public_addr":"172.21.15.144:6817/3439446288","cluster_addr":"172.21.15.144:6819/3439446288","heartbeat_back_addr":"172.21.15.144:6823/3439446288","heartbeat_front_addr":"172.21.15.144:6821/3439446288","state":["exists","up"]},{"osd":7,"uuid":"40100aaf-830d-4137-b037-5cf167cba308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6825","nonce":2371064429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6827","nonce":2371064429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6831","nonce":2371064429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6829","nonce":2371064429}]},"public_addr":"172.21.15.144:6825/2371064429","cluster_addr":"172.21.15.144:6827/2371064429","heartbeat_back_addr":"172.21.15.144:6831/2371064429","heartbeat_front_addr":"172.21.15.144:6829/2371064429","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:07:39.053491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:06.696931+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:07:37.218887+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:07:59.826901+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:21.276126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:07:25.856240+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:07:51.041009+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:16.977128+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/2584267737":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/2513993410":"2024-03-12T22:03:52.557006+0000","172.21.15.45:6800/4256719031":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/2777740355":"2024-03-12T22:03:25.087713+0000","172.21.15.45:0/926341889":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6801/4256719031":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/2331445880":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6800/2299156110":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6801/2299156110":"2024-03-12T22:03:25.087713+0000","172.21.15.45:0/1110781721":"2024-03-12T22:03:52.557006+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:08:23.079 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:23 smithi045 ceph-mon[30178]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T22:08:23.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:23 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/354713580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:08:23.080 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:23 smithi045 ceph-mon[30178]: pgmap v190: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T22:08:23.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:23 smithi045 ceph-mon[39715]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T22:08:23.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:23 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/354713580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:08:23.080 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:23 smithi045 ceph-mon[39715]: pgmap v190: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T22:08:23.109 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T22:06:12.903386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T22:08:23.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T22:08:23.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:23 smithi144 ceph-mon[33984]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T22:08:23.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:23 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/354713580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:08:23.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:23 smithi144 ceph-mon[33984]: pgmap v190: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T22:08:23.349 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:08:24.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:24 smithi045 ceph-mon[39715]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-11T22:08:24.832 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:24 smithi045 ceph-mon[30178]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-11T22:08:25.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:24 smithi144 ceph-mon[33984]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-11T22:08:25.300 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-03-11T22:08:25.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:25 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3896119166' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:08:25.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:25 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3896119166' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:08:25.923 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi045 2024-03-11T22:08:25.923 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi045=foo.a' 2024-03-11T22:08:25.953 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:25 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3896119166' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:08:26.175 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:08:27.246 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:27 smithi144 ceph-mon[33984]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-11T22:08:27.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:27 smithi045 ceph-mon[30178]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-11T22:08:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:27 smithi045 ceph-mon[39715]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-11T22:08:28.008 INFO:teuthology.orchestra.run.smithi144.stdout:Scheduled rgw.foo.a update... 2024-03-11T22:08:28.605 DEBUG:teuthology.orchestra.run.smithi045:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-11T22:08:28.607 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi144 2024-03-11T22:08:28.608 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-11T22:08:28.883 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:08:29.256 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:29 smithi144 ceph-mon[33984]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi045=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:29.256 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:29 smithi144 ceph-mon[33984]: Saving service rgw.foo.a spec with placement smithi045=foo.a;count:1 2024-03-11T22:08:29.256 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:29 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:29.256 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:29 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:29.256 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:29 smithi144 ceph-mon[33984]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:08:29.256 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:29 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:29.256 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:29 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:29.256 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:29 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:29.256 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:29 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:08:29.256 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:29 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:08:29.257 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:29 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:29.257 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:29 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[30178]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi045=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[30178]: Saving service rgw.foo.a spec with placement smithi045=foo.a;count:1 2024-03-11T22:08:29.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[30178]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:08:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:08:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:08:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:29.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[39715]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi045=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:29.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[39715]: Saving service rgw.foo.a spec with placement smithi045=foo.a;count:1 2024-03-11T22:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[39715]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:29.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:29 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:30.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:30 smithi045 ceph-mon[39715]: Deploying daemon rgw.foo.a on smithi045 2024-03-11T22:08:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:30 smithi045 ceph-mon[30178]: Deploying daemon rgw.foo.a on smithi045 2024-03-11T22:08:30.329 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:30 smithi144 ceph-mon[33984]: Deploying daemon rgw.foo.a on smithi045 2024-03-11T22:08:31.015 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:08:30 smithi045 systemd[1]: Starting Ceph rgw.foo.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:08:31.024 INFO:teuthology.orchestra.run.smithi144.stderr:pool 'datapool' created 2024-03-11T22:08:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:31 smithi045 ceph-mon[39715]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:08:31.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:31 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/3193613407' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T22:08:31.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:31 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T22:08:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:31 smithi045 ceph-mon[30178]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:08:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:31 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/3193613407' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T22:08:31.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:31 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T22:08:31.276 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:08:31 smithi045 podman[68444]: 2024-03-11 22:08:30.966525619 +0000 UTC m=+0.018582781 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:08:31.276 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:08:31 smithi045 podman[68444]: 2024-03-11 22:08:31.070001542 +0000 UTC m=+0.122058747 container create 7a0d92e11abc9d7e1dbb25aeb3b1e7ada05f4a30cddec6f3cdbc9c67ced2e232 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-11T22:08:31.290 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:31 smithi144 ceph-mon[33984]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:08:31.290 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:31 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/3193613407' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T22:08:31.290 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:31 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T22:08:31.597 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- rbd pool init datapool 2024-03-11T22:08:31.775 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:08:31 smithi045 podman[68444]: 2024-03-11 22:08:31.298959296 +0000 UTC m=+0.351016456 container init 7a0d92e11abc9d7e1dbb25aeb3b1e7ada05f4a30cddec6f3cdbc9c67ced2e232 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T22:08:31.775 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:08:31 smithi045 podman[68444]: 2024-03-11 22:08:31.304229291 +0000 UTC m=+0.356286450 container start 7a0d92e11abc9d7e1dbb25aeb3b1e7ada05f4a30cddec6f3cdbc9c67ced2e232 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-94a5397) 2024-03-11T22:08:31.775 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:08:31 smithi045 bash[68444]: 7a0d92e11abc9d7e1dbb25aeb3b1e7ada05f4a30cddec6f3cdbc9c67ced2e232 2024-03-11T22:08:31.775 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:08:31 smithi045 systemd[1]: Started Ceph rgw.foo.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:08:31.837 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:08:32.228 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:32 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T22:08:32.228 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:32 smithi144 ceph-mon[33984]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T22:08:32.228 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:32 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.228 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:32 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.229 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:32 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.229 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:32 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.229 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:32 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.229 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:32 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:32.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T22:08:32.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[39715]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T22:08:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T22:08:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[30178]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T22:08:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:32.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:32 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:33.153 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:33 smithi144 ceph-mon[33984]: Saving service rgw.foo.a spec with placement smithi045=foo.a;count:1 2024-03-11T22:08:33.153 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:33 smithi144 ceph-mon[33984]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T22:08:33.153 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:33 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2228576527' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:08:33.153 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:33 smithi144 ceph-mon[33984]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:08:33.153 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:33 smithi144 ceph-mon[33984]: pgmap v197: 36 pgs: 34 unknown, 2 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:08:33.154 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:33 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:33.154 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:33 smithi144 ceph-mon[33984]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:08:33.154 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:33 smithi144 ceph-mon[33984]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T22:08:33.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[30178]: Saving service rgw.foo.a spec with placement smithi045=foo.a;count:1 2024-03-11T22:08:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[30178]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T22:08:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2228576527' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:08:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[30178]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:08:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[30178]: pgmap v197: 36 pgs: 34 unknown, 2 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:08:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[30178]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:08:33.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[30178]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T22:08:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[39715]: Saving service rgw.foo.a spec with placement smithi045=foo.a;count:1 2024-03-11T22:08:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[39715]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T22:08:33.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2228576527' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:08:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[39715]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:08:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[39715]: pgmap v197: 36 pgs: 34 unknown, 2 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:08:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[39715]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:08:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:33 smithi045 ceph-mon[39715]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T22:08:34.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:34 smithi144 ceph-mon[33984]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:08:34.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:34 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/3013497677' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T22:08:34.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:34 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T22:08:34.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:34 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:34.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:34 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:34.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:34 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:34.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:34 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:34.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:34 smithi144 ceph-mon[33984]: Checking dashboard <-> RGW credentials 2024-03-11T22:08:34.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:34 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T22:08:34.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:34 smithi144 ceph-mon[33984]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T22:08:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[39715]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:08:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/3013497677' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T22:08:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T22:08:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[39715]: Checking dashboard <-> RGW credentials 2024-03-11T22:08:34.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T22:08:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[39715]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T22:08:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[30178]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:08:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/3013497677' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T22:08:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T22:08:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[30178]: Checking dashboard <-> RGW credentials 2024-03-11T22:08:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T22:08:34.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:34 smithi045 ceph-mon[30178]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T22:08:35.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:35 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3033943457' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:08:35.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:35 smithi144 ceph-mon[33984]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:08:35.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:35 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4166879095' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:08:35.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:35 smithi144 ceph-mon[33984]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:08:35.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:35 smithi144 ceph-mon[33984]: pgmap v200: 68 pgs: 15 creating+peering, 12 active+clean, 41 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-11T22:08:35.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:35 smithi144 ceph-mon[33984]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:08:35.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:35 smithi144 ceph-mon[33984]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:08:35.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:35 smithi144 ceph-mon[33984]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T22:08:35.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3033943457' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:08:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[39715]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:08:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4166879095' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:08:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[39715]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:08:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[39715]: pgmap v200: 68 pgs: 15 creating+peering, 12 active+clean, 41 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-11T22:08:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[39715]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:08:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[39715]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:08:35.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[39715]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T22:08:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3033943457' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:08:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[30178]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:08:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4166879095' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:08:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[30178]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:08:35.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[30178]: pgmap v200: 68 pgs: 15 creating+peering, 12 active+clean, 41 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-11T22:08:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[30178]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:08:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[30178]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:08:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:35 smithi045 ceph-mon[30178]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T22:08:36.623 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.144 --placement '1;smithi144=iscsi.a' 2024-03-11T22:08:36.861 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:08:37.242 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:37 smithi144 ceph-mon[33984]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T22:08:37.242 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:37 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4166879095' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:08:37.242 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:37 smithi144 ceph-mon[33984]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:08:37.242 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:37 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3033943457' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:08:37.242 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:37 smithi144 ceph-mon[33984]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:08:37.242 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:37 smithi144 ceph-mon[33984]: pgmap v203: 100 pgs: 3 creating+activating, 15 creating+peering, 28 active+clean, 54 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 1 op/s 2024-03-11T22:08:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:37 smithi045 ceph-mon[39715]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T22:08:37.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:37 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4166879095' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:08:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:37 smithi045 ceph-mon[39715]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:08:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:37 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3033943457' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:08:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:37 smithi045 ceph-mon[39715]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:08:37.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:37 smithi045 ceph-mon[39715]: pgmap v203: 100 pgs: 3 creating+activating, 15 creating+peering, 28 active+clean, 54 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 1 op/s 2024-03-11T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:37 smithi045 ceph-mon[30178]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:37 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4166879095' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:37 smithi045 ceph-mon[30178]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:08:37.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:37 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3033943457' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:37 smithi045 ceph-mon[30178]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:08:37.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:37 smithi045 ceph-mon[30178]: pgmap v203: 100 pgs: 3 creating+activating, 15 creating+peering, 28 active+clean, 54 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 767 B/s wr, 1 op/s 2024-03-11T22:08:38.253 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:38 smithi144 ceph-mon[33984]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:08:38.253 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:38 smithi144 ceph-mon[33984]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:08:38.253 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:38 smithi144 ceph-mon[33984]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T22:08:38.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:38 smithi045 ceph-mon[30178]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:08:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:38 smithi045 ceph-mon[30178]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:08:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:38 smithi045 ceph-mon[30178]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T22:08:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:38 smithi045 ceph-mon[39715]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:08:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:38 smithi045 ceph-mon[39715]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:08:38.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:38 smithi045 ceph-mon[39715]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T22:08:38.699 INFO:teuthology.orchestra.run.smithi144.stdout:Scheduled iscsi.datapool update... 2024-03-11T22:08:39.320 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T22:08:39.320 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3033943457' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:08:39.320 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4166879095' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:08:39.320 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:08:39.320 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:08:39.320 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: pgmap v206: 132 pgs: 3 creating+activating, 73 active+clean, 56 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-11T22:08:39.321 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:39.321 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:08:39.321 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:08:39.321 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T22:08:39.321 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3033943457' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:08:39.321 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4166879095' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:08:39.321 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:08:39.321 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:39 smithi144 ceph-mon[33984]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:08:39.346 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-11T22:08:39.346 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:08:39.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-11T22:08:39.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T22:08:39.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3033943457' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:08:39.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4166879095' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:08:39.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:08:39.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:08:39.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: pgmap v206: 132 pgs: 3 creating+activating, 73 active+clean, 56 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-11T22:08:39.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:39.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:08:39.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:08:39.377 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T22:08:39.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3033943457' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:08:39.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4166879095' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:08:39.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:08:39.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[39715]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:08:39.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T22:08:39.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3033943457' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:08:39.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4166879095' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:08:39.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:08:39.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:08:39.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: pgmap v206: 132 pgs: 3 creating+activating, 73 active+clean, 56 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-11T22:08:39.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:39.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:08:39.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:08:39.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T22:08:39.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3033943457' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:08:39.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4166879095' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:08:39.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:08:39.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:39 smithi045 ceph-mon[30178]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:08:39.382 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-03-11T22:08:39.382 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-11T22:08:39.411 DEBUG:teuthology.orchestra.run.smithi144:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@iscsi.iscsi.a.service 2024-03-11T22:08:39.455 INFO:tasks.cephadm:Adding prometheus.a on smithi144 2024-03-11T22:08:39.455 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi144=a' 2024-03-11T22:08:39.730 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:08:40.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:40 smithi144 ceph-mon[33984]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.144", "placement": "1;smithi144=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:40.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:40 smithi144 ceph-mon[33984]: Saving service iscsi.datapool spec with placement smithi144=iscsi.a;count:1 2024-03-11T22:08:40.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:40 smithi144 ceph-mon[33984]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:08:40.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:40 smithi144 ceph-mon[33984]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:08:40.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:40 smithi144 ceph-mon[33984]: osdmap e75: 8 total, 8 up, 8 in 2024-03-11T22:08:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:40 smithi045 ceph-mon[30178]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.144", "placement": "1;smithi144=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:40.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:40 smithi045 ceph-mon[30178]: Saving service iscsi.datapool spec with placement smithi144=iscsi.a;count:1 2024-03-11T22:08:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:40 smithi045 ceph-mon[30178]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:08:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:40 smithi045 ceph-mon[30178]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:08:40.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:40 smithi045 ceph-mon[30178]: osdmap e75: 8 total, 8 up, 8 in 2024-03-11T22:08:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:40 smithi045 ceph-mon[39715]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.144", "placement": "1;smithi144=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:40 smithi045 ceph-mon[39715]: Saving service iscsi.datapool spec with placement smithi144=iscsi.a;count:1 2024-03-11T22:08:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:40 smithi045 ceph-mon[39715]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:08:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:40 smithi045 ceph-mon[39715]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:08:40.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:40 smithi045 ceph-mon[39715]: osdmap e75: 8 total, 8 up, 8 in 2024-03-11T22:08:41.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:41.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:41.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:41.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T22:08:41.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T22:08:41.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:41 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:41.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:41 smithi144 ceph-mon[33984]: Deploying daemon iscsi.iscsi.a on smithi144 2024-03-11T22:08:41.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:41 smithi144 ceph-mon[33984]: pgmap v209: 132 pgs: 7 creating+peering, 108 active+clean, 17 unknown; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 4.2 KiB/s wr, 62 op/s 2024-03-11T22:08:41.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:41 smithi144 ceph-mon[33984]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:08:41.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:41 smithi144 ceph-mon[33984]: osdmap e76: 8 total, 8 up, 8 in 2024-03-11T22:08:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T22:08:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T22:08:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[30178]: Deploying daemon iscsi.iscsi.a on smithi144 2024-03-11T22:08:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[30178]: pgmap v209: 132 pgs: 7 creating+peering, 108 active+clean, 17 unknown; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 4.2 KiB/s wr, 62 op/s 2024-03-11T22:08:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[30178]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:08:41.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[30178]: osdmap e76: 8 total, 8 up, 8 in 2024-03-11T22:08:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T22:08:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T22:08:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[39715]: Deploying daemon iscsi.iscsi.a on smithi144 2024-03-11T22:08:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[39715]: pgmap v209: 132 pgs: 7 creating+peering, 108 active+clean, 17 unknown; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 4.2 KiB/s wr, 62 op/s 2024-03-11T22:08:41.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[39715]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:08:41.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:41 smithi045 ceph-mon[39715]: osdmap e76: 8 total, 8 up, 8 in 2024-03-11T22:08:41.729 INFO:teuthology.orchestra.run.smithi144.stdout:Scheduled prometheus update... 2024-03-11T22:08:42.381 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:42 smithi144 ceph-mon[33984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:08:42.381 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:42 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:08:42.381 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:42 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:42.409 DEBUG:teuthology.orchestra.run.smithi144:prometheus.a> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-11T22:08:42.411 INFO:tasks.cephadm:Adding node-exporter.a on smithi045 2024-03-11T22:08:42.411 INFO:tasks.cephadm:Adding node-exporter.b on smithi144 2024-03-11T22:08:42.411 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '2;smithi045=a;smithi144=b' 2024-03-11T22:08:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:42 smithi045 ceph-mon[39715]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:08:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:42 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:08:42.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:42 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:42.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:42 smithi045 ceph-mon[30178]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:08:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:42 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:08:42.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:42 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:43.120 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:08:43.551 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:43 smithi144 ceph-mon[33984]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi144=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:43.551 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:43 smithi144 ceph-mon[33984]: Saving service prometheus spec with placement smithi144=a;count:1 2024-03-11T22:08:43.551 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:43 smithi144 ceph-mon[33984]: 1.0 scrub starts 2024-03-11T22:08:43.552 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:43 smithi144 ceph-mon[33984]: 1.0 scrub ok 2024-03-11T22:08:43.552 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:43 smithi144 ceph-mon[33984]: pgmap v211: 132 pgs: 7 creating+peering, 110 active+clean, 15 unknown; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 2.9 KiB/s wr, 60 op/s 2024-03-11T22:08:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:43 smithi045 ceph-mon[30178]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi144=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:43 smithi045 ceph-mon[30178]: Saving service prometheus spec with placement smithi144=a;count:1 2024-03-11T22:08:43.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:43 smithi045 ceph-mon[30178]: 1.0 scrub starts 2024-03-11T22:08:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:43 smithi045 ceph-mon[30178]: 1.0 scrub ok 2024-03-11T22:08:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:43 smithi045 ceph-mon[30178]: pgmap v211: 132 pgs: 7 creating+peering, 110 active+clean, 15 unknown; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 2.9 KiB/s wr, 60 op/s 2024-03-11T22:08:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:43 smithi045 ceph-mon[39715]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi144=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:43 smithi045 ceph-mon[39715]: Saving service prometheus spec with placement smithi144=a;count:1 2024-03-11T22:08:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:43 smithi045 ceph-mon[39715]: 1.0 scrub starts 2024-03-11T22:08:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:43 smithi045 ceph-mon[39715]: 1.0 scrub ok 2024-03-11T22:08:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:43 smithi045 ceph-mon[39715]: pgmap v211: 132 pgs: 7 creating+peering, 110 active+clean, 15 unknown; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 2.9 KiB/s wr, 60 op/s 2024-03-11T22:08:44.973 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:44 smithi144 ceph-mon[33984]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 5.5 KiB/s wr, 161 op/s 2024-03-11T22:08:45.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:44 smithi045 ceph-mon[30178]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 5.5 KiB/s wr, 161 op/s 2024-03-11T22:08:45.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:44 smithi045 ceph-mon[39715]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 5.5 KiB/s wr, 161 op/s 2024-03-11T22:08:45.194 INFO:teuthology.orchestra.run.smithi144.stdout:Scheduled node-exporter update... 2024-03-11T22:08:45.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:08:45 smithi144 podman[63604]: 2024-03-11 22:08:45.320942849 +0000 UTC m=+0.804170604 container start 8976c83b2a7cd9359b9c290c75ed525c178cfb59d774390dd2b7b0f5b876fe8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-11T22:08:45.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:08:45 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug Started the configuration object watcher 2024-03-11T22:08:45.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:08:45 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug Checking for config object changes every 1s 2024-03-11T22:08:45.839 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:08:45 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug Processing osd blocklist entries for this node 2024-03-11T22:08:45.872 DEBUG:teuthology.orchestra.run.smithi045:node-exporter.a> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-11T22:08:45.875 DEBUG:teuthology.orchestra.run.smithi144:node-exporter.b> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-11T22:08:45.877 INFO:tasks.cephadm:Adding alertmanager.a on smithi045 2024-03-11T22:08:45.878 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi045=a' 2024-03-11T22:08:46.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:08:45 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug Reading the configuration object to update local LIO configuration 2024-03-11T22:08:46.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:08:45 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug Configuration does not have an entry for this host(smithi144.front.sepia.ceph.com) - nothing to define to LIO 2024-03-11T22:08:46.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:08:45 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-11T22:08:46.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:08:46 smithi144 bash[63604]: 8976c83b2a7cd9359b9c290c75ed525c178cfb59d774390dd2b7b0f5b876fe8e 2024-03-11T22:08:46.096 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:08:46 smithi144 systemd[1]: Started Ceph iscsi.iscsi.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:08:46.173 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:08:46.516 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:46 smithi144 ceph-mon[33984]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi045=a;smithi144=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:46.516 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:46 smithi144 ceph-mon[33984]: Saving service node-exporter spec with placement smithi045=a;smithi144=b;count:2 2024-03-11T22:08:46.516 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:46 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:46.516 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:46 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/1812391597' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T22:08:46.516 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:46 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:46.516 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:46 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:46.516 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:46 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[39715]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi045=a;smithi144=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[39715]: Saving service node-exporter spec with placement smithi045=a;smithi144=b;count:2 2024-03-11T22:08:46.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/1812391597' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T22:08:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[30178]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi045=a;smithi144=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[30178]: Saving service node-exporter spec with placement smithi045=a;smithi144=b;count:2 2024-03-11T22:08:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/1812391597' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T22:08:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:46.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:46 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:47.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:47 smithi144 ceph-mon[33984]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T22:08:47.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:47.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:47 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:47.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:47 smithi144 ceph-mon[33984]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 4.6 KiB/s wr, 148 op/s 2024-03-11T22:08:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:47 smithi045 ceph-mon[30178]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T22:08:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:47 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:47 smithi045 ceph-mon[30178]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 4.6 KiB/s wr, 148 op/s 2024-03-11T22:08:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:47 smithi045 ceph-mon[39715]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T22:08:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:47 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:08:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:47 smithi045 ceph-mon[39715]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 4.6 KiB/s wr, 148 op/s 2024-03-11T22:08:48.306 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:48 smithi144 ceph-mon[33984]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T22:08:48.306 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:48 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:48 smithi045 ceph-mon[30178]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T22:08:48.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:48 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:48 smithi045 ceph-mon[39715]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T22:08:48.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:48 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:48.785 INFO:teuthology.orchestra.run.smithi144.stdout:Scheduled alertmanager update... 2024-03-11T22:08:49.280 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:49 smithi144 ceph-mon[33984]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 132 op/s 2024-03-11T22:08:49.280 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:49 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:49 smithi045 ceph-mon[30178]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 132 op/s 2024-03-11T22:08:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:49 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:49 smithi045 ceph-mon[39715]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 132 op/s 2024-03-11T22:08:49.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:49 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:49.687 DEBUG:teuthology.orchestra.run.smithi045:alertmanager.a> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-11T22:08:49.690 INFO:tasks.cephadm:Adding grafana.a on smithi144 2024-03-11T22:08:49.691 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi144=a' 2024-03-11T22:08:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:50 smithi045 ceph-mon[30178]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi045=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:50.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:50 smithi045 ceph-mon[30178]: Saving service alertmanager spec with placement smithi045=a;count:1 2024-03-11T22:08:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:50 smithi045 ceph-mon[39715]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi045=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:50 smithi045 ceph-mon[39715]: Saving service alertmanager spec with placement smithi045=a;count:1 2024-03-11T22:08:50.568 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:08:50.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:50 smithi144 ceph-mon[33984]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi045=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:50.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:50 smithi144 ceph-mon[33984]: Saving service alertmanager spec with placement smithi045=a;count:1 2024-03-11T22:08:51.762 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:51 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:51.762 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:51 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:51.762 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:51 smithi144 ceph-mon[33984]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 2.5 KiB/s wr, 90 op/s 2024-03-11T22:08:51.762 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:51 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:08:51.763 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:51 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:08:51.763 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:51 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:51.763 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:51 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:51.763 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:51 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:52.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[39715]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 2.5 KiB/s wr, 90 op/s 2024-03-11T22:08:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:08:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:08:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:52.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[30178]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 2.5 KiB/s wr, 90 op/s 2024-03-11T22:08:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:08:52.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:08:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:08:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:08:52.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:51 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:52.533 INFO:teuthology.orchestra.run.smithi144.stdout:Scheduled grafana update... 2024-03-11T22:08:52.835 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:52 smithi144 ceph-mon[33984]: Deploying daemon prometheus.a on smithi144 2024-03-11T22:08:52.835 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:52 smithi144 ceph-mon[33984]: osdmap e77: 8 total, 8 up, 8 in 2024-03-11T22:08:52.835 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:52 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:53.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:52 smithi045 ceph-mon[39715]: Deploying daemon prometheus.a on smithi144 2024-03-11T22:08:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:52 smithi045 ceph-mon[39715]: osdmap e77: 8 total, 8 up, 8 in 2024-03-11T22:08:53.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:52 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:52 smithi045 ceph-mon[30178]: Deploying daemon prometheus.a on smithi144 2024-03-11T22:08:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:52 smithi045 ceph-mon[30178]: osdmap e77: 8 total, 8 up, 8 in 2024-03-11T22:08:53.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:52 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:53.129 DEBUG:teuthology.orchestra.run.smithi144:grafana.a> sudo journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@grafana.a.service 2024-03-11T22:08:53.131 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T22:08:53.132 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:08:53.383 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:08:53.759 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[30178]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi144=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:53.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[30178]: Saving service grafana spec with placement smithi144=a;count:1 2024-03-11T22:08:53.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-03-11T22:08:53.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-11T22:08:53.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-11T22:08:53.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[30178]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.4 KiB/s wr, 86 op/s 2024-03-11T22:08:53.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:53.760 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[39715]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi144=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:53.760 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[39715]: Saving service grafana spec with placement smithi144=a;count:1 2024-03-11T22:08:53.760 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-03-11T22:08:53.760 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-11T22:08:53.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-11T22:08:53.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[39715]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.4 KiB/s wr, 86 op/s 2024-03-11T22:08:53.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:53 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:53 smithi144 ceph-mon[33984]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi144=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:08:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:53 smithi144 ceph-mon[33984]: Saving service grafana spec with placement smithi144=a;count:1 2024-03-11T22:08:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:53 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-03-11T22:08:54.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:53 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-11T22:08:54.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:53 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-11T22:08:54.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:53 smithi144 ceph-mon[33984]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.4 KiB/s wr, 86 op/s 2024-03-11T22:08:54.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:53 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:08:54.907 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:08:54.907 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:08:54.908 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-03-11T22:08:54.908 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-11T22:08:54.908 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-03-11T22:08:54.908 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[30178]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T22:08:54.908 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:08:54.908 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:08:54.908 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:08:54.908 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:08:54.908 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-03-11T22:08:54.908 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-11T22:08:54.908 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-03-11T22:08:54.908 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[39715]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T22:08:54.908 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:08:54.909 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:54 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:08:55.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:54 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:08:55.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:54 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:08:55.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:54 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-03-11T22:08:55.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:54 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-11T22:08:55.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:54 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-03-11T22:08:55.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:54 smithi144 ceph-mon[33984]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T22:08:55.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:54 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:08:55.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:54 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:08:55.447 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-03-11T22:08:55.448 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQD3gO9l7of4GhAAYH3ltEpPVsboSzPQtFBHsg== 2024-03-11T22:08:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:55 smithi045 ceph-mon[30178]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T22:08:55.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:55 smithi045 ceph-mon[30178]: pgmap v220: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:08:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:55 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3741391889' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:08:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:55 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3741391889' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:08:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:55 smithi045 ceph-mon[39715]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T22:08:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:55 smithi045 ceph-mon[39715]: pgmap v220: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:08:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:55 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3741391889' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:08:55.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:55 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3741391889' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:08:55.839 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:08:55 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:08:55.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:55 smithi144 ceph-mon[33984]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T22:08:55.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:55 smithi144 ceph-mon[33984]: pgmap v220: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:08:55.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:55 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3741391889' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:08:55.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:55 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3741391889' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:08:56.146 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-03-11T22:08:56.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T22:08:56.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T22:08:56.188 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:08:56.441 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:08:56.825 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:56 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:08:56.825 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:56 smithi144 ceph-mon[33984]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T22:08:57.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:56 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:08:57.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:56 smithi045 ceph-mon[39715]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T22:08:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:56 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:08:57.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:56 smithi045 ceph-mon[30178]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T22:08:58.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:57 smithi045 ceph-mon[39715]: pgmap v222: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s; 21 B/s, 0 objects/s recovering 2024-03-11T22:08:58.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:57 smithi045 ceph-mon[30178]: pgmap v222: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s; 21 B/s, 0 objects/s recovering 2024-03-11T22:08:58.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:57 smithi144 ceph-mon[33984]: pgmap v222: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s; 21 B/s, 0 objects/s recovering 2024-03-11T22:08:59.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:08:58 smithi045 ceph-mon[39715]: pgmap v223: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 54 B/s, 1 objects/s recovering 2024-03-11T22:08:59.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:08:58 smithi045 ceph-mon[30178]: pgmap v223: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 54 B/s, 1 objects/s recovering 2024-03-11T22:08:59.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:08:58 smithi144 ceph-mon[33984]: pgmap v223: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 54 B/s, 1 objects/s recovering 2024-03-11T22:09:01.007 INFO:teuthology.orchestra.run.smithi144.stdout:[client.1] 2024-03-11T22:09:01.008 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQD9gO9llhyRABAANlybWHWm8U/VMI+1MoY/3w== 2024-03-11T22:09:01.008 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:00 smithi144 ceph-mon[33984]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 0 op/s; 47 B/s, 1 objects/s recovering 2024-03-11T22:09:01.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:00 smithi045 ceph-mon[39715]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 0 op/s; 47 B/s, 1 objects/s recovering 2024-03-11T22:09:01.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:00 smithi045 ceph-mon[30178]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 729 B/s rd, 0 op/s; 47 B/s, 1 objects/s recovering 2024-03-11T22:09:02.006 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:01 smithi144 ceph-mon[33984]: from='client.? 172.21.15.144:0/4193295442' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:09:02.006 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:01 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:09:02.006 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:01 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:09:02.006 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:01 smithi144 ceph-mon[33984]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T22:09:02.006 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:01 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:09:02.006 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:01 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:09:02.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:01 smithi045 ceph-mon[39715]: from='client.? 172.21.15.144:0/4193295442' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:09:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:01 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:09:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:01 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:09:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:01 smithi045 ceph-mon[39715]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T22:09:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:01 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:09:02.026 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:01 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:09:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:01 smithi045 ceph-mon[30178]: from='client.? 172.21.15.144:0/4193295442' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:09:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:01 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:09:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:01 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:09:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:01 smithi045 ceph-mon[30178]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T22:09:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:01 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:09:02.026 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:01 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:09:03.236 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:03 smithi144 ceph-mon[33984]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T22:09:03.236 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:03 smithi144 ceph-mon[33984]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 777 B/s rd, 0 op/s; 33 B/s, 1 objects/s recovering 2024-03-11T22:09:03.264 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-03-11T22:09:03.264 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T22:09:03.264 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T22:09:03.307 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T22:09:03.307 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T22:09:03.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T22:09:03.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:03 smithi045 ceph-mon[39715]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T22:09:03.337 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:03 smithi045 ceph-mon[39715]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 777 B/s rd, 0 op/s; 33 B/s, 1 objects/s recovering 2024-03-11T22:09:03.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:03 smithi045 ceph-mon[30178]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T22:09:03.337 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:03 smithi045 ceph-mon[30178]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 777 B/s rd, 0 op/s; 33 B/s, 1 objects/s recovering 2024-03-11T22:09:03.563 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:04.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:04 smithi144 ceph-mon[33984]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-11T22:09:05.005 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:04 smithi045 ceph-mon[39715]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-11T22:09:05.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:04 smithi045 ceph-mon[30178]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-11T22:09:05.215 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:04 smithi144 podman[65513]: 2024-03-11 22:09:04.949096918 +0000 UTC m=+0.157325616 container create 23c49ad0fa001d3e32ddf4328aaa972fcc8dc58290f3f1382ea76376ac108b95 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:09:05.480 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 podman[65513]: 2024-03-11 22:09:05.353468943 +0000 UTC m=+0.561697606 container init 23c49ad0fa001d3e32ddf4328aaa972fcc8dc58290f3f1382ea76376ac108b95 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:09:05.480 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 podman[65513]: 2024-03-11 22:09:05.358898067 +0000 UTC m=+0.567126723 container start 23c49ad0fa001d3e32ddf4328aaa972fcc8dc58290f3f1382ea76376ac108b95 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:09:05.481 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.383Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T22:09:05.481 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.383Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T22:09:05.481 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.383Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi144 (none))" 2024-03-11T22:09:05.481 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.383Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T22:09:05.481 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.383Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T22:09:05.481 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.384Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T22:09:05.481 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.385Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T22:09:05.481 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.386Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T22:09:05.481 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.386Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T22:09:05.481 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.388Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T22:09:05.481 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.388Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.301µs 2024-03-11T22:09:05.481 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.388Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T22:09:05.481 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.388Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-11T22:09:05.482 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.388Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=29.692µs wal_replay_duration=277.221µs wbl_replay_duration=142ns total_replay_duration=380.169µs 2024-03-11T22:09:05.482 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.390Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T22:09:05.482 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.390Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T22:09:05.482 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.390Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T22:09:05.482 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.407Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.180443ms db_storage=671ns remote_storage=1.088µs web_handler=337ns query_engine=478ns scrape=384.583µs scrape_sd=75.729µs notify=700ns notify_sd=1.413µs rules=16.483403ms tracing=5.608µs 2024-03-11T22:09:05.482 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.407Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T22:09:05.482 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:09:05.407Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T22:09:05.482 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 bash[65513]: 23c49ad0fa001d3e32ddf4328aaa972fcc8dc58290f3f1382ea76376ac108b95 2024-03-11T22:09:05.482 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:09:05 smithi144 systemd[1]: Started Ceph prometheus.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:09:05.839 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:09:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:09:05.885 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:09:06.634 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3882489981},{"type":"v1","addr":"172.21.15.45:6801","nonce":3882489981}]},"active_addr":"172.21.15.45:6801/3882489981","active_change":"2024-03-11T22:03:52.557161+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.45:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":1903379086}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":272906373}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":3509006862}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":1901139892}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":11898002}]}]} 2024-03-11T22:09:06.637 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T22:09:06.637 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T22:09:06.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:09:06.776 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ignoring --setuser ceph since I am not root 2024-03-11T22:09:06.776 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ignoring --setgroup ceph since I am not root 2024-03-11T22:09:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:09:06.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:09:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:09:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:09:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3632437889' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:09:06.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[30178]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T22:09:06.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:09:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:09:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:09:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:09:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3632437889' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:09:06.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:06 smithi045 ceph-mon[39715]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T22:09:06.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:09:06.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:09:06.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' 2024-03-11T22:09:06.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:06 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:09:06.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:06 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:06.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:06 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3632437889' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:09:06.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:06 smithi144 ceph-mon[33984]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T22:09:06.840 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: ignoring --setuser ceph since I am not root 2024-03-11T22:09:06.840 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: ignoring --setgroup ceph since I am not root 2024-03-11T22:09:06.840 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:06 smithi144 ceph-mgr[36089]: -- 172.21.15.144:0/685936867 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ae3954a1a0 con 0x55ae39521000 2024-03-11T22:09:06.928 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:07.339 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:07 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:07.172+0000 7fa28721a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:09:07.339 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:07 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:07.275+0000 7fa28721a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:09:07.559 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:07 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:07.126+0000 7ff9eb9b2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:09:07.559 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:07 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:07.221+0000 7ff9eb9b2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:09:07.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:07 smithi045 ceph-mon[30178]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:09:07.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:07 smithi045 ceph-mon[30178]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-11T22:09:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:07 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:09:07.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:07 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:09:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:07 smithi045 ceph-mon[39715]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:09:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:07 smithi045 ceph-mon[39715]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-11T22:09:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:07 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:09:07.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:07 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:09:07.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:07 smithi144 ceph-mon[33984]: from='mgr.14152 172.21.15.45:0/3453148444' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:09:07.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:07 smithi144 ceph-mon[33984]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-11T22:09:07.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:07 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:09:07.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:07 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:09:08.152 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:07 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:07.946+0000 7fa28721a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:09:08.152 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:08 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:08.042+0000 7fa28721a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:09:08.185 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:07 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:07.858+0000 7ff9eb9b2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:09:08.185 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:07 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:07.951+0000 7ff9eb9b2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:09:08.185 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:08 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:08.094+0000 7ff9eb9b2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:09:08.447 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:08 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:08.186+0000 7fa28721a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:09:08.447 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:08 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:08.290+0000 7fa28721a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:09:08.447 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:08 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:08.377+0000 7fa28721a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:09:08.532 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:08 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:08.190+0000 7ff9eb9b2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:09:08.532 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:08 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:08.273+0000 7ff9eb9b2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:09:08.532 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:08 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:08.537+0000 7ff9eb9b2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:09:08.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:08 smithi045 ceph-mon[39715]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T22:09:08.776 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:08 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:08.665+0000 7fa28721a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:09:08.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:08 smithi045 ceph-mon[30178]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T22:09:08.823 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:08 smithi144 ceph-mon[33984]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T22:09:08.823 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:08 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:08.647+0000 7ff9eb9b2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:09:08.824 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:08 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:08.733+0000 7ff9eb9b2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:09:08.824 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:08 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:08.828+0000 7ff9eb9b2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:09:08.930 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:09:08.930 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":84,"fsid":"f75aafa4-dff2-11ee-95c8-87774f69a715","created":"2024-03-11T22:02:51.075940+0000","modified":"2024-03-11T22:09:07.564168+0000","last_up_change":"2024-03-11T22:08:17.992246+0000","last_in_change":"2024-03-11T22:08:00.853801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:06:12.903386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-11T22:08:31.014060+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-11T22:08:31.591167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-11T22:08:33.606516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T22:08:35.315522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T22:08:37.383856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55a4e633-f2a6-4d27-b10c-de1c0f87cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6803","nonce":2216353820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6805","nonce":2216353820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6809","nonce":2216353820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6807","nonce":2216353820}]},"public_addr":"172.21.15.45:6803/2216353820","cluster_addr":"172.21.15.45:6805/2216353820","heartbeat_back_addr":"172.21.15.45:6809/2216353820","heartbeat_front_addr":"172.21.15.45:6807/2216353820","state":["exists","up"]},{"osd":1,"uuid":"60274eeb-591a-43fe-a579-f13d13631e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6811","nonce":1241486091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6813","nonce":1241486091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6817","nonce":1241486091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6815","nonce":1241486091}]},"public_addr":"172.21.15.45:6811/1241486091","cluster_addr":"172.21.15.45:6813/1241486091","heartbeat_back_addr":"172.21.15.45:6817/1241486091","heartbeat_front_addr":"172.21.15.45:6815/1241486091","state":["exists","up"]},{"osd":2,"uuid":"15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6819","nonce":4251983518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6821","nonce":4251983518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6825","nonce":4251983518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6823","nonce":4251983518}]},"public_addr":"172.21.15.45:6819/4251983518","cluster_addr":"172.21.15.45:6821/4251983518","heartbeat_back_addr":"172.21.15.45:6825/4251983518","heartbeat_front_addr":"172.21.15.45:6823/4251983518","state":["exists","up"]},{"osd":3,"uuid":"e51a65f3-96f6-478c-8d26-df1ace4a5058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6827","nonce":3178755830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6829","nonce":3178755830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6833","nonce":3178755830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6831","nonce":3178755830}]},"public_addr":"172.21.15.45:6827/3178755830","cluster_addr":"172.21.15.45:6829/3178755830","heartbeat_back_addr":"172.21.15.45:6833/3178755830","heartbeat_front_addr":"172.21.15.45:6831/3178755830","state":["exists","up"]},{"osd":4,"uuid":"32d0b71f-c696-4677-bdd3-b88a04c97604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6801","nonce":912745159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6803","nonce":912745159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6807","nonce":912745159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6805","nonce":912745159}]},"public_addr":"172.21.15.144:6801/912745159","cluster_addr":"172.21.15.144:6803/912745159","heartbeat_back_addr":"172.21.15.144:6807/912745159","heartbeat_front_addr":"172.21.15.144:6805/912745159","state":["exists","up"]},{"osd":5,"uuid":"f46b91af-d67d-404c-aad1-75348e35d1c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6809","nonce":3162869174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6811","nonce":3162869174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6815","nonce":3162869174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6813","nonce":3162869174}]},"public_addr":"172.21.15.144:6809/3162869174","cluster_addr":"172.21.15.144:6811/3162869174","heartbeat_back_addr":"172.21.15.144:6815/3162869174","heartbeat_front_addr":"172.21.15.144:6813/3162869174","state":["exists","up"]},{"osd":6,"uuid":"970b128d-cd56-4dff-8345-3196b46c1257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6817","nonce":3439446288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6819","nonce":3439446288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6823","nonce":3439446288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6821","nonce":3439446288}]},"public_addr":"172.21.15.144:6817/3439446288","cluster_addr":"172.21.15.144:6819/3439446288","heartbeat_back_addr":"172.21.15.144:6823/3439446288","heartbeat_front_addr":"172.21.15.144:6821/3439446288","state":["exists","up"]},{"osd":7,"uuid":"40100aaf-830d-4137-b037-5cf167cba308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6825","nonce":2371064429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6827","nonce":2371064429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6831","nonce":2371064429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6829","nonce":2371064429}]},"public_addr":"172.21.15.144:6825/2371064429","cluster_addr":"172.21.15.144:6827/2371064429","heartbeat_back_addr":"172.21.15.144:6831/2371064429","heartbeat_front_addr":"172.21.15.144:6829/2371064429","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:53.169229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:06.696931+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:50.905938+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:09:01.283597+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:21.276126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:53.858762+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:09:06.822665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:16.977128+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/2584267737":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/2513993410":"2024-03-12T22:03:52.557006+0000","172.21.15.45:6800/4256719031":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/2777740355":"2024-03-12T22:03:25.087713+0000","172.21.15.45:0/926341889":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6801/4256719031":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/2331445880":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6800/2299156110":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6801/2299156110":"2024-03-12T22:03:25.087713+0000","172.21.15.45:0/1110781721":"2024-03-12T22:03:52.557006+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:09:09.089 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:09 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:09.008+0000 7ff9eb9b2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:09:09.174 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:08 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:08.807+0000 7fa28721a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:09:09.175 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:08 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:08.895+0000 7fa28721a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:09:09.175 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:08 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:08.997+0000 7fa28721a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:09:09.175 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:09 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:09.179+0000 7fa28721a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:09:09.736 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T22:09:09.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:09:09.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:09 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2943452362' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:09:09.984 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:10.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:09 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2943452362' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:09:10.025 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:09 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2943452362' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:09:10.339 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:09 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:09.968+0000 7ff9eb9b2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:09:10.339 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:10 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:10.098+0000 7ff9eb9b2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:09:10.407 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:10 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:10.186+0000 7fa28721a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:09:10.407 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:10 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:10.326+0000 7fa28721a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:09:10.716 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:10 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:10.435+0000 7ff9eb9b2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:09:10.716 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:10 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:10.524+0000 7ff9eb9b2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:09:10.716 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:10 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:10.721+0000 7ff9eb9b2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:09:10.963 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:10 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:10.670+0000 7fa28721a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:09:10.963 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:10 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:10.763+0000 7fa28721a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:09:11.089 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:10 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:10.906+0000 7ff9eb9b2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:09:11.275 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:10 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:10.968+0000 7fa28721a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:09:11.275 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:11 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:11.163+0000 7fa28721a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:09:11.457 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:11 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:11.195+0000 7ff9eb9b2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:09:11.457 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:11 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:11.461+0000 7ff9eb9b2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:09:11.609 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:11 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:11.467+0000 7fa28721a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:09:11.839 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:11 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:11.699+0000 7ff9eb9b2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:09:12.015 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:11 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:11.734+0000 7fa28721a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:09:12.118 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:09:12.118 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":85,"fsid":"f75aafa4-dff2-11ee-95c8-87774f69a715","created":"2024-03-11T22:02:51.075940+0000","modified":"2024-03-11T22:09:11.038027+0000","last_up_change":"2024-03-11T22:08:17.992246+0000","last_in_change":"2024-03-11T22:08:00.853801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:06:12.903386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-11T22:08:31.014060+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-11T22:08:31.591167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-11T22:08:33.606516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T22:08:35.315522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T22:08:37.383856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55a4e633-f2a6-4d27-b10c-de1c0f87cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6803","nonce":2216353820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6805","nonce":2216353820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6809","nonce":2216353820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6807","nonce":2216353820}]},"public_addr":"172.21.15.45:6803/2216353820","cluster_addr":"172.21.15.45:6805/2216353820","heartbeat_back_addr":"172.21.15.45:6809/2216353820","heartbeat_front_addr":"172.21.15.45:6807/2216353820","state":["exists","up"]},{"osd":1,"uuid":"60274eeb-591a-43fe-a579-f13d13631e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6811","nonce":1241486091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6813","nonce":1241486091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6817","nonce":1241486091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6815","nonce":1241486091}]},"public_addr":"172.21.15.45:6811/1241486091","cluster_addr":"172.21.15.45:6813/1241486091","heartbeat_back_addr":"172.21.15.45:6817/1241486091","heartbeat_front_addr":"172.21.15.45:6815/1241486091","state":["exists","up"]},{"osd":2,"uuid":"15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6819","nonce":4251983518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6821","nonce":4251983518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6825","nonce":4251983518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6823","nonce":4251983518}]},"public_addr":"172.21.15.45:6819/4251983518","cluster_addr":"172.21.15.45:6821/4251983518","heartbeat_back_addr":"172.21.15.45:6825/4251983518","heartbeat_front_addr":"172.21.15.45:6823/4251983518","state":["exists","up"]},{"osd":3,"uuid":"e51a65f3-96f6-478c-8d26-df1ace4a5058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6827","nonce":3178755830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6829","nonce":3178755830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6833","nonce":3178755830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6831","nonce":3178755830}]},"public_addr":"172.21.15.45:6827/3178755830","cluster_addr":"172.21.15.45:6829/3178755830","heartbeat_back_addr":"172.21.15.45:6833/3178755830","heartbeat_front_addr":"172.21.15.45:6831/3178755830","state":["exists","up"]},{"osd":4,"uuid":"32d0b71f-c696-4677-bdd3-b88a04c97604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6801","nonce":912745159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6803","nonce":912745159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6807","nonce":912745159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6805","nonce":912745159}]},"public_addr":"172.21.15.144:6801/912745159","cluster_addr":"172.21.15.144:6803/912745159","heartbeat_back_addr":"172.21.15.144:6807/912745159","heartbeat_front_addr":"172.21.15.144:6805/912745159","state":["exists","up"]},{"osd":5,"uuid":"f46b91af-d67d-404c-aad1-75348e35d1c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6809","nonce":3162869174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6811","nonce":3162869174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6815","nonce":3162869174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6813","nonce":3162869174}]},"public_addr":"172.21.15.144:6809/3162869174","cluster_addr":"172.21.15.144:6811/3162869174","heartbeat_back_addr":"172.21.15.144:6815/3162869174","heartbeat_front_addr":"172.21.15.144:6813/3162869174","state":["exists","up"]},{"osd":6,"uuid":"970b128d-cd56-4dff-8345-3196b46c1257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6817","nonce":3439446288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6819","nonce":3439446288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6823","nonce":3439446288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6821","nonce":3439446288}]},"public_addr":"172.21.15.144:6817/3439446288","cluster_addr":"172.21.15.144:6819/3439446288","heartbeat_back_addr":"172.21.15.144:6823/3439446288","heartbeat_front_addr":"172.21.15.144:6821/3439446288","state":["exists","up"]},{"osd":7,"uuid":"40100aaf-830d-4137-b037-5cf167cba308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6825","nonce":2371064429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6827","nonce":2371064429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6831","nonce":2371064429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6829","nonce":2371064429}]},"public_addr":"172.21.15.144:6825/2371064429","cluster_addr":"172.21.15.144:6827/2371064429","heartbeat_back_addr":"172.21.15.144:6831/2371064429","heartbeat_front_addr":"172.21.15.144:6829/2371064429","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:53.169229+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:06.696931+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:50.905938+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:09:01.283597+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:21.276126+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:53.858762+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:09:06.822665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:08:16.977128+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/2584267737":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/2513993410":"2024-03-12T22:03:52.557006+0000","172.21.15.45:6800/4256719031":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/2777740355":"2024-03-12T22:03:25.087713+0000","172.21.15.45:0/926341889":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6801/4256719031":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/2331445880":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6800/2299156110":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6801/2299156110":"2024-03-12T22:03:25.087713+0000","172.21.15.45:0/1110781721":"2024-03-12T22:03:52.557006+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:09:12.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:12 smithi045 ceph-mon[30178]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T22:09:12.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:12 smithi045 ceph-mon[39715]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T22:09:12.276 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:12.020+0000 7fa28721a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:09:12.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:12 smithi144 ceph-mon[33984]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T22:09:12.839 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:12 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:12.499+0000 7ff9eb9b2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:09:12.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T22:09:12.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T22:09:12.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T22:09:12.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T22:09:12.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T22:09:12.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T22:09:12.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T22:09:12.877 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T22:09:13.275 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:12.865+0000 7fa28721a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:09:13.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:13 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/767504294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:09:13.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:13 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/767504294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:09:13.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:13 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/767504294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:09:13.339 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:13.078+0000 7ff9eb9b2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:09:13.339 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:13.174+0000 7ff9eb9b2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:09:13.400 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:13.405 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:13.406 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:13.450 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:13.455 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:13.797 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:13 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:13.669+0000 7fa28721a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:09:13.797 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:13 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:13.778+0000 7fa28721a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:09:13.839 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:13.390+0000 7ff9eb9b2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:09:13.839 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: [11/Mar/2024:22:09:13] ENGINE Bus STARTING 2024-03-11T22:09:13.839 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: CherryPy Checker: 2024-03-11T22:09:13.840 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: The Application mounted at '' has an empty config. 2024-03-11T22:09:13.840 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: 2024-03-11T22:09:13.840 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: [11/Mar/2024:22:09:13] ENGINE Serving on http://:::9283 2024-03-11T22:09:13.840 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:09:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x[36066]: [11/Mar/2024:22:09:13] ENGINE Bus STARTED 2024-03-11T22:09:14.048 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:14.007+0000 7fa28721a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:09:14.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: Standby manager daemon x restarted 2024-03-11T22:09:14.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: Standby manager daemon x started 2024-03-11T22:09:14.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.? 172.21.15.144:0/3472395254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:09:14.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.? 172.21.15.144:0/3472395254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:09:14.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.? 172.21.15.144:0/3472395254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:09:14.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.? 172.21.15.144:0/3472395254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:09:14.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: Active manager daemon y restarted 2024-03-11T22:09:14.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: Activating manager daemon y 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: mgrmap e18: y(active, starting, since 0.00938766s), standbys: x 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:09:14.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:09:14.342 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:09:14.342 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:09:14.342 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:09:14.342 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:09:14.525 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:09:14] ENGINE Bus STARTING 2024-03-11T22:09:14.526 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: CherryPy Checker: 2024-03-11T22:09:14.526 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: The Application mounted at '' has an empty config. 2024-03-11T22:09:14.526 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: 2024-03-11T22:09:14.526 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:09:14] ENGINE Serving on http://:::9283 2024-03-11T22:09:14.526 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:09:14] ENGINE Bus STARTED 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: Standby manager daemon x restarted 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: Standby manager daemon x started 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.? 172.21.15.144:0/3472395254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.? 172.21.15.144:0/3472395254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.? 172.21.15.144:0/3472395254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.? 172.21.15.144:0/3472395254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: Active manager daemon y restarted 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: Activating manager daemon y 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: mgrmap e18: y(active, starting, since 0.00938766s), standbys: x 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:09:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:09:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:09:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:09:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:09:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:09:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:09:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:09:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:09:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:09:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:09:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:09:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:09:14.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:09:14.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: Standby manager daemon x restarted 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: Standby manager daemon x started 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.? 172.21.15.144:0/3472395254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.? 172.21.15.144:0/3472395254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.? 172.21.15.144:0/3472395254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.? 172.21.15.144:0/3472395254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: Active manager daemon y restarted 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: Activating manager daemon y 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: mgrmap e18: y(active, starting, since 0.00938766s), standbys: x 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:09:14.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:09:14.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:09:14.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:09:14.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:09:14.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:09:14.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:09:14.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T22:09:14.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T22:09:14.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:09:14.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:09:14.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:09:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[30178]: Manager daemon y is now available 2024-03-11T22:09:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:09:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:09:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:09:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:09:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:09:15.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[39715]: Manager daemon y is now available 2024-03-11T22:09:15.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:09:15.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:09:15.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:15.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:09:15.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:09:15.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:09:15.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:15 smithi144 ceph-mon[33984]: Manager daemon y is now available 2024-03-11T22:09:15.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:09:15.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:09:15.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:15.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:09:15.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:09:15.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:09:15.839 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:09:15 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:09:16.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[39715]: mgrmap e19: y(active, since 1.04571s), standbys: x 2024-03-11T22:09:16.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[39715]: [11/Mar/2024:22:09:15] ENGINE Bus STARTING 2024-03-11T22:09:16.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[39715]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:09:16.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[39715]: [11/Mar/2024:22:09:15] ENGINE Serving on http://172.21.15.45:8765 2024-03-11T22:09:16.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[39715]: [11/Mar/2024:22:09:15] ENGINE Serving on https://172.21.15.45:7150 2024-03-11T22:09:16.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[39715]: [11/Mar/2024:22:09:15] ENGINE Bus STARTED 2024-03-11T22:09:16.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:16.329 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[39715]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T22:09:16.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[30178]: mgrmap e19: y(active, since 1.04571s), standbys: x 2024-03-11T22:09:16.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[30178]: [11/Mar/2024:22:09:15] ENGINE Bus STARTING 2024-03-11T22:09:16.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[30178]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:09:16.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[30178]: [11/Mar/2024:22:09:15] ENGINE Serving on http://172.21.15.45:8765 2024-03-11T22:09:16.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[30178]: [11/Mar/2024:22:09:15] ENGINE Serving on https://172.21.15.45:7150 2024-03-11T22:09:16.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[30178]: [11/Mar/2024:22:09:15] ENGINE Bus STARTED 2024-03-11T22:09:16.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:16.330 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:16 smithi045 ceph-mon[30178]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T22:09:16.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:16 smithi144 ceph-mon[33984]: mgrmap e19: y(active, since 1.04571s), standbys: x 2024-03-11T22:09:16.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:16 smithi144 ceph-mon[33984]: [11/Mar/2024:22:09:15] ENGINE Bus STARTING 2024-03-11T22:09:16.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:16 smithi144 ceph-mon[33984]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:09:16.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:16 smithi144 ceph-mon[33984]: [11/Mar/2024:22:09:15] ENGINE Serving on http://172.21.15.45:8765 2024-03-11T22:09:16.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:16 smithi144 ceph-mon[33984]: [11/Mar/2024:22:09:15] ENGINE Serving on https://172.21.15.45:7150 2024-03-11T22:09:16.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:16 smithi144 ceph-mon[33984]: [11/Mar/2024:22:09:15] ENGINE Bus STARTED 2024-03-11T22:09:16.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:16 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:16.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:16 smithi144 ceph-mon[33984]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T22:09:17.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:17 smithi045 ceph-mon[30178]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:09:17.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:17 smithi045 ceph-mon[30178]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-11T22:09:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:17 smithi045 ceph-mon[39715]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:09:17.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:17 smithi045 ceph-mon[39715]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-11T22:09:17.316 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:17 smithi144 ceph-mon[33984]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:09:17.316 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:17 smithi144 ceph-mon[33984]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-11T22:09:18.763 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:18 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:18.763 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:18 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:18.763 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:18 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:18.763 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:18 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:19.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:18 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:19.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:18 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:19.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[39715]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:09:19.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[39715]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-11T22:09:19.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:19.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:19.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:19.694 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:19.694 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:19.694 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:19.694 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:19.694 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[30178]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:09:19.694 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[30178]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-11T22:09:19.694 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:19.694 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:19.694 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:19.694 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:19.694 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:19.694 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:19.695 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:20.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:19 smithi144 ceph-mon[33984]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:09:20.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:19 smithi144 ceph-mon[33984]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-11T22:09:20.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:20.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:20.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:20.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:20.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:20.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:20.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:20.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:20 smithi045 ceph-mon[39715]: Adjusting osd_memory_target on smithi144 to 3760M 2024-03-11T22:09:20.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:20 smithi045 ceph-mon[30178]: Adjusting osd_memory_target on smithi144 to 3760M 2024-03-11T22:09:20.882 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:20 smithi144 ceph-mon[33984]: Adjusting osd_memory_target on smithi144 to 3760M 2024-03-11T22:09:21.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:21 smithi045 ceph-mon[39715]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:09:21.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:21 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:09:21.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:21 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:09:21.946 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:21 smithi045 ceph-mon[30178]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:09:21.946 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:21 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:09:21.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:21 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:09:22.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:21 smithi144 ceph-mon[33984]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:09:22.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:21 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:09:22.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:21 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:09:22.500 INFO:teuthology.orchestra.run.smithi045.stdout:111669149729 2024-03-11T22:09:22.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T22:09:22.759 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:09:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:09:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:23 smithi045 ceph-mon[30178]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T22:09:23.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:23 smithi045 ceph-mon[30178]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T22:09:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:23 smithi045 ceph-mon[39715]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T22:09:23.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:23 smithi045 ceph-mon[39715]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T22:09:23.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:23 smithi144 ceph-mon[33984]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T22:09:23.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:23 smithi144 ceph-mon[33984]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T22:09:24.512 INFO:teuthology.orchestra.run.smithi045.stdout:141733920798 2024-03-11T22:09:24.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T22:09:25.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:25 smithi144 ceph-mon[33984]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-11T22:09:25.452 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:25 smithi045 ceph-mon[39715]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-11T22:09:25.453 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:25 smithi045 ceph-mon[30178]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-11T22:09:26.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:09:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:09:26.268 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:26 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:26.268 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:26 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:09:26.268 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:26 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:09:26.268 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:26 smithi045 ceph-mon[39715]: osdmap e89: 8 total, 8 up, 8 in 2024-03-11T22:09:26.268 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:26 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:26.268 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:26 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:09:26.268 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:26 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:09:26.269 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:26 smithi045 ceph-mon[30178]: osdmap e89: 8 total, 8 up, 8 in 2024-03-11T22:09:26.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:26 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:26.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:26 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:09:26.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:26 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:09:26.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:26 smithi144 ceph-mon[33984]: osdmap e89: 8 total, 8 up, 8 in 2024-03-11T22:09:27.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:27 smithi144 ceph-mon[33984]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T22:09:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:27 smithi045 ceph-mon[39715]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T22:09:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:27 smithi045 ceph-mon[30178]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T22:09:28.179 INFO:teuthology.orchestra.run.smithi045.stdout:77309411368 2024-03-11T22:09:28.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T22:09:28.532 INFO:teuthology.orchestra.run.smithi045.stdout:261993005071 2024-03-11T22:09:28.533 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T22:09:29.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:29 smithi045 ceph-mon[30178]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-11T22:09:29.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:29 smithi045 ceph-mon[39715]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-11T22:09:29.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:29 smithi144 ceph-mon[33984]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-11T22:09:29.942 INFO:teuthology.orchestra.run.smithi045.stdout:55834574894 2024-03-11T22:09:29.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T22:09:30.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:09:30.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:09:30.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:09:30.782 INFO:teuthology.orchestra.run.smithi045.stdout:34359738420 2024-03-11T22:09:30.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T22:09:31.321 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:31 smithi045 ceph-mon[39715]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T22:09:31.321 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:31 smithi045 ceph-mon[30178]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T22:09:31.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:31 smithi144 ceph-mon[33984]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T22:09:31.977 INFO:teuthology.orchestra.run.smithi045.stdout:176093659161 2024-03-11T22:09:31.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T22:09:32.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:32 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:09:32.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:32 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:09:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:32 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:09:32.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:32 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:09:32.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:32 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:09:32.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:32 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:09:32.498 INFO:teuthology.orchestra.run.smithi045.stdout:219043332116 2024-03-11T22:09:32.499 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T22:09:32.536 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:32.645 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:32.645 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:32.651 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:32.775 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:09:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:09:32.789 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:32.894 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:32.894 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:33.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:33 smithi045 ceph-mon[39715]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-11T22:09:33.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:33 smithi045 ceph-mon[39715]: osdmap e90: 8 total, 8 up, 8 in 2024-03-11T22:09:33.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:33 smithi045 ceph-mon[30178]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-11T22:09:33.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:33 smithi045 ceph-mon[30178]: osdmap e90: 8 total, 8 up, 8 in 2024-03-11T22:09:33.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:33 smithi144 ceph-mon[33984]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-11T22:09:33.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:33 smithi144 ceph-mon[33984]: osdmap e90: 8 total, 8 up, 8 in 2024-03-11T22:09:35.197 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:35 smithi045 ceph-mon[39715]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T22:09:35.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:35 smithi045 ceph-mon[30178]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T22:09:35.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:35 smithi144 ceph-mon[33984]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T22:09:35.839 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:09:35 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:09:36.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:36 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:36.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:36 smithi144 ceph-mon[33984]: osdmap e91: 8 total, 8 up, 8 in 2024-03-11T22:09:36.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:36 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:36.371 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:36 smithi045 ceph-mon[30178]: osdmap e91: 8 total, 8 up, 8 in 2024-03-11T22:09:36.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:36 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:36.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:36 smithi045 ceph-mon[39715]: osdmap e91: 8 total, 8 up, 8 in 2024-03-11T22:09:37.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:37 smithi144 ceph-mon[33984]: 3.0 scrub starts 2024-03-11T22:09:37.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:37 smithi144 ceph-mon[33984]: 3.0 scrub ok 2024-03-11T22:09:37.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:37 smithi144 ceph-mon[33984]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-03-11T22:09:37.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:37 smithi144 ceph-mon[33984]: 4.8 deep-scrub starts 2024-03-11T22:09:37.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:37 smithi144 ceph-mon[33984]: 4.8 deep-scrub ok 2024-03-11T22:09:37.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:37 smithi144 ceph-mon[33984]: 3.7 scrub starts 2024-03-11T22:09:37.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:37 smithi144 ceph-mon[33984]: 3.7 scrub ok 2024-03-11T22:09:37.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[39715]: 3.0 scrub starts 2024-03-11T22:09:37.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[39715]: 3.0 scrub ok 2024-03-11T22:09:37.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[39715]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-03-11T22:09:37.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[39715]: 4.8 deep-scrub starts 2024-03-11T22:09:37.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[39715]: 4.8 deep-scrub ok 2024-03-11T22:09:37.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[39715]: 3.7 scrub starts 2024-03-11T22:09:37.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[39715]: 3.7 scrub ok 2024-03-11T22:09:37.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[30178]: 3.0 scrub starts 2024-03-11T22:09:37.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[30178]: 3.0 scrub ok 2024-03-11T22:09:37.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[30178]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-03-11T22:09:37.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[30178]: 4.8 deep-scrub starts 2024-03-11T22:09:37.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[30178]: 4.8 deep-scrub ok 2024-03-11T22:09:37.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[30178]: 3.7 scrub starts 2024-03-11T22:09:37.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:37 smithi045 ceph-mon[30178]: 3.7 scrub ok 2024-03-11T22:09:39.052 INFO:teuthology.orchestra.run.smithi045.stdout:176093659162 2024-03-11T22:09:39.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:39 smithi045 ceph-mon[39715]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:09:39.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:39 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3235678264' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T22:09:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:39 smithi045 ceph-mon[30178]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:09:39.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:39 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3235678264' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T22:09:39.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:39 smithi144 ceph-mon[33984]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:09:39.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:39 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3235678264' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T22:09:40.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:40 smithi045 ceph-mon[30178]: 5.d deep-scrub starts 2024-03-11T22:09:40.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:40 smithi045 ceph-mon[30178]: 5.d deep-scrub ok 2024-03-11T22:09:40.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:40 smithi045 ceph-mon[30178]: 3.1e scrub starts 2024-03-11T22:09:40.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:40 smithi045 ceph-mon[30178]: 3.1e scrub ok 2024-03-11T22:09:40.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:40 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:40.097 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:40 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:40.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:40 smithi045 ceph-mon[39715]: 5.d deep-scrub starts 2024-03-11T22:09:40.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:40 smithi045 ceph-mon[39715]: 5.d deep-scrub ok 2024-03-11T22:09:40.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:40 smithi045 ceph-mon[39715]: 3.1e scrub starts 2024-03-11T22:09:40.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:40 smithi045 ceph-mon[39715]: 3.1e scrub ok 2024-03-11T22:09:40.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:40 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:40.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:40 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:40.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:40 smithi144 ceph-mon[33984]: 5.d deep-scrub starts 2024-03-11T22:09:40.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:40 smithi144 ceph-mon[33984]: 5.d deep-scrub ok 2024-03-11T22:09:40.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:40 smithi144 ceph-mon[33984]: 3.1e scrub starts 2024-03-11T22:09:40.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:40 smithi144 ceph-mon[33984]: 3.1e scrub ok 2024-03-11T22:09:40.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:40 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:40.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:40 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:41.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:41 smithi045 ceph-mon[39715]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:09:41.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:41 smithi045 ceph-mon[39715]: 3.1d scrub starts 2024-03-11T22:09:41.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:41 smithi045 ceph-mon[39715]: 3.1d scrub ok 2024-03-11T22:09:41.312 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:41 smithi045 ceph-mon[39715]: 5.11 scrub starts 2024-03-11T22:09:41.313 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:41 smithi045 ceph-mon[39715]: 5.11 scrub ok 2024-03-11T22:09:41.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:41 smithi045 ceph-mon[30178]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:09:41.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:41 smithi045 ceph-mon[30178]: 3.1d scrub starts 2024-03-11T22:09:41.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:41 smithi045 ceph-mon[30178]: 3.1d scrub ok 2024-03-11T22:09:41.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:41 smithi045 ceph-mon[30178]: 5.11 scrub starts 2024-03-11T22:09:41.313 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:41 smithi045 ceph-mon[30178]: 5.11 scrub ok 2024-03-11T22:09:41.339 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659161 got 176093659162 for osd.5 2024-03-11T22:09:41.339 DEBUG:teuthology.parallel:result is None 2024-03-11T22:09:41.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:41 smithi144 ceph-mon[33984]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:09:41.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:41 smithi144 ceph-mon[33984]: 3.1d scrub starts 2024-03-11T22:09:41.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:41 smithi144 ceph-mon[33984]: 3.1d scrub ok 2024-03-11T22:09:41.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:41 smithi144 ceph-mon[33984]: 5.11 scrub starts 2024-03-11T22:09:41.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:41 smithi144 ceph-mon[33984]: 5.11 scrub ok 2024-03-11T22:09:42.006 INFO:teuthology.orchestra.run.smithi045.stdout:261993005073 2024-03-11T22:09:42.091 INFO:teuthology.orchestra.run.smithi045.stdout:141733920802 2024-03-11T22:09:42.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[39715]: 4.e scrub starts 2024-03-11T22:09:42.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[39715]: 4.e scrub ok 2024-03-11T22:09:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[39715]: 3.15 scrub starts 2024-03-11T22:09:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[39715]: 3.15 scrub ok 2024-03-11T22:09:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[39715]: 3.2 scrub starts 2024-03-11T22:09:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[39715]: 3.2 scrub ok 2024-03-11T22:09:42.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1450973527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T22:09:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[30178]: 4.e scrub starts 2024-03-11T22:09:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[30178]: 4.e scrub ok 2024-03-11T22:09:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[30178]: 3.15 scrub starts 2024-03-11T22:09:42.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[30178]: 3.15 scrub ok 2024-03-11T22:09:42.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[30178]: 3.2 scrub starts 2024-03-11T22:09:42.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[30178]: 3.2 scrub ok 2024-03-11T22:09:42.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1450973527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T22:09:42.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:42 smithi144 ceph-mon[33984]: 4.e scrub starts 2024-03-11T22:09:42.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:42 smithi144 ceph-mon[33984]: 4.e scrub ok 2024-03-11T22:09:42.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:42 smithi144 ceph-mon[33984]: 3.15 scrub starts 2024-03-11T22:09:42.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:42 smithi144 ceph-mon[33984]: 3.15 scrub ok 2024-03-11T22:09:42.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:42 smithi144 ceph-mon[33984]: 3.2 scrub starts 2024-03-11T22:09:42.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:42 smithi144 ceph-mon[33984]: 3.2 scrub ok 2024-03-11T22:09:42.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:42 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1450973527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T22:09:42.613 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:09:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:09:42.638 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005071 got 261993005073 for osd.7 2024-03-11T22:09:42.638 DEBUG:teuthology.parallel:result is None 2024-03-11T22:09:43.251 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[39715]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:09:43.252 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[39715]: 3.d scrub starts 2024-03-11T22:09:43.252 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[39715]: 3.d scrub ok 2024-03-11T22:09:43.252 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/197301756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T22:09:43.252 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/528062811' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:09:43.252 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[39715]: 4.10 scrub starts 2024-03-11T22:09:43.252 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[39715]: 4.10 scrub ok 2024-03-11T22:09:43.252 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[30178]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:09:43.252 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[30178]: 3.d scrub starts 2024-03-11T22:09:43.252 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[30178]: 3.d scrub ok 2024-03-11T22:09:43.252 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/197301756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T22:09:43.252 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/528062811' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:09:43.253 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[30178]: 4.10 scrub starts 2024-03-11T22:09:43.253 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:43 smithi045 ceph-mon[30178]: 4.10 scrub ok 2024-03-11T22:09:43.278 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920798 got 141733920802 for osd.4 2024-03-11T22:09:43.278 DEBUG:teuthology.parallel:result is None 2024-03-11T22:09:43.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:43 smithi144 ceph-mon[33984]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:09:43.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:43 smithi144 ceph-mon[33984]: 3.d scrub starts 2024-03-11T22:09:43.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:43 smithi144 ceph-mon[33984]: 3.d scrub ok 2024-03-11T22:09:43.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:43 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/197301756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T22:09:43.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:43 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/528062811' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:09:43.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:43 smithi144 ceph-mon[33984]: 4.10 scrub starts 2024-03-11T22:09:43.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:43 smithi144 ceph-mon[33984]: 4.10 scrub ok 2024-03-11T22:09:43.368 INFO:teuthology.orchestra.run.smithi045.stdout:55834574897 2024-03-11T22:09:44.435 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:44 smithi045 ceph-mon[39715]: 3.5 scrub starts 2024-03-11T22:09:44.435 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:44 smithi045 ceph-mon[39715]: 3.5 scrub ok 2024-03-11T22:09:44.435 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:44 smithi045 ceph-mon[39715]: 5.13 scrub starts 2024-03-11T22:09:44.435 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:44 smithi045 ceph-mon[39715]: 5.13 scrub ok 2024-03-11T22:09:44.435 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:44 smithi045 ceph-mon[30178]: 3.5 scrub starts 2024-03-11T22:09:44.435 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:44 smithi045 ceph-mon[30178]: 3.5 scrub ok 2024-03-11T22:09:44.435 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:44 smithi045 ceph-mon[30178]: 5.13 scrub starts 2024-03-11T22:09:44.435 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:44 smithi045 ceph-mon[30178]: 5.13 scrub ok 2024-03-11T22:09:44.570 INFO:teuthology.orchestra.run.smithi045.stdout:34359738423 2024-03-11T22:09:44.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:44 smithi144 ceph-mon[33984]: 3.5 scrub starts 2024-03-11T22:09:44.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:44 smithi144 ceph-mon[33984]: 3.5 scrub ok 2024-03-11T22:09:44.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:44 smithi144 ceph-mon[33984]: 5.13 scrub starts 2024-03-11T22:09:44.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:44 smithi144 ceph-mon[33984]: 5.13 scrub ok 2024-03-11T22:09:45.218 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:09:45.218 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:09:45.218 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:45.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:45.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:45.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:45.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:45.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:45.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: Adjusting osd_memory_target on smithi045 to 3761M 2024-03-11T22:09:45.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:45.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:09:45.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:09:45.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-11T22:09:45.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: Updating smithi144:/etc/ceph/ceph.conf 2024-03-11T22:09:45.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: Updating smithi045:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:09:45.219 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:09:45.220 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1168773365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:09:45.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:09:45.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:09:45.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:45.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:45.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:45.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:45.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:45.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:45.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: Adjusting osd_memory_target on smithi045 to 3761M 2024-03-11T22:09:45.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:45.220 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:09:45.221 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:09:45.221 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-11T22:09:45.221 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: Updating smithi144:/etc/ceph/ceph.conf 2024-03-11T22:09:45.221 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: Updating smithi045:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:09:45.221 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:09:45.221 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:45 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1168773365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:09:45.298 INFO:teuthology.orchestra.run.smithi045.stdout:77309411372 2024-03-11T22:09:45.555 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:45.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:09:45.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:09:45.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:45.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:45.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:45.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:45.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:45.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:09:45.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: Adjusting osd_memory_target on smithi045 to 3761M 2024-03-11T22:09:45.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:45.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:09:45.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:09:45.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: Updating smithi045:/etc/ceph/ceph.conf 2024-03-11T22:09:45.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: Updating smithi144:/etc/ceph/ceph.conf 2024-03-11T22:09:45.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: Updating smithi045:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:09:45.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:09:45.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1168773365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:09:46.089 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:09:45 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:09:46.092 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574894 got 55834574897 for osd.1 2024-03-11T22:09:46.093 DEBUG:teuthology.parallel:result is None 2024-03-11T22:09:46.371 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: 5.5 scrub starts 2024-03-11T22:09:46.371 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: 5.5 scrub ok 2024-03-11T22:09:46.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: 6.14 scrub starts 2024-03-11T22:09:46.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: 6.14 scrub ok 2024-03-11T22:09:46.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:09:46.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:09:46.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: Updating smithi045:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:09:46.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:09:46.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: 3.13 scrub starts 2024-03-11T22:09:46.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: 3.13 scrub ok 2024-03-11T22:09:46.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.372 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3035591747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:09:46.373 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.373 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.373 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.373 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: Deploying daemon alertmanager.a on smithi045 2024-03-11T22:09:46.373 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: 5.17 scrub starts 2024-03-11T22:09:46.373 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[39715]: 5.17 scrub ok 2024-03-11T22:09:46.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: 5.5 scrub starts 2024-03-11T22:09:46.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: 5.5 scrub ok 2024-03-11T22:09:46.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: 6.14 scrub starts 2024-03-11T22:09:46.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: 6.14 scrub ok 2024-03-11T22:09:46.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:09:46.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:09:46.373 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: Updating smithi045:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:09:46.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:09:46.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: 3.13 scrub starts 2024-03-11T22:09:46.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: 3.13 scrub ok 2024-03-11T22:09:46.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3035591747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:09:46.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: Deploying daemon alertmanager.a on smithi045 2024-03-11T22:09:46.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: 5.17 scrub starts 2024-03-11T22:09:46.374 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:46 smithi045 ceph-mon[30178]: 5.17 scrub ok 2024-03-11T22:09:46.413 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: 5.5 scrub starts 2024-03-11T22:09:46.413 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: 5.5 scrub ok 2024-03-11T22:09:46.413 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: 6.14 scrub starts 2024-03-11T22:09:46.413 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: 6.14 scrub ok 2024-03-11T22:09:46.413 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:09:46.413 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:09:46.413 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: Updating smithi045:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:09:46.413 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: Updating smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:09:46.413 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: 3.13 scrub starts 2024-03-11T22:09:46.413 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: 3.13 scrub ok 2024-03-11T22:09:46.413 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.413 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.413 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3035591747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:09:46.414 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.414 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.414 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:09:46.414 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: Deploying daemon alertmanager.a on smithi045 2024-03-11T22:09:46.414 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: 5.17 scrub starts 2024-03-11T22:09:46.414 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:46 smithi144 ceph-mon[33984]: 5.17 scrub ok 2024-03-11T22:09:46.540 INFO:teuthology.orchestra.run.smithi045.stdout:111669149735 2024-03-11T22:09:47.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:47.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[39715]: 4.1c scrub starts 2024-03-11T22:09:47.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[39715]: 4.1c scrub ok 2024-03-11T22:09:47.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[39715]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:09:47.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[39715]: 3.a scrub starts 2024-03-11T22:09:47.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[39715]: 3.a scrub ok 2024-03-11T22:09:47.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2758339756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T22:09:47.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:47.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[30178]: 4.1c scrub starts 2024-03-11T22:09:47.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[30178]: 4.1c scrub ok 2024-03-11T22:09:47.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[30178]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:09:47.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[30178]: 3.a scrub starts 2024-03-11T22:09:47.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[30178]: 3.a scrub ok 2024-03-11T22:09:47.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2758339756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T22:09:47.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:47 smithi144 ceph-mon[33984]: 4.1c scrub starts 2024-03-11T22:09:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:47 smithi144 ceph-mon[33984]: 4.1c scrub ok 2024-03-11T22:09:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:47 smithi144 ceph-mon[33984]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:09:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:47 smithi144 ceph-mon[33984]: 3.a scrub starts 2024-03-11T22:09:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:47 smithi144 ceph-mon[33984]: 3.a scrub ok 2024-03-11T22:09:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:47 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2758339756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T22:09:47.697 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738420 got 34359738423 for osd.0 2024-03-11T22:09:47.697 DEBUG:teuthology.parallel:result is None 2024-03-11T22:09:48.592 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411372 for osd.2 2024-03-11T22:09:48.592 DEBUG:teuthology.parallel:result is None 2024-03-11T22:09:49.279 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149729 got 111669149735 for osd.3 2024-03-11T22:09:49.279 DEBUG:teuthology.parallel:result is None 2024-03-11T22:09:49.416 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:49 smithi045 ceph-mon[39715]: 4.4 scrub starts 2024-03-11T22:09:49.417 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:49 smithi045 ceph-mon[39715]: 4.4 scrub ok 2024-03-11T22:09:49.417 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:49 smithi045 ceph-mon[39715]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:09:49.417 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:49 smithi045 ceph-mon[39715]: 3.14 scrub starts 2024-03-11T22:09:49.417 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:49 smithi045 ceph-mon[39715]: 3.14 scrub ok 2024-03-11T22:09:49.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:49 smithi045 ceph-mon[30178]: 4.4 scrub starts 2024-03-11T22:09:49.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:49 smithi045 ceph-mon[30178]: 4.4 scrub ok 2024-03-11T22:09:49.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:49 smithi045 ceph-mon[30178]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:09:49.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:49 smithi045 ceph-mon[30178]: 3.14 scrub starts 2024-03-11T22:09:49.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:49 smithi045 ceph-mon[30178]: 3.14 scrub ok 2024-03-11T22:09:49.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:49 smithi144 ceph-mon[33984]: 4.4 scrub starts 2024-03-11T22:09:49.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:49 smithi144 ceph-mon[33984]: 4.4 scrub ok 2024-03-11T22:09:49.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:49 smithi144 ceph-mon[33984]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:09:49.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:49 smithi144 ceph-mon[33984]: 3.14 scrub starts 2024-03-11T22:09:49.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:49 smithi144 ceph-mon[33984]: 3.14 scrub ok 2024-03-11T22:09:49.743 INFO:teuthology.orchestra.run.smithi045.stdout:219043332120 2024-03-11T22:09:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[39715]: 5.3 scrub starts 2024-03-11T22:09:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[39715]: 5.3 scrub ok 2024-03-11T22:09:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[39715]: 5.8 scrub starts 2024-03-11T22:09:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[39715]: 5.8 scrub ok 2024-03-11T22:09:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[39715]: 3.16 scrub starts 2024-03-11T22:09:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[39715]: 3.16 scrub ok 2024-03-11T22:09:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[39715]: 4.1f scrub starts 2024-03-11T22:09:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[39715]: 4.1f scrub ok 2024-03-11T22:09:50.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/554746106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T22:09:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[30178]: 5.3 scrub starts 2024-03-11T22:09:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[30178]: 5.3 scrub ok 2024-03-11T22:09:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[30178]: 5.8 scrub starts 2024-03-11T22:09:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[30178]: 5.8 scrub ok 2024-03-11T22:09:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[30178]: 3.16 scrub starts 2024-03-11T22:09:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[30178]: 3.16 scrub ok 2024-03-11T22:09:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[30178]: 4.1f scrub starts 2024-03-11T22:09:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[30178]: 4.1f scrub ok 2024-03-11T22:09:50.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:50 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/554746106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T22:09:50.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:50 smithi144 ceph-mon[33984]: 5.3 scrub starts 2024-03-11T22:09:50.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:50 smithi144 ceph-mon[33984]: 5.3 scrub ok 2024-03-11T22:09:50.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:50 smithi144 ceph-mon[33984]: 5.8 scrub starts 2024-03-11T22:09:50.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:50 smithi144 ceph-mon[33984]: 5.8 scrub ok 2024-03-11T22:09:50.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:50 smithi144 ceph-mon[33984]: 3.16 scrub starts 2024-03-11T22:09:50.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:50 smithi144 ceph-mon[33984]: 3.16 scrub ok 2024-03-11T22:09:50.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:50 smithi144 ceph-mon[33984]: 4.1f scrub starts 2024-03-11T22:09:50.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:50 smithi144 ceph-mon[33984]: 4.1f scrub ok 2024-03-11T22:09:50.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:50 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/554746106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T22:09:50.625 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332116 got 219043332120 for osd.6 2024-03-11T22:09:50.625 DEBUG:teuthology.parallel:result is None 2024-03-11T22:09:50.625 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T22:09:50.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:09:50.871 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:51.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:51 smithi045 ceph-mon[39715]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:09:51.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:51 smithi045 ceph-mon[30178]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:09:51.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:51 smithi144 ceph-mon[33984]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:09:52.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:52 smithi045 ceph-mon[39715]: 4.6 scrub starts 2024-03-11T22:09:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:52 smithi045 ceph-mon[39715]: 4.6 scrub ok 2024-03-11T22:09:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:52 smithi045 ceph-mon[39715]: 6.1e scrub starts 2024-03-11T22:09:52.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:52 smithi045 ceph-mon[39715]: 6.1e scrub ok 2024-03-11T22:09:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:52 smithi045 ceph-mon[30178]: 4.6 scrub starts 2024-03-11T22:09:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:52 smithi045 ceph-mon[30178]: 4.6 scrub ok 2024-03-11T22:09:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:52 smithi045 ceph-mon[30178]: 6.1e scrub starts 2024-03-11T22:09:52.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:52 smithi045 ceph-mon[30178]: 6.1e scrub ok 2024-03-11T22:09:52.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:52 smithi144 ceph-mon[33984]: 4.6 scrub starts 2024-03-11T22:09:52.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:52 smithi144 ceph-mon[33984]: 4.6 scrub ok 2024-03-11T22:09:52.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:52 smithi144 ceph-mon[33984]: 6.1e scrub starts 2024-03-11T22:09:52.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:52 smithi144 ceph-mon[33984]: 6.1e scrub ok 2024-03-11T22:09:52.826 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:09:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:09:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:09:53.506 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:09:53.507 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:53 smithi045 ceph-mon[39715]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:09:53.507 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:53 smithi045 ceph-mon[39715]: 4.15 scrub starts 2024-03-11T22:09:53.507 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:53 smithi045 ceph-mon[39715]: 4.15 scrub ok 2024-03-11T22:09:53.507 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:53 smithi045 ceph-mon[30178]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:09:53.507 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:53 smithi045 ceph-mon[30178]: 4.15 scrub starts 2024-03-11T22:09:53.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:53 smithi045 ceph-mon[30178]: 4.15 scrub ok 2024-03-11T22:09:53.513 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:09:53.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:53 smithi144 ceph-mon[33984]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:09:53.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:53 smithi144 ceph-mon[33984]: 4.15 scrub starts 2024-03-11T22:09:53.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:53 smithi144 ceph-mon[33984]: 4.15 scrub ok 2024-03-11T22:09:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:54 smithi045 ceph-mon[39715]: 6.1c scrub starts 2024-03-11T22:09:54.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:54 smithi045 ceph-mon[39715]: 6.1c scrub ok 2024-03-11T22:09:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:54 smithi045 ceph-mon[39715]: 4.1 scrub starts 2024-03-11T22:09:54.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:54 smithi045 ceph-mon[39715]: from='client.24581 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:09:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:54 smithi045 ceph-mon[30178]: 6.1c scrub starts 2024-03-11T22:09:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:54 smithi045 ceph-mon[30178]: 6.1c scrub ok 2024-03-11T22:09:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:54 smithi045 ceph-mon[30178]: 4.1 scrub starts 2024-03-11T22:09:54.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:54 smithi045 ceph-mon[30178]: from='client.24581 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:09:54.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:54 smithi144 ceph-mon[33984]: 6.1c scrub starts 2024-03-11T22:09:54.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:54 smithi144 ceph-mon[33984]: 6.1c scrub ok 2024-03-11T22:09:54.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:54 smithi144 ceph-mon[33984]: 4.1 scrub starts 2024-03-11T22:09:54.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:54 smithi144 ceph-mon[33984]: from='client.24581 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:09:55.341 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-03-11T22:09:52.043769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1004,"num_read_kb":850,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":608,"ondisk_log_size":608,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":241756,"kb_used_data":8988,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749719460,"statfs":{"total":767960285184,"available":767712727040,"internally_reserved":0,"allocated":9203712,"data_stored":5240788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":238276147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002632"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048103+0000","last_change":"2024-03-11T22:08:39.077833+0000","last_active":"2024-03-11T22:09:36.048103+0000","last_peered":"2024-03-11T22:09:36.048103+0000","last_clean":"2024-03-11T22:09:36.048103+0000","last_became_active":"2024-03-11T22:08:39.074695+0000","last_became_peered":"2024-03-11T22:08:39.074695+0000","last_unstale":"2024-03-11T22:09:36.048103+0000","last_undegraded":"2024-03-11T22:09:36.048103+0000","last_fullsized":"2024-03-11T22:09:36.048103+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:05.446193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621456+0000","last_change":"2024-03-11T22:08:37.074138+0000","last_active":"2024-03-11T22:09:36.621456+0000","last_peered":"2024-03-11T22:09:36.621456+0000","last_clean":"2024-03-11T22:09:36.621456+0000","last_became_active":"2024-03-11T22:08:37.073775+0000","last_became_peered":"2024-03-11T22:08:37.073775+0000","last_unstale":"2024-03-11T22:09:36.621456+0000","last_undegraded":"2024-03-11T22:09:36.621456+0000","last_fullsized":"2024-03-11T22:09:36.621456+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:02.401482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049526+0000","last_change":"2024-03-11T22:08:33.298497+0000","last_active":"2024-03-11T22:09:36.049526+0000","last_peered":"2024-03-11T22:09:36.049526+0000","last_clean":"2024-03-11T22:09:36.049526+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:09:36.049526+0000","last_undegraded":"2024-03-11T22:09:36.049526+0000","last_fullsized":"2024-03-11T22:09:36.049526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:46.682751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047777+0000","last_change":"2024-03-11T22:08:35.053644+0000","last_active":"2024-03-11T22:09:36.047777+0000","last_peered":"2024-03-11T22:09:36.047777+0000","last_clean":"2024-03-11T22:09:36.047777+0000","last_became_active":"2024-03-11T22:08:35.053179+0000","last_became_peered":"2024-03-11T22:08:35.053179+0000","last_unstale":"2024-03-11T22:09:36.047777+0000","last_undegraded":"2024-03-11T22:09:36.047777+0000","last_fullsized":"2024-03-11T22:09:36.047777+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:48.806815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048530+0000","last_change":"2024-03-11T22:08:39.079762+0000","last_active":"2024-03-11T22:09:36.048530+0000","last_peered":"2024-03-11T22:09:36.048530+0000","last_clean":"2024-03-11T22:09:36.048530+0000","last_became_active":"2024-03-11T22:08:39.079045+0000","last_became_peered":"2024-03-11T22:08:39.079045+0000","last_unstale":"2024-03-11T22:09:36.048530+0000","last_undegraded":"2024-03-11T22:09:36.048530+0000","last_fullsized":"2024-03-11T22:09:36.048530+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.065191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047777+0000","last_change":"2024-03-11T22:08:37.082192+0000","last_active":"2024-03-11T22:09:36.047777+0000","last_peered":"2024-03-11T22:09:36.047777+0000","last_clean":"2024-03-11T22:09:36.047777+0000","last_became_active":"2024-03-11T22:08:37.081861+0000","last_became_peered":"2024-03-11T22:08:37.081861+0000","last_unstale":"2024-03-11T22:09:36.047777+0000","last_undegraded":"2024-03-11T22:09:36.047777+0000","last_fullsized":"2024-03-11T22:09:36.047777+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:47.606115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:39.608956+0000","last_change":"2024-03-11T22:09:39.608984+0000","last_active":"2024-03-11T22:09:39.608956+0000","last_peered":"2024-03-11T22:09:39.608956+0000","last_clean":"2024-03-11T22:09:39.608956+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:09:39.608956+0000","last_undegraded":"2024-03-11T22:09:39.608956+0000","last_fullsized":"2024-03-11T22:09:39.608956+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:09:39.608890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:39.608890+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:49.086609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047962+0000","last_change":"2024-03-11T22:08:35.061181+0000","last_active":"2024-03-11T22:09:36.047962+0000","last_peered":"2024-03-11T22:09:36.047962+0000","last_clean":"2024-03-11T22:09:36.047962+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:09:36.047962+0000","last_undegraded":"2024-03-11T22:09:36.047962+0000","last_fullsized":"2024-03-11T22:09:36.047962+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.748787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048226+0000","last_change":"2024-03-11T22:08:39.083128+0000","last_active":"2024-03-11T22:09:36.048226+0000","last_peered":"2024-03-11T22:09:36.048226+0000","last_clean":"2024-03-11T22:09:36.048226+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:09:36.048226+0000","last_undegraded":"2024-03-11T22:09:36.048226+0000","last_fullsized":"2024-03-11T22:09:36.048226+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.592922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267136+0000","last_change":"2024-03-11T22:08:37.082294+0000","last_active":"2024-03-11T22:09:36.267136+0000","last_peered":"2024-03-11T22:09:36.267136+0000","last_clean":"2024-03-11T22:09:36.267136+0000","last_became_active":"2024-03-11T22:08:37.081996+0000","last_became_peered":"2024-03-11T22:08:37.081996+0000","last_unstale":"2024-03-11T22:09:36.267136+0000","last_undegraded":"2024-03-11T22:09:36.267136+0000","last_fullsized":"2024-03-11T22:09:36.267136+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.530415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:40.301247+0000","last_change":"2024-03-11T22:09:40.301289+0000","last_active":"2024-03-11T22:09:40.301247+0000","last_peered":"2024-03-11T22:09:40.301247+0000","last_clean":"2024-03-11T22:09:40.301247+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:09:40.301247+0000","last_undegraded":"2024-03-11T22:09:40.301247+0000","last_fullsized":"2024-03-11T22:09:40.301247+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:40.301162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:40.301162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:40.904990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048314+0000","last_change":"2024-03-11T22:08:35.061181+0000","last_active":"2024-03-11T22:09:36.048314+0000","last_peered":"2024-03-11T22:09:36.048314+0000","last_clean":"2024-03-11T22:09:36.048314+0000","last_became_active":"2024-03-11T22:08:35.060802+0000","last_became_peered":"2024-03-11T22:08:35.060802+0000","last_unstale":"2024-03-11T22:09:36.048314+0000","last_undegraded":"2024-03-11T22:09:36.048314+0000","last_fullsized":"2024-03-11T22:09:36.048314+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:44.218392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.266787+0000","last_change":"2024-03-11T22:08:39.076507+0000","last_active":"2024-03-11T22:09:36.266787+0000","last_peered":"2024-03-11T22:09:36.266787+0000","last_clean":"2024-03-11T22:09:36.266787+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:09:36.266787+0000","last_undegraded":"2024-03-11T22:09:36.266787+0000","last_fullsized":"2024-03-11T22:09:36.266787+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:46.781818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.268834+0000","last_change":"2024-03-11T22:08:37.077641+0000","last_active":"2024-03-11T22:09:36.268834+0000","last_peered":"2024-03-11T22:09:36.268834+0000","last_clean":"2024-03-11T22:09:36.268834+0000","last_became_active":"2024-03-11T22:08:37.077279+0000","last_became_peered":"2024-03-11T22:08:37.077279+0000","last_unstale":"2024-03-11T22:09:36.268834+0000","last_undegraded":"2024-03-11T22:09:36.268834+0000","last_fullsized":"2024-03-11T22:09:36.268834+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:44.368118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267049+0000","last_change":"2024-03-11T22:08:33.040601+0000","last_active":"2024-03-11T22:09:36.267049+0000","last_peered":"2024-03-11T22:09:36.267049+0000","last_clean":"2024-03-11T22:09:36.267049+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:09:36.267049+0000","last_undegraded":"2024-03-11T22:09:36.267049+0000","last_fullsized":"2024-03-11T22:09:36.267049+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:48.720858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048153+0000","last_change":"2024-03-11T22:08:35.061039+0000","last_active":"2024-03-11T22:09:36.048153+0000","last_peered":"2024-03-11T22:09:36.048153+0000","last_clean":"2024-03-11T22:09:36.048153+0000","last_became_active":"2024-03-11T22:08:35.060777+0000","last_became_peered":"2024-03-11T22:08:35.060777+0000","last_unstale":"2024-03-11T22:09:36.048153+0000","last_undegraded":"2024-03-11T22:09:36.048153+0000","last_fullsized":"2024-03-11T22:09:36.048153+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:51.210939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048938+0000","last_change":"2024-03-11T22:08:39.076623+0000","last_active":"2024-03-11T22:09:36.048938+0000","last_peered":"2024-03-11T22:09:36.048938+0000","last_clean":"2024-03-11T22:09:36.048938+0000","last_became_active":"2024-03-11T22:08:39.075609+0000","last_became_peered":"2024-03-11T22:08:39.075609+0000","last_unstale":"2024-03-11T22:09:36.048938+0000","last_undegraded":"2024-03-11T22:09:36.048938+0000","last_fullsized":"2024-03-11T22:09:36.048938+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:41.256023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621925+0000","last_change":"2024-03-11T22:08:37.077878+0000","last_active":"2024-03-11T22:09:36.621925+0000","last_peered":"2024-03-11T22:09:36.621925+0000","last_clean":"2024-03-11T22:09:36.621925+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:09:36.621925+0000","last_undegraded":"2024-03-11T22:09:36.621925+0000","last_fullsized":"2024-03-11T22:09:36.621925+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.718775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049612+0000","last_change":"2024-03-11T22:08:33.036036+0000","last_active":"2024-03-11T22:09:36.049612+0000","last_peered":"2024-03-11T22:09:36.049612+0000","last_clean":"2024-03-11T22:09:36.049612+0000","last_became_active":"2024-03-11T22:08:33.035624+0000","last_became_peered":"2024-03-11T22:08:33.035624+0000","last_unstale":"2024-03-11T22:09:36.049612+0000","last_undegraded":"2024-03-11T22:09:36.049612+0000","last_fullsized":"2024-03-11T22:09:36.049612+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:48.949714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:45.731324+0000","last_change":"2024-03-11T22:09:45.731375+0000","last_active":"2024-03-11T22:09:45.731324+0000","last_peered":"2024-03-11T22:09:45.731324+0000","last_clean":"2024-03-11T22:09:45.731324+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:09:45.731324+0000","last_undegraded":"2024-03-11T22:09:45.731324+0000","last_fullsized":"2024-03-11T22:09:45.731324+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:09:45.731248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:45.731248+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:08.445140+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049326+0000","last_change":"2024-03-11T22:08:39.078269+0000","last_active":"2024-03-11T22:09:36.049326+0000","last_peered":"2024-03-11T22:09:36.049326+0000","last_clean":"2024-03-11T22:09:36.049326+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:09:36.049326+0000","last_undegraded":"2024-03-11T22:09:36.049326+0000","last_fullsized":"2024-03-11T22:09:36.049326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.509768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048453+0000","last_change":"2024-03-11T22:08:37.080325+0000","last_active":"2024-03-11T22:09:36.048453+0000","last_peered":"2024-03-11T22:09:36.048453+0000","last_clean":"2024-03-11T22:09:36.048453+0000","last_became_active":"2024-03-11T22:08:37.080014+0000","last_became_peered":"2024-03-11T22:08:37.080014+0000","last_unstale":"2024-03-11T22:09:36.048453+0000","last_undegraded":"2024-03-11T22:09:36.048453+0000","last_fullsized":"2024-03-11T22:09:36.048453+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:04.066121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048415+0000","last_change":"2024-03-11T22:08:33.299536+0000","last_active":"2024-03-11T22:09:36.048415+0000","last_peered":"2024-03-11T22:09:36.048415+0000","last_clean":"2024-03-11T22:09:36.048415+0000","last_became_active":"2024-03-11T22:08:33.299189+0000","last_became_peered":"2024-03-11T22:08:33.299189+0000","last_unstale":"2024-03-11T22:09:36.048415+0000","last_undegraded":"2024-03-11T22:09:36.048415+0000","last_fullsized":"2024-03-11T22:09:36.048415+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:45.295342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048507+0000","last_change":"2024-03-11T22:08:35.066453+0000","last_active":"2024-03-11T22:09:36.048507+0000","last_peered":"2024-03-11T22:09:36.048507+0000","last_clean":"2024-03-11T22:09:36.048507+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:09:36.048507+0000","last_undegraded":"2024-03-11T22:09:36.048507+0000","last_fullsized":"2024-03-11T22:09:36.048507+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:49.876100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.268968+0000","last_change":"2024-03-11T22:08:39.073282+0000","last_active":"2024-03-11T22:09:36.268968+0000","last_peered":"2024-03-11T22:09:36.268968+0000","last_clean":"2024-03-11T22:09:36.268968+0000","last_became_active":"2024-03-11T22:08:39.072956+0000","last_became_peered":"2024-03-11T22:08:39.072956+0000","last_unstale":"2024-03-11T22:09:36.268968+0000","last_undegraded":"2024-03-11T22:09:36.268968+0000","last_fullsized":"2024-03-11T22:09:36.268968+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:38.519286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:43.350666+0000","last_change":"2024-03-11T22:08:37.078899+0000","last_active":"2024-03-11T22:09:43.350666+0000","last_peered":"2024-03-11T22:09:43.350666+0000","last_clean":"2024-03-11T22:09:43.350666+0000","last_became_active":"2024-03-11T22:08:37.078588+0000","last_became_peered":"2024-03-11T22:08:37.078588+0000","last_unstale":"2024-03-11T22:09:43.350666+0000","last_undegraded":"2024-03-11T22:09:43.350666+0000","last_fullsized":"2024-03-11T22:09:43.350666+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:55.590469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621574+0000","last_change":"2024-03-11T22:08:33.030653+0000","last_active":"2024-03-11T22:09:36.621574+0000","last_peered":"2024-03-11T22:09:36.621574+0000","last_clean":"2024-03-11T22:09:36.621574+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:09:36.621574+0000","last_undegraded":"2024-03-11T22:09:36.621574+0000","last_fullsized":"2024-03-11T22:09:36.621574+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.528647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049953+0000","last_change":"2024-03-11T22:08:35.086209+0000","last_active":"2024-03-11T22:09:36.049953+0000","last_peered":"2024-03-11T22:09:36.049953+0000","last_clean":"2024-03-11T22:09:36.049953+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:09:36.049953+0000","last_undegraded":"2024-03-11T22:09:36.049953+0000","last_fullsized":"2024-03-11T22:09:36.049953+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:02.938827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622083+0000","last_change":"2024-03-11T22:08:39.075560+0000","last_active":"2024-03-11T22:09:36.622083+0000","last_peered":"2024-03-11T22:09:36.622083+0000","last_clean":"2024-03-11T22:09:36.622083+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:09:36.622083+0000","last_undegraded":"2024-03-11T22:09:36.622083+0000","last_fullsized":"2024-03-11T22:09:36.622083+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:52.374594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.050110+0000","last_change":"2024-03-11T22:08:37.069965+0000","last_active":"2024-03-11T22:09:36.050110+0000","last_peered":"2024-03-11T22:09:36.050110+0000","last_clean":"2024-03-11T22:09:36.050110+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:09:36.050110+0000","last_undegraded":"2024-03-11T22:09:36.050110+0000","last_fullsized":"2024-03-11T22:09:36.050110+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.292950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047578+0000","last_change":"2024-03-11T22:08:33.038259+0000","last_active":"2024-03-11T22:09:36.047578+0000","last_peered":"2024-03-11T22:09:36.047578+0000","last_clean":"2024-03-11T22:09:36.047578+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:09:36.047578+0000","last_undegraded":"2024-03-11T22:09:36.047578+0000","last_fullsized":"2024-03-11T22:09:36.047578+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.852988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:49.567242+0000","last_change":"2024-03-11T22:09:49.567290+0000","last_active":"2024-03-11T22:09:49.567242+0000","last_peered":"2024-03-11T22:09:49.567242+0000","last_clean":"2024-03-11T22:09:49.567242+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:09:49.567242+0000","last_undegraded":"2024-03-11T22:09:49.567242+0000","last_fullsized":"2024-03-11T22:09:49.567242+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:09:49.567170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:49.567170+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:01.624956+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.268733+0000","last_change":"2024-03-11T22:08:39.074473+0000","last_active":"2024-03-11T22:09:36.268733+0000","last_peered":"2024-03-11T22:09:36.268733+0000","last_clean":"2024-03-11T22:09:36.268733+0000","last_became_active":"2024-03-11T22:08:39.074179+0000","last_became_peered":"2024-03-11T22:08:39.074179+0000","last_unstale":"2024-03-11T22:09:36.268733+0000","last_undegraded":"2024-03-11T22:09:36.268733+0000","last_fullsized":"2024-03-11T22:09:36.268733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.168751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:40.669281+0000","last_change":"2024-03-11T22:09:40.669305+0000","last_active":"2024-03-11T22:09:40.669281+0000","last_peered":"2024-03-11T22:09:40.669281+0000","last_clean":"2024-03-11T22:09:40.669281+0000","last_became_active":"2024-03-11T22:08:37.075341+0000","last_became_peered":"2024-03-11T22:08:37.075341+0000","last_unstale":"2024-03-11T22:09:40.669281+0000","last_undegraded":"2024-03-11T22:09:40.669281+0000","last_fullsized":"2024-03-11T22:09:40.669281+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:40.669213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:40.669213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:05.240222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049221+0000","last_change":"2024-03-11T22:08:33.040896+0000","last_active":"2024-03-11T22:09:36.049221+0000","last_peered":"2024-03-11T22:09:36.049221+0000","last_clean":"2024-03-11T22:09:36.049221+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:09:36.049221+0000","last_undegraded":"2024-03-11T22:09:36.049221+0000","last_fullsized":"2024-03-11T22:09:36.049221+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.000587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:42.568177+0000","last_change":"2024-03-11T22:09:42.568204+0000","last_active":"2024-03-11T22:09:42.568177+0000","last_peered":"2024-03-11T22:09:42.568177+0000","last_clean":"2024-03-11T22:09:42.568177+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:09:42.568177+0000","last_undegraded":"2024-03-11T22:09:42.568177+0000","last_fullsized":"2024-03-11T22:09:42.568177+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-11T22:09:42.568114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:42.568114+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.634829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048411+0000","last_change":"2024-03-11T22:08:39.081827+0000","last_active":"2024-03-11T22:09:36.048411+0000","last_peered":"2024-03-11T22:09:36.048411+0000","last_clean":"2024-03-11T22:09:36.048411+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:09:36.048411+0000","last_undegraded":"2024-03-11T22:09:36.048411+0000","last_fullsized":"2024-03-11T22:09:36.048411+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:49.990435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269551+0000","last_change":"2024-03-11T22:08:37.079252+0000","last_active":"2024-03-11T22:09:36.269551+0000","last_peered":"2024-03-11T22:09:36.269551+0000","last_clean":"2024-03-11T22:09:36.269551+0000","last_became_active":"2024-03-11T22:08:37.078863+0000","last_became_peered":"2024-03-11T22:08:37.078863+0000","last_unstale":"2024-03-11T22:09:36.269551+0000","last_undegraded":"2024-03-11T22:09:36.269551+0000","last_fullsized":"2024-03-11T22:09:36.269551+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:43.735583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:49.271515+0000","last_change":"2024-03-11T22:09:49.271541+0000","last_active":"2024-03-11T22:09:49.271515+0000","last_peered":"2024-03-11T22:09:49.271515+0000","last_clean":"2024-03-11T22:09:49.271515+0000","last_became_active":"2024-03-11T22:08:33.039349+0000","last_became_peered":"2024-03-11T22:08:33.039349+0000","last_unstale":"2024-03-11T22:09:49.271515+0000","last_undegraded":"2024-03-11T22:09:49.271515+0000","last_fullsized":"2024-03-11T22:09:49.271515+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:49.271431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:49.271431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:01.697571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047870+0000","last_change":"2024-03-11T22:08:35.086272+0000","last_active":"2024-03-11T22:09:36.047870+0000","last_peered":"2024-03-11T22:09:36.047870+0000","last_clean":"2024-03-11T22:09:36.047870+0000","last_became_active":"2024-03-11T22:08:35.085928+0000","last_became_peered":"2024-03-11T22:08:35.085928+0000","last_unstale":"2024-03-11T22:09:36.047870+0000","last_undegraded":"2024-03-11T22:09:36.047870+0000","last_fullsized":"2024-03-11T22:09:36.047870+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.863565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047674+0000","last_change":"2024-03-11T22:08:39.083049+0000","last_active":"2024-03-11T22:09:36.047674+0000","last_peered":"2024-03-11T22:09:36.047674+0000","last_clean":"2024-03-11T22:09:36.047674+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:09:36.047674+0000","last_undegraded":"2024-03-11T22:09:36.047674+0000","last_fullsized":"2024-03-11T22:09:36.047674+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.375813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:43.569403+0000","last_change":"2024-03-11T22:09:43.569426+0000","last_active":"2024-03-11T22:09:43.569403+0000","last_peered":"2024-03-11T22:09:43.569403+0000","last_clean":"2024-03-11T22:09:43.569403+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:09:43.569403+0000","last_undegraded":"2024-03-11T22:09:43.569403+0000","last_fullsized":"2024-03-11T22:09:43.569403+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:43.569328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:43.569328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:05.155162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:41.032420+0000","last_change":"2024-03-11T22:09:41.032444+0000","last_active":"2024-03-11T22:09:41.032420+0000","last_peered":"2024-03-11T22:09:41.032420+0000","last_clean":"2024-03-11T22:09:41.032420+0000","last_became_active":"2024-03-11T22:08:33.038783+0000","last_became_peered":"2024-03-11T22:08:33.038783+0000","last_unstale":"2024-03-11T22:09:41.032420+0000","last_undegraded":"2024-03-11T22:09:41.032420+0000","last_fullsized":"2024-03-11T22:09:41.032420+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:41.032339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:41.032339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:01.424543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622312+0000","last_change":"2024-03-11T22:08:35.053322+0000","last_active":"2024-03-11T22:09:36.622312+0000","last_peered":"2024-03-11T22:09:36.622312+0000","last_clean":"2024-03-11T22:09:36.622312+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:09:36.622312+0000","last_undegraded":"2024-03-11T22:09:36.622312+0000","last_fullsized":"2024-03-11T22:09:36.622312+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:50.812560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047456+0000","last_change":"2024-03-11T22:08:39.080670+0000","last_active":"2024-03-11T22:09:36.047456+0000","last_peered":"2024-03-11T22:09:36.047456+0000","last_clean":"2024-03-11T22:09:36.047456+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:09:36.047456+0000","last_undegraded":"2024-03-11T22:09:36.047456+0000","last_fullsized":"2024-03-11T22:09:36.047456+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.573443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622083+0000","last_change":"2024-03-11T22:08:37.075322+0000","last_active":"2024-03-11T22:09:36.622083+0000","last_peered":"2024-03-11T22:09:36.622083+0000","last_clean":"2024-03-11T22:09:36.622083+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:09:36.622083+0000","last_undegraded":"2024-03-11T22:09:36.622083+0000","last_fullsized":"2024-03-11T22:09:36.622083+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.827062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.362087+0000","last_change":"2024-03-11T22:09:48.362144+0000","last_active":"2024-03-11T22:09:48.362087+0000","last_peered":"2024-03-11T22:09:48.362087+0000","last_clean":"2024-03-11T22:09:48.362087+0000","last_became_active":"2024-03-11T22:08:33.036083+0000","last_became_peered":"2024-03-11T22:08:33.036083+0000","last_unstale":"2024-03-11T22:09:48.362087+0000","last_undegraded":"2024-03-11T22:09:48.362087+0000","last_fullsized":"2024-03-11T22:09:48.362087+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:09:48.361967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:48.361967+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:05.666698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048296+0000","last_change":"2024-03-11T22:08:35.054987+0000","last_active":"2024-03-11T22:09:36.048296+0000","last_peered":"2024-03-11T22:09:36.048296+0000","last_clean":"2024-03-11T22:09:36.048296+0000","last_became_active":"2024-03-11T22:08:35.054726+0000","last_became_peered":"2024-03-11T22:08:35.054726+0000","last_unstale":"2024-03-11T22:09:36.048296+0000","last_undegraded":"2024-03-11T22:09:36.048296+0000","last_fullsized":"2024-03-11T22:09:36.048296+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.886570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049756+0000","last_change":"2024-03-11T22:08:39.083736+0000","last_active":"2024-03-11T22:09:36.049756+0000","last_peered":"2024-03-11T22:09:36.049756+0000","last_clean":"2024-03-11T22:09:36.049756+0000","last_became_active":"2024-03-11T22:08:39.082938+0000","last_became_peered":"2024-03-11T22:08:39.082938+0000","last_unstale":"2024-03-11T22:09:36.049756+0000","last_undegraded":"2024-03-11T22:09:36.049756+0000","last_fullsized":"2024-03-11T22:09:36.049756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:06.944689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:43.350548+0000","last_change":"2024-03-11T22:08:37.076601+0000","last_active":"2024-03-11T22:09:43.350548+0000","last_peered":"2024-03-11T22:09:43.350548+0000","last_clean":"2024-03-11T22:09:43.350548+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:09:43.350548+0000","last_undegraded":"2024-03-11T22:09:43.350548+0000","last_fullsized":"2024-03-11T22:09:43.350548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:45.198461+0000","last_change":"2024-03-11T22:09:45.198482+0000","last_active":"2024-03-11T22:09:45.198461+0000","last_peered":"2024-03-11T22:09:45.198461+0000","last_clean":"2024-03-11T22:09:45.198461+0000","last_became_active":"2024-03-11T22:08:33.300181+0000","last_became_peered":"2024-03-11T22:08:33.300181+0000","last_unstale":"2024-03-11T22:09:45.198461+0000","last_undegraded":"2024-03-11T22:09:45.198461+0000","last_fullsized":"2024-03-11T22:09:45.198461+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:45.198397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:45.198397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:12.972849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049076+0000","last_change":"2024-03-11T22:08:35.086113+0000","last_active":"2024-03-11T22:09:36.049076+0000","last_peered":"2024-03-11T22:09:36.049076+0000","last_clean":"2024-03-11T22:09:36.049076+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:09:36.049076+0000","last_undegraded":"2024-03-11T22:09:36.049076+0000","last_fullsized":"2024-03-11T22:09:36.049076+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:40.519001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049197+0000","last_change":"2024-03-11T22:08:39.072508+0000","last_active":"2024-03-11T22:09:36.049197+0000","last_peered":"2024-03-11T22:09:36.049197+0000","last_clean":"2024-03-11T22:09:36.049197+0000","last_became_active":"2024-03-11T22:08:39.072089+0000","last_became_peered":"2024-03-11T22:08:39.072089+0000","last_unstale":"2024-03-11T22:09:36.049197+0000","last_undegraded":"2024-03-11T22:09:36.049197+0000","last_fullsized":"2024-03-11T22:09:36.049197+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:06.283027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:43.350808+0000","last_change":"2024-03-11T22:08:37.071570+0000","last_active":"2024-03-11T22:09:43.350808+0000","last_peered":"2024-03-11T22:09:43.350808+0000","last_clean":"2024-03-11T22:09:43.350808+0000","last_became_active":"2024-03-11T22:08:37.071202+0000","last_became_peered":"2024-03-11T22:08:37.071202+0000","last_unstale":"2024-03-11T22:09:43.350808+0000","last_undegraded":"2024-03-11T22:09:43.350808+0000","last_fullsized":"2024-03-11T22:09:43.350808+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:49.713520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048004+0000","last_change":"2024-03-11T22:08:33.033199+0000","last_active":"2024-03-11T22:09:36.048004+0000","last_peered":"2024-03-11T22:09:36.048004+0000","last_clean":"2024-03-11T22:09:36.048004+0000","last_became_active":"2024-03-11T22:08:33.032850+0000","last_became_peered":"2024-03-11T22:08:33.032850+0000","last_unstale":"2024-03-11T22:09:36.048004+0000","last_undegraded":"2024-03-11T22:09:36.048004+0000","last_fullsized":"2024-03-11T22:09:36.048004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:51.370651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267477+0000","last_change":"2024-03-11T22:08:35.086614+0000","last_active":"2024-03-11T22:09:36.267477+0000","last_peered":"2024-03-11T22:09:36.267477+0000","last_clean":"2024-03-11T22:09:36.267477+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:09:36.267477+0000","last_undegraded":"2024-03-11T22:09:36.267477+0000","last_fullsized":"2024-03-11T22:09:36.267477+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:40.286009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:44.743673+0000","last_change":"2024-03-11T22:09:44.743698+0000","last_active":"2024-03-11T22:09:44.743673+0000","last_peered":"2024-03-11T22:09:44.743673+0000","last_clean":"2024-03-11T22:09:44.743673+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:09:44.743673+0000","last_undegraded":"2024-03-11T22:09:44.743673+0000","last_fullsized":"2024-03-11T22:09:44.743673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:44.743600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:44.743600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:03.506943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:45.582545+0000","last_change":"2024-03-11T22:09:45.582583+0000","last_active":"2024-03-11T22:09:45.582545+0000","last_peered":"2024-03-11T22:09:45.582545+0000","last_clean":"2024-03-11T22:09:45.582545+0000","last_became_active":"2024-03-11T22:08:37.069611+0000","last_became_peered":"2024-03-11T22:08:37.069611+0000","last_unstale":"2024-03-11T22:09:45.582545+0000","last_undegraded":"2024-03-11T22:09:45.582545+0000","last_fullsized":"2024-03-11T22:09:45.582545+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:45.582469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:45.582469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.713402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.268777+0000","last_change":"2024-03-11T22:08:33.038436+0000","last_active":"2024-03-11T22:09:36.268777+0000","last_peered":"2024-03-11T22:09:36.268777+0000","last_clean":"2024-03-11T22:09:36.268777+0000","last_became_active":"2024-03-11T22:08:33.038132+0000","last_became_peered":"2024-03-11T22:08:33.038132+0000","last_unstale":"2024-03-11T22:09:36.268777+0000","last_undegraded":"2024-03-11T22:09:36.268777+0000","last_fullsized":"2024-03-11T22:09:36.268777+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:53.291744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.050020+0000","last_change":"2024-03-11T22:08:35.086149+0000","last_active":"2024-03-11T22:09:36.050020+0000","last_peered":"2024-03-11T22:09:36.050020+0000","last_clean":"2024-03-11T22:09:36.050020+0000","last_became_active":"2024-03-11T22:08:35.085787+0000","last_became_peered":"2024-03-11T22:08:35.085787+0000","last_unstale":"2024-03-11T22:09:36.050020+0000","last_undegraded":"2024-03-11T22:09:36.050020+0000","last_fullsized":"2024-03-11T22:09:36.050020+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:53.984069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269004+0000","last_change":"2024-03-11T22:08:39.075607+0000","last_active":"2024-03-11T22:09:36.269004+0000","last_peered":"2024-03-11T22:09:36.269004+0000","last_clean":"2024-03-11T22:09:36.269004+0000","last_became_active":"2024-03-11T22:08:39.075305+0000","last_became_peered":"2024-03-11T22:08:39.075305+0000","last_unstale":"2024-03-11T22:09:36.269004+0000","last_undegraded":"2024-03-11T22:09:36.269004+0000","last_fullsized":"2024-03-11T22:09:36.269004+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.166687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267282+0000","last_change":"2024-03-11T22:08:37.077768+0000","last_active":"2024-03-11T22:09:36.267282+0000","last_peered":"2024-03-11T22:09:36.267282+0000","last_clean":"2024-03-11T22:09:36.267282+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:09:36.267282+0000","last_undegraded":"2024-03-11T22:09:36.267282+0000","last_fullsized":"2024-03-11T22:09:36.267282+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:48.146238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.120856+0000","last_change":"2024-03-11T22:08:33.035663+0000","last_active":"2024-03-11T22:09:36.120856+0000","last_peered":"2024-03-11T22:09:36.120856+0000","last_clean":"2024-03-11T22:09:36.120856+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:09:36.120856+0000","last_undegraded":"2024-03-11T22:09:36.120856+0000","last_fullsized":"2024-03-11T22:09:36.120856+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.335803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049802+0000","last_change":"2024-03-11T22:08:35.086263+0000","last_active":"2024-03-11T22:09:36.049802+0000","last_peered":"2024-03-11T22:09:36.049802+0000","last_clean":"2024-03-11T22:09:36.049802+0000","last_became_active":"2024-03-11T22:08:35.085914+0000","last_became_peered":"2024-03-11T22:08:35.085914+0000","last_unstale":"2024-03-11T22:09:36.049802+0000","last_undegraded":"2024-03-11T22:09:36.049802+0000","last_fullsized":"2024-03-11T22:09:36.049802+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:53.031835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267056+0000","last_change":"2024-03-11T22:08:39.079848+0000","last_active":"2024-03-11T22:09:36.267056+0000","last_peered":"2024-03-11T22:09:36.267056+0000","last_clean":"2024-03-11T22:09:36.267056+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:09:36.267056+0000","last_undegraded":"2024-03-11T22:09:36.267056+0000","last_fullsized":"2024-03-11T22:09:36.267056+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.299384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269220+0000","last_change":"2024-03-11T22:08:33.039060+0000","last_active":"2024-03-11T22:09:36.269220+0000","last_peered":"2024-03-11T22:09:36.269220+0000","last_clean":"2024-03-11T22:09:36.269220+0000","last_became_active":"2024-03-11T22:08:33.037662+0000","last_became_peered":"2024-03-11T22:08:33.037662+0000","last_unstale":"2024-03-11T22:09:36.269220+0000","last_undegraded":"2024-03-11T22:09:36.269220+0000","last_fullsized":"2024-03-11T22:09:36.269220+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:49.450289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"91'22","reported_seq":93,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.274283+0000","last_change":"2024-03-11T22:09:36.274318+0000","last_active":"2024-03-11T22:09:36.274283+0000","last_peered":"2024-03-11T22:09:36.274283+0000","last_clean":"2024-03-11T22:09:36.274283+0000","last_became_active":"2024-03-11T22:08:35.085916+0000","last_became_peered":"2024-03-11T22:08:35.085916+0000","last_unstale":"2024-03-11T22:09:36.274283+0000","last_undegraded":"2024-03-11T22:09:36.274283+0000","last_fullsized":"2024-03-11T22:09:36.274283+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'22","last_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:09:36.274225+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:38.387952+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:43.350676+0000","last_change":"2024-03-11T22:08:37.078410+0000","last_active":"2024-03-11T22:09:43.350676+0000","last_peered":"2024-03-11T22:09:43.350676+0000","last_clean":"2024-03-11T22:09:43.350676+0000","last_became_active":"2024-03-11T22:08:37.078114+0000","last_became_peered":"2024-03-11T22:08:37.078114+0000","last_unstale":"2024-03-11T22:09:43.350676+0000","last_undegraded":"2024-03-11T22:09:43.350676+0000","last_fullsized":"2024-03-11T22:09:43.350676+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:52.415942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"75'1","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048453+0000","last_change":"2024-03-11T22:08:39.083458+0000","last_active":"2024-03-11T22:09:36.048453+0000","last_peered":"2024-03-11T22:09:36.048453+0000","last_clean":"2024-03-11T22:09:36.048453+0000","last_became_active":"2024-03-11T22:08:39.083141+0000","last_became_peered":"2024-03-11T22:08:39.083141+0000","last_unstale":"2024-03-11T22:09:36.048453+0000","last_undegraded":"2024-03-11T22:09:36.048453+0000","last_fullsized":"2024-03-11T22:09:36.048453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:54.422237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269155+0000","last_change":"2024-03-11T22:08:33.041804+0000","last_active":"2024-03-11T22:09:36.269155+0000","last_peered":"2024-03-11T22:09:36.269155+0000","last_clean":"2024-03-11T22:09:36.269155+0000","last_became_active":"2024-03-11T22:08:33.041492+0000","last_became_peered":"2024-03-11T22:08:33.041492+0000","last_unstale":"2024-03-11T22:09:36.269155+0000","last_undegraded":"2024-03-11T22:09:36.269155+0000","last_fullsized":"2024-03-11T22:09:36.269155+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:43.260561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047993+0000","last_change":"2024-03-11T22:08:54.614766+0000","last_active":"2024-03-11T22:09:36.047993+0000","last_peered":"2024-03-11T22:09:36.047993+0000","last_clean":"2024-03-11T22:09:36.047993+0000","last_became_active":"2024-03-11T22:08:54.607153+0000","last_became_peered":"2024-03-11T22:08:54.607153+0000","last_unstale":"2024-03-11T22:09:36.047993+0000","last_undegraded":"2024-03-11T22:09:36.047993+0000","last_fullsized":"2024-03-11T22:09:36.047993+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:58.241457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.721722+0000","last_change":"2024-03-11T22:09:48.721747+0000","last_active":"2024-03-11T22:09:48.721722+0000","last_peered":"2024-03-11T22:09:48.721722+0000","last_clean":"2024-03-11T22:09:48.721722+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:09:48.721722+0000","last_undegraded":"2024-03-11T22:09:48.721722+0000","last_fullsized":"2024-03-11T22:09:48.721722+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:48.721656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:48.721656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.253690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.268601+0000","last_change":"2024-03-11T22:08:39.079520+0000","last_active":"2024-03-11T22:09:36.268601+0000","last_peered":"2024-03-11T22:09:36.268601+0000","last_clean":"2024-03-11T22:09:36.268601+0000","last_became_active":"2024-03-11T22:08:39.079238+0000","last_became_peered":"2024-03-11T22:08:39.079238+0000","last_unstale":"2024-03-11T22:09:36.268601+0000","last_undegraded":"2024-03-11T22:09:36.268601+0000","last_fullsized":"2024-03-11T22:09:36.268601+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:49.698176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622427+0000","last_change":"2024-03-11T22:08:37.075286+0000","last_active":"2024-03-11T22:09:36.622427+0000","last_peered":"2024-03-11T22:09:36.622427+0000","last_clean":"2024-03-11T22:09:36.622427+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:09:36.622427+0000","last_undegraded":"2024-03-11T22:09:36.622427+0000","last_fullsized":"2024-03-11T22:09:36.622427+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:54.723130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:42.079785+0000","last_change":"2024-03-11T22:09:42.079812+0000","last_active":"2024-03-11T22:09:42.079785+0000","last_peered":"2024-03-11T22:09:42.079785+0000","last_clean":"2024-03-11T22:09:42.079785+0000","last_became_active":"2024-03-11T22:08:33.035188+0000","last_became_peered":"2024-03-11T22:08:33.035188+0000","last_unstale":"2024-03-11T22:09:42.079785+0000","last_undegraded":"2024-03-11T22:09:42.079785+0000","last_fullsized":"2024-03-11T22:09:42.079785+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:42.079695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:42.079695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:45.595806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.121027+0000","last_change":"2024-03-11T22:08:35.087388+0000","last_active":"2024-03-11T22:09:36.121027+0000","last_peered":"2024-03-11T22:09:36.121027+0000","last_clean":"2024-03-11T22:09:36.121027+0000","last_became_active":"2024-03-11T22:08:35.087046+0000","last_became_peered":"2024-03-11T22:08:35.087046+0000","last_unstale":"2024-03-11T22:09:36.121027+0000","last_undegraded":"2024-03-11T22:09:36.121027+0000","last_fullsized":"2024-03-11T22:09:36.121027+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:51.800776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047796+0000","last_change":"2024-03-11T22:08:39.085891+0000","last_active":"2024-03-11T22:09:36.047796+0000","last_peered":"2024-03-11T22:09:36.047796+0000","last_clean":"2024-03-11T22:09:36.047796+0000","last_became_active":"2024-03-11T22:08:39.085496+0000","last_became_peered":"2024-03-11T22:08:39.085496+0000","last_unstale":"2024-03-11T22:09:36.047796+0000","last_undegraded":"2024-03-11T22:09:36.047796+0000","last_fullsized":"2024-03-11T22:09:36.047796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:47.594130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622153+0000","last_change":"2024-03-11T22:08:37.073732+0000","last_active":"2024-03-11T22:09:36.622153+0000","last_peered":"2024-03-11T22:09:36.622153+0000","last_clean":"2024-03-11T22:09:36.622153+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:09:36.622153+0000","last_undegraded":"2024-03-11T22:09:36.622153+0000","last_fullsized":"2024-03-11T22:09:36.622153+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:54.219715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267080+0000","last_change":"2024-03-11T22:08:33.038820+0000","last_active":"2024-03-11T22:09:36.267080+0000","last_peered":"2024-03-11T22:09:36.267080+0000","last_clean":"2024-03-11T22:09:36.267080+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:09:36.267080+0000","last_undegraded":"2024-03-11T22:09:36.267080+0000","last_fullsized":"2024-03-11T22:09:36.267080+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:54.045701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049729+0000","last_change":"2024-03-11T22:08:54.617268+0000","last_active":"2024-03-11T22:09:36.049729+0000","last_peered":"2024-03-11T22:09:36.049729+0000","last_clean":"2024-03-11T22:09:36.049729+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:09:36.049729+0000","last_undegraded":"2024-03-11T22:09:36.049729+0000","last_fullsized":"2024-03-11T22:09:36.049729+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.997317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048925+0000","last_change":"2024-03-11T22:08:39.084054+0000","last_active":"2024-03-11T22:09:36.048925+0000","last_peered":"2024-03-11T22:09:36.048925+0000","last_clean":"2024-03-11T22:09:36.048925+0000","last_became_active":"2024-03-11T22:08:39.083738+0000","last_became_peered":"2024-03-11T22:08:39.083738+0000","last_unstale":"2024-03-11T22:09:36.048925+0000","last_undegraded":"2024-03-11T22:09:36.048925+0000","last_fullsized":"2024-03-11T22:09:36.048925+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.602554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.266879+0000","last_change":"2024-03-11T22:08:36.055999+0000","last_active":"2024-03-11T22:09:36.266879+0000","last_peered":"2024-03-11T22:09:36.266879+0000","last_clean":"2024-03-11T22:09:36.266879+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:09:36.266879+0000","last_undegraded":"2024-03-11T22:09:36.266879+0000","last_fullsized":"2024-03-11T22:09:36.266879+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_clean_scrub_stamp":"2024-03-11T22:08:31.015772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:44.756728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047958+0000","last_change":"2024-03-11T22:08:33.035748+0000","last_active":"2024-03-11T22:09:36.047958+0000","last_peered":"2024-03-11T22:09:36.047958+0000","last_clean":"2024-03-11T22:09:36.047958+0000","last_became_active":"2024-03-11T22:08:33.035469+0000","last_became_peered":"2024-03-11T22:08:33.035469+0000","last_unstale":"2024-03-11T22:09:36.047958+0000","last_undegraded":"2024-03-11T22:09:36.047958+0000","last_fullsized":"2024-03-11T22:09:36.047958+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:43.474016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":125,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:47.779316+0000","last_change":"2024-03-11T22:09:47.779316+0000","last_active":"2024-03-11T22:09:47.779316+0000","last_peered":"2024-03-11T22:09:47.779316+0000","last_clean":"2024-03-11T22:09:47.779316+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:09:47.779316+0000","last_undegraded":"2024-03-11T22:09:47.779316+0000","last_fullsized":"2024-03-11T22:09:47.779316+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-11T22:09:47.779209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:47.779209+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:10.865609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:44.165039+0000","last_change":"2024-03-11T22:09:44.165077+0000","last_active":"2024-03-11T22:09:44.165039+0000","last_peered":"2024-03-11T22:09:44.165039+0000","last_clean":"2024-03-11T22:09:44.165039+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:09:44.165039+0000","last_undegraded":"2024-03-11T22:09:44.165039+0000","last_fullsized":"2024-03-11T22:09:44.165039+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:44.164939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:44.164939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:55.636705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269397+0000","last_change":"2024-03-11T22:08:39.077855+0000","last_active":"2024-03-11T22:09:36.269397+0000","last_peered":"2024-03-11T22:09:36.269397+0000","last_clean":"2024-03-11T22:09:36.269397+0000","last_became_active":"2024-03-11T22:08:39.077566+0000","last_became_peered":"2024-03-11T22:08:39.077566+0000","last_unstale":"2024-03-11T22:09:36.269397+0000","last_undegraded":"2024-03-11T22:09:36.269397+0000","last_fullsized":"2024-03-11T22:09:36.269397+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:49.972071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621579+0000","last_change":"2024-03-11T22:09:35.724312+0000","last_active":"2024-03-11T22:09:36.621579+0000","last_peered":"2024-03-11T22:09:36.621579+0000","last_clean":"2024-03-11T22:09:36.621579+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:09:36.621579+0000","last_undegraded":"2024-03-11T22:09:36.621579+0000","last_fullsized":"2024-03-11T22:09:36.621579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:35.724224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:35.724224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:59.123602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621974+0000","last_change":"2024-03-11T22:08:36.055357+0000","last_active":"2024-03-11T22:09:36.621974+0000","last_peered":"2024-03-11T22:09:36.621974+0000","last_clean":"2024-03-11T22:09:36.621974+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:09:36.621974+0000","last_undegraded":"2024-03-11T22:09:36.621974+0000","last_fullsized":"2024-03-11T22:09:36.621974+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_clean_scrub_stamp":"2024-03-11T22:08:31.015772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:52.926784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622491+0000","last_change":"2024-03-11T22:08:35.058141+0000","last_active":"2024-03-11T22:09:36.622491+0000","last_peered":"2024-03-11T22:09:36.622491+0000","last_clean":"2024-03-11T22:09:36.622491+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:09:36.622491+0000","last_undegraded":"2024-03-11T22:09:36.622491+0000","last_fullsized":"2024-03-11T22:09:36.622491+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:51.069549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622607+0000","last_change":"2024-03-11T22:08:37.076581+0000","last_active":"2024-03-11T22:09:36.622607+0000","last_peered":"2024-03-11T22:09:36.622607+0000","last_clean":"2024-03-11T22:09:36.622607+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:09:36.622607+0000","last_undegraded":"2024-03-11T22:09:36.622607+0000","last_fullsized":"2024-03-11T22:09:36.622607+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:05.137609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267346+0000","last_change":"2024-03-11T22:08:39.073180+0000","last_active":"2024-03-11T22:09:36.267346+0000","last_peered":"2024-03-11T22:09:36.267346+0000","last_clean":"2024-03-11T22:09:36.267346+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:09:36.267346+0000","last_undegraded":"2024-03-11T22:09:36.267346+0000","last_fullsized":"2024-03-11T22:09:36.267346+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:54.803103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":310,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.268866+0000","last_change":"2024-03-11T22:08:42.030878+0000","last_active":"2024-03-11T22:09:36.268866+0000","last_peered":"2024-03-11T22:09:36.268866+0000","last_clean":"2024-03-11T22:09:36.268866+0000","last_became_active":"2024-03-11T22:08:22.015394+0000","last_became_peered":"2024-03-11T22:08:22.015394+0000","last_unstale":"2024-03-11T22:09:36.268866+0000","last_undegraded":"2024-03-11T22:09:36.268866+0000","last_fullsized":"2024-03-11T22:09:36.268866+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-03-11T22:08:42.030749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:06:13.216089+0000","last_clean_scrub_stamp":"2024-03-11T22:08:42.030749+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:44.064265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:41.615904+0000","last_change":"2024-03-11T22:09:41.615925+0000","last_active":"2024-03-11T22:09:41.615904+0000","last_peered":"2024-03-11T22:09:41.615904+0000","last_clean":"2024-03-11T22:09:41.615904+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:09:41.615904+0000","last_undegraded":"2024-03-11T22:09:41.615904+0000","last_fullsized":"2024-03-11T22:09:41.615904+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:41.615829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:41.615829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:04.514072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":91,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:43.350562+0000","last_change":"2024-03-11T22:08:35.086266+0000","last_active":"2024-03-11T22:09:43.350562+0000","last_peered":"2024-03-11T22:09:43.350562+0000","last_clean":"2024-03-11T22:09:43.350562+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:09:43.350562+0000","last_undegraded":"2024-03-11T22:09:43.350562+0000","last_fullsized":"2024-03-11T22:09:43.350562+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.950025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269112+0000","last_change":"2024-03-11T22:08:37.076167+0000","last_active":"2024-03-11T22:09:36.269112+0000","last_peered":"2024-03-11T22:09:36.269112+0000","last_clean":"2024-03-11T22:09:36.269112+0000","last_became_active":"2024-03-11T22:08:37.075876+0000","last_became_peered":"2024-03-11T22:08:37.075876+0000","last_unstale":"2024-03-11T22:09:36.269112+0000","last_undegraded":"2024-03-11T22:09:36.269112+0000","last_fullsized":"2024-03-11T22:09:36.269112+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:52.209442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621576+0000","last_change":"2024-03-11T22:08:39.077599+0000","last_active":"2024-03-11T22:09:36.621576+0000","last_peered":"2024-03-11T22:09:36.621576+0000","last_clean":"2024-03-11T22:09:36.621576+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:09:36.621576+0000","last_undegraded":"2024-03-11T22:09:36.621576+0000","last_fullsized":"2024-03-11T22:09:36.621576+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:48.216094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048026+0000","last_change":"2024-03-11T22:08:33.035933+0000","last_active":"2024-03-11T22:09:36.048026+0000","last_peered":"2024-03-11T22:09:36.048026+0000","last_clean":"2024-03-11T22:09:36.048026+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:09:36.048026+0000","last_undegraded":"2024-03-11T22:09:36.048026+0000","last_fullsized":"2024-03-11T22:09:36.048026+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.049951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"76'5","reported_seq":139,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.675121+0000","last_change":"2024-03-11T22:08:36.060461+0000","last_active":"2024-03-11T22:09:48.675121+0000","last_peered":"2024-03-11T22:09:48.675121+0000","last_clean":"2024-03-11T22:09:48.675121+0000","last_became_active":"2024-03-11T22:08:32.033652+0000","last_became_peered":"2024-03-11T22:08:32.033652+0000","last_unstale":"2024-03-11T22:09:48.675121+0000","last_undegraded":"2024-03-11T22:09:48.675121+0000","last_fullsized":"2024-03-11T22:09:48.675121+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_clean_scrub_stamp":"2024-03-11T22:08:31.015772+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:58.101329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":70,"num_read_kb":65,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049851+0000","last_change":"2024-03-11T22:08:35.058544+0000","last_active":"2024-03-11T22:09:36.049851+0000","last_peered":"2024-03-11T22:09:36.049851+0000","last_clean":"2024-03-11T22:09:36.049851+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:09:36.049851+0000","last_undegraded":"2024-03-11T22:09:36.049851+0000","last_fullsized":"2024-03-11T22:09:36.049851+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:45.368059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267521+0000","last_change":"2024-03-11T22:08:37.082329+0000","last_active":"2024-03-11T22:09:36.267521+0000","last_peered":"2024-03-11T22:09:36.267521+0000","last_clean":"2024-03-11T22:09:36.267521+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:09:36.267521+0000","last_undegraded":"2024-03-11T22:09:36.267521+0000","last_fullsized":"2024-03-11T22:09:36.267521+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:04.541129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621436+0000","last_change":"2024-03-11T22:08:39.076335+0000","last_active":"2024-03-11T22:09:36.621436+0000","last_peered":"2024-03-11T22:09:36.621436+0000","last_clean":"2024-03-11T22:09:36.621436+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:09:36.621436+0000","last_undegraded":"2024-03-11T22:09:36.621436+0000","last_fullsized":"2024-03-11T22:09:36.621436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:49.460991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621916+0000","last_change":"2024-03-11T22:08:33.298533+0000","last_active":"2024-03-11T22:09:36.621916+0000","last_peered":"2024-03-11T22:09:36.621916+0000","last_clean":"2024-03-11T22:09:36.621916+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:09:36.621916+0000","last_undegraded":"2024-03-11T22:09:36.621916+0000","last_fullsized":"2024-03-11T22:09:36.621916+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.310011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":89,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049619+0000","last_change":"2024-03-11T22:08:35.087309+0000","last_active":"2024-03-11T22:09:36.049619+0000","last_peered":"2024-03-11T22:09:36.049619+0000","last_clean":"2024-03-11T22:09:36.049619+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:09:36.049619+0000","last_undegraded":"2024-03-11T22:09:36.049619+0000","last_fullsized":"2024-03-11T22:09:36.049619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:02.382775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.121074+0000","last_change":"2024-03-11T22:08:37.074393+0000","last_active":"2024-03-11T22:09:36.121074+0000","last_peered":"2024-03-11T22:09:36.121074+0000","last_clean":"2024-03-11T22:09:36.121074+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:09:36.121074+0000","last_undegraded":"2024-03-11T22:09:36.121074+0000","last_fullsized":"2024-03-11T22:09:36.121074+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:02.379574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048168+0000","last_change":"2024-03-11T22:08:39.085179+0000","last_active":"2024-03-11T22:09:36.048168+0000","last_peered":"2024-03-11T22:09:36.048168+0000","last_clean":"2024-03-11T22:09:36.048168+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:09:36.048168+0000","last_undegraded":"2024-03-11T22:09:36.048168+0000","last_fullsized":"2024-03-11T22:09:36.048168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:05.345954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:43.271383+0000","last_change":"2024-03-11T22:09:43.271410+0000","last_active":"2024-03-11T22:09:43.271383+0000","last_peered":"2024-03-11T22:09:43.271383+0000","last_clean":"2024-03-11T22:09:43.271383+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:09:43.271383+0000","last_undegraded":"2024-03-11T22:09:43.271383+0000","last_fullsized":"2024-03-11T22:09:43.271383+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:43.271304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:43.271304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:50.753986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622130+0000","last_change":"2024-03-11T22:08:35.058060+0000","last_active":"2024-03-11T22:09:36.622130+0000","last_peered":"2024-03-11T22:09:36.622130+0000","last_clean":"2024-03-11T22:09:36.622130+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:09:36.622130+0000","last_undegraded":"2024-03-11T22:09:36.622130+0000","last_fullsized":"2024-03-11T22:09:36.622130+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.703549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.187735+0000","last_change":"2024-03-11T22:09:48.187735+0000","last_active":"2024-03-11T22:09:48.187735+0000","last_peered":"2024-03-11T22:09:48.187735+0000","last_clean":"2024-03-11T22:09:48.187735+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:09:48.187735+0000","last_undegraded":"2024-03-11T22:09:48.187735+0000","last_fullsized":"2024-03-11T22:09:48.187735+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:09:48.187641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:48.187641+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:56.947907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269149+0000","last_change":"2024-03-11T22:08:39.076424+0000","last_active":"2024-03-11T22:09:36.269149+0000","last_peered":"2024-03-11T22:09:36.269149+0000","last_clean":"2024-03-11T22:09:36.269149+0000","last_became_active":"2024-03-11T22:08:39.076114+0000","last_became_peered":"2024-03-11T22:08:39.076114+0000","last_unstale":"2024-03-11T22:09:36.269149+0000","last_undegraded":"2024-03-11T22:09:36.269149+0000","last_fullsized":"2024-03-11T22:09:36.269149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:48.446160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049921+0000","last_change":"2024-03-11T22:08:33.039271+0000","last_active":"2024-03-11T22:09:36.049921+0000","last_peered":"2024-03-11T22:09:36.049921+0000","last_clean":"2024-03-11T22:09:36.049921+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:09:36.049921+0000","last_undegraded":"2024-03-11T22:09:36.049921+0000","last_fullsized":"2024-03-11T22:09:36.049921+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:52.098180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047924+0000","last_change":"2024-03-11T22:08:35.053883+0000","last_active":"2024-03-11T22:09:36.047924+0000","last_peered":"2024-03-11T22:09:36.047924+0000","last_clean":"2024-03-11T22:09:36.047924+0000","last_became_active":"2024-03-11T22:08:35.053452+0000","last_became_peered":"2024-03-11T22:08:35.053452+0000","last_unstale":"2024-03-11T22:09:36.047924+0000","last_undegraded":"2024-03-11T22:09:36.047924+0000","last_fullsized":"2024-03-11T22:09:36.047924+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:42.807757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:43.350422+0000","last_change":"2024-03-11T22:08:37.078918+0000","last_active":"2024-03-11T22:09:43.350422+0000","last_peered":"2024-03-11T22:09:43.350422+0000","last_clean":"2024-03-11T22:09:43.350422+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:09:43.350422+0000","last_undegraded":"2024-03-11T22:09:43.350422+0000","last_fullsized":"2024-03-11T22:09:43.350422+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:50.568134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048188+0000","last_change":"2024-03-11T22:08:39.078715+0000","last_active":"2024-03-11T22:09:36.048188+0000","last_peered":"2024-03-11T22:09:36.048188+0000","last_clean":"2024-03-11T22:09:36.048188+0000","last_became_active":"2024-03-11T22:08:39.078244+0000","last_became_peered":"2024-03-11T22:08:39.078244+0000","last_unstale":"2024-03-11T22:09:36.048188+0000","last_undegraded":"2024-03-11T22:09:36.048188+0000","last_fullsized":"2024-03-11T22:09:36.048188+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:07.090388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.564984+0000","last_change":"2024-03-11T22:09:36.565005+0000","last_active":"2024-03-11T22:09:36.564984+0000","last_peered":"2024-03-11T22:09:36.564984+0000","last_clean":"2024-03-11T22:09:36.564984+0000","last_became_active":"2024-03-11T22:08:33.036469+0000","last_became_peered":"2024-03-11T22:08:33.036469+0000","last_unstale":"2024-03-11T22:09:36.564984+0000","last_undegraded":"2024-03-11T22:09:36.564984+0000","last_fullsized":"2024-03-11T22:09:36.564984+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:36.564912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:36.564912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:49.963865+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047627+0000","last_change":"2024-03-11T22:08:35.086576+0000","last_active":"2024-03-11T22:09:36.047627+0000","last_peered":"2024-03-11T22:09:36.047627+0000","last_clean":"2024-03-11T22:09:36.047627+0000","last_became_active":"2024-03-11T22:08:35.085979+0000","last_became_peered":"2024-03-11T22:08:35.085979+0000","last_unstale":"2024-03-11T22:09:36.047627+0000","last_undegraded":"2024-03-11T22:09:36.047627+0000","last_fullsized":"2024-03-11T22:09:36.047627+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.168181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047777+0000","last_change":"2024-03-11T22:08:37.076797+0000","last_active":"2024-03-11T22:09:36.047777+0000","last_peered":"2024-03-11T22:09:36.047777+0000","last_clean":"2024-03-11T22:09:36.047777+0000","last_became_active":"2024-03-11T22:08:37.076522+0000","last_became_peered":"2024-03-11T22:08:37.076522+0000","last_unstale":"2024-03-11T22:09:36.047777+0000","last_undegraded":"2024-03-11T22:09:36.047777+0000","last_fullsized":"2024-03-11T22:09:36.047777+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:43.451980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267135+0000","last_change":"2024-03-11T22:08:39.080281+0000","last_active":"2024-03-11T22:09:36.267135+0000","last_peered":"2024-03-11T22:09:36.267135+0000","last_clean":"2024-03-11T22:09:36.267135+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:09:36.267135+0000","last_undegraded":"2024-03-11T22:09:36.267135+0000","last_fullsized":"2024-03-11T22:09:36.267135+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.517616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:43.350667+0000","last_change":"2024-03-11T22:08:37.077452+0000","last_active":"2024-03-11T22:09:43.350667+0000","last_peered":"2024-03-11T22:09:43.350667+0000","last_clean":"2024-03-11T22:09:43.350667+0000","last_became_active":"2024-03-11T22:08:37.077142+0000","last_became_peered":"2024-03-11T22:08:37.077142+0000","last_unstale":"2024-03-11T22:09:43.350667+0000","last_undegraded":"2024-03-11T22:09:43.350667+0000","last_fullsized":"2024-03-11T22:09:43.350667+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:49.795619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047538+0000","last_change":"2024-03-11T22:08:33.037839+0000","last_active":"2024-03-11T22:09:36.047538+0000","last_peered":"2024-03-11T22:09:36.047538+0000","last_clean":"2024-03-11T22:09:36.047538+0000","last_became_active":"2024-03-11T22:08:33.037537+0000","last_became_peered":"2024-03-11T22:08:33.037537+0000","last_unstale":"2024-03-11T22:09:36.047538+0000","last_undegraded":"2024-03-11T22:09:36.047538+0000","last_fullsized":"2024-03-11T22:09:36.047538+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:51.539803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622582+0000","last_change":"2024-03-11T22:08:35.060718+0000","last_active":"2024-03-11T22:09:36.622582+0000","last_peered":"2024-03-11T22:09:36.622582+0000","last_clean":"2024-03-11T22:09:36.622582+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:09:36.622582+0000","last_undegraded":"2024-03-11T22:09:36.622582+0000","last_fullsized":"2024-03-11T22:09:36.622582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:53.393792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048865+0000","last_change":"2024-03-11T22:08:39.077317+0000","last_active":"2024-03-11T22:09:36.048865+0000","last_peered":"2024-03-11T22:09:36.048865+0000","last_clean":"2024-03-11T22:09:36.048865+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:09:36.048865+0000","last_undegraded":"2024-03-11T22:09:36.048865+0000","last_fullsized":"2024-03-11T22:09:36.048865+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.786808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267508+0000","last_change":"2024-03-11T22:08:37.082228+0000","last_active":"2024-03-11T22:09:36.267508+0000","last_peered":"2024-03-11T22:09:36.267508+0000","last_clean":"2024-03-11T22:09:36.267508+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:09:36.267508+0000","last_undegraded":"2024-03-11T22:09:36.267508+0000","last_fullsized":"2024-03-11T22:09:36.267508+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.417015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047823+0000","last_change":"2024-03-11T22:08:33.300125+0000","last_active":"2024-03-11T22:09:36.047823+0000","last_peered":"2024-03-11T22:09:36.047823+0000","last_clean":"2024-03-11T22:09:36.047823+0000","last_became_active":"2024-03-11T22:08:33.299778+0000","last_became_peered":"2024-03-11T22:08:33.299778+0000","last_unstale":"2024-03-11T22:09:36.047823+0000","last_undegraded":"2024-03-11T22:09:36.047823+0000","last_fullsized":"2024-03-11T22:09:36.047823+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:58.833176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:40.448823+0000","last_change":"2024-03-11T22:09:40.448862+0000","last_active":"2024-03-11T22:09:40.448823+0000","last_peered":"2024-03-11T22:09:40.448823+0000","last_clean":"2024-03-11T22:09:40.448823+0000","last_became_active":"2024-03-11T22:08:35.053123+0000","last_became_peered":"2024-03-11T22:08:35.053123+0000","last_unstale":"2024-03-11T22:09:40.448823+0000","last_undegraded":"2024-03-11T22:09:40.448823+0000","last_fullsized":"2024-03-11T22:09:40.448823+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:09:40.448751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:40.448751+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:07.077279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622708+0000","last_change":"2024-03-11T22:08:39.074701+0000","last_active":"2024-03-11T22:09:36.622708+0000","last_peered":"2024-03-11T22:09:36.622708+0000","last_clean":"2024-03-11T22:09:36.622708+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:09:36.622708+0000","last_undegraded":"2024-03-11T22:09:36.622708+0000","last_fullsized":"2024-03-11T22:09:36.622708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.861450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622128+0000","last_change":"2024-03-11T22:08:37.078886+0000","last_active":"2024-03-11T22:09:36.622128+0000","last_peered":"2024-03-11T22:09:36.622128+0000","last_clean":"2024-03-11T22:09:36.622128+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:09:36.622128+0000","last_undegraded":"2024-03-11T22:09:36.622128+0000","last_fullsized":"2024-03-11T22:09:36.622128+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:55.775643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:46.542124+0000","last_change":"2024-03-11T22:09:46.542145+0000","last_active":"2024-03-11T22:09:46.542124+0000","last_peered":"2024-03-11T22:09:46.542124+0000","last_clean":"2024-03-11T22:09:46.542124+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:09:46.542124+0000","last_undegraded":"2024-03-11T22:09:46.542124+0000","last_fullsized":"2024-03-11T22:09:46.542124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:46.542054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:46.542054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.271294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048879+0000","last_change":"2024-03-11T22:08:35.055974+0000","last_active":"2024-03-11T22:09:36.048879+0000","last_peered":"2024-03-11T22:09:36.048879+0000","last_clean":"2024-03-11T22:09:36.048879+0000","last_became_active":"2024-03-11T22:08:35.055770+0000","last_became_peered":"2024-03-11T22:08:35.055770+0000","last_unstale":"2024-03-11T22:09:36.048879+0000","last_undegraded":"2024-03-11T22:09:36.048879+0000","last_fullsized":"2024-03-11T22:09:36.048879+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:54.010930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049081+0000","last_change":"2024-03-11T22:08:39.073197+0000","last_active":"2024-03-11T22:09:36.049081+0000","last_peered":"2024-03-11T22:09:36.049081+0000","last_clean":"2024-03-11T22:09:36.049081+0000","last_became_active":"2024-03-11T22:08:39.072422+0000","last_became_peered":"2024-03-11T22:08:39.072422+0000","last_unstale":"2024-03-11T22:09:36.049081+0000","last_undegraded":"2024-03-11T22:09:36.049081+0000","last_fullsized":"2024-03-11T22:09:36.049081+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:51.710195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:43.350414+0000","last_change":"2024-03-11T22:09:38.769937+0000","last_active":"2024-03-11T22:09:43.350414+0000","last_peered":"2024-03-11T22:09:43.350414+0000","last_clean":"2024-03-11T22:09:43.350414+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:09:43.350414+0000","last_undegraded":"2024-03-11T22:09:43.350414+0000","last_fullsized":"2024-03-11T22:09:43.350414+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_clean_scrub_stamp":"2024-03-11T22:09:38.769825+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:58.315026+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048625+0000","last_change":"2024-03-11T22:08:33.036116+0000","last_active":"2024-03-11T22:09:36.048625+0000","last_peered":"2024-03-11T22:09:36.048625+0000","last_clean":"2024-03-11T22:09:36.048625+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:09:36.048625+0000","last_undegraded":"2024-03-11T22:09:36.048625+0000","last_fullsized":"2024-03-11T22:09:36.048625+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:40.981310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048532+0000","last_change":"2024-03-11T22:08:35.057039+0000","last_active":"2024-03-11T22:09:36.048532+0000","last_peered":"2024-03-11T22:09:36.048532+0000","last_clean":"2024-03-11T22:09:36.048532+0000","last_became_active":"2024-03-11T22:08:35.056698+0000","last_became_peered":"2024-03-11T22:08:35.056698+0000","last_unstale":"2024-03-11T22:09:36.048532+0000","last_undegraded":"2024-03-11T22:09:36.048532+0000","last_fullsized":"2024-03-11T22:09:36.048532+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:51.765344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":400,"ondisk_log_size":400,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":65,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005076,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1523712,"data_stored":1024363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332121,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28484,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716668,"statfs":{"total":95995035648,"available":95965868032,"internally_reserved":0,"allocated":1503232,"data_stored":1024173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":434001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32660,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712492,"statfs":{"total":95995035648,"available":95961591808,"internally_reserved":0,"allocated":1519616,"data_stored":1023695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":905216,"data_stored":433631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":434311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920804,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":433536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659165,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":433078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:09:55.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:09:55.375 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:55 smithi045 ceph-mon[39715]: 4.1 scrub ok 2024-03-11T22:09:55.375 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:55 smithi045 ceph-mon[39715]: 6.4 scrub starts 2024-03-11T22:09:55.375 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:55 smithi045 ceph-mon[39715]: 6.4 scrub ok 2024-03-11T22:09:55.375 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:55 smithi045 ceph-mon[39715]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:09:55.375 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:55 smithi045 ceph-mon[39715]: 4.a scrub starts 2024-03-11T22:09:55.375 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:55 smithi045 ceph-mon[39715]: 4.a scrub ok 2024-03-11T22:09:55.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:55 smithi045 ceph-mon[30178]: 4.1 scrub ok 2024-03-11T22:09:55.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:55 smithi045 ceph-mon[30178]: 6.4 scrub starts 2024-03-11T22:09:55.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:55 smithi045 ceph-mon[30178]: 6.4 scrub ok 2024-03-11T22:09:55.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:55 smithi045 ceph-mon[30178]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:09:55.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:55 smithi045 ceph-mon[30178]: 4.a scrub starts 2024-03-11T22:09:55.375 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:55 smithi045 ceph-mon[30178]: 4.a scrub ok 2024-03-11T22:09:55.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:55 smithi144 ceph-mon[33984]: 4.1 scrub ok 2024-03-11T22:09:55.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:55 smithi144 ceph-mon[33984]: 6.4 scrub starts 2024-03-11T22:09:55.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:55 smithi144 ceph-mon[33984]: 6.4 scrub ok 2024-03-11T22:09:55.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:55 smithi144 ceph-mon[33984]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:09:55.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:55 smithi144 ceph-mon[33984]: 4.a scrub starts 2024-03-11T22:09:55.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:55 smithi144 ceph-mon[33984]: 4.a scrub ok 2024-03-11T22:09:56.089 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:09:55 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:09:56.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: 5.1 scrub starts 2024-03-11T22:09:56.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: 5.1 scrub ok 2024-03-11T22:09:56.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: 6.1 scrub starts 2024-03-11T22:09:56.439 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: 6.1 scrub ok 2024-03-11T22:09:56.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:09:56.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:09:56.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T22:09:56.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: 3.1f scrub starts 2024-03-11T22:09:56.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: 3.1f scrub ok 2024-03-11T22:09:56.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: 3.e scrub starts 2024-03-11T22:09:56.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: 3.e scrub ok 2024-03-11T22:09:56.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:09:56.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:09:56.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: 4.14 scrub starts 2024-03-11T22:09:56.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[30178]: 4.14 scrub ok 2024-03-11T22:09:56.440 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: 5.1 scrub starts 2024-03-11T22:09:56.440 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: 5.1 scrub ok 2024-03-11T22:09:56.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: 6.1 scrub starts 2024-03-11T22:09:56.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: 6.1 scrub ok 2024-03-11T22:09:56.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:09:56.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:09:56.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T22:09:56.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: 3.1f scrub starts 2024-03-11T22:09:56.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: 3.1f scrub ok 2024-03-11T22:09:56.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: 3.e scrub starts 2024-03-11T22:09:56.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: 3.e scrub ok 2024-03-11T22:09:56.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:09:56.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:09:56.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: 4.14 scrub starts 2024-03-11T22:09:56.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:56 smithi045 ceph-mon[39715]: 4.14 scrub ok 2024-03-11T22:09:56.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: 5.1 scrub starts 2024-03-11T22:09:56.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: 5.1 scrub ok 2024-03-11T22:09:56.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: 6.1 scrub starts 2024-03-11T22:09:56.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: 6.1 scrub ok 2024-03-11T22:09:56.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:09:56.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:09:56.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T22:09:56.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: 3.1f scrub starts 2024-03-11T22:09:56.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: 3.1f scrub ok 2024-03-11T22:09:56.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: 3.e scrub starts 2024-03-11T22:09:56.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: 3.e scrub ok 2024-03-11T22:09:56.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:09:56.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:09:56.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: 4.14 scrub starts 2024-03-11T22:09:56.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:56 smithi144 ceph-mon[33984]: 4.14 scrub ok 2024-03-11T22:09:57.433 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:57 smithi045 ceph-mon[39715]: 3.3 scrub starts 2024-03-11T22:09:57.434 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:57 smithi045 ceph-mon[39715]: 3.3 scrub ok 2024-03-11T22:09:57.434 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:57.434 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:57 smithi045 ceph-mon[39715]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:09:57.434 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:57 smithi045 ceph-mon[39715]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T22:09:57.434 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:57 smithi045 ceph-mon[30178]: 3.3 scrub starts 2024-03-11T22:09:57.434 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:57 smithi045 ceph-mon[30178]: 3.3 scrub ok 2024-03-11T22:09:57.434 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:57.434 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:57 smithi045 ceph-mon[30178]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:09:57.434 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:57 smithi045 ceph-mon[30178]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T22:09:57.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:57 smithi144 ceph-mon[33984]: 3.3 scrub starts 2024-03-11T22:09:57.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:57 smithi144 ceph-mon[33984]: 3.3 scrub ok 2024-03-11T22:09:57.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:09:57.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:57 smithi144 ceph-mon[33984]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:09:57.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:57 smithi144 ceph-mon[33984]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T22:09:58.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:58 smithi045 ceph-mon[39715]: 6.9 scrub starts 2024-03-11T22:09:58.275 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:58 smithi045 ceph-mon[39715]: 6.9 scrub ok 2024-03-11T22:09:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:58 smithi045 ceph-mon[30178]: 6.9 scrub starts 2024-03-11T22:09:58.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:58 smithi045 ceph-mon[30178]: 6.9 scrub ok 2024-03-11T22:09:58.500 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:09:58.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:58 smithi144 ceph-mon[33984]: 6.9 scrub starts 2024-03-11T22:09:58.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:58 smithi144 ceph-mon[33984]: 6.9 scrub ok 2024-03-11T22:09:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[30178]: 4.12 scrub starts 2024-03-11T22:09:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[30178]: 4.12 scrub ok 2024-03-11T22:09:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[30178]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:09:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[30178]: 5.15 scrub starts 2024-03-11T22:09:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[30178]: 5.15 scrub ok 2024-03-11T22:09:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[30178]: 3.1b scrub starts 2024-03-11T22:09:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[30178]: 3.1b scrub ok 2024-03-11T22:09:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[30178]: 5.10 deep-scrub starts 2024-03-11T22:09:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[30178]: 5.10 deep-scrub ok 2024-03-11T22:09:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[30178]: 3.b scrub starts 2024-03-11T22:09:59.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[30178]: 3.b scrub ok 2024-03-11T22:09:59.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[39715]: 4.12 scrub starts 2024-03-11T22:09:59.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[39715]: 4.12 scrub ok 2024-03-11T22:09:59.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[39715]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:09:59.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[39715]: 5.15 scrub starts 2024-03-11T22:09:59.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[39715]: 5.15 scrub ok 2024-03-11T22:09:59.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[39715]: 3.1b scrub starts 2024-03-11T22:09:59.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[39715]: 3.1b scrub ok 2024-03-11T22:09:59.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[39715]: 5.10 deep-scrub starts 2024-03-11T22:09:59.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[39715]: 5.10 deep-scrub ok 2024-03-11T22:09:59.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[39715]: 3.b scrub starts 2024-03-11T22:09:59.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:09:59 smithi045 ceph-mon[39715]: 3.b scrub ok 2024-03-11T22:09:59.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:59 smithi144 ceph-mon[33984]: 4.12 scrub starts 2024-03-11T22:09:59.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:59 smithi144 ceph-mon[33984]: 4.12 scrub ok 2024-03-11T22:09:59.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:59 smithi144 ceph-mon[33984]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:09:59.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:59 smithi144 ceph-mon[33984]: 5.15 scrub starts 2024-03-11T22:09:59.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:59 smithi144 ceph-mon[33984]: 5.15 scrub ok 2024-03-11T22:09:59.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:59 smithi144 ceph-mon[33984]: 3.1b scrub starts 2024-03-11T22:09:59.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:59 smithi144 ceph-mon[33984]: 3.1b scrub ok 2024-03-11T22:09:59.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:59 smithi144 ceph-mon[33984]: 5.10 deep-scrub starts 2024-03-11T22:09:59.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:59 smithi144 ceph-mon[33984]: 5.10 deep-scrub ok 2024-03-11T22:09:59.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:59 smithi144 ceph-mon[33984]: 3.b scrub starts 2024-03-11T22:09:59.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:09:59 smithi144 ceph-mon[33984]: 3.b scrub ok 2024-03-11T22:10:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:10:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[30178]: 3.12 scrub starts 2024-03-11T22:10:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[30178]: 3.12 scrub ok 2024-03-11T22:10:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[30178]: 1.0 deep-scrub starts 2024-03-11T22:10:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[30178]: 1.0 deep-scrub ok 2024-03-11T22:10:00.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[30178]: 5.1f scrub starts 2024-03-11T22:10:00.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[30178]: 5.1f scrub ok 2024-03-11T22:10:00.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[30178]: overall HEALTH_OK 2024-03-11T22:10:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:10:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[39715]: 3.12 scrub starts 2024-03-11T22:10:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[39715]: 3.12 scrub ok 2024-03-11T22:10:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[39715]: 1.0 deep-scrub starts 2024-03-11T22:10:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[39715]: 1.0 deep-scrub ok 2024-03-11T22:10:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[39715]: 5.1f scrub starts 2024-03-11T22:10:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[39715]: 5.1f scrub ok 2024-03-11T22:10:00.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:00 smithi045 ceph-mon[39715]: overall HEALTH_OK 2024-03-11T22:10:00.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:10:00.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:00 smithi144 ceph-mon[33984]: 3.12 scrub starts 2024-03-11T22:10:00.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:00 smithi144 ceph-mon[33984]: 3.12 scrub ok 2024-03-11T22:10:00.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:00 smithi144 ceph-mon[33984]: 1.0 deep-scrub starts 2024-03-11T22:10:00.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:00 smithi144 ceph-mon[33984]: 1.0 deep-scrub ok 2024-03-11T22:10:00.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:00 smithi144 ceph-mon[33984]: 5.1f scrub starts 2024-03-11T22:10:00.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:00 smithi144 ceph-mon[33984]: 5.1f scrub ok 2024-03-11T22:10:00.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:00 smithi144 ceph-mon[33984]: overall HEALTH_OK 2024-03-11T22:10:00.947 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:10:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:10:01.383 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[39715]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:10:01.384 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[39715]: 2.2 deep-scrub starts 2024-03-11T22:10:01.384 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[39715]: 2.2 deep-scrub ok 2024-03-11T22:10:01.384 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[39715]: 5.1a deep-scrub starts 2024-03-11T22:10:01.384 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[39715]: 5.1a deep-scrub ok 2024-03-11T22:10:01.384 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[39715]: 3.10 deep-scrub starts 2024-03-11T22:10:01.384 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[39715]: 3.10 deep-scrub ok 2024-03-11T22:10:01.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[30178]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:10:01.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[30178]: 2.2 deep-scrub starts 2024-03-11T22:10:01.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[30178]: 2.2 deep-scrub ok 2024-03-11T22:10:01.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[30178]: 5.1a deep-scrub starts 2024-03-11T22:10:01.384 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[30178]: 5.1a deep-scrub ok 2024-03-11T22:10:01.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[30178]: 3.10 deep-scrub starts 2024-03-11T22:10:01.385 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-mon[30178]: 3.10 deep-scrub ok 2024-03-11T22:10:01.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:01 smithi144 ceph-mon[33984]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:10:01.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:01 smithi144 ceph-mon[33984]: 2.2 deep-scrub starts 2024-03-11T22:10:01.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:01 smithi144 ceph-mon[33984]: 2.2 deep-scrub ok 2024-03-11T22:10:01.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:01 smithi144 ceph-mon[33984]: 5.1a deep-scrub starts 2024-03-11T22:10:01.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:01 smithi144 ceph-mon[33984]: 5.1a deep-scrub ok 2024-03-11T22:10:01.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:01 smithi144 ceph-mon[33984]: 3.10 deep-scrub starts 2024-03-11T22:10:01.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:01 smithi144 ceph-mon[33984]: 3.10 deep-scrub ok 2024-03-11T22:10:01.714 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-03-11T22:10:00.045565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1009,"num_read_kb":855,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":608,"ondisk_log_size":608,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":241756,"kb_used_data":8988,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749719460,"statfs":{"total":767960285184,"available":767712727040,"internally_reserved":0,"allocated":9203712,"data_stored":5240788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":238276147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001795"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048103+0000","last_change":"2024-03-11T22:08:39.077833+0000","last_active":"2024-03-11T22:09:36.048103+0000","last_peered":"2024-03-11T22:09:36.048103+0000","last_clean":"2024-03-11T22:09:36.048103+0000","last_became_active":"2024-03-11T22:08:39.074695+0000","last_became_peered":"2024-03-11T22:08:39.074695+0000","last_unstale":"2024-03-11T22:09:36.048103+0000","last_undegraded":"2024-03-11T22:09:36.048103+0000","last_fullsized":"2024-03-11T22:09:36.048103+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:05.446193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621456+0000","last_change":"2024-03-11T22:08:37.074138+0000","last_active":"2024-03-11T22:09:36.621456+0000","last_peered":"2024-03-11T22:09:36.621456+0000","last_clean":"2024-03-11T22:09:36.621456+0000","last_became_active":"2024-03-11T22:08:37.073775+0000","last_became_peered":"2024-03-11T22:08:37.073775+0000","last_unstale":"2024-03-11T22:09:36.621456+0000","last_undegraded":"2024-03-11T22:09:36.621456+0000","last_fullsized":"2024-03-11T22:09:36.621456+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:02.401482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049526+0000","last_change":"2024-03-11T22:08:33.298497+0000","last_active":"2024-03-11T22:09:36.049526+0000","last_peered":"2024-03-11T22:09:36.049526+0000","last_clean":"2024-03-11T22:09:36.049526+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:09:36.049526+0000","last_undegraded":"2024-03-11T22:09:36.049526+0000","last_fullsized":"2024-03-11T22:09:36.049526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:46.682751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047777+0000","last_change":"2024-03-11T22:08:35.053644+0000","last_active":"2024-03-11T22:09:36.047777+0000","last_peered":"2024-03-11T22:09:36.047777+0000","last_clean":"2024-03-11T22:09:36.047777+0000","last_became_active":"2024-03-11T22:08:35.053179+0000","last_became_peered":"2024-03-11T22:08:35.053179+0000","last_unstale":"2024-03-11T22:09:36.047777+0000","last_undegraded":"2024-03-11T22:09:36.047777+0000","last_fullsized":"2024-03-11T22:09:36.047777+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:48.806815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048530+0000","last_change":"2024-03-11T22:08:39.079762+0000","last_active":"2024-03-11T22:09:36.048530+0000","last_peered":"2024-03-11T22:09:36.048530+0000","last_clean":"2024-03-11T22:09:36.048530+0000","last_became_active":"2024-03-11T22:08:39.079045+0000","last_became_peered":"2024-03-11T22:08:39.079045+0000","last_unstale":"2024-03-11T22:09:36.048530+0000","last_undegraded":"2024-03-11T22:09:36.048530+0000","last_fullsized":"2024-03-11T22:09:36.048530+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.065191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047777+0000","last_change":"2024-03-11T22:08:37.082192+0000","last_active":"2024-03-11T22:09:36.047777+0000","last_peered":"2024-03-11T22:09:36.047777+0000","last_clean":"2024-03-11T22:09:36.047777+0000","last_became_active":"2024-03-11T22:08:37.081861+0000","last_became_peered":"2024-03-11T22:08:37.081861+0000","last_unstale":"2024-03-11T22:09:36.047777+0000","last_undegraded":"2024-03-11T22:09:36.047777+0000","last_fullsized":"2024-03-11T22:09:36.047777+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:47.606115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:39.608956+0000","last_change":"2024-03-11T22:09:39.608984+0000","last_active":"2024-03-11T22:09:39.608956+0000","last_peered":"2024-03-11T22:09:39.608956+0000","last_clean":"2024-03-11T22:09:39.608956+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:09:39.608956+0000","last_undegraded":"2024-03-11T22:09:39.608956+0000","last_fullsized":"2024-03-11T22:09:39.608956+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:09:39.608890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:39.608890+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:49.086609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047962+0000","last_change":"2024-03-11T22:08:35.061181+0000","last_active":"2024-03-11T22:09:36.047962+0000","last_peered":"2024-03-11T22:09:36.047962+0000","last_clean":"2024-03-11T22:09:36.047962+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:09:36.047962+0000","last_undegraded":"2024-03-11T22:09:36.047962+0000","last_fullsized":"2024-03-11T22:09:36.047962+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.748787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048226+0000","last_change":"2024-03-11T22:08:39.083128+0000","last_active":"2024-03-11T22:09:36.048226+0000","last_peered":"2024-03-11T22:09:36.048226+0000","last_clean":"2024-03-11T22:09:36.048226+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:09:36.048226+0000","last_undegraded":"2024-03-11T22:09:36.048226+0000","last_fullsized":"2024-03-11T22:09:36.048226+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.592922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267136+0000","last_change":"2024-03-11T22:08:37.082294+0000","last_active":"2024-03-11T22:09:36.267136+0000","last_peered":"2024-03-11T22:09:36.267136+0000","last_clean":"2024-03-11T22:09:36.267136+0000","last_became_active":"2024-03-11T22:08:37.081996+0000","last_became_peered":"2024-03-11T22:08:37.081996+0000","last_unstale":"2024-03-11T22:09:36.267136+0000","last_undegraded":"2024-03-11T22:09:36.267136+0000","last_fullsized":"2024-03-11T22:09:36.267136+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.530415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:40.301247+0000","last_change":"2024-03-11T22:09:40.301289+0000","last_active":"2024-03-11T22:09:40.301247+0000","last_peered":"2024-03-11T22:09:40.301247+0000","last_clean":"2024-03-11T22:09:40.301247+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:09:40.301247+0000","last_undegraded":"2024-03-11T22:09:40.301247+0000","last_fullsized":"2024-03-11T22:09:40.301247+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:40.301162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:40.301162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:40.904990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048314+0000","last_change":"2024-03-11T22:08:35.061181+0000","last_active":"2024-03-11T22:09:36.048314+0000","last_peered":"2024-03-11T22:09:36.048314+0000","last_clean":"2024-03-11T22:09:36.048314+0000","last_became_active":"2024-03-11T22:08:35.060802+0000","last_became_peered":"2024-03-11T22:08:35.060802+0000","last_unstale":"2024-03-11T22:09:36.048314+0000","last_undegraded":"2024-03-11T22:09:36.048314+0000","last_fullsized":"2024-03-11T22:09:36.048314+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:44.218392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.266787+0000","last_change":"2024-03-11T22:08:39.076507+0000","last_active":"2024-03-11T22:09:36.266787+0000","last_peered":"2024-03-11T22:09:36.266787+0000","last_clean":"2024-03-11T22:09:36.266787+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:09:36.266787+0000","last_undegraded":"2024-03-11T22:09:36.266787+0000","last_fullsized":"2024-03-11T22:09:36.266787+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:46.781818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.268834+0000","last_change":"2024-03-11T22:08:37.077641+0000","last_active":"2024-03-11T22:09:36.268834+0000","last_peered":"2024-03-11T22:09:36.268834+0000","last_clean":"2024-03-11T22:09:36.268834+0000","last_became_active":"2024-03-11T22:08:37.077279+0000","last_became_peered":"2024-03-11T22:08:37.077279+0000","last_unstale":"2024-03-11T22:09:36.268834+0000","last_undegraded":"2024-03-11T22:09:36.268834+0000","last_fullsized":"2024-03-11T22:09:36.268834+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:44.368118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267049+0000","last_change":"2024-03-11T22:08:33.040601+0000","last_active":"2024-03-11T22:09:36.267049+0000","last_peered":"2024-03-11T22:09:36.267049+0000","last_clean":"2024-03-11T22:09:36.267049+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:09:36.267049+0000","last_undegraded":"2024-03-11T22:09:36.267049+0000","last_fullsized":"2024-03-11T22:09:36.267049+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:48.720858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048153+0000","last_change":"2024-03-11T22:08:35.061039+0000","last_active":"2024-03-11T22:09:36.048153+0000","last_peered":"2024-03-11T22:09:36.048153+0000","last_clean":"2024-03-11T22:09:36.048153+0000","last_became_active":"2024-03-11T22:08:35.060777+0000","last_became_peered":"2024-03-11T22:08:35.060777+0000","last_unstale":"2024-03-11T22:09:36.048153+0000","last_undegraded":"2024-03-11T22:09:36.048153+0000","last_fullsized":"2024-03-11T22:09:36.048153+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:51.210939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:51.367666+0000","last_change":"2024-03-11T22:09:51.367715+0000","last_active":"2024-03-11T22:09:51.367666+0000","last_peered":"2024-03-11T22:09:51.367666+0000","last_clean":"2024-03-11T22:09:51.367666+0000","last_became_active":"2024-03-11T22:08:39.075609+0000","last_became_peered":"2024-03-11T22:08:39.075609+0000","last_unstale":"2024-03-11T22:09:51.367666+0000","last_undegraded":"2024-03-11T22:09:51.367666+0000","last_fullsized":"2024-03-11T22:09:51.367666+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:51.367570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:51.367570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:10.802427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621925+0000","last_change":"2024-03-11T22:08:37.077878+0000","last_active":"2024-03-11T22:09:36.621925+0000","last_peered":"2024-03-11T22:09:36.621925+0000","last_clean":"2024-03-11T22:09:36.621925+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:09:36.621925+0000","last_undegraded":"2024-03-11T22:09:36.621925+0000","last_fullsized":"2024-03-11T22:09:36.621925+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.718775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049612+0000","last_change":"2024-03-11T22:08:33.036036+0000","last_active":"2024-03-11T22:09:36.049612+0000","last_peered":"2024-03-11T22:09:36.049612+0000","last_clean":"2024-03-11T22:09:36.049612+0000","last_became_active":"2024-03-11T22:08:33.035624+0000","last_became_peered":"2024-03-11T22:08:33.035624+0000","last_unstale":"2024-03-11T22:09:36.049612+0000","last_undegraded":"2024-03-11T22:09:36.049612+0000","last_fullsized":"2024-03-11T22:09:36.049612+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:48.949714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:45.731324+0000","last_change":"2024-03-11T22:09:45.731375+0000","last_active":"2024-03-11T22:09:45.731324+0000","last_peered":"2024-03-11T22:09:45.731324+0000","last_clean":"2024-03-11T22:09:45.731324+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:09:45.731324+0000","last_undegraded":"2024-03-11T22:09:45.731324+0000","last_fullsized":"2024-03-11T22:09:45.731324+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:09:45.731248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:45.731248+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:08.445140+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049326+0000","last_change":"2024-03-11T22:08:39.078269+0000","last_active":"2024-03-11T22:09:36.049326+0000","last_peered":"2024-03-11T22:09:36.049326+0000","last_clean":"2024-03-11T22:09:36.049326+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:09:36.049326+0000","last_undegraded":"2024-03-11T22:09:36.049326+0000","last_fullsized":"2024-03-11T22:09:36.049326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.509768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048453+0000","last_change":"2024-03-11T22:08:37.080325+0000","last_active":"2024-03-11T22:09:36.048453+0000","last_peered":"2024-03-11T22:09:36.048453+0000","last_clean":"2024-03-11T22:09:36.048453+0000","last_became_active":"2024-03-11T22:08:37.080014+0000","last_became_peered":"2024-03-11T22:08:37.080014+0000","last_unstale":"2024-03-11T22:09:36.048453+0000","last_undegraded":"2024-03-11T22:09:36.048453+0000","last_fullsized":"2024-03-11T22:09:36.048453+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:04.066121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048415+0000","last_change":"2024-03-11T22:08:33.299536+0000","last_active":"2024-03-11T22:09:36.048415+0000","last_peered":"2024-03-11T22:09:36.048415+0000","last_clean":"2024-03-11T22:09:36.048415+0000","last_became_active":"2024-03-11T22:08:33.299189+0000","last_became_peered":"2024-03-11T22:08:33.299189+0000","last_unstale":"2024-03-11T22:09:36.048415+0000","last_undegraded":"2024-03-11T22:09:36.048415+0000","last_fullsized":"2024-03-11T22:09:36.048415+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:45.295342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048507+0000","last_change":"2024-03-11T22:08:35.066453+0000","last_active":"2024-03-11T22:09:36.048507+0000","last_peered":"2024-03-11T22:09:36.048507+0000","last_clean":"2024-03-11T22:09:36.048507+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:09:36.048507+0000","last_undegraded":"2024-03-11T22:09:36.048507+0000","last_fullsized":"2024-03-11T22:09:36.048507+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:49.876100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:53.160224+0000","last_change":"2024-03-11T22:09:53.160256+0000","last_active":"2024-03-11T22:09:53.160224+0000","last_peered":"2024-03-11T22:09:53.160224+0000","last_clean":"2024-03-11T22:09:53.160224+0000","last_became_active":"2024-03-11T22:08:39.072956+0000","last_became_peered":"2024-03-11T22:08:39.072956+0000","last_unstale":"2024-03-11T22:09:53.160224+0000","last_undegraded":"2024-03-11T22:09:53.160224+0000","last_fullsized":"2024-03-11T22:09:53.160224+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:09:53.160150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:53.160150+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:21.225798+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.351689+0000","last_change":"2024-03-11T22:08:37.078899+0000","last_active":"2024-03-11T22:09:48.351689+0000","last_peered":"2024-03-11T22:09:48.351689+0000","last_clean":"2024-03-11T22:09:48.351689+0000","last_became_active":"2024-03-11T22:08:37.078588+0000","last_became_peered":"2024-03-11T22:08:37.078588+0000","last_unstale":"2024-03-11T22:09:48.351689+0000","last_undegraded":"2024-03-11T22:09:48.351689+0000","last_fullsized":"2024-03-11T22:09:48.351689+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:55.590469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621574+0000","last_change":"2024-03-11T22:08:33.030653+0000","last_active":"2024-03-11T22:09:36.621574+0000","last_peered":"2024-03-11T22:09:36.621574+0000","last_clean":"2024-03-11T22:09:36.621574+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:09:36.621574+0000","last_undegraded":"2024-03-11T22:09:36.621574+0000","last_fullsized":"2024-03-11T22:09:36.621574+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.528647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049953+0000","last_change":"2024-03-11T22:08:35.086209+0000","last_active":"2024-03-11T22:09:36.049953+0000","last_peered":"2024-03-11T22:09:36.049953+0000","last_clean":"2024-03-11T22:09:36.049953+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:09:36.049953+0000","last_undegraded":"2024-03-11T22:09:36.049953+0000","last_fullsized":"2024-03-11T22:09:36.049953+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:02.938827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622083+0000","last_change":"2024-03-11T22:08:39.075560+0000","last_active":"2024-03-11T22:09:36.622083+0000","last_peered":"2024-03-11T22:09:36.622083+0000","last_clean":"2024-03-11T22:09:36.622083+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:09:36.622083+0000","last_undegraded":"2024-03-11T22:09:36.622083+0000","last_fullsized":"2024-03-11T22:09:36.622083+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:52.374594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.050110+0000","last_change":"2024-03-11T22:08:37.069965+0000","last_active":"2024-03-11T22:09:36.050110+0000","last_peered":"2024-03-11T22:09:36.050110+0000","last_clean":"2024-03-11T22:09:36.050110+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:09:36.050110+0000","last_undegraded":"2024-03-11T22:09:36.050110+0000","last_fullsized":"2024-03-11T22:09:36.050110+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.292950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047578+0000","last_change":"2024-03-11T22:08:33.038259+0000","last_active":"2024-03-11T22:09:36.047578+0000","last_peered":"2024-03-11T22:09:36.047578+0000","last_clean":"2024-03-11T22:09:36.047578+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:09:36.047578+0000","last_undegraded":"2024-03-11T22:09:36.047578+0000","last_fullsized":"2024-03-11T22:09:36.047578+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.852988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:49.567242+0000","last_change":"2024-03-11T22:09:49.567290+0000","last_active":"2024-03-11T22:09:49.567242+0000","last_peered":"2024-03-11T22:09:49.567242+0000","last_clean":"2024-03-11T22:09:49.567242+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:09:49.567242+0000","last_undegraded":"2024-03-11T22:09:49.567242+0000","last_fullsized":"2024-03-11T22:09:49.567242+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:09:49.567170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:49.567170+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:01.624956+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.268733+0000","last_change":"2024-03-11T22:08:39.074473+0000","last_active":"2024-03-11T22:09:36.268733+0000","last_peered":"2024-03-11T22:09:36.268733+0000","last_clean":"2024-03-11T22:09:36.268733+0000","last_became_active":"2024-03-11T22:08:39.074179+0000","last_became_peered":"2024-03-11T22:08:39.074179+0000","last_unstale":"2024-03-11T22:09:36.268733+0000","last_undegraded":"2024-03-11T22:09:36.268733+0000","last_fullsized":"2024-03-11T22:09:36.268733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.168751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:40.669281+0000","last_change":"2024-03-11T22:09:40.669305+0000","last_active":"2024-03-11T22:09:40.669281+0000","last_peered":"2024-03-11T22:09:40.669281+0000","last_clean":"2024-03-11T22:09:40.669281+0000","last_became_active":"2024-03-11T22:08:37.075341+0000","last_became_peered":"2024-03-11T22:08:37.075341+0000","last_unstale":"2024-03-11T22:09:40.669281+0000","last_undegraded":"2024-03-11T22:09:40.669281+0000","last_fullsized":"2024-03-11T22:09:40.669281+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:40.669213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:40.669213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:05.240222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049221+0000","last_change":"2024-03-11T22:08:33.040896+0000","last_active":"2024-03-11T22:09:36.049221+0000","last_peered":"2024-03-11T22:09:36.049221+0000","last_clean":"2024-03-11T22:09:36.049221+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:09:36.049221+0000","last_undegraded":"2024-03-11T22:09:36.049221+0000","last_fullsized":"2024-03-11T22:09:36.049221+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.000587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:42.568177+0000","last_change":"2024-03-11T22:09:42.568204+0000","last_active":"2024-03-11T22:09:42.568177+0000","last_peered":"2024-03-11T22:09:42.568177+0000","last_clean":"2024-03-11T22:09:42.568177+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:09:42.568177+0000","last_undegraded":"2024-03-11T22:09:42.568177+0000","last_fullsized":"2024-03-11T22:09:42.568177+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-11T22:09:42.568114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:42.568114+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.634829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048411+0000","last_change":"2024-03-11T22:08:39.081827+0000","last_active":"2024-03-11T22:09:36.048411+0000","last_peered":"2024-03-11T22:09:36.048411+0000","last_clean":"2024-03-11T22:09:36.048411+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:09:36.048411+0000","last_undegraded":"2024-03-11T22:09:36.048411+0000","last_fullsized":"2024-03-11T22:09:36.048411+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:49.990435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269551+0000","last_change":"2024-03-11T22:08:37.079252+0000","last_active":"2024-03-11T22:09:36.269551+0000","last_peered":"2024-03-11T22:09:36.269551+0000","last_clean":"2024-03-11T22:09:36.269551+0000","last_became_active":"2024-03-11T22:08:37.078863+0000","last_became_peered":"2024-03-11T22:08:37.078863+0000","last_unstale":"2024-03-11T22:09:36.269551+0000","last_undegraded":"2024-03-11T22:09:36.269551+0000","last_fullsized":"2024-03-11T22:09:36.269551+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:43.735583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:49.271515+0000","last_change":"2024-03-11T22:09:49.271541+0000","last_active":"2024-03-11T22:09:49.271515+0000","last_peered":"2024-03-11T22:09:49.271515+0000","last_clean":"2024-03-11T22:09:49.271515+0000","last_became_active":"2024-03-11T22:08:33.039349+0000","last_became_peered":"2024-03-11T22:08:33.039349+0000","last_unstale":"2024-03-11T22:09:49.271515+0000","last_undegraded":"2024-03-11T22:09:49.271515+0000","last_fullsized":"2024-03-11T22:09:49.271515+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:49.271431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:49.271431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:01.697571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047870+0000","last_change":"2024-03-11T22:08:35.086272+0000","last_active":"2024-03-11T22:09:36.047870+0000","last_peered":"2024-03-11T22:09:36.047870+0000","last_clean":"2024-03-11T22:09:36.047870+0000","last_became_active":"2024-03-11T22:08:35.085928+0000","last_became_peered":"2024-03-11T22:08:35.085928+0000","last_unstale":"2024-03-11T22:09:36.047870+0000","last_undegraded":"2024-03-11T22:09:36.047870+0000","last_fullsized":"2024-03-11T22:09:36.047870+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.863565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047674+0000","last_change":"2024-03-11T22:08:39.083049+0000","last_active":"2024-03-11T22:09:36.047674+0000","last_peered":"2024-03-11T22:09:36.047674+0000","last_clean":"2024-03-11T22:09:36.047674+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:09:36.047674+0000","last_undegraded":"2024-03-11T22:09:36.047674+0000","last_fullsized":"2024-03-11T22:09:36.047674+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.375813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:43.569403+0000","last_change":"2024-03-11T22:09:43.569426+0000","last_active":"2024-03-11T22:09:43.569403+0000","last_peered":"2024-03-11T22:09:43.569403+0000","last_clean":"2024-03-11T22:09:43.569403+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:09:43.569403+0000","last_undegraded":"2024-03-11T22:09:43.569403+0000","last_fullsized":"2024-03-11T22:09:43.569403+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:43.569328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:43.569328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:05.155162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:41.032420+0000","last_change":"2024-03-11T22:09:41.032444+0000","last_active":"2024-03-11T22:09:41.032420+0000","last_peered":"2024-03-11T22:09:41.032420+0000","last_clean":"2024-03-11T22:09:41.032420+0000","last_became_active":"2024-03-11T22:08:33.038783+0000","last_became_peered":"2024-03-11T22:08:33.038783+0000","last_unstale":"2024-03-11T22:09:41.032420+0000","last_undegraded":"2024-03-11T22:09:41.032420+0000","last_fullsized":"2024-03-11T22:09:41.032420+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:41.032339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:41.032339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:01.424543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622312+0000","last_change":"2024-03-11T22:08:35.053322+0000","last_active":"2024-03-11T22:09:36.622312+0000","last_peered":"2024-03-11T22:09:36.622312+0000","last_clean":"2024-03-11T22:09:36.622312+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:09:36.622312+0000","last_undegraded":"2024-03-11T22:09:36.622312+0000","last_fullsized":"2024-03-11T22:09:36.622312+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:50.812560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047456+0000","last_change":"2024-03-11T22:08:39.080670+0000","last_active":"2024-03-11T22:09:36.047456+0000","last_peered":"2024-03-11T22:09:36.047456+0000","last_clean":"2024-03-11T22:09:36.047456+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:09:36.047456+0000","last_undegraded":"2024-03-11T22:09:36.047456+0000","last_fullsized":"2024-03-11T22:09:36.047456+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.573443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622083+0000","last_change":"2024-03-11T22:08:37.075322+0000","last_active":"2024-03-11T22:09:36.622083+0000","last_peered":"2024-03-11T22:09:36.622083+0000","last_clean":"2024-03-11T22:09:36.622083+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:09:36.622083+0000","last_undegraded":"2024-03-11T22:09:36.622083+0000","last_fullsized":"2024-03-11T22:09:36.622083+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.827062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.362087+0000","last_change":"2024-03-11T22:09:48.362144+0000","last_active":"2024-03-11T22:09:48.362087+0000","last_peered":"2024-03-11T22:09:48.362087+0000","last_clean":"2024-03-11T22:09:48.362087+0000","last_became_active":"2024-03-11T22:08:33.036083+0000","last_became_peered":"2024-03-11T22:08:33.036083+0000","last_unstale":"2024-03-11T22:09:48.362087+0000","last_undegraded":"2024-03-11T22:09:48.362087+0000","last_fullsized":"2024-03-11T22:09:48.362087+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:09:48.361967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:48.361967+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:05.666698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048296+0000","last_change":"2024-03-11T22:08:35.054987+0000","last_active":"2024-03-11T22:09:36.048296+0000","last_peered":"2024-03-11T22:09:36.048296+0000","last_clean":"2024-03-11T22:09:36.048296+0000","last_became_active":"2024-03-11T22:08:35.054726+0000","last_became_peered":"2024-03-11T22:08:35.054726+0000","last_unstale":"2024-03-11T22:09:36.048296+0000","last_undegraded":"2024-03-11T22:09:36.048296+0000","last_fullsized":"2024-03-11T22:09:36.048296+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.886570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049756+0000","last_change":"2024-03-11T22:08:39.083736+0000","last_active":"2024-03-11T22:09:36.049756+0000","last_peered":"2024-03-11T22:09:36.049756+0000","last_clean":"2024-03-11T22:09:36.049756+0000","last_became_active":"2024-03-11T22:08:39.082938+0000","last_became_peered":"2024-03-11T22:08:39.082938+0000","last_unstale":"2024-03-11T22:09:36.049756+0000","last_undegraded":"2024-03-11T22:09:36.049756+0000","last_fullsized":"2024-03-11T22:09:36.049756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:06.944689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.351666+0000","last_change":"2024-03-11T22:08:37.076601+0000","last_active":"2024-03-11T22:09:48.351666+0000","last_peered":"2024-03-11T22:09:48.351666+0000","last_clean":"2024-03-11T22:09:48.351666+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:09:48.351666+0000","last_undegraded":"2024-03-11T22:09:48.351666+0000","last_fullsized":"2024-03-11T22:09:48.351666+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:45.198461+0000","last_change":"2024-03-11T22:09:45.198482+0000","last_active":"2024-03-11T22:09:45.198461+0000","last_peered":"2024-03-11T22:09:45.198461+0000","last_clean":"2024-03-11T22:09:45.198461+0000","last_became_active":"2024-03-11T22:08:33.300181+0000","last_became_peered":"2024-03-11T22:08:33.300181+0000","last_unstale":"2024-03-11T22:09:45.198461+0000","last_undegraded":"2024-03-11T22:09:45.198461+0000","last_fullsized":"2024-03-11T22:09:45.198461+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:45.198397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:45.198397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:12.972849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049076+0000","last_change":"2024-03-11T22:08:35.086113+0000","last_active":"2024-03-11T22:09:36.049076+0000","last_peered":"2024-03-11T22:09:36.049076+0000","last_clean":"2024-03-11T22:09:36.049076+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:09:36.049076+0000","last_undegraded":"2024-03-11T22:09:36.049076+0000","last_fullsized":"2024-03-11T22:09:36.049076+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:40.519001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049197+0000","last_change":"2024-03-11T22:08:39.072508+0000","last_active":"2024-03-11T22:09:36.049197+0000","last_peered":"2024-03-11T22:09:36.049197+0000","last_clean":"2024-03-11T22:09:36.049197+0000","last_became_active":"2024-03-11T22:08:39.072089+0000","last_became_peered":"2024-03-11T22:08:39.072089+0000","last_unstale":"2024-03-11T22:09:36.049197+0000","last_undegraded":"2024-03-11T22:09:36.049197+0000","last_fullsized":"2024-03-11T22:09:36.049197+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:06.283027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.351531+0000","last_change":"2024-03-11T22:08:37.071570+0000","last_active":"2024-03-11T22:09:48.351531+0000","last_peered":"2024-03-11T22:09:48.351531+0000","last_clean":"2024-03-11T22:09:48.351531+0000","last_became_active":"2024-03-11T22:08:37.071202+0000","last_became_peered":"2024-03-11T22:08:37.071202+0000","last_unstale":"2024-03-11T22:09:48.351531+0000","last_undegraded":"2024-03-11T22:09:48.351531+0000","last_fullsized":"2024-03-11T22:09:48.351531+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048004+0000","last_change":"2024-03-11T22:08:33.033199+0000","last_active":"2024-03-11T22:09:36.048004+0000","last_peered":"2024-03-11T22:09:36.048004+0000","last_clean":"2024-03-11T22:09:36.048004+0000","last_became_active":"2024-03-11T22:08:33.032850+0000","last_became_peered":"2024-03-11T22:08:33.032850+0000","last_unstale":"2024-03-11T22:09:36.048004+0000","last_undegraded":"2024-03-11T22:09:36.048004+0000","last_fullsized":"2024-03-11T22:09:36.048004+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:51.370651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:52.219783+0000","last_change":"2024-03-11T22:09:52.219839+0000","last_active":"2024-03-11T22:09:52.219783+0000","last_peered":"2024-03-11T22:09:52.219783+0000","last_clean":"2024-03-11T22:09:52.219783+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:09:52.219783+0000","last_undegraded":"2024-03-11T22:09:52.219783+0000","last_fullsized":"2024-03-11T22:09:52.219783+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:09:52.219687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:52.219687+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:06.397728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:44.743673+0000","last_change":"2024-03-11T22:09:44.743698+0000","last_active":"2024-03-11T22:09:44.743673+0000","last_peered":"2024-03-11T22:09:44.743673+0000","last_clean":"2024-03-11T22:09:44.743673+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:09:44.743673+0000","last_undegraded":"2024-03-11T22:09:44.743673+0000","last_fullsized":"2024-03-11T22:09:44.743673+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:44.743600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:44.743600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:03.506943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:45.582545+0000","last_change":"2024-03-11T22:09:45.582583+0000","last_active":"2024-03-11T22:09:45.582545+0000","last_peered":"2024-03-11T22:09:45.582545+0000","last_clean":"2024-03-11T22:09:45.582545+0000","last_became_active":"2024-03-11T22:08:37.069611+0000","last_became_peered":"2024-03-11T22:08:37.069611+0000","last_unstale":"2024-03-11T22:09:45.582545+0000","last_undegraded":"2024-03-11T22:09:45.582545+0000","last_fullsized":"2024-03-11T22:09:45.582545+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:45.582469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:45.582469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.713402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.268777+0000","last_change":"2024-03-11T22:08:33.038436+0000","last_active":"2024-03-11T22:09:36.268777+0000","last_peered":"2024-03-11T22:09:36.268777+0000","last_clean":"2024-03-11T22:09:36.268777+0000","last_became_active":"2024-03-11T22:08:33.038132+0000","last_became_peered":"2024-03-11T22:08:33.038132+0000","last_unstale":"2024-03-11T22:09:36.268777+0000","last_undegraded":"2024-03-11T22:09:36.268777+0000","last_fullsized":"2024-03-11T22:09:36.268777+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:53.291744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.050020+0000","last_change":"2024-03-11T22:08:35.086149+0000","last_active":"2024-03-11T22:09:36.050020+0000","last_peered":"2024-03-11T22:09:36.050020+0000","last_clean":"2024-03-11T22:09:36.050020+0000","last_became_active":"2024-03-11T22:08:35.085787+0000","last_became_peered":"2024-03-11T22:08:35.085787+0000","last_unstale":"2024-03-11T22:09:36.050020+0000","last_undegraded":"2024-03-11T22:09:36.050020+0000","last_fullsized":"2024-03-11T22:09:36.050020+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:53.984069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269004+0000","last_change":"2024-03-11T22:08:39.075607+0000","last_active":"2024-03-11T22:09:36.269004+0000","last_peered":"2024-03-11T22:09:36.269004+0000","last_clean":"2024-03-11T22:09:36.269004+0000","last_became_active":"2024-03-11T22:08:39.075305+0000","last_became_peered":"2024-03-11T22:08:39.075305+0000","last_unstale":"2024-03-11T22:09:36.269004+0000","last_undegraded":"2024-03-11T22:09:36.269004+0000","last_fullsized":"2024-03-11T22:09:36.269004+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.166687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267282+0000","last_change":"2024-03-11T22:08:37.077768+0000","last_active":"2024-03-11T22:09:36.267282+0000","last_peered":"2024-03-11T22:09:36.267282+0000","last_clean":"2024-03-11T22:09:36.267282+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:09:36.267282+0000","last_undegraded":"2024-03-11T22:09:36.267282+0000","last_fullsized":"2024-03-11T22:09:36.267282+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:48.146238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.120856+0000","last_change":"2024-03-11T22:08:33.035663+0000","last_active":"2024-03-11T22:09:36.120856+0000","last_peered":"2024-03-11T22:09:36.120856+0000","last_clean":"2024-03-11T22:09:36.120856+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:09:36.120856+0000","last_undegraded":"2024-03-11T22:09:36.120856+0000","last_fullsized":"2024-03-11T22:09:36.120856+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.335803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049802+0000","last_change":"2024-03-11T22:08:35.086263+0000","last_active":"2024-03-11T22:09:36.049802+0000","last_peered":"2024-03-11T22:09:36.049802+0000","last_clean":"2024-03-11T22:09:36.049802+0000","last_became_active":"2024-03-11T22:08:35.085914+0000","last_became_peered":"2024-03-11T22:08:35.085914+0000","last_unstale":"2024-03-11T22:09:36.049802+0000","last_undegraded":"2024-03-11T22:09:36.049802+0000","last_fullsized":"2024-03-11T22:09:36.049802+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:53.031835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267056+0000","last_change":"2024-03-11T22:08:39.079848+0000","last_active":"2024-03-11T22:09:36.267056+0000","last_peered":"2024-03-11T22:09:36.267056+0000","last_clean":"2024-03-11T22:09:36.267056+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:09:36.267056+0000","last_undegraded":"2024-03-11T22:09:36.267056+0000","last_fullsized":"2024-03-11T22:09:36.267056+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.299384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269220+0000","last_change":"2024-03-11T22:08:33.039060+0000","last_active":"2024-03-11T22:09:36.269220+0000","last_peered":"2024-03-11T22:09:36.269220+0000","last_clean":"2024-03-11T22:09:36.269220+0000","last_became_active":"2024-03-11T22:08:33.037662+0000","last_became_peered":"2024-03-11T22:08:33.037662+0000","last_unstale":"2024-03-11T22:09:36.269220+0000","last_undegraded":"2024-03-11T22:09:36.269220+0000","last_fullsized":"2024-03-11T22:09:36.269220+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:49.450289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"91'22","reported_seq":93,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.274283+0000","last_change":"2024-03-11T22:09:36.274318+0000","last_active":"2024-03-11T22:09:36.274283+0000","last_peered":"2024-03-11T22:09:36.274283+0000","last_clean":"2024-03-11T22:09:36.274283+0000","last_became_active":"2024-03-11T22:08:35.085916+0000","last_became_peered":"2024-03-11T22:08:35.085916+0000","last_unstale":"2024-03-11T22:09:36.274283+0000","last_undegraded":"2024-03-11T22:09:36.274283+0000","last_fullsized":"2024-03-11T22:09:36.274283+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'22","last_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:09:36.274225+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:38.387952+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.351666+0000","last_change":"2024-03-11T22:08:37.078410+0000","last_active":"2024-03-11T22:09:48.351666+0000","last_peered":"2024-03-11T22:09:48.351666+0000","last_clean":"2024-03-11T22:09:48.351666+0000","last_became_active":"2024-03-11T22:08:37.078114+0000","last_became_peered":"2024-03-11T22:08:37.078114+0000","last_unstale":"2024-03-11T22:09:48.351666+0000","last_undegraded":"2024-03-11T22:09:48.351666+0000","last_fullsized":"2024-03-11T22:09:48.351666+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"75'1","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048453+0000","last_change":"2024-03-11T22:08:39.083458+0000","last_active":"2024-03-11T22:09:36.048453+0000","last_peered":"2024-03-11T22:09:36.048453+0000","last_clean":"2024-03-11T22:09:36.048453+0000","last_became_active":"2024-03-11T22:08:39.083141+0000","last_became_peered":"2024-03-11T22:08:39.083141+0000","last_unstale":"2024-03-11T22:09:36.048453+0000","last_undegraded":"2024-03-11T22:09:36.048453+0000","last_fullsized":"2024-03-11T22:09:36.048453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:54.422237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269155+0000","last_change":"2024-03-11T22:08:33.041804+0000","last_active":"2024-03-11T22:09:36.269155+0000","last_peered":"2024-03-11T22:09:36.269155+0000","last_clean":"2024-03-11T22:09:36.269155+0000","last_became_active":"2024-03-11T22:08:33.041492+0000","last_became_peered":"2024-03-11T22:08:33.041492+0000","last_unstale":"2024-03-11T22:09:36.269155+0000","last_undegraded":"2024-03-11T22:09:36.269155+0000","last_fullsized":"2024-03-11T22:09:36.269155+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:43.260561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047993+0000","last_change":"2024-03-11T22:08:54.614766+0000","last_active":"2024-03-11T22:09:36.047993+0000","last_peered":"2024-03-11T22:09:36.047993+0000","last_clean":"2024-03-11T22:09:36.047993+0000","last_became_active":"2024-03-11T22:08:54.607153+0000","last_became_peered":"2024-03-11T22:08:54.607153+0000","last_unstale":"2024-03-11T22:09:36.047993+0000","last_undegraded":"2024-03-11T22:09:36.047993+0000","last_fullsized":"2024-03-11T22:09:36.047993+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:58.241457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.721722+0000","last_change":"2024-03-11T22:09:48.721747+0000","last_active":"2024-03-11T22:09:48.721722+0000","last_peered":"2024-03-11T22:09:48.721722+0000","last_clean":"2024-03-11T22:09:48.721722+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:09:48.721722+0000","last_undegraded":"2024-03-11T22:09:48.721722+0000","last_fullsized":"2024-03-11T22:09:48.721722+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:48.721656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:48.721656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.253690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.268601+0000","last_change":"2024-03-11T22:08:39.079520+0000","last_active":"2024-03-11T22:09:36.268601+0000","last_peered":"2024-03-11T22:09:36.268601+0000","last_clean":"2024-03-11T22:09:36.268601+0000","last_became_active":"2024-03-11T22:08:39.079238+0000","last_became_peered":"2024-03-11T22:08:39.079238+0000","last_unstale":"2024-03-11T22:09:36.268601+0000","last_undegraded":"2024-03-11T22:09:36.268601+0000","last_fullsized":"2024-03-11T22:09:36.268601+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:49.698176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622427+0000","last_change":"2024-03-11T22:08:37.075286+0000","last_active":"2024-03-11T22:09:36.622427+0000","last_peered":"2024-03-11T22:09:36.622427+0000","last_clean":"2024-03-11T22:09:36.622427+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:09:36.622427+0000","last_undegraded":"2024-03-11T22:09:36.622427+0000","last_fullsized":"2024-03-11T22:09:36.622427+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:54.723130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:42.079785+0000","last_change":"2024-03-11T22:09:42.079812+0000","last_active":"2024-03-11T22:09:42.079785+0000","last_peered":"2024-03-11T22:09:42.079785+0000","last_clean":"2024-03-11T22:09:42.079785+0000","last_became_active":"2024-03-11T22:08:33.035188+0000","last_became_peered":"2024-03-11T22:08:33.035188+0000","last_unstale":"2024-03-11T22:09:42.079785+0000","last_undegraded":"2024-03-11T22:09:42.079785+0000","last_fullsized":"2024-03-11T22:09:42.079785+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:42.079695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:42.079695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:45.595806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":92,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:54.637564+0000","last_change":"2024-03-11T22:09:54.637633+0000","last_active":"2024-03-11T22:09:54.637564+0000","last_peered":"2024-03-11T22:09:54.637564+0000","last_clean":"2024-03-11T22:09:54.637564+0000","last_became_active":"2024-03-11T22:08:35.087046+0000","last_became_peered":"2024-03-11T22:08:35.087046+0000","last_unstale":"2024-03-11T22:09:54.637564+0000","last_undegraded":"2024-03-11T22:09:54.637564+0000","last_fullsized":"2024-03-11T22:09:54.637564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:09:54.637462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:54.637462+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:00.008092+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047796+0000","last_change":"2024-03-11T22:08:39.085891+0000","last_active":"2024-03-11T22:09:36.047796+0000","last_peered":"2024-03-11T22:09:36.047796+0000","last_clean":"2024-03-11T22:09:36.047796+0000","last_became_active":"2024-03-11T22:08:39.085496+0000","last_became_peered":"2024-03-11T22:08:39.085496+0000","last_unstale":"2024-03-11T22:09:36.047796+0000","last_undegraded":"2024-03-11T22:09:36.047796+0000","last_fullsized":"2024-03-11T22:09:36.047796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:47.594130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622153+0000","last_change":"2024-03-11T22:08:37.073732+0000","last_active":"2024-03-11T22:09:36.622153+0000","last_peered":"2024-03-11T22:09:36.622153+0000","last_clean":"2024-03-11T22:09:36.622153+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:09:36.622153+0000","last_undegraded":"2024-03-11T22:09:36.622153+0000","last_fullsized":"2024-03-11T22:09:36.622153+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:54.219715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267080+0000","last_change":"2024-03-11T22:08:33.038820+0000","last_active":"2024-03-11T22:09:36.267080+0000","last_peered":"2024-03-11T22:09:36.267080+0000","last_clean":"2024-03-11T22:09:36.267080+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:09:36.267080+0000","last_undegraded":"2024-03-11T22:09:36.267080+0000","last_fullsized":"2024-03-11T22:09:36.267080+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:54.045701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049729+0000","last_change":"2024-03-11T22:08:54.617268+0000","last_active":"2024-03-11T22:09:36.049729+0000","last_peered":"2024-03-11T22:09:36.049729+0000","last_clean":"2024-03-11T22:09:36.049729+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:09:36.049729+0000","last_undegraded":"2024-03-11T22:09:36.049729+0000","last_fullsized":"2024-03-11T22:09:36.049729+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.997317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048925+0000","last_change":"2024-03-11T22:08:39.084054+0000","last_active":"2024-03-11T22:09:36.048925+0000","last_peered":"2024-03-11T22:09:36.048925+0000","last_clean":"2024-03-11T22:09:36.048925+0000","last_became_active":"2024-03-11T22:08:39.083738+0000","last_became_peered":"2024-03-11T22:08:39.083738+0000","last_unstale":"2024-03-11T22:09:36.048925+0000","last_undegraded":"2024-03-11T22:09:36.048925+0000","last_fullsized":"2024-03-11T22:09:36.048925+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.602554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.266879+0000","last_change":"2024-03-11T22:08:36.055999+0000","last_active":"2024-03-11T22:09:36.266879+0000","last_peered":"2024-03-11T22:09:36.266879+0000","last_clean":"2024-03-11T22:09:36.266879+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:09:36.266879+0000","last_undegraded":"2024-03-11T22:09:36.266879+0000","last_fullsized":"2024-03-11T22:09:36.266879+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_clean_scrub_stamp":"2024-03-11T22:08:31.015772+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:44.756728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047958+0000","last_change":"2024-03-11T22:08:33.035748+0000","last_active":"2024-03-11T22:09:36.047958+0000","last_peered":"2024-03-11T22:09:36.047958+0000","last_clean":"2024-03-11T22:09:36.047958+0000","last_became_active":"2024-03-11T22:08:33.035469+0000","last_became_peered":"2024-03-11T22:08:33.035469+0000","last_unstale":"2024-03-11T22:09:36.047958+0000","last_undegraded":"2024-03-11T22:09:36.047958+0000","last_fullsized":"2024-03-11T22:09:36.047958+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:43.474016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":126,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.351325+0000","last_change":"2024-03-11T22:09:47.779316+0000","last_active":"2024-03-11T22:09:48.351325+0000","last_peered":"2024-03-11T22:09:48.351325+0000","last_clean":"2024-03-11T22:09:48.351325+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:09:48.351325+0000","last_undegraded":"2024-03-11T22:09:48.351325+0000","last_fullsized":"2024-03-11T22:09:48.351325+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-11T22:09:47.779209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:47.779209+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:10.865609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:44.165039+0000","last_change":"2024-03-11T22:09:44.165077+0000","last_active":"2024-03-11T22:09:44.165039+0000","last_peered":"2024-03-11T22:09:44.165039+0000","last_clean":"2024-03-11T22:09:44.165039+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:09:44.165039+0000","last_undegraded":"2024-03-11T22:09:44.165039+0000","last_fullsized":"2024-03-11T22:09:44.165039+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:44.164939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:44.164939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:55.636705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269397+0000","last_change":"2024-03-11T22:08:39.077855+0000","last_active":"2024-03-11T22:09:36.269397+0000","last_peered":"2024-03-11T22:09:36.269397+0000","last_clean":"2024-03-11T22:09:36.269397+0000","last_became_active":"2024-03-11T22:08:39.077566+0000","last_became_peered":"2024-03-11T22:08:39.077566+0000","last_unstale":"2024-03-11T22:09:36.269397+0000","last_undegraded":"2024-03-11T22:09:36.269397+0000","last_fullsized":"2024-03-11T22:09:36.269397+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:49.972071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621579+0000","last_change":"2024-03-11T22:09:35.724312+0000","last_active":"2024-03-11T22:09:36.621579+0000","last_peered":"2024-03-11T22:09:36.621579+0000","last_clean":"2024-03-11T22:09:36.621579+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:09:36.621579+0000","last_undegraded":"2024-03-11T22:09:36.621579+0000","last_fullsized":"2024-03-11T22:09:36.621579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:35.724224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:35.724224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:59.123602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621974+0000","last_change":"2024-03-11T22:08:36.055357+0000","last_active":"2024-03-11T22:09:36.621974+0000","last_peered":"2024-03-11T22:09:36.621974+0000","last_clean":"2024-03-11T22:09:36.621974+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:09:36.621974+0000","last_undegraded":"2024-03-11T22:09:36.621974+0000","last_fullsized":"2024-03-11T22:09:36.621974+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_clean_scrub_stamp":"2024-03-11T22:08:31.015772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:52.926784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622491+0000","last_change":"2024-03-11T22:08:35.058141+0000","last_active":"2024-03-11T22:09:36.622491+0000","last_peered":"2024-03-11T22:09:36.622491+0000","last_clean":"2024-03-11T22:09:36.622491+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:09:36.622491+0000","last_undegraded":"2024-03-11T22:09:36.622491+0000","last_fullsized":"2024-03-11T22:09:36.622491+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:51.069549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622607+0000","last_change":"2024-03-11T22:08:37.076581+0000","last_active":"2024-03-11T22:09:36.622607+0000","last_peered":"2024-03-11T22:09:36.622607+0000","last_clean":"2024-03-11T22:09:36.622607+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:09:36.622607+0000","last_undegraded":"2024-03-11T22:09:36.622607+0000","last_fullsized":"2024-03-11T22:09:36.622607+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:05.137609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267346+0000","last_change":"2024-03-11T22:08:39.073180+0000","last_active":"2024-03-11T22:09:36.267346+0000","last_peered":"2024-03-11T22:09:36.267346+0000","last_clean":"2024-03-11T22:09:36.267346+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:09:36.267346+0000","last_undegraded":"2024-03-11T22:09:36.267346+0000","last_fullsized":"2024-03-11T22:09:36.267346+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:54.803103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":310,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.268866+0000","last_change":"2024-03-11T22:08:42.030878+0000","last_active":"2024-03-11T22:09:36.268866+0000","last_peered":"2024-03-11T22:09:36.268866+0000","last_clean":"2024-03-11T22:09:36.268866+0000","last_became_active":"2024-03-11T22:08:22.015394+0000","last_became_peered":"2024-03-11T22:08:22.015394+0000","last_unstale":"2024-03-11T22:09:36.268866+0000","last_undegraded":"2024-03-11T22:09:36.268866+0000","last_fullsized":"2024-03-11T22:09:36.268866+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-03-11T22:08:42.030749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:06:13.216089+0000","last_clean_scrub_stamp":"2024-03-11T22:08:42.030749+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:44.064265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:41.615904+0000","last_change":"2024-03-11T22:09:41.615925+0000","last_active":"2024-03-11T22:09:41.615904+0000","last_peered":"2024-03-11T22:09:41.615904+0000","last_clean":"2024-03-11T22:09:41.615904+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:09:41.615904+0000","last_undegraded":"2024-03-11T22:09:41.615904+0000","last_fullsized":"2024-03-11T22:09:41.615904+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:41.615829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:41.615829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:04.514072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":92,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.351753+0000","last_change":"2024-03-11T22:08:35.086266+0000","last_active":"2024-03-11T22:09:48.351753+0000","last_peered":"2024-03-11T22:09:48.351753+0000","last_clean":"2024-03-11T22:09:48.351753+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:09:48.351753+0000","last_undegraded":"2024-03-11T22:09:48.351753+0000","last_fullsized":"2024-03-11T22:09:48.351753+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.950025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269112+0000","last_change":"2024-03-11T22:08:37.076167+0000","last_active":"2024-03-11T22:09:36.269112+0000","last_peered":"2024-03-11T22:09:36.269112+0000","last_clean":"2024-03-11T22:09:36.269112+0000","last_became_active":"2024-03-11T22:08:37.075876+0000","last_became_peered":"2024-03-11T22:08:37.075876+0000","last_unstale":"2024-03-11T22:09:36.269112+0000","last_undegraded":"2024-03-11T22:09:36.269112+0000","last_fullsized":"2024-03-11T22:09:36.269112+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:52.209442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:53.690806+0000","last_change":"2024-03-11T22:09:53.690828+0000","last_active":"2024-03-11T22:09:53.690806+0000","last_peered":"2024-03-11T22:09:53.690806+0000","last_clean":"2024-03-11T22:09:53.690806+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:09:53.690806+0000","last_undegraded":"2024-03-11T22:09:53.690806+0000","last_fullsized":"2024-03-11T22:09:53.690806+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:53.690738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:53.690738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:11.065631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048026+0000","last_change":"2024-03-11T22:08:33.035933+0000","last_active":"2024-03-11T22:09:36.048026+0000","last_peered":"2024-03-11T22:09:36.048026+0000","last_clean":"2024-03-11T22:09:36.048026+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:09:36.048026+0000","last_undegraded":"2024-03-11T22:09:36.048026+0000","last_fullsized":"2024-03-11T22:09:36.048026+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.049951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"76'5","reported_seq":144,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:53.683385+0000","last_change":"2024-03-11T22:08:36.060461+0000","last_active":"2024-03-11T22:09:53.683385+0000","last_peered":"2024-03-11T22:09:53.683385+0000","last_clean":"2024-03-11T22:09:53.683385+0000","last_became_active":"2024-03-11T22:08:32.033652+0000","last_became_peered":"2024-03-11T22:08:32.033652+0000","last_unstale":"2024-03-11T22:09:53.683385+0000","last_undegraded":"2024-03-11T22:09:53.683385+0000","last_fullsized":"2024-03-11T22:09:53.683385+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_clean_scrub_stamp":"2024-03-11T22:08:31.015772+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:58.101329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:51.213608+0000","last_change":"2024-03-11T22:09:51.213643+0000","last_active":"2024-03-11T22:09:51.213608+0000","last_peered":"2024-03-11T22:09:51.213608+0000","last_clean":"2024-03-11T22:09:51.213608+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:09:51.213608+0000","last_undegraded":"2024-03-11T22:09:51.213608+0000","last_fullsized":"2024-03-11T22:09:51.213608+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:09:51.213532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:51.213532+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:51.292156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267521+0000","last_change":"2024-03-11T22:08:37.082329+0000","last_active":"2024-03-11T22:09:36.267521+0000","last_peered":"2024-03-11T22:09:36.267521+0000","last_clean":"2024-03-11T22:09:36.267521+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:09:36.267521+0000","last_undegraded":"2024-03-11T22:09:36.267521+0000","last_fullsized":"2024-03-11T22:09:36.267521+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:04.541129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:54.681419+0000","last_change":"2024-03-11T22:09:54.681445+0000","last_active":"2024-03-11T22:09:54.681419+0000","last_peered":"2024-03-11T22:09:54.681419+0000","last_clean":"2024-03-11T22:09:54.681419+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:09:54.681419+0000","last_undegraded":"2024-03-11T22:09:54.681419+0000","last_fullsized":"2024-03-11T22:09:54.681419+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:54.681338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:54.681338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:22.214871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.621916+0000","last_change":"2024-03-11T22:08:33.298533+0000","last_active":"2024-03-11T22:09:36.621916+0000","last_peered":"2024-03-11T22:09:36.621916+0000","last_clean":"2024-03-11T22:09:36.621916+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:09:36.621916+0000","last_undegraded":"2024-03-11T22:09:36.621916+0000","last_fullsized":"2024-03-11T22:09:36.621916+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.310011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":89,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049619+0000","last_change":"2024-03-11T22:08:35.087309+0000","last_active":"2024-03-11T22:09:36.049619+0000","last_peered":"2024-03-11T22:09:36.049619+0000","last_clean":"2024-03-11T22:09:36.049619+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:09:36.049619+0000","last_undegraded":"2024-03-11T22:09:36.049619+0000","last_fullsized":"2024-03-11T22:09:36.049619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:02.382775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.121074+0000","last_change":"2024-03-11T22:08:37.074393+0000","last_active":"2024-03-11T22:09:36.121074+0000","last_peered":"2024-03-11T22:09:36.121074+0000","last_clean":"2024-03-11T22:09:36.121074+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:09:36.121074+0000","last_undegraded":"2024-03-11T22:09:36.121074+0000","last_fullsized":"2024-03-11T22:09:36.121074+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:02.379574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048168+0000","last_change":"2024-03-11T22:08:39.085179+0000","last_active":"2024-03-11T22:09:36.048168+0000","last_peered":"2024-03-11T22:09:36.048168+0000","last_clean":"2024-03-11T22:09:36.048168+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:09:36.048168+0000","last_undegraded":"2024-03-11T22:09:36.048168+0000","last_fullsized":"2024-03-11T22:09:36.048168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:05.345954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:43.271383+0000","last_change":"2024-03-11T22:09:43.271410+0000","last_active":"2024-03-11T22:09:43.271383+0000","last_peered":"2024-03-11T22:09:43.271383+0000","last_clean":"2024-03-11T22:09:43.271383+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:09:43.271383+0000","last_undegraded":"2024-03-11T22:09:43.271383+0000","last_fullsized":"2024-03-11T22:09:43.271383+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:43.271304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:43.271304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:50.753986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622130+0000","last_change":"2024-03-11T22:08:35.058060+0000","last_active":"2024-03-11T22:09:36.622130+0000","last_peered":"2024-03-11T22:09:36.622130+0000","last_clean":"2024-03-11T22:09:36.622130+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:09:36.622130+0000","last_undegraded":"2024-03-11T22:09:36.622130+0000","last_fullsized":"2024-03-11T22:09:36.622130+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.703549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.351323+0000","last_change":"2024-03-11T22:09:48.187735+0000","last_active":"2024-03-11T22:09:48.351323+0000","last_peered":"2024-03-11T22:09:48.351323+0000","last_clean":"2024-03-11T22:09:48.351323+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:09:48.351323+0000","last_undegraded":"2024-03-11T22:09:48.351323+0000","last_fullsized":"2024-03-11T22:09:48.351323+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:09:48.187641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:48.187641+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:56.947907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.269149+0000","last_change":"2024-03-11T22:08:39.076424+0000","last_active":"2024-03-11T22:09:36.269149+0000","last_peered":"2024-03-11T22:09:36.269149+0000","last_clean":"2024-03-11T22:09:36.269149+0000","last_became_active":"2024-03-11T22:08:39.076114+0000","last_became_peered":"2024-03-11T22:08:39.076114+0000","last_unstale":"2024-03-11T22:09:36.269149+0000","last_undegraded":"2024-03-11T22:09:36.269149+0000","last_fullsized":"2024-03-11T22:09:36.269149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:48.446160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049921+0000","last_change":"2024-03-11T22:08:33.039271+0000","last_active":"2024-03-11T22:09:36.049921+0000","last_peered":"2024-03-11T22:09:36.049921+0000","last_clean":"2024-03-11T22:09:36.049921+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:09:36.049921+0000","last_undegraded":"2024-03-11T22:09:36.049921+0000","last_fullsized":"2024-03-11T22:09:36.049921+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:52.098180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:53.414201+0000","last_change":"2024-03-11T22:09:53.414248+0000","last_active":"2024-03-11T22:09:53.414201+0000","last_peered":"2024-03-11T22:09:53.414201+0000","last_clean":"2024-03-11T22:09:53.414201+0000","last_became_active":"2024-03-11T22:08:35.053452+0000","last_became_peered":"2024-03-11T22:08:35.053452+0000","last_unstale":"2024-03-11T22:09:53.414201+0000","last_undegraded":"2024-03-11T22:09:53.414201+0000","last_fullsized":"2024-03-11T22:09:53.414201+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:09:53.414127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:53.414127+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:17.661472+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.351353+0000","last_change":"2024-03-11T22:08:37.078918+0000","last_active":"2024-03-11T22:09:48.351353+0000","last_peered":"2024-03-11T22:09:48.351353+0000","last_clean":"2024-03-11T22:09:48.351353+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:09:48.351353+0000","last_undegraded":"2024-03-11T22:09:48.351353+0000","last_fullsized":"2024-03-11T22:09:48.351353+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048188+0000","last_change":"2024-03-11T22:08:39.078715+0000","last_active":"2024-03-11T22:09:36.048188+0000","last_peered":"2024-03-11T22:09:36.048188+0000","last_clean":"2024-03-11T22:09:36.048188+0000","last_became_active":"2024-03-11T22:08:39.078244+0000","last_became_peered":"2024-03-11T22:08:39.078244+0000","last_unstale":"2024-03-11T22:09:36.048188+0000","last_undegraded":"2024-03-11T22:09:36.048188+0000","last_fullsized":"2024-03-11T22:09:36.048188+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:07.090388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.564984+0000","last_change":"2024-03-11T22:09:36.565005+0000","last_active":"2024-03-11T22:09:36.564984+0000","last_peered":"2024-03-11T22:09:36.564984+0000","last_clean":"2024-03-11T22:09:36.564984+0000","last_became_active":"2024-03-11T22:08:33.036469+0000","last_became_peered":"2024-03-11T22:08:33.036469+0000","last_unstale":"2024-03-11T22:09:36.564984+0000","last_undegraded":"2024-03-11T22:09:36.564984+0000","last_fullsized":"2024-03-11T22:09:36.564984+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:36.564912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:36.564912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:49.963865+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047627+0000","last_change":"2024-03-11T22:08:35.086576+0000","last_active":"2024-03-11T22:09:36.047627+0000","last_peered":"2024-03-11T22:09:36.047627+0000","last_clean":"2024-03-11T22:09:36.047627+0000","last_became_active":"2024-03-11T22:08:35.085979+0000","last_became_peered":"2024-03-11T22:08:35.085979+0000","last_unstale":"2024-03-11T22:09:36.047627+0000","last_undegraded":"2024-03-11T22:09:36.047627+0000","last_fullsized":"2024-03-11T22:09:36.047627+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.168181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:54.458828+0000","last_change":"2024-03-11T22:09:54.458851+0000","last_active":"2024-03-11T22:09:54.458828+0000","last_peered":"2024-03-11T22:09:54.458828+0000","last_clean":"2024-03-11T22:09:54.458828+0000","last_became_active":"2024-03-11T22:08:37.076522+0000","last_became_peered":"2024-03-11T22:08:37.076522+0000","last_unstale":"2024-03-11T22:09:54.458828+0000","last_undegraded":"2024-03-11T22:09:54.458828+0000","last_fullsized":"2024-03-11T22:09:54.458828+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:54.458752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:54.458752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.358836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267135+0000","last_change":"2024-03-11T22:08:39.080281+0000","last_active":"2024-03-11T22:09:36.267135+0000","last_peered":"2024-03-11T22:09:36.267135+0000","last_clean":"2024-03-11T22:09:36.267135+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:09:36.267135+0000","last_undegraded":"2024-03-11T22:09:36.267135+0000","last_fullsized":"2024-03-11T22:09:36.267135+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.517616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.351684+0000","last_change":"2024-03-11T22:08:37.077452+0000","last_active":"2024-03-11T22:09:48.351684+0000","last_peered":"2024-03-11T22:09:48.351684+0000","last_clean":"2024-03-11T22:09:48.351684+0000","last_became_active":"2024-03-11T22:08:37.077142+0000","last_became_peered":"2024-03-11T22:08:37.077142+0000","last_unstale":"2024-03-11T22:09:48.351684+0000","last_undegraded":"2024-03-11T22:09:48.351684+0000","last_fullsized":"2024-03-11T22:09:48.351684+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047538+0000","last_change":"2024-03-11T22:08:33.037839+0000","last_active":"2024-03-11T22:09:36.047538+0000","last_peered":"2024-03-11T22:09:36.047538+0000","last_clean":"2024-03-11T22:09:36.047538+0000","last_became_active":"2024-03-11T22:08:33.037537+0000","last_became_peered":"2024-03-11T22:08:33.037537+0000","last_unstale":"2024-03-11T22:09:36.047538+0000","last_undegraded":"2024-03-11T22:09:36.047538+0000","last_fullsized":"2024-03-11T22:09:36.047538+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:51.539803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622582+0000","last_change":"2024-03-11T22:08:35.060718+0000","last_active":"2024-03-11T22:09:36.622582+0000","last_peered":"2024-03-11T22:09:36.622582+0000","last_clean":"2024-03-11T22:09:36.622582+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:09:36.622582+0000","last_undegraded":"2024-03-11T22:09:36.622582+0000","last_fullsized":"2024-03-11T22:09:36.622582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:53.393792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048865+0000","last_change":"2024-03-11T22:08:39.077317+0000","last_active":"2024-03-11T22:09:36.048865+0000","last_peered":"2024-03-11T22:09:36.048865+0000","last_clean":"2024-03-11T22:09:36.048865+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:09:36.048865+0000","last_undegraded":"2024-03-11T22:09:36.048865+0000","last_fullsized":"2024-03-11T22:09:36.048865+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.786808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.267508+0000","last_change":"2024-03-11T22:08:37.082228+0000","last_active":"2024-03-11T22:09:36.267508+0000","last_peered":"2024-03-11T22:09:36.267508+0000","last_clean":"2024-03-11T22:09:36.267508+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:09:36.267508+0000","last_undegraded":"2024-03-11T22:09:36.267508+0000","last_fullsized":"2024-03-11T22:09:36.267508+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.417015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.047823+0000","last_change":"2024-03-11T22:08:33.300125+0000","last_active":"2024-03-11T22:09:36.047823+0000","last_peered":"2024-03-11T22:09:36.047823+0000","last_clean":"2024-03-11T22:09:36.047823+0000","last_became_active":"2024-03-11T22:08:33.299778+0000","last_became_peered":"2024-03-11T22:08:33.299778+0000","last_unstale":"2024-03-11T22:09:36.047823+0000","last_undegraded":"2024-03-11T22:09:36.047823+0000","last_fullsized":"2024-03-11T22:09:36.047823+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:58.833176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":80,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:40.448823+0000","last_change":"2024-03-11T22:09:40.448862+0000","last_active":"2024-03-11T22:09:40.448823+0000","last_peered":"2024-03-11T22:09:40.448823+0000","last_clean":"2024-03-11T22:09:40.448823+0000","last_became_active":"2024-03-11T22:08:35.053123+0000","last_became_peered":"2024-03-11T22:08:35.053123+0000","last_unstale":"2024-03-11T22:09:40.448823+0000","last_undegraded":"2024-03-11T22:09:40.448823+0000","last_fullsized":"2024-03-11T22:09:40.448823+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:09:40.448751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:40.448751+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:07.077279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622708+0000","last_change":"2024-03-11T22:08:39.074701+0000","last_active":"2024-03-11T22:09:36.622708+0000","last_peered":"2024-03-11T22:09:36.622708+0000","last_clean":"2024-03-11T22:09:36.622708+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:09:36.622708+0000","last_undegraded":"2024-03-11T22:09:36.622708+0000","last_fullsized":"2024-03-11T22:09:36.622708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.861450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":49,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.622128+0000","last_change":"2024-03-11T22:08:37.078886+0000","last_active":"2024-03-11T22:09:36.622128+0000","last_peered":"2024-03-11T22:09:36.622128+0000","last_clean":"2024-03-11T22:09:36.622128+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:09:36.622128+0000","last_undegraded":"2024-03-11T22:09:36.622128+0000","last_fullsized":"2024-03-11T22:09:36.622128+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:55.775643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:46.542124+0000","last_change":"2024-03-11T22:09:46.542145+0000","last_active":"2024-03-11T22:09:46.542124+0000","last_peered":"2024-03-11T22:09:46.542124+0000","last_clean":"2024-03-11T22:09:46.542124+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:09:46.542124+0000","last_undegraded":"2024-03-11T22:09:46.542124+0000","last_fullsized":"2024-03-11T22:09:46.542124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:46.542054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:46.542054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.271294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048879+0000","last_change":"2024-03-11T22:08:35.055974+0000","last_active":"2024-03-11T22:09:36.048879+0000","last_peered":"2024-03-11T22:09:36.048879+0000","last_clean":"2024-03-11T22:09:36.048879+0000","last_became_active":"2024-03-11T22:08:35.055770+0000","last_became_peered":"2024-03-11T22:08:35.055770+0000","last_unstale":"2024-03-11T22:09:36.048879+0000","last_undegraded":"2024-03-11T22:09:36.048879+0000","last_fullsized":"2024-03-11T22:09:36.048879+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:54.010930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.049081+0000","last_change":"2024-03-11T22:08:39.073197+0000","last_active":"2024-03-11T22:09:36.049081+0000","last_peered":"2024-03-11T22:09:36.049081+0000","last_clean":"2024-03-11T22:09:36.049081+0000","last_became_active":"2024-03-11T22:08:39.072422+0000","last_became_peered":"2024-03-11T22:08:39.072422+0000","last_unstale":"2024-03-11T22:09:36.049081+0000","last_undegraded":"2024-03-11T22:09:36.049081+0000","last_fullsized":"2024-03-11T22:09:36.049081+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:51.710195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:48.351324+0000","last_change":"2024-03-11T22:09:38.769937+0000","last_active":"2024-03-11T22:09:48.351324+0000","last_peered":"2024-03-11T22:09:48.351324+0000","last_clean":"2024-03-11T22:09:48.351324+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:09:48.351324+0000","last_undegraded":"2024-03-11T22:09:48.351324+0000","last_fullsized":"2024-03-11T22:09:48.351324+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_clean_scrub_stamp":"2024-03-11T22:09:38.769825+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:58.315026+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048625+0000","last_change":"2024-03-11T22:08:33.036116+0000","last_active":"2024-03-11T22:09:36.048625+0000","last_peered":"2024-03-11T22:09:36.048625+0000","last_clean":"2024-03-11T22:09:36.048625+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:09:36.048625+0000","last_undegraded":"2024-03-11T22:09:36.048625+0000","last_fullsized":"2024-03-11T22:09:36.048625+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:40.981310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-11T22:09:36.048532+0000","last_change":"2024-03-11T22:08:35.057039+0000","last_active":"2024-03-11T22:09:36.048532+0000","last_peered":"2024-03-11T22:09:36.048532+0000","last_clean":"2024-03-11T22:09:36.048532+0000","last_became_active":"2024-03-11T22:08:35.056698+0000","last_became_peered":"2024-03-11T22:08:35.056698+0000","last_unstale":"2024-03-11T22:09:36.048532+0000","last_undegraded":"2024-03-11T22:09:36.048532+0000","last_fullsized":"2024-03-11T22:09:36.048532+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:51.765344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":400,"ondisk_log_size":400,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005077,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1523712,"data_stored":1024363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332122,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28484,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716668,"statfs":{"total":95995035648,"available":95965868032,"internally_reserved":0,"allocated":1503232,"data_stored":1024173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":434001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32660,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712492,"statfs":{"total":95995035648,"available":95961591808,"internally_reserved":0,"allocated":1519616,"data_stored":1023695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":905216,"data_stored":433631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":434311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920805,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":433536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659166,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":433078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:10:01.719 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T22:10:01.720 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T22:10:01.720 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T22:10:01.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T22:10:01.936 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:10:01 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[77897]: ts=2024-03-11T22:10:01.884Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T22:10:02.017 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:02.257 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:10:01 smithi045 bash[77831]: 92b3721cb5b78104493f3eabcf0691942a890dfc48da95f86ba3f45f430d247b 2024-03-11T22:10:02.257 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:10:01 smithi045 systemd[1]: Started Ceph alertmanager.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:10:02.258 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:10:02] ENGINE Bus STOPPING 2024-03-11T22:10:02.258 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:10:02] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:10:02.258 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:10:02] ENGINE Bus STOPPED 2024-03-11T22:10:02.258 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:10:02] ENGINE Bus STARTING 2024-03-11T22:10:02.258 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:10:02] ENGINE Serving on http://:::9283 2024-03-11T22:10:02.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[39715]: 2.1 deep-scrub starts 2024-03-11T22:10:02.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[39715]: 2.1 deep-scrub ok 2024-03-11T22:10:02.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[39715]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:10:02.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:10:02.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.259 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[30178]: 2.1 deep-scrub starts 2024-03-11T22:10:02.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[30178]: 2.1 deep-scrub ok 2024-03-11T22:10:02.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[30178]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:10:02.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:10:02.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.526 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:10:02] ENGINE Bus STARTED 2024-03-11T22:10:02.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:02 smithi144 ceph-mon[33984]: 2.1 deep-scrub starts 2024-03-11T22:10:02.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:02 smithi144 ceph-mon[33984]: 2.1 deep-scrub ok 2024-03-11T22:10:02.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:02 smithi144 ceph-mon[33984]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:10:02.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:02 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:02 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:02 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:02 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:02 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:02 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:02 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:10:02.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:02 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:02.910 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:10:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:10:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[30178]: 4.1a scrub starts 2024-03-11T22:10:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[30178]: 4.1a scrub ok 2024-03-11T22:10:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[30178]: 5.a deep-scrub starts 2024-03-11T22:10:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[30178]: 5.a deep-scrub ok 2024-03-11T22:10:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[30178]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:10:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[30178]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:10:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[30178]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:10:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[30178]: Deploying daemon grafana.a on smithi144 2024-03-11T22:10:03.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[30178]: 3.6 scrub starts 2024-03-11T22:10:03.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[30178]: 3.6 scrub ok 2024-03-11T22:10:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[39715]: 4.1a scrub starts 2024-03-11T22:10:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[39715]: 4.1a scrub ok 2024-03-11T22:10:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[39715]: 5.a deep-scrub starts 2024-03-11T22:10:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[39715]: 5.a deep-scrub ok 2024-03-11T22:10:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[39715]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:10:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[39715]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:10:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[39715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:10:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[39715]: Deploying daemon grafana.a on smithi144 2024-03-11T22:10:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[39715]: 3.6 scrub starts 2024-03-11T22:10:03.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-mon[39715]: 3.6 scrub ok 2024-03-11T22:10:03.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:03 smithi144 ceph-mon[33984]: 4.1a scrub starts 2024-03-11T22:10:03.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:03 smithi144 ceph-mon[33984]: 4.1a scrub ok 2024-03-11T22:10:03.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:03 smithi144 ceph-mon[33984]: 5.a deep-scrub starts 2024-03-11T22:10:03.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:03 smithi144 ceph-mon[33984]: 5.a deep-scrub ok 2024-03-11T22:10:03.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:03 smithi144 ceph-mon[33984]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:10:03.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:03 smithi144 ceph-mon[33984]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:10:03.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:03 smithi144 ceph-mon[33984]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:10:03.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:03 smithi144 ceph-mon[33984]: Deploying daemon grafana.a on smithi144 2024-03-11T22:10:03.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:03 smithi144 ceph-mon[33984]: 3.6 scrub starts 2024-03-11T22:10:03.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:03 smithi144 ceph-mon[33984]: 3.6 scrub ok 2024-03-11T22:10:03.882 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:10:03 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[77897]: ts=2024-03-11T22:10:03.853Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000197735s 2024-03-11T22:10:04.154 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:10:04.154 INFO:teuthology.orchestra.run.smithi045.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T22:10:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[30178]: 4.7 scrub starts 2024-03-11T22:10:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[30178]: 4.7 scrub ok 2024-03-11T22:10:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[30178]: 6.3 deep-scrub starts 2024-03-11T22:10:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[30178]: 6.3 deep-scrub ok 2024-03-11T22:10:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[30178]: 5.14 scrub starts 2024-03-11T22:10:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[30178]: 5.14 scrub ok 2024-03-11T22:10:04.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[39715]: 4.7 scrub starts 2024-03-11T22:10:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[39715]: 4.7 scrub ok 2024-03-11T22:10:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[39715]: 6.3 deep-scrub starts 2024-03-11T22:10:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[39715]: 6.3 deep-scrub ok 2024-03-11T22:10:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[39715]: 5.14 scrub starts 2024-03-11T22:10:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[39715]: 5.14 scrub ok 2024-03-11T22:10:04.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:04 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:04.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:04 smithi144 ceph-mon[33984]: 4.7 scrub starts 2024-03-11T22:10:04.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:04 smithi144 ceph-mon[33984]: 4.7 scrub ok 2024-03-11T22:10:04.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:04 smithi144 ceph-mon[33984]: 6.3 deep-scrub starts 2024-03-11T22:10:04.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:04 smithi144 ceph-mon[33984]: 6.3 deep-scrub ok 2024-03-11T22:10:04.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:04 smithi144 ceph-mon[33984]: 5.14 scrub starts 2024-03-11T22:10:04.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:04 smithi144 ceph-mon[33984]: 5.14 scrub ok 2024-03-11T22:10:04.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:04 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:04.864 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T22:10:04.864 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T22:10:04.865 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-11T22:10:04.875 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-11T22:10:04.875 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'squid', 'sha1': 'fbcdfe674e56d1ea7a0810744e5915bb07961be5'}} 2024-03-11T22:10:04.876 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-11T22:10:04.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-11T22:10:05.120 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:05.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[39715]: 3.1a scrub starts 2024-03-11T22:10:05.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[39715]: 3.1a scrub ok 2024-03-11T22:10:05.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[39715]: 6.1d scrub starts 2024-03-11T22:10:05.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[39715]: 6.1d scrub ok 2024-03-11T22:10:05.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[39715]: 5.b deep-scrub starts 2024-03-11T22:10:05.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[39715]: 5.b deep-scrub ok 2024-03-11T22:10:05.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[39715]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 574 B/s rd, 0 op/s 2024-03-11T22:10:05.407 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[39715]: 4.16 deep-scrub starts 2024-03-11T22:10:05.407 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[39715]: 4.16 deep-scrub ok 2024-03-11T22:10:05.407 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1106861780' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:10:05.407 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[39715]: 4.1d scrub starts 2024-03-11T22:10:05.407 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[39715]: 4.1d scrub ok 2024-03-11T22:10:05.407 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[30178]: 3.1a scrub starts 2024-03-11T22:10:05.407 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[30178]: 3.1a scrub ok 2024-03-11T22:10:05.407 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[30178]: 6.1d scrub starts 2024-03-11T22:10:05.407 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[30178]: 6.1d scrub ok 2024-03-11T22:10:05.407 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[30178]: 5.b deep-scrub starts 2024-03-11T22:10:05.407 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[30178]: 5.b deep-scrub ok 2024-03-11T22:10:05.407 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[30178]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 574 B/s rd, 0 op/s 2024-03-11T22:10:05.408 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[30178]: 4.16 deep-scrub starts 2024-03-11T22:10:05.408 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[30178]: 4.16 deep-scrub ok 2024-03-11T22:10:05.408 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1106861780' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:10:05.408 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[30178]: 4.1d scrub starts 2024-03-11T22:10:05.408 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:05 smithi045 ceph-mon[30178]: 4.1d scrub ok 2024-03-11T22:10:05.490 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:05 smithi144 ceph-mon[33984]: 3.1a scrub starts 2024-03-11T22:10:05.490 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:05 smithi144 ceph-mon[33984]: 3.1a scrub ok 2024-03-11T22:10:05.490 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:05 smithi144 ceph-mon[33984]: 6.1d scrub starts 2024-03-11T22:10:05.490 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:05 smithi144 ceph-mon[33984]: 6.1d scrub ok 2024-03-11T22:10:05.490 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:05 smithi144 ceph-mon[33984]: 5.b deep-scrub starts 2024-03-11T22:10:05.491 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:05 smithi144 ceph-mon[33984]: 5.b deep-scrub ok 2024-03-11T22:10:05.491 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:05 smithi144 ceph-mon[33984]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 574 B/s rd, 0 op/s 2024-03-11T22:10:05.491 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:05 smithi144 ceph-mon[33984]: 4.16 deep-scrub starts 2024-03-11T22:10:05.491 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:05 smithi144 ceph-mon[33984]: 4.16 deep-scrub ok 2024-03-11T22:10:05.491 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:05 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1106861780' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:10:05.491 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:05 smithi144 ceph-mon[33984]: 4.1d scrub starts 2024-03-11T22:10:05.491 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:05 smithi144 ceph-mon[33984]: 4.1d scrub ok 2024-03-11T22:10:05.839 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:10:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:10:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:06 smithi045 ceph-mon[39715]: 5.18 scrub starts 2024-03-11T22:10:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:06 smithi045 ceph-mon[39715]: 5.18 scrub ok 2024-03-11T22:10:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:06 smithi045 ceph-mon[39715]: 6.f scrub starts 2024-03-11T22:10:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:06 smithi045 ceph-mon[39715]: 6.f scrub ok 2024-03-11T22:10:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:06 smithi045 ceph-mon[39715]: 3.f scrub starts 2024-03-11T22:10:06.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:06 smithi045 ceph-mon[39715]: 3.f scrub ok 2024-03-11T22:10:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:06 smithi045 ceph-mon[30178]: 5.18 scrub starts 2024-03-11T22:10:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:06 smithi045 ceph-mon[30178]: 5.18 scrub ok 2024-03-11T22:10:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:06 smithi045 ceph-mon[30178]: 6.f scrub starts 2024-03-11T22:10:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:06 smithi045 ceph-mon[30178]: 6.f scrub ok 2024-03-11T22:10:06.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:06 smithi045 ceph-mon[30178]: 3.f scrub starts 2024-03-11T22:10:06.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:06 smithi045 ceph-mon[30178]: 3.f scrub ok 2024-03-11T22:10:06.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:06 smithi144 ceph-mon[33984]: 5.18 scrub starts 2024-03-11T22:10:06.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:06 smithi144 ceph-mon[33984]: 5.18 scrub ok 2024-03-11T22:10:06.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:06 smithi144 ceph-mon[33984]: 6.f scrub starts 2024-03-11T22:10:06.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:06 smithi144 ceph-mon[33984]: 6.f scrub ok 2024-03-11T22:10:06.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:06 smithi144 ceph-mon[33984]: 3.f scrub starts 2024-03-11T22:10:06.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:06 smithi144 ceph-mon[33984]: 3.f scrub ok 2024-03-11T22:10:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[30178]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[30178]: 6.19 scrub starts 2024-03-11T22:10:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[30178]: 6.19 scrub ok 2024-03-11T22:10:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[30178]: 6.8 scrub starts 2024-03-11T22:10:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[30178]: 6.8 scrub ok 2024-03-11T22:10:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:10:07.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:10:07.527 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:10:07] ENGINE Bus STOPPING 2024-03-11T22:10:07.527 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:10:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:10:07.527 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:10:07] ENGINE Bus STOPPED 2024-03-11T22:10:07.527 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:10:07] ENGINE Bus STARTING 2024-03-11T22:10:07.527 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:10:07] ENGINE Serving on http://:::9283 2024-03-11T22:10:07.527 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:10:07] ENGINE Bus STARTED 2024-03-11T22:10:07.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:07.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[39715]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:07.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[39715]: 6.19 scrub starts 2024-03-11T22:10:07.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[39715]: 6.19 scrub ok 2024-03-11T22:10:07.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[39715]: 6.8 scrub starts 2024-03-11T22:10:07.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[39715]: 6.8 scrub ok 2024-03-11T22:10:07.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:10:07.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:07 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:10:07.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:07.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:07 smithi144 ceph-mon[33984]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:07.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:07 smithi144 ceph-mon[33984]: 6.19 scrub starts 2024-03-11T22:10:07.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:07 smithi144 ceph-mon[33984]: 6.19 scrub ok 2024-03-11T22:10:07.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:07 smithi144 ceph-mon[33984]: 6.8 scrub starts 2024-03-11T22:10:07.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:07 smithi144 ceph-mon[33984]: 6.8 scrub ok 2024-03-11T22:10:07.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:07 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:10:07.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:07 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:10:07.881 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-11T22:10:07.882 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T22:10:07.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:10:08.129 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:08.415 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:08 smithi045 ceph-mon[39715]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T22:10:08.416 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:08 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' 2024-03-11T22:10:08.416 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:08 smithi045 ceph-mon[39715]: 5.16 deep-scrub starts 2024-03-11T22:10:08.416 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:08 smithi045 ceph-mon[39715]: 5.16 deep-scrub ok 2024-03-11T22:10:08.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:08 smithi045 ceph-mon[30178]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T22:10:08.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:08 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' 2024-03-11T22:10:08.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:08 smithi045 ceph-mon[30178]: 5.16 deep-scrub starts 2024-03-11T22:10:08.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:08 smithi045 ceph-mon[30178]: 5.16 deep-scrub ok 2024-03-11T22:10:08.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:08 smithi144 ceph-mon[33984]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T22:10:08.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:08 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' 2024-03-11T22:10:08.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:08 smithi144 ceph-mon[33984]: 5.16 deep-scrub starts 2024-03-11T22:10:08.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:08 smithi144 ceph-mon[33984]: 5.16 deep-scrub ok 2024-03-11T22:10:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:09 smithi045 ceph-mon[30178]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:09 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:10:09.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:09 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:10:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:09 smithi045 ceph-mon[39715]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:09 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:10:09.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:09 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:10:09.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:09 smithi144 ceph-mon[33984]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:09.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:09 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:10:09.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:09 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:10:10.365 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:10:10.367 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:10:10.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:10 smithi045 ceph-mon[30178]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T22:10:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:10 smithi045 ceph-mon[30178]: 6.5 scrub starts 2024-03-11T22:10:10.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:10 smithi045 ceph-mon[30178]: 6.5 scrub ok 2024-03-11T22:10:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:10 smithi045 ceph-mon[39715]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T22:10:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:10 smithi045 ceph-mon[39715]: 6.5 scrub starts 2024-03-11T22:10:10.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:10 smithi045 ceph-mon[39715]: 6.5 scrub ok 2024-03-11T22:10:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:10 smithi144 ceph-mon[33984]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T22:10:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:10 smithi144 ceph-mon[33984]: 6.5 scrub starts 2024-03-11T22:10:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:10 smithi144 ceph-mon[33984]: 6.5 scrub ok 2024-03-11T22:10:11.031 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-11T22:10:10.049301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1019,"num_read_kb":865,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":615,"ondisk_log_size":615,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240188,"kb_used_data":7420,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749721028,"statfs":{"total":767960285184,"available":767714332672,"internally_reserved":0,"allocated":7598080,"data_stored":4524844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":238276147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002090"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135327+0000","last_change":"2024-03-11T22:08:39.077833+0000","last_active":"2024-03-11T22:09:56.135327+0000","last_peered":"2024-03-11T22:09:56.135327+0000","last_clean":"2024-03-11T22:09:56.135327+0000","last_became_active":"2024-03-11T22:08:39.074695+0000","last_became_peered":"2024-03-11T22:08:39.074695+0000","last_unstale":"2024-03-11T22:09:56.135327+0000","last_undegraded":"2024-03-11T22:09:56.135327+0000","last_fullsized":"2024-03-11T22:09:56.135327+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:05.446193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135662+0000","last_change":"2024-03-11T22:08:37.074138+0000","last_active":"2024-03-11T22:09:56.135662+0000","last_peered":"2024-03-11T22:09:56.135662+0000","last_clean":"2024-03-11T22:09:56.135662+0000","last_became_active":"2024-03-11T22:08:37.073775+0000","last_became_peered":"2024-03-11T22:08:37.073775+0000","last_unstale":"2024-03-11T22:09:56.135662+0000","last_undegraded":"2024-03-11T22:09:56.135662+0000","last_fullsized":"2024-03-11T22:09:56.135662+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:02.401482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516145+0000","last_change":"2024-03-11T22:09:55.162082+0000","last_active":"2024-03-11T22:09:56.516145+0000","last_peered":"2024-03-11T22:09:56.516145+0000","last_clean":"2024-03-11T22:09:56.516145+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:09:56.516145+0000","last_undegraded":"2024-03-11T22:09:56.516145+0000","last_fullsized":"2024-03-11T22:09:56.516145+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:55.161978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:55.161978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:14.539455+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135239+0000","last_change":"2024-03-11T22:08:35.053644+0000","last_active":"2024-03-11T22:09:56.135239+0000","last_peered":"2024-03-11T22:09:56.135239+0000","last_clean":"2024-03-11T22:09:56.135239+0000","last_became_active":"2024-03-11T22:08:35.053179+0000","last_became_peered":"2024-03-11T22:08:35.053179+0000","last_unstale":"2024-03-11T22:09:56.135239+0000","last_undegraded":"2024-03-11T22:09:56.135239+0000","last_fullsized":"2024-03-11T22:09:56.135239+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515719+0000","last_change":"2024-03-11T22:08:39.079762+0000","last_active":"2024-03-11T22:09:56.515719+0000","last_peered":"2024-03-11T22:09:56.515719+0000","last_clean":"2024-03-11T22:09:56.515719+0000","last_became_active":"2024-03-11T22:08:39.079045+0000","last_became_peered":"2024-03-11T22:08:39.079045+0000","last_unstale":"2024-03-11T22:09:56.515719+0000","last_undegraded":"2024-03-11T22:09:56.515719+0000","last_fullsized":"2024-03-11T22:09:56.515719+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.065191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:04.453985+0000","last_change":"2024-03-11T22:10:04.454026+0000","last_active":"2024-03-11T22:10:04.453985+0000","last_peered":"2024-03-11T22:10:04.453985+0000","last_clean":"2024-03-11T22:10:04.453985+0000","last_became_active":"2024-03-11T22:08:37.081861+0000","last_became_peered":"2024-03-11T22:08:37.081861+0000","last_unstale":"2024-03-11T22:10:04.453985+0000","last_undegraded":"2024-03-11T22:10:04.453985+0000","last_fullsized":"2024-03-11T22:10:04.453985+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:04.453912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:10:04.453912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:04.768471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":75,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515693+0000","last_change":"2024-03-11T22:09:39.608984+0000","last_active":"2024-03-11T22:09:56.515693+0000","last_peered":"2024-03-11T22:09:56.515693+0000","last_clean":"2024-03-11T22:09:56.515693+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:09:56.515693+0000","last_undegraded":"2024-03-11T22:09:56.515693+0000","last_fullsized":"2024-03-11T22:09:56.515693+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:09:39.608890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:39.608890+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:49.086609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":82,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516054+0000","last_change":"2024-03-11T22:08:35.061181+0000","last_active":"2024-03-11T22:09:56.516054+0000","last_peered":"2024-03-11T22:09:56.516054+0000","last_clean":"2024-03-11T22:09:56.516054+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:09:56.516054+0000","last_undegraded":"2024-03-11T22:09:56.516054+0000","last_fullsized":"2024-03-11T22:09:56.516054+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.748787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515254+0000","last_change":"2024-03-11T22:08:39.083128+0000","last_active":"2024-03-11T22:09:56.515254+0000","last_peered":"2024-03-11T22:09:56.515254+0000","last_clean":"2024-03-11T22:09:56.515254+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:09:56.515254+0000","last_undegraded":"2024-03-11T22:09:56.515254+0000","last_fullsized":"2024-03-11T22:09:56.515254+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.592922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515582+0000","last_change":"2024-03-11T22:08:37.082294+0000","last_active":"2024-03-11T22:09:56.515582+0000","last_peered":"2024-03-11T22:09:56.515582+0000","last_clean":"2024-03-11T22:09:56.515582+0000","last_became_active":"2024-03-11T22:08:37.081996+0000","last_became_peered":"2024-03-11T22:08:37.081996+0000","last_unstale":"2024-03-11T22:09:56.515582+0000","last_undegraded":"2024-03-11T22:09:56.515582+0000","last_fullsized":"2024-03-11T22:09:56.515582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.530415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515675+0000","last_change":"2024-03-11T22:09:40.301289+0000","last_active":"2024-03-11T22:09:56.515675+0000","last_peered":"2024-03-11T22:09:56.515675+0000","last_clean":"2024-03-11T22:09:56.515675+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:09:56.515675+0000","last_undegraded":"2024-03-11T22:09:56.515675+0000","last_fullsized":"2024-03-11T22:09:56.515675+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:40.301162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:40.301162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:40.904990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":81,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:01.441788+0000","last_change":"2024-03-11T22:10:01.441822+0000","last_active":"2024-03-11T22:10:01.441788+0000","last_peered":"2024-03-11T22:10:01.441788+0000","last_clean":"2024-03-11T22:10:01.441788+0000","last_became_active":"2024-03-11T22:08:35.060802+0000","last_became_peered":"2024-03-11T22:08:35.060802+0000","last_unstale":"2024-03-11T22:10:01.441788+0000","last_undegraded":"2024-03-11T22:10:01.441788+0000","last_fullsized":"2024-03-11T22:10:01.441788+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:10:01.441711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:01.441711+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:04.897877+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516226+0000","last_change":"2024-03-11T22:08:39.076507+0000","last_active":"2024-03-11T22:09:56.516226+0000","last_peered":"2024-03-11T22:09:56.516226+0000","last_clean":"2024-03-11T22:09:56.516226+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:09:56.516226+0000","last_undegraded":"2024-03-11T22:09:56.516226+0000","last_fullsized":"2024-03-11T22:09:56.516226+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:00.270238+0000","last_change":"2024-03-11T22:10:00.270261+0000","last_active":"2024-03-11T22:10:00.270238+0000","last_peered":"2024-03-11T22:10:00.270238+0000","last_clean":"2024-03-11T22:10:00.270238+0000","last_became_active":"2024-03-11T22:08:37.077279+0000","last_became_peered":"2024-03-11T22:08:37.077279+0000","last_unstale":"2024-03-11T22:10:00.270238+0000","last_undegraded":"2024-03-11T22:10:00.270238+0000","last_fullsized":"2024-03-11T22:10:00.270238+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:00.270166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.270166+0000","last_clean_scrub_stamp":"2024-03-11T22:10:00.270166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:16.941652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516095+0000","last_change":"2024-03-11T22:08:33.040601+0000","last_active":"2024-03-11T22:09:56.516095+0000","last_peered":"2024-03-11T22:09:56.516095+0000","last_clean":"2024-03-11T22:09:56.516095+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:09:56.516095+0000","last_undegraded":"2024-03-11T22:09:56.516095+0000","last_fullsized":"2024-03-11T22:09:56.516095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135828+0000","last_change":"2024-03-11T22:08:35.061039+0000","last_active":"2024-03-11T22:09:56.135828+0000","last_peered":"2024-03-11T22:09:56.135828+0000","last_clean":"2024-03-11T22:09:56.135828+0000","last_became_active":"2024-03-11T22:08:35.060777+0000","last_became_peered":"2024-03-11T22:08:35.060777+0000","last_unstale":"2024-03-11T22:09:56.135828+0000","last_undegraded":"2024-03-11T22:09:56.135828+0000","last_fullsized":"2024-03-11T22:09:56.135828+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.136230+0000","last_change":"2024-03-11T22:09:51.367715+0000","last_active":"2024-03-11T22:09:56.136230+0000","last_peered":"2024-03-11T22:09:56.136230+0000","last_clean":"2024-03-11T22:09:56.136230+0000","last_became_active":"2024-03-11T22:08:39.075609+0000","last_became_peered":"2024-03-11T22:08:39.075609+0000","last_unstale":"2024-03-11T22:09:56.136230+0000","last_undegraded":"2024-03-11T22:09:56.136230+0000","last_fullsized":"2024-03-11T22:09:56.136230+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:51.367570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:51.367570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:10.802427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.136547+0000","last_change":"2024-03-11T22:08:37.077878+0000","last_active":"2024-03-11T22:09:56.136547+0000","last_peered":"2024-03-11T22:09:56.136547+0000","last_clean":"2024-03-11T22:09:56.136547+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:09:56.136547+0000","last_undegraded":"2024-03-11T22:09:56.136547+0000","last_fullsized":"2024-03-11T22:09:56.136547+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.718775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:58.215549+0000","last_change":"2024-03-11T22:09:58.215572+0000","last_active":"2024-03-11T22:09:58.215549+0000","last_peered":"2024-03-11T22:09:58.215549+0000","last_clean":"2024-03-11T22:09:58.215549+0000","last_became_active":"2024-03-11T22:08:33.035624+0000","last_became_peered":"2024-03-11T22:08:33.035624+0000","last_unstale":"2024-03-11T22:09:58.215549+0000","last_undegraded":"2024-03-11T22:09:58.215549+0000","last_fullsized":"2024-03-11T22:09:58.215549+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:58.215488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:58.215488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:16.044592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":90,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135649+0000","last_change":"2024-03-11T22:09:45.731375+0000","last_active":"2024-03-11T22:09:56.135649+0000","last_peered":"2024-03-11T22:09:56.135649+0000","last_clean":"2024-03-11T22:09:56.135649+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:09:56.135649+0000","last_undegraded":"2024-03-11T22:09:56.135649+0000","last_fullsized":"2024-03-11T22:09:56.135649+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:09:45.731248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:45.731248+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:08.445140+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515461+0000","last_change":"2024-03-11T22:08:39.078269+0000","last_active":"2024-03-11T22:09:56.515461+0000","last_peered":"2024-03-11T22:09:56.515461+0000","last_clean":"2024-03-11T22:09:56.515461+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:09:56.515461+0000","last_undegraded":"2024-03-11T22:09:56.515461+0000","last_fullsized":"2024-03-11T22:09:56.515461+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.509768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135816+0000","last_change":"2024-03-11T22:08:37.080325+0000","last_active":"2024-03-11T22:09:56.135816+0000","last_peered":"2024-03-11T22:09:56.135816+0000","last_clean":"2024-03-11T22:09:56.135816+0000","last_became_active":"2024-03-11T22:08:37.080014+0000","last_became_peered":"2024-03-11T22:08:37.080014+0000","last_unstale":"2024-03-11T22:09:56.135816+0000","last_undegraded":"2024-03-11T22:09:56.135816+0000","last_fullsized":"2024-03-11T22:09:56.135816+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:04.066121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:03.469416+0000","last_change":"2024-03-11T22:10:03.469439+0000","last_active":"2024-03-11T22:10:03.469416+0000","last_peered":"2024-03-11T22:10:03.469416+0000","last_clean":"2024-03-11T22:10:03.469416+0000","last_became_active":"2024-03-11T22:08:33.299189+0000","last_became_peered":"2024-03-11T22:08:33.299189+0000","last_unstale":"2024-03-11T22:10:03.469416+0000","last_undegraded":"2024-03-11T22:10:03.469416+0000","last_fullsized":"2024-03-11T22:10:03.469416+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:03.469346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.469346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:11.637778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":88,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:04.596648+0000","last_change":"2024-03-11T22:10:04.596677+0000","last_active":"2024-03-11T22:10:04.596648+0000","last_peered":"2024-03-11T22:10:04.596648+0000","last_clean":"2024-03-11T22:10:04.596648+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:10:04.596648+0000","last_undegraded":"2024-03-11T22:10:04.596648+0000","last_fullsized":"2024-03-11T22:10:04.596648+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:10:04.596588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:04.596588+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:27.560340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515730+0000","last_change":"2024-03-11T22:09:53.160256+0000","last_active":"2024-03-11T22:09:56.515730+0000","last_peered":"2024-03-11T22:09:56.515730+0000","last_clean":"2024-03-11T22:09:56.515730+0000","last_became_active":"2024-03-11T22:08:39.072956+0000","last_became_peered":"2024-03-11T22:08:39.072956+0000","last_unstale":"2024-03-11T22:09:56.515730+0000","last_undegraded":"2024-03-11T22:09:56.515730+0000","last_fullsized":"2024-03-11T22:09:56.515730+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:09:53.160150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:53.160150+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:21.225798+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":91,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:59.504774+0000","last_change":"2024-03-11T22:09:59.504774+0000","last_active":"2024-03-11T22:09:59.504774+0000","last_peered":"2024-03-11T22:09:59.504774+0000","last_clean":"2024-03-11T22:09:59.504774+0000","last_became_active":"2024-03-11T22:08:37.078588+0000","last_became_peered":"2024-03-11T22:08:37.078588+0000","last_unstale":"2024-03-11T22:09:59.504774+0000","last_undegraded":"2024-03-11T22:09:59.504774+0000","last_fullsized":"2024-03-11T22:09:59.504774+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:09:59.504668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:59.504668+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:06.422336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135768+0000","last_change":"2024-03-11T22:08:33.030653+0000","last_active":"2024-03-11T22:09:56.135768+0000","last_peered":"2024-03-11T22:09:56.135768+0000","last_clean":"2024-03-11T22:09:56.135768+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:09:56.135768+0000","last_undegraded":"2024-03-11T22:09:56.135768+0000","last_fullsized":"2024-03-11T22:09:56.135768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.528647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516221+0000","last_change":"2024-03-11T22:08:35.086209+0000","last_active":"2024-03-11T22:09:56.516221+0000","last_peered":"2024-03-11T22:09:56.516221+0000","last_clean":"2024-03-11T22:09:56.516221+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:09:56.516221+0000","last_undegraded":"2024-03-11T22:09:56.516221+0000","last_fullsized":"2024-03-11T22:09:56.516221+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:02.938827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:03.601932+0000","last_change":"2024-03-11T22:10:03.601956+0000","last_active":"2024-03-11T22:10:03.601932+0000","last_peered":"2024-03-11T22:10:03.601932+0000","last_clean":"2024-03-11T22:10:03.601932+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:10:03.601932+0000","last_undegraded":"2024-03-11T22:10:03.601932+0000","last_fullsized":"2024-03-11T22:10:03.601932+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:03.601863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.601863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:25.688404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516415+0000","last_change":"2024-03-11T22:08:37.069965+0000","last_active":"2024-03-11T22:09:56.516415+0000","last_peered":"2024-03-11T22:09:56.516415+0000","last_clean":"2024-03-11T22:09:56.516415+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:09:56.516415+0000","last_undegraded":"2024-03-11T22:09:56.516415+0000","last_fullsized":"2024-03-11T22:09:56.516415+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.292950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515797+0000","last_change":"2024-03-11T22:08:33.038259+0000","last_active":"2024-03-11T22:09:56.515797+0000","last_peered":"2024-03-11T22:09:56.515797+0000","last_clean":"2024-03-11T22:09:56.515797+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:09:56.515797+0000","last_undegraded":"2024-03-11T22:09:56.515797+0000","last_fullsized":"2024-03-11T22:09:56.515797+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.852988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.517737+0000","last_change":"2024-03-11T22:09:49.567290+0000","last_active":"2024-03-11T22:09:56.517737+0000","last_peered":"2024-03-11T22:09:56.517737+0000","last_clean":"2024-03-11T22:09:56.517737+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:09:56.517737+0000","last_undegraded":"2024-03-11T22:09:56.517737+0000","last_fullsized":"2024-03-11T22:09:56.517737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:09:49.567170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:49.567170+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:01.624956+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516035+0000","last_change":"2024-03-11T22:08:39.074473+0000","last_active":"2024-03-11T22:09:56.516035+0000","last_peered":"2024-03-11T22:09:56.516035+0000","last_clean":"2024-03-11T22:09:56.516035+0000","last_became_active":"2024-03-11T22:08:39.074179+0000","last_became_peered":"2024-03-11T22:08:39.074179+0000","last_unstale":"2024-03-11T22:09:56.516035+0000","last_undegraded":"2024-03-11T22:09:56.516035+0000","last_fullsized":"2024-03-11T22:09:56.516035+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.168751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515181+0000","last_change":"2024-03-11T22:09:40.669305+0000","last_active":"2024-03-11T22:09:56.515181+0000","last_peered":"2024-03-11T22:09:56.515181+0000","last_clean":"2024-03-11T22:09:56.515181+0000","last_became_active":"2024-03-11T22:08:37.075341+0000","last_became_peered":"2024-03-11T22:08:37.075341+0000","last_unstale":"2024-03-11T22:09:56.515181+0000","last_undegraded":"2024-03-11T22:09:56.515181+0000","last_fullsized":"2024-03-11T22:09:56.515181+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:40.669213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:40.669213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:05.240222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516052+0000","last_change":"2024-03-11T22:08:33.040896+0000","last_active":"2024-03-11T22:09:56.516052+0000","last_peered":"2024-03-11T22:09:56.516052+0000","last_clean":"2024-03-11T22:09:56.516052+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:09:56.516052+0000","last_undegraded":"2024-03-11T22:09:56.516052+0000","last_fullsized":"2024-03-11T22:09:56.516052+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.000587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515119+0000","last_change":"2024-03-11T22:09:42.568204+0000","last_active":"2024-03-11T22:09:56.515119+0000","last_peered":"2024-03-11T22:09:56.515119+0000","last_clean":"2024-03-11T22:09:56.515119+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:09:56.515119+0000","last_undegraded":"2024-03-11T22:09:56.515119+0000","last_fullsized":"2024-03-11T22:09:56.515119+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-11T22:09:42.568114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:42.568114+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.634829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515335+0000","last_change":"2024-03-11T22:08:39.081827+0000","last_active":"2024-03-11T22:09:56.515335+0000","last_peered":"2024-03-11T22:09:56.515335+0000","last_clean":"2024-03-11T22:09:56.515335+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:09:56.515335+0000","last_undegraded":"2024-03-11T22:09:56.515335+0000","last_fullsized":"2024-03-11T22:09:56.515335+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:58.233177+0000","last_change":"2024-03-11T22:09:58.233204+0000","last_active":"2024-03-11T22:09:58.233177+0000","last_peered":"2024-03-11T22:09:58.233177+0000","last_clean":"2024-03-11T22:09:58.233177+0000","last_became_active":"2024-03-11T22:08:37.078863+0000","last_became_peered":"2024-03-11T22:08:37.078863+0000","last_unstale":"2024-03-11T22:09:58.233177+0000","last_undegraded":"2024-03-11T22:09:58.233177+0000","last_fullsized":"2024-03-11T22:09:58.233177+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:58.233081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:09:58.233081+0000","last_clean_scrub_stamp":"2024-03-11T22:09:58.233081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:08.891585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515932+0000","last_change":"2024-03-11T22:09:49.271541+0000","last_active":"2024-03-11T22:09:56.515932+0000","last_peered":"2024-03-11T22:09:56.515932+0000","last_clean":"2024-03-11T22:09:56.515932+0000","last_became_active":"2024-03-11T22:08:33.039349+0000","last_became_peered":"2024-03-11T22:08:33.039349+0000","last_unstale":"2024-03-11T22:09:56.515932+0000","last_undegraded":"2024-03-11T22:09:56.515932+0000","last_fullsized":"2024-03-11T22:09:56.515932+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:49.271431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:49.271431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:01.697571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516058+0000","last_change":"2024-03-11T22:08:35.086272+0000","last_active":"2024-03-11T22:09:56.516058+0000","last_peered":"2024-03-11T22:09:56.516058+0000","last_clean":"2024-03-11T22:09:56.516058+0000","last_became_active":"2024-03-11T22:08:35.085928+0000","last_became_peered":"2024-03-11T22:08:35.085928+0000","last_unstale":"2024-03-11T22:09:56.516058+0000","last_undegraded":"2024-03-11T22:09:56.516058+0000","last_fullsized":"2024-03-11T22:09:56.516058+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.863565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.514991+0000","last_change":"2024-03-11T22:08:39.083049+0000","last_active":"2024-03-11T22:09:56.514991+0000","last_peered":"2024-03-11T22:09:56.514991+0000","last_clean":"2024-03-11T22:09:56.514991+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:09:56.514991+0000","last_undegraded":"2024-03-11T22:09:56.514991+0000","last_fullsized":"2024-03-11T22:09:56.514991+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.375813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515845+0000","last_change":"2024-03-11T22:09:43.569426+0000","last_active":"2024-03-11T22:09:56.515845+0000","last_peered":"2024-03-11T22:09:56.515845+0000","last_clean":"2024-03-11T22:09:56.515845+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:09:56.515845+0000","last_undegraded":"2024-03-11T22:09:56.515845+0000","last_fullsized":"2024-03-11T22:09:56.515845+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:43.569328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:43.569328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:05.155162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516797+0000","last_change":"2024-03-11T22:09:41.032444+0000","last_active":"2024-03-11T22:09:56.516797+0000","last_peered":"2024-03-11T22:09:56.516797+0000","last_clean":"2024-03-11T22:09:56.516797+0000","last_became_active":"2024-03-11T22:08:33.038783+0000","last_became_peered":"2024-03-11T22:08:33.038783+0000","last_unstale":"2024-03-11T22:09:56.516797+0000","last_undegraded":"2024-03-11T22:09:56.516797+0000","last_fullsized":"2024-03-11T22:09:56.516797+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:41.032339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:41.032339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:01.424543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":81,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:57.633737+0000","last_change":"2024-03-11T22:09:57.633770+0000","last_active":"2024-03-11T22:09:57.633737+0000","last_peered":"2024-03-11T22:09:57.633737+0000","last_clean":"2024-03-11T22:09:57.633737+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:09:57.633737+0000","last_undegraded":"2024-03-11T22:09:57.633737+0000","last_fullsized":"2024-03-11T22:09:57.633737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:09:57.633671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:57.633671+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:15.549351+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515153+0000","last_change":"2024-03-11T22:08:39.080670+0000","last_active":"2024-03-11T22:09:56.515153+0000","last_peered":"2024-03-11T22:09:56.515153+0000","last_clean":"2024-03-11T22:09:56.515153+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:09:56.515153+0000","last_undegraded":"2024-03-11T22:09:56.515153+0000","last_fullsized":"2024-03-11T22:09:56.515153+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.573443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.136148+0000","last_change":"2024-03-11T22:08:37.075322+0000","last_active":"2024-03-11T22:09:56.136148+0000","last_peered":"2024-03-11T22:09:56.136148+0000","last_clean":"2024-03-11T22:09:56.136148+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:09:56.136148+0000","last_undegraded":"2024-03-11T22:09:56.136148+0000","last_fullsized":"2024-03-11T22:09:56.136148+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.827062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":75,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135988+0000","last_change":"2024-03-11T22:09:48.362144+0000","last_active":"2024-03-11T22:09:56.135988+0000","last_peered":"2024-03-11T22:09:56.135988+0000","last_clean":"2024-03-11T22:09:56.135988+0000","last_became_active":"2024-03-11T22:08:33.036083+0000","last_became_peered":"2024-03-11T22:08:33.036083+0000","last_unstale":"2024-03-11T22:09:56.135988+0000","last_undegraded":"2024-03-11T22:09:56.135988+0000","last_fullsized":"2024-03-11T22:09:56.135988+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:09:48.361967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:48.361967+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:05.666698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":76,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135911+0000","last_change":"2024-03-11T22:08:35.054987+0000","last_active":"2024-03-11T22:09:56.135911+0000","last_peered":"2024-03-11T22:09:56.135911+0000","last_clean":"2024-03-11T22:09:56.135911+0000","last_became_active":"2024-03-11T22:08:35.054726+0000","last_became_peered":"2024-03-11T22:08:35.054726+0000","last_unstale":"2024-03-11T22:09:56.135911+0000","last_undegraded":"2024-03-11T22:09:56.135911+0000","last_fullsized":"2024-03-11T22:09:56.135911+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.886570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515904+0000","last_change":"2024-03-11T22:08:39.083736+0000","last_active":"2024-03-11T22:09:56.515904+0000","last_peered":"2024-03-11T22:09:56.515904+0000","last_clean":"2024-03-11T22:09:56.515904+0000","last_became_active":"2024-03-11T22:08:39.082938+0000","last_became_peered":"2024-03-11T22:08:39.082938+0000","last_unstale":"2024-03-11T22:09:56.515904+0000","last_undegraded":"2024-03-11T22:09:56.515904+0000","last_fullsized":"2024-03-11T22:09:56.515904+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:06.944689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":88,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:58.352797+0000","last_change":"2024-03-11T22:09:58.187539+0000","last_active":"2024-03-11T22:09:58.352797+0000","last_peered":"2024-03-11T22:09:58.352797+0000","last_clean":"2024-03-11T22:09:58.352797+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:09:58.352797+0000","last_undegraded":"2024-03-11T22:09:58.352797+0000","last_fullsized":"2024-03-11T22:09:58.352797+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:09:58.187420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:58.187420+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:59.523954+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515855+0000","last_change":"2024-03-11T22:09:45.198482+0000","last_active":"2024-03-11T22:09:56.515855+0000","last_peered":"2024-03-11T22:09:56.515855+0000","last_clean":"2024-03-11T22:09:56.515855+0000","last_became_active":"2024-03-11T22:08:33.300181+0000","last_became_peered":"2024-03-11T22:08:33.300181+0000","last_unstale":"2024-03-11T22:09:56.515855+0000","last_undegraded":"2024-03-11T22:09:56.515855+0000","last_fullsized":"2024-03-11T22:09:56.515855+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:45.198397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:45.198397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:12.972849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":80,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515759+0000","last_change":"2024-03-11T22:09:55.647845+0000","last_active":"2024-03-11T22:09:56.515759+0000","last_peered":"2024-03-11T22:09:56.515759+0000","last_clean":"2024-03-11T22:09:56.515759+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:09:56.515759+0000","last_undegraded":"2024-03-11T22:09:56.515759+0000","last_fullsized":"2024-03-11T22:09:56.515759+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:09:55.647703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:55.647703+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:22.462130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.136422+0000","last_change":"2024-03-11T22:08:39.072508+0000","last_active":"2024-03-11T22:09:56.136422+0000","last_peered":"2024-03-11T22:09:56.136422+0000","last_clean":"2024-03-11T22:09:56.136422+0000","last_became_active":"2024-03-11T22:08:39.072089+0000","last_became_peered":"2024-03-11T22:08:39.072089+0000","last_unstale":"2024-03-11T22:09:56.136422+0000","last_undegraded":"2024-03-11T22:09:56.136422+0000","last_fullsized":"2024-03-11T22:09:56.136422+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:06.283027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":91,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:03.640729+0000","last_change":"2024-03-11T22:10:03.640781+0000","last_active":"2024-03-11T22:10:03.640729+0000","last_peered":"2024-03-11T22:10:03.640729+0000","last_clean":"2024-03-11T22:10:03.640729+0000","last_became_active":"2024-03-11T22:08:37.071202+0000","last_became_peered":"2024-03-11T22:08:37.071202+0000","last_unstale":"2024-03-11T22:10:03.640729+0000","last_undegraded":"2024-03-11T22:10:03.640729+0000","last_fullsized":"2024-03-11T22:10:03.640729+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:10:03.640604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.640604+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:13.116374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:59.229513+0000","last_change":"2024-03-11T22:09:59.229533+0000","last_active":"2024-03-11T22:09:59.229513+0000","last_peered":"2024-03-11T22:09:59.229513+0000","last_clean":"2024-03-11T22:09:59.229513+0000","last_became_active":"2024-03-11T22:08:33.032850+0000","last_became_peered":"2024-03-11T22:08:33.032850+0000","last_unstale":"2024-03-11T22:09:59.229513+0000","last_undegraded":"2024-03-11T22:09:59.229513+0000","last_fullsized":"2024-03-11T22:09:59.229513+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:59.229452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:59.229452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:23.364611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":81,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516060+0000","last_change":"2024-03-11T22:09:52.219839+0000","last_active":"2024-03-11T22:09:56.516060+0000","last_peered":"2024-03-11T22:09:56.516060+0000","last_clean":"2024-03-11T22:09:56.516060+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:09:56.516060+0000","last_undegraded":"2024-03-11T22:09:56.516060+0000","last_fullsized":"2024-03-11T22:09:56.516060+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:09:52.219687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:52.219687+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:06.397728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135814+0000","last_change":"2024-03-11T22:09:44.743698+0000","last_active":"2024-03-11T22:09:56.135814+0000","last_peered":"2024-03-11T22:09:56.135814+0000","last_clean":"2024-03-11T22:09:56.135814+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:09:56.135814+0000","last_undegraded":"2024-03-11T22:09:56.135814+0000","last_fullsized":"2024-03-11T22:09:56.135814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:44.743600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:44.743600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:03.506943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516025+0000","last_change":"2024-03-11T22:09:45.582583+0000","last_active":"2024-03-11T22:09:56.516025+0000","last_peered":"2024-03-11T22:09:56.516025+0000","last_clean":"2024-03-11T22:09:56.516025+0000","last_became_active":"2024-03-11T22:08:37.069611+0000","last_became_peered":"2024-03-11T22:08:37.069611+0000","last_unstale":"2024-03-11T22:09:56.516025+0000","last_undegraded":"2024-03-11T22:09:56.516025+0000","last_fullsized":"2024-03-11T22:09:56.516025+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:45.582469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:45.582469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.713402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515883+0000","last_change":"2024-03-11T22:08:33.038436+0000","last_active":"2024-03-11T22:09:56.515883+0000","last_peered":"2024-03-11T22:09:56.515883+0000","last_clean":"2024-03-11T22:09:56.515883+0000","last_became_active":"2024-03-11T22:08:33.038132+0000","last_became_peered":"2024-03-11T22:08:33.038132+0000","last_unstale":"2024-03-11T22:09:56.515883+0000","last_undegraded":"2024-03-11T22:09:56.515883+0000","last_fullsized":"2024-03-11T22:09:56.515883+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"93'13","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:04.123492+0000","last_change":"2024-03-11T22:10:04.123522+0000","last_active":"2024-03-11T22:10:04.123492+0000","last_peered":"2024-03-11T22:10:04.123492+0000","last_clean":"2024-03-11T22:10:04.123492+0000","last_became_active":"2024-03-11T22:08:35.085787+0000","last_became_peered":"2024-03-11T22:08:35.085787+0000","last_unstale":"2024-03-11T22:10:04.123492+0000","last_undegraded":"2024-03-11T22:10:04.123492+0000","last_fullsized":"2024-03-11T22:10:04.123492+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:10:04.123429+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:11.598479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516204+0000","last_change":"2024-03-11T22:08:39.075607+0000","last_active":"2024-03-11T22:09:56.516204+0000","last_peered":"2024-03-11T22:09:56.516204+0000","last_clean":"2024-03-11T22:09:56.516204+0000","last_became_active":"2024-03-11T22:08:39.075305+0000","last_became_peered":"2024-03-11T22:08:39.075305+0000","last_unstale":"2024-03-11T22:09:56.516204+0000","last_undegraded":"2024-03-11T22:09:56.516204+0000","last_fullsized":"2024-03-11T22:09:56.516204+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.166687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516046+0000","last_change":"2024-03-11T22:08:37.077768+0000","last_active":"2024-03-11T22:09:56.516046+0000","last_peered":"2024-03-11T22:09:56.516046+0000","last_clean":"2024-03-11T22:09:56.516046+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:09:56.516046+0000","last_undegraded":"2024-03-11T22:09:56.516046+0000","last_fullsized":"2024-03-11T22:09:56.516046+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:00.499477+0000","last_change":"2024-03-11T22:10:00.499503+0000","last_active":"2024-03-11T22:10:00.499477+0000","last_peered":"2024-03-11T22:10:00.499477+0000","last_clean":"2024-03-11T22:10:00.499477+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:10:00.499477+0000","last_undegraded":"2024-03-11T22:10:00.499477+0000","last_fullsized":"2024-03-11T22:10:00.499477+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:10:00.499404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:24.856919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516326+0000","last_change":"2024-03-11T22:08:35.086263+0000","last_active":"2024-03-11T22:09:56.516326+0000","last_peered":"2024-03-11T22:09:56.516326+0000","last_clean":"2024-03-11T22:09:56.516326+0000","last_became_active":"2024-03-11T22:08:35.085914+0000","last_became_peered":"2024-03-11T22:08:35.085914+0000","last_unstale":"2024-03-11T22:09:56.516326+0000","last_undegraded":"2024-03-11T22:09:56.516326+0000","last_fullsized":"2024-03-11T22:09:56.516326+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515692+0000","last_change":"2024-03-11T22:08:39.079848+0000","last_active":"2024-03-11T22:09:56.515692+0000","last_peered":"2024-03-11T22:09:56.515692+0000","last_clean":"2024-03-11T22:09:56.515692+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:09:56.515692+0000","last_undegraded":"2024-03-11T22:09:56.515692+0000","last_fullsized":"2024-03-11T22:09:56.515692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.299384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515816+0000","last_change":"2024-03-11T22:08:33.039060+0000","last_active":"2024-03-11T22:09:56.515816+0000","last_peered":"2024-03-11T22:09:56.515816+0000","last_clean":"2024-03-11T22:09:56.515816+0000","last_became_active":"2024-03-11T22:08:33.037662+0000","last_became_peered":"2024-03-11T22:08:33.037662+0000","last_unstale":"2024-03-11T22:09:56.515816+0000","last_undegraded":"2024-03-11T22:09:56.515816+0000","last_fullsized":"2024-03-11T22:09:56.515816+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"91'22","reported_seq":97,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515320+0000","last_change":"2024-03-11T22:09:36.274318+0000","last_active":"2024-03-11T22:09:56.515320+0000","last_peered":"2024-03-11T22:09:56.515320+0000","last_clean":"2024-03-11T22:09:56.515320+0000","last_became_active":"2024-03-11T22:08:35.085916+0000","last_became_peered":"2024-03-11T22:08:35.085916+0000","last_unstale":"2024-03-11T22:09:56.515320+0000","last_undegraded":"2024-03-11T22:09:56.515320+0000","last_fullsized":"2024-03-11T22:09:56.515320+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'22","last_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:09:36.274225+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:38.387952+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":80,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:58.352639+0000","last_change":"2024-03-11T22:08:37.078410+0000","last_active":"2024-03-11T22:09:58.352639+0000","last_peered":"2024-03-11T22:09:58.352639+0000","last_clean":"2024-03-11T22:09:58.352639+0000","last_became_active":"2024-03-11T22:08:37.078114+0000","last_became_peered":"2024-03-11T22:08:37.078114+0000","last_unstale":"2024-03-11T22:09:58.352639+0000","last_undegraded":"2024-03-11T22:09:58.352639+0000","last_fullsized":"2024-03-11T22:09:58.352639+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"75'1","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515449+0000","last_change":"2024-03-11T22:08:39.083458+0000","last_active":"2024-03-11T22:09:56.515449+0000","last_peered":"2024-03-11T22:09:56.515449+0000","last_clean":"2024-03-11T22:09:56.515449+0000","last_became_active":"2024-03-11T22:08:39.083141+0000","last_became_peered":"2024-03-11T22:08:39.083141+0000","last_unstale":"2024-03-11T22:09:56.515449+0000","last_undegraded":"2024-03-11T22:09:56.515449+0000","last_fullsized":"2024-03-11T22:09:56.515449+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516767+0000","last_change":"2024-03-11T22:09:55.169720+0000","last_active":"2024-03-11T22:09:56.516767+0000","last_peered":"2024-03-11T22:09:56.516767+0000","last_clean":"2024-03-11T22:09:56.516767+0000","last_became_active":"2024-03-11T22:08:33.041492+0000","last_became_peered":"2024-03-11T22:08:33.041492+0000","last_unstale":"2024-03-11T22:09:56.516767+0000","last_undegraded":"2024-03-11T22:09:56.516767+0000","last_fullsized":"2024-03-11T22:09:56.516767+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:55.169587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:55.169587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:18.640477+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":98,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135566+0000","last_change":"2024-03-11T22:08:54.614766+0000","last_active":"2024-03-11T22:09:56.135566+0000","last_peered":"2024-03-11T22:09:56.135566+0000","last_clean":"2024-03-11T22:09:56.135566+0000","last_became_active":"2024-03-11T22:08:54.607153+0000","last_became_peered":"2024-03-11T22:08:54.607153+0000","last_unstale":"2024-03-11T22:09:56.135566+0000","last_undegraded":"2024-03-11T22:09:56.135566+0000","last_fullsized":"2024-03-11T22:09:56.135566+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:58.241457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135034+0000","last_change":"2024-03-11T22:09:48.721747+0000","last_active":"2024-03-11T22:09:56.135034+0000","last_peered":"2024-03-11T22:09:56.135034+0000","last_clean":"2024-03-11T22:09:56.135034+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:09:56.135034+0000","last_undegraded":"2024-03-11T22:09:56.135034+0000","last_fullsized":"2024-03-11T22:09:56.135034+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:48.721656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:48.721656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.253690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516404+0000","last_change":"2024-03-11T22:08:39.079520+0000","last_active":"2024-03-11T22:09:56.516404+0000","last_peered":"2024-03-11T22:09:56.516404+0000","last_clean":"2024-03-11T22:09:56.516404+0000","last_became_active":"2024-03-11T22:08:39.079238+0000","last_became_peered":"2024-03-11T22:08:39.079238+0000","last_unstale":"2024-03-11T22:09:56.516404+0000","last_undegraded":"2024-03-11T22:09:56.516404+0000","last_fullsized":"2024-03-11T22:09:56.516404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:03.710383+0000","last_change":"2024-03-11T22:10:03.710406+0000","last_active":"2024-03-11T22:10:03.710383+0000","last_peered":"2024-03-11T22:10:03.710383+0000","last_clean":"2024-03-11T22:10:03.710383+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:10:03.710383+0000","last_undegraded":"2024-03-11T22:10:03.710383+0000","last_fullsized":"2024-03-11T22:10:03.710383+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:03.710319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.710319+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.710319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:24.719061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516497+0000","last_change":"2024-03-11T22:09:42.079812+0000","last_active":"2024-03-11T22:09:56.516497+0000","last_peered":"2024-03-11T22:09:56.516497+0000","last_clean":"2024-03-11T22:09:56.516497+0000","last_became_active":"2024-03-11T22:08:33.035188+0000","last_became_peered":"2024-03-11T22:08:33.035188+0000","last_unstale":"2024-03-11T22:09:56.516497+0000","last_undegraded":"2024-03-11T22:09:56.516497+0000","last_fullsized":"2024-03-11T22:09:56.516497+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:42.079695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:42.079695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:45.595806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":96,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516891+0000","last_change":"2024-03-11T22:09:54.637633+0000","last_active":"2024-03-11T22:09:56.516891+0000","last_peered":"2024-03-11T22:09:56.516891+0000","last_clean":"2024-03-11T22:09:56.516891+0000","last_became_active":"2024-03-11T22:08:35.087046+0000","last_became_peered":"2024-03-11T22:08:35.087046+0000","last_unstale":"2024-03-11T22:09:56.516891+0000","last_undegraded":"2024-03-11T22:09:56.516891+0000","last_fullsized":"2024-03-11T22:09:56.516891+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:09:54.637462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:54.637462+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:00.008092+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:57.233747+0000","last_change":"2024-03-11T22:09:57.233798+0000","last_active":"2024-03-11T22:09:57.233747+0000","last_peered":"2024-03-11T22:09:57.233747+0000","last_clean":"2024-03-11T22:09:57.233747+0000","last_became_active":"2024-03-11T22:08:39.085496+0000","last_became_peered":"2024-03-11T22:08:39.085496+0000","last_unstale":"2024-03-11T22:09:57.233747+0000","last_undegraded":"2024-03-11T22:09:57.233747+0000","last_fullsized":"2024-03-11T22:09:57.233747+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:57.233610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:57.233610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:19.937525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:01.692548+0000","last_change":"2024-03-11T22:10:01.692571+0000","last_active":"2024-03-11T22:10:01.692548+0000","last_peered":"2024-03-11T22:10:01.692548+0000","last_clean":"2024-03-11T22:10:01.692548+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:10:01.692548+0000","last_undegraded":"2024-03-11T22:10:01.692548+0000","last_fullsized":"2024-03-11T22:10:01.692548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:01.692464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:01.692464+0000","last_clean_scrub_stamp":"2024-03-11T22:10:01.692464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:27.908248+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516159+0000","last_change":"2024-03-11T22:08:33.038820+0000","last_active":"2024-03-11T22:09:56.516159+0000","last_peered":"2024-03-11T22:09:56.516159+0000","last_clean":"2024-03-11T22:09:56.516159+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:09:56.516159+0000","last_undegraded":"2024-03-11T22:09:56.516159+0000","last_fullsized":"2024-03-11T22:09:56.516159+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":89,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516696+0000","last_change":"2024-03-11T22:08:54.617268+0000","last_active":"2024-03-11T22:09:56.516696+0000","last_peered":"2024-03-11T22:09:56.516696+0000","last_clean":"2024-03-11T22:09:56.516696+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:09:56.516696+0000","last_undegraded":"2024-03-11T22:09:56.516696+0000","last_fullsized":"2024-03-11T22:09:56.516696+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:59.997317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515498+0000","last_change":"2024-03-11T22:08:39.084054+0000","last_active":"2024-03-11T22:09:56.515498+0000","last_peered":"2024-03-11T22:09:56.515498+0000","last_clean":"2024-03-11T22:09:56.515498+0000","last_became_active":"2024-03-11T22:08:39.083738+0000","last_became_peered":"2024-03-11T22:08:39.083738+0000","last_unstale":"2024-03-11T22:09:56.515498+0000","last_undegraded":"2024-03-11T22:09:56.515498+0000","last_fullsized":"2024-03-11T22:09:56.515498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:03.602554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":81,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:00.178068+0000","last_change":"2024-03-11T22:10:00.178117+0000","last_active":"2024-03-11T22:10:00.178068+0000","last_peered":"2024-03-11T22:10:00.178068+0000","last_clean":"2024-03-11T22:10:00.178068+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:10:00.178068+0000","last_undegraded":"2024-03-11T22:10:00.178068+0000","last_fullsized":"2024-03-11T22:10:00.178068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:10:00.177978+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:18.053454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135683+0000","last_change":"2024-03-11T22:09:55.458484+0000","last_active":"2024-03-11T22:09:56.135683+0000","last_peered":"2024-03-11T22:09:56.135683+0000","last_clean":"2024-03-11T22:09:56.135683+0000","last_became_active":"2024-03-11T22:08:33.035469+0000","last_became_peered":"2024-03-11T22:08:33.035469+0000","last_unstale":"2024-03-11T22:09:56.135683+0000","last_undegraded":"2024-03-11T22:09:56.135683+0000","last_fullsized":"2024-03-11T22:09:56.135683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:55.458385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:55.458385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:19.365475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":132,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:58.352357+0000","last_change":"2024-03-11T22:09:47.779316+0000","last_active":"2024-03-11T22:09:58.352357+0000","last_peered":"2024-03-11T22:09:58.352357+0000","last_clean":"2024-03-11T22:09:58.352357+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:09:58.352357+0000","last_undegraded":"2024-03-11T22:09:58.352357+0000","last_fullsized":"2024-03-11T22:09:58.352357+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-11T22:09:47.779209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:47.779209+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:10.865609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515786+0000","last_change":"2024-03-11T22:09:44.165077+0000","last_active":"2024-03-11T22:09:56.515786+0000","last_peered":"2024-03-11T22:09:56.515786+0000","last_clean":"2024-03-11T22:09:56.515786+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:09:56.515786+0000","last_undegraded":"2024-03-11T22:09:56.515786+0000","last_fullsized":"2024-03-11T22:09:56.515786+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:44.164939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:44.164939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:55.636705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516702+0000","last_change":"2024-03-11T22:08:39.077855+0000","last_active":"2024-03-11T22:09:56.516702+0000","last_peered":"2024-03-11T22:09:56.516702+0000","last_clean":"2024-03-11T22:09:56.516702+0000","last_became_active":"2024-03-11T22:08:39.077566+0000","last_became_peered":"2024-03-11T22:08:39.077566+0000","last_unstale":"2024-03-11T22:09:56.516702+0000","last_undegraded":"2024-03-11T22:09:56.516702+0000","last_fullsized":"2024-03-11T22:09:56.516702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135278+0000","last_change":"2024-03-11T22:09:35.724312+0000","last_active":"2024-03-11T22:09:56.135278+0000","last_peered":"2024-03-11T22:09:56.135278+0000","last_clean":"2024-03-11T22:09:56.135278+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:09:56.135278+0000","last_undegraded":"2024-03-11T22:09:56.135278+0000","last_fullsized":"2024-03-11T22:09:56.135278+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:35.724224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:35.724224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:59.123602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:00.733314+0000","last_change":"2024-03-11T22:10:00.733336+0000","last_active":"2024-03-11T22:10:00.733314+0000","last_peered":"2024-03-11T22:10:00.733314+0000","last_clean":"2024-03-11T22:10:00.733314+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:10:00.733314+0000","last_undegraded":"2024-03-11T22:10:00.733314+0000","last_fullsized":"2024-03-11T22:10:00.733314+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:10:00.733239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:12.509827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":92,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:02.604303+0000","last_change":"2024-03-11T22:10:02.604337+0000","last_active":"2024-03-11T22:10:02.604303+0000","last_peered":"2024-03-11T22:10:02.604303+0000","last_clean":"2024-03-11T22:10:02.604303+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:10:02.604303+0000","last_undegraded":"2024-03-11T22:10:02.604303+0000","last_fullsized":"2024-03-11T22:10:02.604303+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-03-11T22:10:02.604241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:02.604241+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:06.403890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135749+0000","last_change":"2024-03-11T22:08:37.076581+0000","last_active":"2024-03-11T22:09:56.135749+0000","last_peered":"2024-03-11T22:09:56.135749+0000","last_clean":"2024-03-11T22:09:56.135749+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:09:56.135749+0000","last_undegraded":"2024-03-11T22:09:56.135749+0000","last_fullsized":"2024-03-11T22:09:56.135749+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:05.137609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516449+0000","last_change":"2024-03-11T22:08:39.073180+0000","last_active":"2024-03-11T22:09:56.516449+0000","last_peered":"2024-03-11T22:09:56.516449+0000","last_clean":"2024-03-11T22:09:56.516449+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:09:56.516449+0000","last_undegraded":"2024-03-11T22:09:56.516449+0000","last_fullsized":"2024-03-11T22:09:56.516449+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":324,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:59.243230+0000","last_change":"2024-03-11T22:09:59.243266+0000","last_active":"2024-03-11T22:09:59.243230+0000","last_peered":"2024-03-11T22:09:59.243230+0000","last_clean":"2024-03-11T22:09:59.243230+0000","last_became_active":"2024-03-11T22:08:22.015394+0000","last_became_peered":"2024-03-11T22:08:22.015394+0000","last_unstale":"2024-03-11T22:09:59.243230+0000","last_undegraded":"2024-03-11T22:09:59.243230+0000","last_fullsized":"2024-03-11T22:09:59.243230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:09:59.243159+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:59.365375+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515477+0000","last_change":"2024-03-11T22:09:41.615925+0000","last_active":"2024-03-11T22:09:56.515477+0000","last_peered":"2024-03-11T22:09:56.515477+0000","last_clean":"2024-03-11T22:09:56.515477+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:09:56.515477+0000","last_undegraded":"2024-03-11T22:09:56.515477+0000","last_fullsized":"2024-03-11T22:09:56.515477+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:41.615829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:41.615829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:04.514072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":98,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:58.352716+0000","last_change":"2024-03-11T22:08:35.086266+0000","last_active":"2024-03-11T22:09:58.352716+0000","last_peered":"2024-03-11T22:09:58.352716+0000","last_clean":"2024-03-11T22:09:58.352716+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:09:58.352716+0000","last_undegraded":"2024-03-11T22:09:58.352716+0000","last_fullsized":"2024-03-11T22:09:58.352716+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516368+0000","last_change":"2024-03-11T22:08:37.076167+0000","last_active":"2024-03-11T22:09:56.516368+0000","last_peered":"2024-03-11T22:09:56.516368+0000","last_clean":"2024-03-11T22:09:56.516368+0000","last_became_active":"2024-03-11T22:08:37.075876+0000","last_became_peered":"2024-03-11T22:08:37.075876+0000","last_unstale":"2024-03-11T22:09:56.516368+0000","last_undegraded":"2024-03-11T22:09:56.516368+0000","last_fullsized":"2024-03-11T22:09:56.516368+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.136323+0000","last_change":"2024-03-11T22:09:53.690828+0000","last_active":"2024-03-11T22:09:56.136323+0000","last_peered":"2024-03-11T22:09:56.136323+0000","last_clean":"2024-03-11T22:09:56.136323+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:09:56.136323+0000","last_undegraded":"2024-03-11T22:09:56.136323+0000","last_fullsized":"2024-03-11T22:09:56.136323+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:53.690738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:53.690738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:11.065631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515630+0000","last_change":"2024-03-11T22:08:33.035933+0000","last_active":"2024-03-11T22:09:56.515630+0000","last_peered":"2024-03-11T22:09:56.515630+0000","last_clean":"2024-03-11T22:09:56.515630+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:09:56.515630+0000","last_undegraded":"2024-03-11T22:09:56.515630+0000","last_fullsized":"2024-03-11T22:09:56.515630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"76'5","reported_seq":158,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:03.697352+0000","last_change":"2024-03-11T22:08:36.060461+0000","last_active":"2024-03-11T22:10:03.697352+0000","last_peered":"2024-03-11T22:10:03.697352+0000","last_clean":"2024-03-11T22:10:03.697352+0000","last_became_active":"2024-03-11T22:08:32.033652+0000","last_became_peered":"2024-03-11T22:08:32.033652+0000","last_unstale":"2024-03-11T22:10:03.697352+0000","last_undegraded":"2024-03-11T22:10:03.697352+0000","last_fullsized":"2024-03-11T22:10:03.697352+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_clean_scrub_stamp":"2024-03-11T22:08:31.015772+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":80,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516012+0000","last_change":"2024-03-11T22:09:51.213643+0000","last_active":"2024-03-11T22:09:56.516012+0000","last_peered":"2024-03-11T22:09:56.516012+0000","last_clean":"2024-03-11T22:09:56.516012+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:09:56.516012+0000","last_undegraded":"2024-03-11T22:09:56.516012+0000","last_fullsized":"2024-03-11T22:09:56.516012+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:09:51.213532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:51.213532+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:51.292156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516794+0000","last_change":"2024-03-11T22:08:37.082329+0000","last_active":"2024-03-11T22:09:56.516794+0000","last_peered":"2024-03-11T22:09:56.516794+0000","last_clean":"2024-03-11T22:09:56.516794+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:09:56.516794+0000","last_undegraded":"2024-03-11T22:09:56.516794+0000","last_fullsized":"2024-03-11T22:09:56.516794+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:04.541129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135570+0000","last_change":"2024-03-11T22:09:54.681445+0000","last_active":"2024-03-11T22:09:56.135570+0000","last_peered":"2024-03-11T22:09:56.135570+0000","last_clean":"2024-03-11T22:09:56.135570+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:09:56.135570+0000","last_undegraded":"2024-03-11T22:09:56.135570+0000","last_fullsized":"2024-03-11T22:09:56.135570+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:54.681338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:54.681338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:22.214871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.136460+0000","last_change":"2024-03-11T22:08:33.298533+0000","last_active":"2024-03-11T22:09:56.136460+0000","last_peered":"2024-03-11T22:09:56.136460+0000","last_clean":"2024-03-11T22:09:56.136460+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:09:56.136460+0000","last_undegraded":"2024-03-11T22:09:56.136460+0000","last_fullsized":"2024-03-11T22:09:56.136460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:00.310011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":93,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516538+0000","last_change":"2024-03-11T22:08:35.087309+0000","last_active":"2024-03-11T22:09:56.516538+0000","last_peered":"2024-03-11T22:09:56.516538+0000","last_clean":"2024-03-11T22:09:56.516538+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:09:56.516538+0000","last_undegraded":"2024-03-11T22:09:56.516538+0000","last_fullsized":"2024-03-11T22:09:56.516538+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:02.382775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516924+0000","last_change":"2024-03-11T22:08:37.074393+0000","last_active":"2024-03-11T22:09:56.516924+0000","last_peered":"2024-03-11T22:09:56.516924+0000","last_clean":"2024-03-11T22:09:56.516924+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:09:56.516924+0000","last_undegraded":"2024-03-11T22:09:56.516924+0000","last_fullsized":"2024-03-11T22:09:56.516924+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:02.379574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.514991+0000","last_change":"2024-03-11T22:08:39.085179+0000","last_active":"2024-03-11T22:09:56.514991+0000","last_peered":"2024-03-11T22:09:56.514991+0000","last_clean":"2024-03-11T22:09:56.514991+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:09:56.514991+0000","last_undegraded":"2024-03-11T22:09:56.514991+0000","last_fullsized":"2024-03-11T22:09:56.514991+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:05.345954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515905+0000","last_change":"2024-03-11T22:09:43.271410+0000","last_active":"2024-03-11T22:09:56.515905+0000","last_peered":"2024-03-11T22:09:56.515905+0000","last_clean":"2024-03-11T22:09:56.515905+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:09:56.515905+0000","last_undegraded":"2024-03-11T22:09:56.515905+0000","last_fullsized":"2024-03-11T22:09:56.515905+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:43.271304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:43.271304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:50.753986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.136164+0000","last_change":"2024-03-11T22:08:35.058060+0000","last_active":"2024-03-11T22:09:56.136164+0000","last_peered":"2024-03-11T22:09:56.136164+0000","last_clean":"2024-03-11T22:09:56.136164+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:09:56.136164+0000","last_undegraded":"2024-03-11T22:09:56.136164+0000","last_fullsized":"2024-03-11T22:09:56.136164+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.703549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":88,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:58.352357+0000","last_change":"2024-03-11T22:09:48.187735+0000","last_active":"2024-03-11T22:09:58.352357+0000","last_peered":"2024-03-11T22:09:58.352357+0000","last_clean":"2024-03-11T22:09:58.352357+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:09:58.352357+0000","last_undegraded":"2024-03-11T22:09:58.352357+0000","last_fullsized":"2024-03-11T22:09:58.352357+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:09:48.187641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:48.187641+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:56.947907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:03.279179+0000","last_change":"2024-03-11T22:10:03.279201+0000","last_active":"2024-03-11T22:10:03.279179+0000","last_peered":"2024-03-11T22:10:03.279179+0000","last_clean":"2024-03-11T22:10:03.279179+0000","last_became_active":"2024-03-11T22:08:39.076114+0000","last_became_peered":"2024-03-11T22:08:39.076114+0000","last_unstale":"2024-03-11T22:10:03.279179+0000","last_undegraded":"2024-03-11T22:10:03.279179+0000","last_fullsized":"2024-03-11T22:10:03.279179+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.279112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:24.270748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:03.109999+0000","last_change":"2024-03-11T22:10:03.110027+0000","last_active":"2024-03-11T22:10:03.109999+0000","last_peered":"2024-03-11T22:10:03.109999+0000","last_clean":"2024-03-11T22:10:03.109999+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:10:03.109999+0000","last_undegraded":"2024-03-11T22:10:03.109999+0000","last_fullsized":"2024-03-11T22:10:03.109999+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:10:03.109931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.109931+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:28.184077+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":86,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135438+0000","last_change":"2024-03-11T22:09:53.414248+0000","last_active":"2024-03-11T22:09:56.135438+0000","last_peered":"2024-03-11T22:09:56.135438+0000","last_clean":"2024-03-11T22:09:56.135438+0000","last_became_active":"2024-03-11T22:08:35.053452+0000","last_became_peered":"2024-03-11T22:08:35.053452+0000","last_unstale":"2024-03-11T22:09:56.135438+0000","last_undegraded":"2024-03-11T22:09:56.135438+0000","last_fullsized":"2024-03-11T22:09:56.135438+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:09:53.414127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:53.414127+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:17.661472+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":80,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:58.352378+0000","last_change":"2024-03-11T22:08:37.078918+0000","last_active":"2024-03-11T22:09:58.352378+0000","last_peered":"2024-03-11T22:09:58.352378+0000","last_clean":"2024-03-11T22:09:58.352378+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:09:58.352378+0000","last_undegraded":"2024-03-11T22:09:58.352378+0000","last_fullsized":"2024-03-11T22:09:58.352378+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135498+0000","last_change":"2024-03-11T22:08:39.078715+0000","last_active":"2024-03-11T22:09:56.135498+0000","last_peered":"2024-03-11T22:09:56.135498+0000","last_clean":"2024-03-11T22:09:56.135498+0000","last_became_active":"2024-03-11T22:08:39.078244+0000","last_became_peered":"2024-03-11T22:08:39.078244+0000","last_unstale":"2024-03-11T22:09:56.135498+0000","last_undegraded":"2024-03-11T22:09:56.135498+0000","last_fullsized":"2024-03-11T22:09:56.135498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:07.090388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515722+0000","last_change":"2024-03-11T22:09:36.565005+0000","last_active":"2024-03-11T22:09:56.515722+0000","last_peered":"2024-03-11T22:09:56.515722+0000","last_clean":"2024-03-11T22:09:56.515722+0000","last_became_active":"2024-03-11T22:08:33.036469+0000","last_became_peered":"2024-03-11T22:08:33.036469+0000","last_unstale":"2024-03-11T22:09:56.515722+0000","last_undegraded":"2024-03-11T22:09:56.515722+0000","last_fullsized":"2024-03-11T22:09:56.515722+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:36.564912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:36.564912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:49.963865+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":89,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515818+0000","last_change":"2024-03-11T22:08:35.086576+0000","last_active":"2024-03-11T22:09:56.515818+0000","last_peered":"2024-03-11T22:09:56.515818+0000","last_clean":"2024-03-11T22:09:56.515818+0000","last_became_active":"2024-03-11T22:08:35.085979+0000","last_became_peered":"2024-03-11T22:08:35.085979+0000","last_unstale":"2024-03-11T22:09:56.515818+0000","last_undegraded":"2024-03-11T22:09:56.515818+0000","last_fullsized":"2024-03-11T22:09:56.515818+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135250+0000","last_change":"2024-03-11T22:09:54.458851+0000","last_active":"2024-03-11T22:09:56.135250+0000","last_peered":"2024-03-11T22:09:56.135250+0000","last_clean":"2024-03-11T22:09:56.135250+0000","last_became_active":"2024-03-11T22:08:37.076522+0000","last_became_peered":"2024-03-11T22:08:37.076522+0000","last_unstale":"2024-03-11T22:09:56.135250+0000","last_undegraded":"2024-03-11T22:09:56.135250+0000","last_fullsized":"2024-03-11T22:09:56.135250+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:54.458752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:54.458752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.358836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515552+0000","last_change":"2024-03-11T22:08:39.080281+0000","last_active":"2024-03-11T22:09:56.515552+0000","last_peered":"2024-03-11T22:09:56.515552+0000","last_clean":"2024-03-11T22:09:56.515552+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:09:56.515552+0000","last_undegraded":"2024-03-11T22:09:56.515552+0000","last_fullsized":"2024-03-11T22:09:56.515552+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:57.517616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":80,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:58.352636+0000","last_change":"2024-03-11T22:08:37.077452+0000","last_active":"2024-03-11T22:09:58.352636+0000","last_peered":"2024-03-11T22:09:58.352636+0000","last_clean":"2024-03-11T22:09:58.352636+0000","last_became_active":"2024-03-11T22:08:37.077142+0000","last_became_peered":"2024-03-11T22:08:37.077142+0000","last_unstale":"2024-03-11T22:09:58.352636+0000","last_undegraded":"2024-03-11T22:09:58.352636+0000","last_fullsized":"2024-03-11T22:09:58.352636+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515234+0000","last_change":"2024-03-11T22:08:33.037839+0000","last_active":"2024-03-11T22:09:56.515234+0000","last_peered":"2024-03-11T22:09:56.515234+0000","last_clean":"2024-03-11T22:09:56.515234+0000","last_became_active":"2024-03-11T22:08:33.037537+0000","last_became_peered":"2024-03-11T22:08:33.037537+0000","last_unstale":"2024-03-11T22:09:56.515234+0000","last_undegraded":"2024-03-11T22:09:56.515234+0000","last_fullsized":"2024-03-11T22:09:56.515234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":82,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.136278+0000","last_change":"2024-03-11T22:08:35.060718+0000","last_active":"2024-03-11T22:09:56.136278+0000","last_peered":"2024-03-11T22:09:56.136278+0000","last_clean":"2024-03-11T22:09:56.136278+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:09:56.136278+0000","last_undegraded":"2024-03-11T22:09:56.136278+0000","last_fullsized":"2024-03-11T22:09:56.136278+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.515538+0000","last_change":"2024-03-11T22:08:39.077317+0000","last_active":"2024-03-11T22:09:56.515538+0000","last_peered":"2024-03-11T22:09:56.515538+0000","last_clean":"2024-03-11T22:09:56.515538+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:09:56.515538+0000","last_undegraded":"2024-03-11T22:09:56.515538+0000","last_fullsized":"2024-03-11T22:09:56.515538+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:56.786808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.516686+0000","last_change":"2024-03-11T22:08:37.082228+0000","last_active":"2024-03-11T22:09:56.516686+0000","last_peered":"2024-03-11T22:09:56.516686+0000","last_clean":"2024-03-11T22:09:56.516686+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:09:56.516686+0000","last_undegraded":"2024-03-11T22:09:56.516686+0000","last_fullsized":"2024-03-11T22:09:56.516686+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135316+0000","last_change":"2024-03-11T22:08:33.300125+0000","last_active":"2024-03-11T22:09:56.135316+0000","last_peered":"2024-03-11T22:09:56.135316+0000","last_clean":"2024-03-11T22:09:56.135316+0000","last_became_active":"2024-03-11T22:08:33.299778+0000","last_became_peered":"2024-03-11T22:08:33.299778+0000","last_unstale":"2024-03-11T22:09:56.135316+0000","last_undegraded":"2024-03-11T22:09:56.135316+0000","last_fullsized":"2024-03-11T22:09:56.135316+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:09:58.833176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.136167+0000","last_change":"2024-03-11T22:09:40.448862+0000","last_active":"2024-03-11T22:09:56.136167+0000","last_peered":"2024-03-11T22:09:56.136167+0000","last_clean":"2024-03-11T22:09:56.136167+0000","last_became_active":"2024-03-11T22:08:35.053123+0000","last_became_peered":"2024-03-11T22:08:35.053123+0000","last_unstale":"2024-03-11T22:09:56.136167+0000","last_undegraded":"2024-03-11T22:09:56.136167+0000","last_fullsized":"2024-03-11T22:09:56.136167+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:09:40.448751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:40.448751+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:07.077279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:10:04.693919+0000","last_change":"2024-03-11T22:10:04.693960+0000","last_active":"2024-03-11T22:10:04.693919+0000","last_peered":"2024-03-11T22:10:04.693919+0000","last_clean":"2024-03-11T22:10:04.693919+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:10:04.693919+0000","last_undegraded":"2024-03-11T22:10:04.693919+0000","last_fullsized":"2024-03-11T22:10:04.693919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:04.693849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:04.693849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:24.922133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.136621+0000","last_change":"2024-03-11T22:08:37.078886+0000","last_active":"2024-03-11T22:09:56.136621+0000","last_peered":"2024-03-11T22:09:56.136621+0000","last_clean":"2024-03-11T22:09:56.136621+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:09:56.136621+0000","last_undegraded":"2024-03-11T22:09:56.136621+0000","last_fullsized":"2024-03-11T22:09:56.136621+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.517011+0000","last_change":"2024-03-11T22:09:46.542145+0000","last_active":"2024-03-11T22:09:56.517011+0000","last_peered":"2024-03-11T22:09:56.517011+0000","last_clean":"2024-03-11T22:09:56.517011+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:09:56.517011+0000","last_undegraded":"2024-03-11T22:09:56.517011+0000","last_fullsized":"2024-03-11T22:09:56.517011+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:46.542054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:46.542054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.271294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.136229+0000","last_change":"2024-03-11T22:08:35.055974+0000","last_active":"2024-03-11T22:09:56.136229+0000","last_peered":"2024-03-11T22:09:56.136229+0000","last_clean":"2024-03-11T22:09:56.136229+0000","last_became_active":"2024-03-11T22:08:35.055770+0000","last_became_peered":"2024-03-11T22:08:35.055770+0000","last_unstale":"2024-03-11T22:09:56.136229+0000","last_undegraded":"2024-03-11T22:09:56.136229+0000","last_fullsized":"2024-03-11T22:09:56.136229+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.136376+0000","last_change":"2024-03-11T22:08:39.073197+0000","last_active":"2024-03-11T22:09:56.136376+0000","last_peered":"2024-03-11T22:09:56.136376+0000","last_clean":"2024-03-11T22:09:56.136376+0000","last_became_active":"2024-03-11T22:08:39.072422+0000","last_became_peered":"2024-03-11T22:08:39.072422+0000","last_unstale":"2024-03-11T22:09:56.136376+0000","last_undegraded":"2024-03-11T22:09:56.136376+0000","last_fullsized":"2024-03-11T22:09:56.136376+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":89,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:58.352379+0000","last_change":"2024-03-11T22:09:38.769937+0000","last_active":"2024-03-11T22:09:58.352379+0000","last_peered":"2024-03-11T22:09:58.352379+0000","last_clean":"2024-03-11T22:09:58.352379+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:09:58.352379+0000","last_undegraded":"2024-03-11T22:09:58.352379+0000","last_fullsized":"2024-03-11T22:09:58.352379+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_clean_scrub_stamp":"2024-03-11T22:09:38.769825+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:58.315026+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":69,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:58.618325+0000","last_change":"2024-03-11T22:09:58.618349+0000","last_active":"2024-03-11T22:09:58.618325+0000","last_peered":"2024-03-11T22:09:58.618325+0000","last_clean":"2024-03-11T22:09:58.618325+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:09:58.618325+0000","last_undegraded":"2024-03-11T22:09:58.618325+0000","last_fullsized":"2024-03-11T22:09:58.618325+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:58.618258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:58.618258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:08.349075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-11T22:09:56.135858+0000","last_change":"2024-03-11T22:08:35.057039+0000","last_active":"2024-03-11T22:09:56.135858+0000","last_peered":"2024-03-11T22:09:56.135858+0000","last_clean":"2024-03-11T22:09:56.135858+0000","last_became_active":"2024-03-11T22:08:35.056698+0000","last_became_peered":"2024-03-11T22:08:35.056698+0000","last_unstale":"2024-03-11T22:09:56.135858+0000","last_undegraded":"2024-03-11T22:09:56.135858+0000","last_fullsized":"2024-03-11T22:09:56.135858+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":80,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005079,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28308,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716844,"statfs":{"total":95995035648,"available":95966048256,"internally_reserved":0,"allocated":1323008,"data_stored":934870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332124,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28288,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716864,"statfs":{"total":95995035648,"available":95966068736,"internally_reserved":0,"allocated":1302528,"data_stored":934680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":757760,"data_stored":344508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32464,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712688,"statfs":{"total":95995035648,"available":95961792512,"internally_reserved":0,"allocated":1318912,"data_stored":934202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31864,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713288,"statfs":{"total":95995035648,"available":95962406912,"internally_reserved":0,"allocated":704512,"data_stored":344138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27744,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717408,"statfs":{"total":95995035648,"available":95966625792,"internally_reserved":0,"allocated":745472,"data_stored":344818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920807,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":733184,"data_stored":344043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659168,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27712,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717440,"statfs":{"total":95995035648,"available":95966658560,"internally_reserved":0,"allocated":712704,"data_stored":343585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:10:11.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:10:11.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[30178]: 4.f scrub starts 2024-03-11T22:10:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[30178]: 4.f scrub ok 2024-03-11T22:10:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[30178]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:10:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[30178]: 3.1 scrub starts 2024-03-11T22:10:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[30178]: 3.1 scrub ok 2024-03-11T22:10:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[30178]: 5.4 deep-scrub starts 2024-03-11T22:10:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[30178]: 5.4 deep-scrub ok 2024-03-11T22:10:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[30178]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:10:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[30178]: 4.5 scrub starts 2024-03-11T22:10:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[30178]: 4.5 scrub ok 2024-03-11T22:10:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[30178]: 6.13 scrub starts 2024-03-11T22:10:11.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[30178]: 6.13 scrub ok 2024-03-11T22:10:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[39715]: 4.f scrub starts 2024-03-11T22:10:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[39715]: 4.f scrub ok 2024-03-11T22:10:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[39715]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:10:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[39715]: 3.1 scrub starts 2024-03-11T22:10:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[39715]: 3.1 scrub ok 2024-03-11T22:10:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[39715]: 5.4 deep-scrub starts 2024-03-11T22:10:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[39715]: 5.4 deep-scrub ok 2024-03-11T22:10:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[39715]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:10:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[39715]: 4.5 scrub starts 2024-03-11T22:10:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[39715]: 4.5 scrub ok 2024-03-11T22:10:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[39715]: 6.13 scrub starts 2024-03-11T22:10:11.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-mon[39715]: 6.13 scrub ok 2024-03-11T22:10:11.286 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:11 smithi144 ceph-mon[33984]: 4.f scrub starts 2024-03-11T22:10:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:11 smithi144 ceph-mon[33984]: 4.f scrub ok 2024-03-11T22:10:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:11 smithi144 ceph-mon[33984]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:10:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:11 smithi144 ceph-mon[33984]: 3.1 scrub starts 2024-03-11T22:10:11.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:11 smithi144 ceph-mon[33984]: 3.1 scrub ok 2024-03-11T22:10:11.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:11 smithi144 ceph-mon[33984]: 5.4 deep-scrub starts 2024-03-11T22:10:11.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:11 smithi144 ceph-mon[33984]: 5.4 deep-scrub ok 2024-03-11T22:10:11.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:11 smithi144 ceph-mon[33984]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:10:11.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:11 smithi144 ceph-mon[33984]: 4.5 scrub starts 2024-03-11T22:10:11.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:11 smithi144 ceph-mon[33984]: 4.5 scrub ok 2024-03-11T22:10:11.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:11 smithi144 ceph-mon[33984]: 6.13 scrub starts 2024-03-11T22:10:11.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:11 smithi144 ceph-mon[33984]: 6.13 scrub ok 2024-03-11T22:10:12.025 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:10:11 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[77897]: ts=2024-03-11T22:10:11.854Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001205472s 2024-03-11T22:10:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[39715]: 5.6 scrub starts 2024-03-11T22:10:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[39715]: 5.6 scrub ok 2024-03-11T22:10:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[39715]: 3.1c scrub starts 2024-03-11T22:10:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[39715]: 3.1c scrub ok 2024-03-11T22:10:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[39715]: 5.1e scrub starts 2024-03-11T22:10:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[39715]: 5.1e scrub ok 2024-03-11T22:10:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[39715]: 5.9 scrub starts 2024-03-11T22:10:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[39715]: 5.9 scrub ok 2024-03-11T22:10:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[39715]: 5.0 scrub starts 2024-03-11T22:10:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[39715]: 5.0 scrub ok 2024-03-11T22:10:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[30178]: 5.6 scrub starts 2024-03-11T22:10:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[30178]: 5.6 scrub ok 2024-03-11T22:10:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[30178]: 3.1c scrub starts 2024-03-11T22:10:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[30178]: 3.1c scrub ok 2024-03-11T22:10:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[30178]: 5.1e scrub starts 2024-03-11T22:10:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[30178]: 5.1e scrub ok 2024-03-11T22:10:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[30178]: 5.9 scrub starts 2024-03-11T22:10:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[30178]: 5.9 scrub ok 2024-03-11T22:10:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[30178]: 5.0 scrub starts 2024-03-11T22:10:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-mon[30178]: 5.0 scrub ok 2024-03-11T22:10:12.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:12 smithi144 ceph-mon[33984]: 5.6 scrub starts 2024-03-11T22:10:12.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:12 smithi144 ceph-mon[33984]: 5.6 scrub ok 2024-03-11T22:10:12.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:12 smithi144 ceph-mon[33984]: 3.1c scrub starts 2024-03-11T22:10:12.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:12 smithi144 ceph-mon[33984]: 3.1c scrub ok 2024-03-11T22:10:12.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:12 smithi144 ceph-mon[33984]: 5.1e scrub starts 2024-03-11T22:10:12.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:12 smithi144 ceph-mon[33984]: 5.1e scrub ok 2024-03-11T22:10:12.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:12 smithi144 ceph-mon[33984]: 5.9 scrub starts 2024-03-11T22:10:12.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:12 smithi144 ceph-mon[33984]: 5.9 scrub ok 2024-03-11T22:10:12.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:12 smithi144 ceph-mon[33984]: 5.0 scrub starts 2024-03-11T22:10:12.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:12 smithi144 ceph-mon[33984]: 5.0 scrub ok 2024-03-11T22:10:12.829 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:10:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:10:13.410 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:10:13.411 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[30178]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:10:13.411 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[30178]: 3.c scrub starts 2024-03-11T22:10:13.411 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[30178]: 3.c scrub ok 2024-03-11T22:10:13.411 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[30178]: 3.11 scrub starts 2024-03-11T22:10:13.411 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[30178]: 3.11 scrub ok 2024-03-11T22:10:13.411 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[30178]: 3.8 scrub starts 2024-03-11T22:10:13.411 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[30178]: 3.8 scrub ok 2024-03-11T22:10:13.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[39715]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:10:13.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[39715]: 3.c scrub starts 2024-03-11T22:10:13.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[39715]: 3.c scrub ok 2024-03-11T22:10:13.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[39715]: 3.11 scrub starts 2024-03-11T22:10:13.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[39715]: 3.11 scrub ok 2024-03-11T22:10:13.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[39715]: 3.8 scrub starts 2024-03-11T22:10:13.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:13 smithi045 ceph-mon[39715]: 3.8 scrub ok 2024-03-11T22:10:13.418 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:10:13.442 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:13 smithi144 ceph-mon[33984]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:10:13.442 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:13 smithi144 ceph-mon[33984]: 3.c scrub starts 2024-03-11T22:10:13.442 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:13 smithi144 ceph-mon[33984]: 3.c scrub ok 2024-03-11T22:10:13.443 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:13 smithi144 ceph-mon[33984]: 3.11 scrub starts 2024-03-11T22:10:13.443 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:13 smithi144 ceph-mon[33984]: 3.11 scrub ok 2024-03-11T22:10:13.443 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:13 smithi144 ceph-mon[33984]: 3.8 scrub starts 2024-03-11T22:10:13.443 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:13 smithi144 ceph-mon[33984]: 3.8 scrub ok 2024-03-11T22:10:14.176 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-03-11T22:10:12.049708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1024,"num_read_kb":870,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":615,"ondisk_log_size":615,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240368,"kb_used_data":7564,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720848,"statfs":{"total":767960285184,"available":767714148352,"internally_reserved":0,"allocated":7745536,"data_stored":4617940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":238276147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001858"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418311+0000","last_change":"2024-03-11T22:08:39.077833+0000","last_active":"2024-03-11T22:10:09.418311+0000","last_peered":"2024-03-11T22:10:09.418311+0000","last_clean":"2024-03-11T22:10:09.418311+0000","last_became_active":"2024-03-11T22:08:39.074695+0000","last_became_peered":"2024-03-11T22:08:39.074695+0000","last_unstale":"2024-03-11T22:10:09.418311+0000","last_undegraded":"2024-03-11T22:10:09.418311+0000","last_fullsized":"2024-03-11T22:10:09.418311+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.316423+0000","last_change":"2024-03-11T22:08:37.074138+0000","last_active":"2024-03-11T22:10:09.316423+0000","last_peered":"2024-03-11T22:10:09.316423+0000","last_clean":"2024-03-11T22:10:09.316423+0000","last_became_active":"2024-03-11T22:08:37.073775+0000","last_became_peered":"2024-03-11T22:08:37.073775+0000","last_unstale":"2024-03-11T22:10:09.316423+0000","last_undegraded":"2024-03-11T22:10:09.316423+0000","last_fullsized":"2024-03-11T22:10:09.316423+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.180091+0000","last_change":"2024-03-11T22:09:55.162082+0000","last_active":"2024-03-11T22:10:09.180091+0000","last_peered":"2024-03-11T22:10:09.180091+0000","last_clean":"2024-03-11T22:10:09.180091+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:10:09.180091+0000","last_undegraded":"2024-03-11T22:10:09.180091+0000","last_fullsized":"2024-03-11T22:10:09.180091+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:55.161978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:55.161978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:14.539455+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":77,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418207+0000","last_change":"2024-03-11T22:08:35.053644+0000","last_active":"2024-03-11T22:10:09.418207+0000","last_peered":"2024-03-11T22:10:09.418207+0000","last_clean":"2024-03-11T22:10:09.418207+0000","last_became_active":"2024-03-11T22:08:35.053179+0000","last_became_peered":"2024-03-11T22:08:35.053179+0000","last_unstale":"2024-03-11T22:10:09.418207+0000","last_undegraded":"2024-03-11T22:10:09.418207+0000","last_fullsized":"2024-03-11T22:10:09.418207+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276355+0000","last_change":"2024-03-11T22:08:39.079762+0000","last_active":"2024-03-11T22:10:09.276355+0000","last_peered":"2024-03-11T22:10:09.276355+0000","last_clean":"2024-03-11T22:10:09.276355+0000","last_became_active":"2024-03-11T22:08:39.079045+0000","last_became_peered":"2024-03-11T22:08:39.079045+0000","last_unstale":"2024-03-11T22:10:09.276355+0000","last_undegraded":"2024-03-11T22:10:09.276355+0000","last_fullsized":"2024-03-11T22:10:09.276355+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418233+0000","last_change":"2024-03-11T22:10:04.454026+0000","last_active":"2024-03-11T22:10:09.418233+0000","last_peered":"2024-03-11T22:10:09.418233+0000","last_clean":"2024-03-11T22:10:09.418233+0000","last_became_active":"2024-03-11T22:08:37.081861+0000","last_became_peered":"2024-03-11T22:08:37.081861+0000","last_unstale":"2024-03-11T22:10:09.418233+0000","last_undegraded":"2024-03-11T22:10:09.418233+0000","last_fullsized":"2024-03-11T22:10:09.418233+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:04.453912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:10:04.453912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:04.768471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":79,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276784+0000","last_change":"2024-03-11T22:09:39.608984+0000","last_active":"2024-03-11T22:10:09.276784+0000","last_peered":"2024-03-11T22:10:09.276784+0000","last_clean":"2024-03-11T22:10:09.276784+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:10:09.276784+0000","last_undegraded":"2024-03-11T22:10:09.276784+0000","last_fullsized":"2024-03-11T22:10:09.276784+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:09:39.608890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:39.608890+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:49.086609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":86,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276627+0000","last_change":"2024-03-11T22:08:35.061181+0000","last_active":"2024-03-11T22:10:09.276627+0000","last_peered":"2024-03-11T22:10:09.276627+0000","last_clean":"2024-03-11T22:10:09.276627+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:10:09.276627+0000","last_undegraded":"2024-03-11T22:10:09.276627+0000","last_fullsized":"2024-03-11T22:10:09.276627+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178768+0000","last_change":"2024-03-11T22:08:39.083128+0000","last_active":"2024-03-11T22:10:09.178768+0000","last_peered":"2024-03-11T22:10:09.178768+0000","last_clean":"2024-03-11T22:10:09.178768+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:10:09.178768+0000","last_undegraded":"2024-03-11T22:10:09.178768+0000","last_fullsized":"2024-03-11T22:10:09.178768+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484311+0000","last_change":"2024-03-11T22:08:37.082294+0000","last_active":"2024-03-11T22:10:09.484311+0000","last_peered":"2024-03-11T22:10:09.484311+0000","last_clean":"2024-03-11T22:10:09.484311+0000","last_became_active":"2024-03-11T22:08:37.081996+0000","last_became_peered":"2024-03-11T22:08:37.081996+0000","last_unstale":"2024-03-11T22:10:09.484311+0000","last_undegraded":"2024-03-11T22:10:09.484311+0000","last_fullsized":"2024-03-11T22:10:09.484311+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484521+0000","last_change":"2024-03-11T22:09:40.301289+0000","last_active":"2024-03-11T22:10:09.484521+0000","last_peered":"2024-03-11T22:10:09.484521+0000","last_clean":"2024-03-11T22:10:09.484521+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:10:09.484521+0000","last_undegraded":"2024-03-11T22:10:09.484521+0000","last_fullsized":"2024-03-11T22:10:09.484521+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:40.301162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:40.301162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:40.904990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":85,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418439+0000","last_change":"2024-03-11T22:10:01.441822+0000","last_active":"2024-03-11T22:10:09.418439+0000","last_peered":"2024-03-11T22:10:09.418439+0000","last_clean":"2024-03-11T22:10:09.418439+0000","last_became_active":"2024-03-11T22:08:35.060802+0000","last_became_peered":"2024-03-11T22:08:35.060802+0000","last_unstale":"2024-03-11T22:10:09.418439+0000","last_undegraded":"2024-03-11T22:10:09.418439+0000","last_fullsized":"2024-03-11T22:10:09.418439+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:10:01.441711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:01.441711+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:04.897877+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":61,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484152+0000","last_change":"2024-03-11T22:10:06.182839+0000","last_active":"2024-03-11T22:10:09.484152+0000","last_peered":"2024-03-11T22:10:09.484152+0000","last_clean":"2024-03-11T22:10:09.484152+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:10:09.484152+0000","last_undegraded":"2024-03-11T22:10:09.484152+0000","last_fullsized":"2024-03-11T22:10:09.484152+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:06.182757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:06.182757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:30.294631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.274184+0000","last_change":"2024-03-11T22:10:00.270261+0000","last_active":"2024-03-11T22:10:09.274184+0000","last_peered":"2024-03-11T22:10:09.274184+0000","last_clean":"2024-03-11T22:10:09.274184+0000","last_became_active":"2024-03-11T22:08:37.077279+0000","last_became_peered":"2024-03-11T22:08:37.077279+0000","last_unstale":"2024-03-11T22:10:09.274184+0000","last_undegraded":"2024-03-11T22:10:09.274184+0000","last_fullsized":"2024-03-11T22:10:09.274184+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:00.270166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.270166+0000","last_clean_scrub_stamp":"2024-03-11T22:10:00.270166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:16.941652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484414+0000","last_change":"2024-03-11T22:08:33.040601+0000","last_active":"2024-03-11T22:10:09.484414+0000","last_peered":"2024-03-11T22:10:09.484414+0000","last_clean":"2024-03-11T22:10:09.484414+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:10:09.484414+0000","last_undegraded":"2024-03-11T22:10:09.484414+0000","last_fullsized":"2024-03-11T22:10:09.484414+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":71,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418798+0000","last_change":"2024-03-11T22:08:35.061039+0000","last_active":"2024-03-11T22:10:09.418798+0000","last_peered":"2024-03-11T22:10:09.418798+0000","last_clean":"2024-03-11T22:10:09.418798+0000","last_became_active":"2024-03-11T22:08:35.060777+0000","last_became_peered":"2024-03-11T22:08:35.060777+0000","last_unstale":"2024-03-11T22:10:09.418798+0000","last_undegraded":"2024-03-11T22:10:09.418798+0000","last_fullsized":"2024-03-11T22:10:09.418798+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":61,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.419103+0000","last_change":"2024-03-11T22:09:51.367715+0000","last_active":"2024-03-11T22:10:09.419103+0000","last_peered":"2024-03-11T22:10:09.419103+0000","last_clean":"2024-03-11T22:10:09.419103+0000","last_became_active":"2024-03-11T22:08:39.075609+0000","last_became_peered":"2024-03-11T22:08:39.075609+0000","last_unstale":"2024-03-11T22:10:09.419103+0000","last_undegraded":"2024-03-11T22:10:09.419103+0000","last_fullsized":"2024-03-11T22:10:09.419103+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:51.367570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:51.367570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:10.802427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.316994+0000","last_change":"2024-03-11T22:08:37.077878+0000","last_active":"2024-03-11T22:10:09.316994+0000","last_peered":"2024-03-11T22:10:09.316994+0000","last_clean":"2024-03-11T22:10:09.316994+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:10:09.316994+0000","last_undegraded":"2024-03-11T22:10:09.316994+0000","last_fullsized":"2024-03-11T22:10:09.316994+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179673+0000","last_change":"2024-03-11T22:09:58.215572+0000","last_active":"2024-03-11T22:10:09.179673+0000","last_peered":"2024-03-11T22:10:09.179673+0000","last_clean":"2024-03-11T22:10:09.179673+0000","last_became_active":"2024-03-11T22:08:33.035624+0000","last_became_peered":"2024-03-11T22:08:33.035624+0000","last_unstale":"2024-03-11T22:10:09.179673+0000","last_undegraded":"2024-03-11T22:10:09.179673+0000","last_fullsized":"2024-03-11T22:10:09.179673+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:58.215488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:58.215488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:16.044592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":94,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178641+0000","last_change":"2024-03-11T22:09:45.731375+0000","last_active":"2024-03-11T22:10:09.178641+0000","last_peered":"2024-03-11T22:10:09.178641+0000","last_clean":"2024-03-11T22:10:09.178641+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:10:09.178641+0000","last_undegraded":"2024-03-11T22:10:09.178641+0000","last_fullsized":"2024-03-11T22:10:09.178641+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:09:45.731248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:45.731248+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:08.445140+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276825+0000","last_change":"2024-03-11T22:08:39.078269+0000","last_active":"2024-03-11T22:10:09.276825+0000","last_peered":"2024-03-11T22:10:09.276825+0000","last_clean":"2024-03-11T22:10:09.276825+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:10:09.276825+0000","last_undegraded":"2024-03-11T22:10:09.276825+0000","last_fullsized":"2024-03-11T22:10:09.276825+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418638+0000","last_change":"2024-03-11T22:08:37.080325+0000","last_active":"2024-03-11T22:10:09.418638+0000","last_peered":"2024-03-11T22:10:09.418638+0000","last_clean":"2024-03-11T22:10:09.418638+0000","last_became_active":"2024-03-11T22:08:37.080014+0000","last_became_peered":"2024-03-11T22:08:37.080014+0000","last_unstale":"2024-03-11T22:10:09.418638+0000","last_undegraded":"2024-03-11T22:10:09.418638+0000","last_fullsized":"2024-03-11T22:10:09.418638+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418068+0000","last_change":"2024-03-11T22:10:03.469439+0000","last_active":"2024-03-11T22:10:09.418068+0000","last_peered":"2024-03-11T22:10:09.418068+0000","last_clean":"2024-03-11T22:10:09.418068+0000","last_became_active":"2024-03-11T22:08:33.299189+0000","last_became_peered":"2024-03-11T22:08:33.299189+0000","last_unstale":"2024-03-11T22:10:09.418068+0000","last_undegraded":"2024-03-11T22:10:09.418068+0000","last_fullsized":"2024-03-11T22:10:09.418068+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:03.469346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.469346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:11.637778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":92,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276362+0000","last_change":"2024-03-11T22:10:04.596677+0000","last_active":"2024-03-11T22:10:09.276362+0000","last_peered":"2024-03-11T22:10:09.276362+0000","last_clean":"2024-03-11T22:10:09.276362+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:10:09.276362+0000","last_undegraded":"2024-03-11T22:10:09.276362+0000","last_fullsized":"2024-03-11T22:10:09.276362+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:10:04.596588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:04.596588+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:27.560340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":62,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.274094+0000","last_change":"2024-03-11T22:09:53.160256+0000","last_active":"2024-03-11T22:10:09.274094+0000","last_peered":"2024-03-11T22:10:09.274094+0000","last_clean":"2024-03-11T22:10:09.274094+0000","last_became_active":"2024-03-11T22:08:39.072956+0000","last_became_peered":"2024-03-11T22:08:39.072956+0000","last_unstale":"2024-03-11T22:10:09.274094+0000","last_undegraded":"2024-03-11T22:10:09.274094+0000","last_fullsized":"2024-03-11T22:10:09.274094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:09:53.160150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:53.160150+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:21.225798+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":96,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179383+0000","last_change":"2024-03-11T22:09:59.504774+0000","last_active":"2024-03-11T22:10:09.179383+0000","last_peered":"2024-03-11T22:10:09.179383+0000","last_clean":"2024-03-11T22:10:09.179383+0000","last_became_active":"2024-03-11T22:08:37.078588+0000","last_became_peered":"2024-03-11T22:08:37.078588+0000","last_unstale":"2024-03-11T22:10:09.179383+0000","last_undegraded":"2024-03-11T22:10:09.179383+0000","last_fullsized":"2024-03-11T22:10:09.179383+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:09:59.504668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:59.504668+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:06.422336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":66,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.316489+0000","last_change":"2024-03-11T22:08:33.030653+0000","last_active":"2024-03-11T22:10:09.316489+0000","last_peered":"2024-03-11T22:10:09.316489+0000","last_clean":"2024-03-11T22:10:09.316489+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:10:09.316489+0000","last_undegraded":"2024-03-11T22:10:09.316489+0000","last_fullsized":"2024-03-11T22:10:09.316489+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":76,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.180132+0000","last_change":"2024-03-11T22:08:35.086209+0000","last_active":"2024-03-11T22:10:09.180132+0000","last_peered":"2024-03-11T22:10:09.180132+0000","last_clean":"2024-03-11T22:10:09.180132+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:10:09.180132+0000","last_undegraded":"2024-03-11T22:10:09.180132+0000","last_fullsized":"2024-03-11T22:10:09.180132+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":61,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.317059+0000","last_change":"2024-03-11T22:10:03.601956+0000","last_active":"2024-03-11T22:10:09.317059+0000","last_peered":"2024-03-11T22:10:09.317059+0000","last_clean":"2024-03-11T22:10:09.317059+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:10:09.317059+0000","last_undegraded":"2024-03-11T22:10:09.317059+0000","last_fullsized":"2024-03-11T22:10:09.317059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:03.601863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.601863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:25.688404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.180284+0000","last_change":"2024-03-11T22:08:37.069965+0000","last_active":"2024-03-11T22:10:09.180284+0000","last_peered":"2024-03-11T22:10:09.180284+0000","last_clean":"2024-03-11T22:10:09.180284+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:10:09.180284+0000","last_undegraded":"2024-03-11T22:10:09.180284+0000","last_fullsized":"2024-03-11T22:10:09.180284+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276035+0000","last_change":"2024-03-11T22:08:33.038259+0000","last_active":"2024-03-11T22:10:09.276035+0000","last_peered":"2024-03-11T22:10:09.276035+0000","last_clean":"2024-03-11T22:10:09.276035+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:10:09.276035+0000","last_undegraded":"2024-03-11T22:10:09.276035+0000","last_fullsized":"2024-03-11T22:10:09.276035+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":88,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179372+0000","last_change":"2024-03-11T22:09:49.567290+0000","last_active":"2024-03-11T22:10:09.179372+0000","last_peered":"2024-03-11T22:10:09.179372+0000","last_clean":"2024-03-11T22:10:09.179372+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:10:09.179372+0000","last_undegraded":"2024-03-11T22:10:09.179372+0000","last_fullsized":"2024-03-11T22:10:09.179372+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:09:49.567170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:49.567170+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:01.624956+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.274075+0000","last_change":"2024-03-11T22:08:39.074473+0000","last_active":"2024-03-11T22:10:09.274075+0000","last_peered":"2024-03-11T22:10:09.274075+0000","last_clean":"2024-03-11T22:10:09.274075+0000","last_became_active":"2024-03-11T22:08:39.074179+0000","last_became_peered":"2024-03-11T22:08:39.074179+0000","last_unstale":"2024-03-11T22:10:09.274075+0000","last_undegraded":"2024-03-11T22:10:09.274075+0000","last_fullsized":"2024-03-11T22:10:09.274075+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178565+0000","last_change":"2024-03-11T22:09:40.669305+0000","last_active":"2024-03-11T22:10:09.178565+0000","last_peered":"2024-03-11T22:10:09.178565+0000","last_clean":"2024-03-11T22:10:09.178565+0000","last_became_active":"2024-03-11T22:08:37.075341+0000","last_became_peered":"2024-03-11T22:08:37.075341+0000","last_unstale":"2024-03-11T22:10:09.178565+0000","last_undegraded":"2024-03-11T22:10:09.178565+0000","last_fullsized":"2024-03-11T22:10:09.178565+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:40.669213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:40.669213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:05.240222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179990+0000","last_change":"2024-03-11T22:08:33.040896+0000","last_active":"2024-03-11T22:10:09.179990+0000","last_peered":"2024-03-11T22:10:09.179990+0000","last_clean":"2024-03-11T22:10:09.179990+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:10:09.179990+0000","last_undegraded":"2024-03-11T22:10:09.179990+0000","last_fullsized":"2024-03-11T22:10:09.179990+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":87,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276035+0000","last_change":"2024-03-11T22:09:42.568204+0000","last_active":"2024-03-11T22:10:09.276035+0000","last_peered":"2024-03-11T22:10:09.276035+0000","last_clean":"2024-03-11T22:10:09.276035+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:10:09.276035+0000","last_undegraded":"2024-03-11T22:10:09.276035+0000","last_fullsized":"2024-03-11T22:10:09.276035+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-11T22:09:42.568114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:42.568114+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.634829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276244+0000","last_change":"2024-03-11T22:08:39.081827+0000","last_active":"2024-03-11T22:10:09.276244+0000","last_peered":"2024-03-11T22:10:09.276244+0000","last_clean":"2024-03-11T22:10:09.276244+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:10:09.276244+0000","last_undegraded":"2024-03-11T22:10:09.276244+0000","last_fullsized":"2024-03-11T22:10:09.276244+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.275215+0000","last_change":"2024-03-11T22:09:58.233204+0000","last_active":"2024-03-11T22:10:09.275215+0000","last_peered":"2024-03-11T22:10:09.275215+0000","last_clean":"2024-03-11T22:10:09.275215+0000","last_became_active":"2024-03-11T22:08:37.078863+0000","last_became_peered":"2024-03-11T22:08:37.078863+0000","last_unstale":"2024-03-11T22:10:09.275215+0000","last_undegraded":"2024-03-11T22:10:09.275215+0000","last_fullsized":"2024-03-11T22:10:09.275215+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:58.233081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:09:58.233081+0000","last_clean_scrub_stamp":"2024-03-11T22:09:58.233081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:08.891585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484625+0000","last_change":"2024-03-11T22:09:49.271541+0000","last_active":"2024-03-11T22:10:09.484625+0000","last_peered":"2024-03-11T22:10:09.484625+0000","last_clean":"2024-03-11T22:10:09.484625+0000","last_became_active":"2024-03-11T22:08:33.039349+0000","last_became_peered":"2024-03-11T22:08:33.039349+0000","last_unstale":"2024-03-11T22:10:09.484625+0000","last_undegraded":"2024-03-11T22:10:09.484625+0000","last_fullsized":"2024-03-11T22:10:09.484625+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:49.271431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:49.271431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:01.697571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":80,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276516+0000","last_change":"2024-03-11T22:08:35.086272+0000","last_active":"2024-03-11T22:10:09.276516+0000","last_peered":"2024-03-11T22:10:09.276516+0000","last_clean":"2024-03-11T22:10:09.276516+0000","last_became_active":"2024-03-11T22:08:35.085928+0000","last_became_peered":"2024-03-11T22:08:35.085928+0000","last_unstale":"2024-03-11T22:10:09.276516+0000","last_undegraded":"2024-03-11T22:10:09.276516+0000","last_fullsized":"2024-03-11T22:10:09.276516+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178426+0000","last_change":"2024-03-11T22:08:39.083049+0000","last_active":"2024-03-11T22:10:09.178426+0000","last_peered":"2024-03-11T22:10:09.178426+0000","last_clean":"2024-03-11T22:10:09.178426+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:10:09.178426+0000","last_undegraded":"2024-03-11T22:10:09.178426+0000","last_fullsized":"2024-03-11T22:10:09.178426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276498+0000","last_change":"2024-03-11T22:09:43.569426+0000","last_active":"2024-03-11T22:10:09.276498+0000","last_peered":"2024-03-11T22:10:09.276498+0000","last_clean":"2024-03-11T22:10:09.276498+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:10:09.276498+0000","last_undegraded":"2024-03-11T22:10:09.276498+0000","last_fullsized":"2024-03-11T22:10:09.276498+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:43.569328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:43.569328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:05.155162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.274335+0000","last_change":"2024-03-11T22:09:41.032444+0000","last_active":"2024-03-11T22:10:09.274335+0000","last_peered":"2024-03-11T22:10:09.274335+0000","last_clean":"2024-03-11T22:10:09.274335+0000","last_became_active":"2024-03-11T22:08:33.038783+0000","last_became_peered":"2024-03-11T22:08:33.038783+0000","last_unstale":"2024-03-11T22:10:09.274335+0000","last_undegraded":"2024-03-11T22:10:09.274335+0000","last_fullsized":"2024-03-11T22:10:09.274335+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:41.032339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:41.032339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:01.424543+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":85,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.316820+0000","last_change":"2024-03-11T22:09:57.633770+0000","last_active":"2024-03-11T22:10:09.316820+0000","last_peered":"2024-03-11T22:10:09.316820+0000","last_clean":"2024-03-11T22:10:09.316820+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:10:09.316820+0000","last_undegraded":"2024-03-11T22:10:09.316820+0000","last_fullsized":"2024-03-11T22:10:09.316820+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:09:57.633671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:57.633671+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:15.549351+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276127+0000","last_change":"2024-03-11T22:08:39.080670+0000","last_active":"2024-03-11T22:10:09.276127+0000","last_peered":"2024-03-11T22:10:09.276127+0000","last_clean":"2024-03-11T22:10:09.276127+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:10:09.276127+0000","last_undegraded":"2024-03-11T22:10:09.276127+0000","last_fullsized":"2024-03-11T22:10:09.276127+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.316694+0000","last_change":"2024-03-11T22:08:37.075322+0000","last_active":"2024-03-11T22:10:09.316694+0000","last_peered":"2024-03-11T22:10:09.316694+0000","last_clean":"2024-03-11T22:10:09.316694+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:10:09.316694+0000","last_undegraded":"2024-03-11T22:10:09.316694+0000","last_fullsized":"2024-03-11T22:10:09.316694+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":79,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418873+0000","last_change":"2024-03-11T22:09:48.362144+0000","last_active":"2024-03-11T22:10:09.418873+0000","last_peered":"2024-03-11T22:10:09.418873+0000","last_clean":"2024-03-11T22:10:09.418873+0000","last_became_active":"2024-03-11T22:08:33.036083+0000","last_became_peered":"2024-03-11T22:08:33.036083+0000","last_unstale":"2024-03-11T22:10:09.418873+0000","last_undegraded":"2024-03-11T22:10:09.418873+0000","last_fullsized":"2024-03-11T22:10:09.418873+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:09:48.361967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:48.361967+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:05.666698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":80,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.419010+0000","last_change":"2024-03-11T22:08:35.054987+0000","last_active":"2024-03-11T22:10:09.419010+0000","last_peered":"2024-03-11T22:10:09.419010+0000","last_clean":"2024-03-11T22:10:09.419010+0000","last_became_active":"2024-03-11T22:08:35.054726+0000","last_became_peered":"2024-03-11T22:08:35.054726+0000","last_unstale":"2024-03-11T22:10:09.419010+0000","last_undegraded":"2024-03-11T22:10:09.419010+0000","last_fullsized":"2024-03-11T22:10:09.419010+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179938+0000","last_change":"2024-03-11T22:08:39.083736+0000","last_active":"2024-03-11T22:10:09.179938+0000","last_peered":"2024-03-11T22:10:09.179938+0000","last_clean":"2024-03-11T22:10:09.179938+0000","last_became_active":"2024-03-11T22:08:39.082938+0000","last_became_peered":"2024-03-11T22:08:39.082938+0000","last_unstale":"2024-03-11T22:10:09.179938+0000","last_undegraded":"2024-03-11T22:10:09.179938+0000","last_fullsized":"2024-03-11T22:10:09.179938+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":93,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484567+0000","last_change":"2024-03-11T22:09:58.187539+0000","last_active":"2024-03-11T22:10:09.484567+0000","last_peered":"2024-03-11T22:10:09.484567+0000","last_clean":"2024-03-11T22:10:09.484567+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:10:09.484567+0000","last_undegraded":"2024-03-11T22:10:09.484567+0000","last_fullsized":"2024-03-11T22:10:09.484567+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:09:58.187420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:58.187420+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:59.523954+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.274473+0000","last_change":"2024-03-11T22:09:45.198482+0000","last_active":"2024-03-11T22:10:09.274473+0000","last_peered":"2024-03-11T22:10:09.274473+0000","last_clean":"2024-03-11T22:10:09.274473+0000","last_became_active":"2024-03-11T22:08:33.300181+0000","last_became_peered":"2024-03-11T22:08:33.300181+0000","last_unstale":"2024-03-11T22:10:09.274473+0000","last_undegraded":"2024-03-11T22:10:09.274473+0000","last_fullsized":"2024-03-11T22:10:09.274473+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:45.198397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:45.198397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:12.972849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":84,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276570+0000","last_change":"2024-03-11T22:09:55.647845+0000","last_active":"2024-03-11T22:10:09.276570+0000","last_peered":"2024-03-11T22:10:09.276570+0000","last_clean":"2024-03-11T22:10:09.276570+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:10:09.276570+0000","last_undegraded":"2024-03-11T22:10:09.276570+0000","last_fullsized":"2024-03-11T22:10:09.276570+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:09:55.647703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:55.647703+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:22.462130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.419206+0000","last_change":"2024-03-11T22:08:39.072508+0000","last_active":"2024-03-11T22:10:09.419206+0000","last_peered":"2024-03-11T22:10:09.419206+0000","last_clean":"2024-03-11T22:10:09.419206+0000","last_became_active":"2024-03-11T22:08:39.072089+0000","last_became_peered":"2024-03-11T22:08:39.072089+0000","last_unstale":"2024-03-11T22:10:09.419206+0000","last_undegraded":"2024-03-11T22:10:09.419206+0000","last_fullsized":"2024-03-11T22:10:09.419206+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":96,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276612+0000","last_change":"2024-03-11T22:10:03.640781+0000","last_active":"2024-03-11T22:10:09.276612+0000","last_peered":"2024-03-11T22:10:09.276612+0000","last_clean":"2024-03-11T22:10:09.276612+0000","last_became_active":"2024-03-11T22:08:37.071202+0000","last_became_peered":"2024-03-11T22:08:37.071202+0000","last_unstale":"2024-03-11T22:10:09.276612+0000","last_undegraded":"2024-03-11T22:10:09.276612+0000","last_fullsized":"2024-03-11T22:10:09.276612+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:10:03.640604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.640604+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:13.116374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179422+0000","last_change":"2024-03-11T22:09:59.229533+0000","last_active":"2024-03-11T22:10:09.179422+0000","last_peered":"2024-03-11T22:10:09.179422+0000","last_clean":"2024-03-11T22:10:09.179422+0000","last_became_active":"2024-03-11T22:08:33.032850+0000","last_became_peered":"2024-03-11T22:08:33.032850+0000","last_unstale":"2024-03-11T22:10:09.179422+0000","last_undegraded":"2024-03-11T22:10:09.179422+0000","last_fullsized":"2024-03-11T22:10:09.179422+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:59.229452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:59.229452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:23.364611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":85,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484685+0000","last_change":"2024-03-11T22:09:52.219839+0000","last_active":"2024-03-11T22:10:09.484685+0000","last_peered":"2024-03-11T22:10:09.484685+0000","last_clean":"2024-03-11T22:10:09.484685+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:10:09.484685+0000","last_undegraded":"2024-03-11T22:10:09.484685+0000","last_fullsized":"2024-03-11T22:10:09.484685+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:09:52.219687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:52.219687+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:06.397728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":61,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178930+0000","last_change":"2024-03-11T22:09:44.743698+0000","last_active":"2024-03-11T22:10:09.178930+0000","last_peered":"2024-03-11T22:10:09.178930+0000","last_clean":"2024-03-11T22:10:09.178930+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:10:09.178930+0000","last_undegraded":"2024-03-11T22:10:09.178930+0000","last_fullsized":"2024-03-11T22:10:09.178930+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:44.743600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:44.743600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:03.506943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.277063+0000","last_change":"2024-03-11T22:09:45.582583+0000","last_active":"2024-03-11T22:10:09.277063+0000","last_peered":"2024-03-11T22:10:09.277063+0000","last_clean":"2024-03-11T22:10:09.277063+0000","last_became_active":"2024-03-11T22:08:37.069611+0000","last_became_peered":"2024-03-11T22:08:37.069611+0000","last_unstale":"2024-03-11T22:10:09.277063+0000","last_undegraded":"2024-03-11T22:10:09.277063+0000","last_fullsized":"2024-03-11T22:10:09.277063+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:45.582469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:45.582469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.713402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.274946+0000","last_change":"2024-03-11T22:08:33.038436+0000","last_active":"2024-03-11T22:10:09.274946+0000","last_peered":"2024-03-11T22:10:09.274946+0000","last_clean":"2024-03-11T22:10:09.274946+0000","last_became_active":"2024-03-11T22:08:33.038132+0000","last_became_peered":"2024-03-11T22:08:33.038132+0000","last_unstale":"2024-03-11T22:10:09.274946+0000","last_undegraded":"2024-03-11T22:10:09.274946+0000","last_fullsized":"2024-03-11T22:10:09.274946+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"93'13","reported_seq":89,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.180151+0000","last_change":"2024-03-11T22:10:04.123522+0000","last_active":"2024-03-11T22:10:09.180151+0000","last_peered":"2024-03-11T22:10:09.180151+0000","last_clean":"2024-03-11T22:10:09.180151+0000","last_became_active":"2024-03-11T22:08:35.085787+0000","last_became_peered":"2024-03-11T22:08:35.085787+0000","last_unstale":"2024-03-11T22:10:09.180151+0000","last_undegraded":"2024-03-11T22:10:09.180151+0000","last_fullsized":"2024-03-11T22:10:09.180151+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:10:04.123429+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:11.598479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.274234+0000","last_change":"2024-03-11T22:08:39.075607+0000","last_active":"2024-03-11T22:10:09.274234+0000","last_peered":"2024-03-11T22:10:09.274234+0000","last_clean":"2024-03-11T22:10:09.274234+0000","last_became_active":"2024-03-11T22:08:39.075305+0000","last_became_peered":"2024-03-11T22:08:39.075305+0000","last_unstale":"2024-03-11T22:10:09.274234+0000","last_undegraded":"2024-03-11T22:10:09.274234+0000","last_fullsized":"2024-03-11T22:10:09.274234+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484681+0000","last_change":"2024-03-11T22:10:07.195413+0000","last_active":"2024-03-11T22:10:09.484681+0000","last_peered":"2024-03-11T22:10:09.484681+0000","last_clean":"2024-03-11T22:10:09.484681+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:10:09.484681+0000","last_undegraded":"2024-03-11T22:10:09.484681+0000","last_fullsized":"2024-03-11T22:10:09.484681+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:07.195289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:07.195289+0000","last_clean_scrub_stamp":"2024-03-11T22:10:07.195289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:13.206483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178556+0000","last_change":"2024-03-11T22:10:00.499503+0000","last_active":"2024-03-11T22:10:09.178556+0000","last_peered":"2024-03-11T22:10:09.178556+0000","last_clean":"2024-03-11T22:10:09.178556+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:10:09.178556+0000","last_undegraded":"2024-03-11T22:10:09.178556+0000","last_fullsized":"2024-03-11T22:10:09.178556+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:10:00.499404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:24.856919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":70,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.277067+0000","last_change":"2024-03-11T22:08:35.086263+0000","last_active":"2024-03-11T22:10:09.277067+0000","last_peered":"2024-03-11T22:10:09.277067+0000","last_clean":"2024-03-11T22:10:09.277067+0000","last_became_active":"2024-03-11T22:08:35.085914+0000","last_became_peered":"2024-03-11T22:08:35.085914+0000","last_unstale":"2024-03-11T22:10:09.277067+0000","last_undegraded":"2024-03-11T22:10:09.277067+0000","last_fullsized":"2024-03-11T22:10:09.277067+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484364+0000","last_change":"2024-03-11T22:08:39.079848+0000","last_active":"2024-03-11T22:10:09.484364+0000","last_peered":"2024-03-11T22:10:09.484364+0000","last_clean":"2024-03-11T22:10:09.484364+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:10:09.484364+0000","last_undegraded":"2024-03-11T22:10:09.484364+0000","last_fullsized":"2024-03-11T22:10:09.484364+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":85,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.274684+0000","last_change":"2024-03-11T22:10:05.284618+0000","last_active":"2024-03-11T22:10:09.274684+0000","last_peered":"2024-03-11T22:10:09.274684+0000","last_clean":"2024-03-11T22:10:09.274684+0000","last_became_active":"2024-03-11T22:08:33.037662+0000","last_became_peered":"2024-03-11T22:08:33.037662+0000","last_unstale":"2024-03-11T22:10:09.274684+0000","last_undegraded":"2024-03-11T22:10:09.274684+0000","last_fullsized":"2024-03-11T22:10:09.274684+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:10:05.284508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:05.284508+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:06.642547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"91'22","reported_seq":101,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484100+0000","last_change":"2024-03-11T22:09:36.274318+0000","last_active":"2024-03-11T22:10:09.484100+0000","last_peered":"2024-03-11T22:10:09.484100+0000","last_clean":"2024-03-11T22:10:09.484100+0000","last_became_active":"2024-03-11T22:08:35.085916+0000","last_became_peered":"2024-03-11T22:08:35.085916+0000","last_unstale":"2024-03-11T22:10:09.484100+0000","last_undegraded":"2024-03-11T22:10:09.484100+0000","last_fullsized":"2024-03-11T22:10:09.484100+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'22","last_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:09:36.274225+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:38.387952+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":85,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.275126+0000","last_change":"2024-03-11T22:08:37.078410+0000","last_active":"2024-03-11T22:10:09.275126+0000","last_peered":"2024-03-11T22:10:09.275126+0000","last_clean":"2024-03-11T22:10:09.275126+0000","last_became_active":"2024-03-11T22:08:37.078114+0000","last_became_peered":"2024-03-11T22:08:37.078114+0000","last_unstale":"2024-03-11T22:10:09.275126+0000","last_undegraded":"2024-03-11T22:10:09.275126+0000","last_fullsized":"2024-03-11T22:10:09.275126+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"75'1","reported_seq":54,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276359+0000","last_change":"2024-03-11T22:08:39.083458+0000","last_active":"2024-03-11T22:10:09.276359+0000","last_peered":"2024-03-11T22:10:09.276359+0000","last_clean":"2024-03-11T22:10:09.276359+0000","last_became_active":"2024-03-11T22:08:39.083141+0000","last_became_peered":"2024-03-11T22:08:39.083141+0000","last_unstale":"2024-03-11T22:10:09.276359+0000","last_undegraded":"2024-03-11T22:10:09.276359+0000","last_fullsized":"2024-03-11T22:10:09.276359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.274472+0000","last_change":"2024-03-11T22:09:55.169720+0000","last_active":"2024-03-11T22:10:09.274472+0000","last_peered":"2024-03-11T22:10:09.274472+0000","last_clean":"2024-03-11T22:10:09.274472+0000","last_became_active":"2024-03-11T22:08:33.041492+0000","last_became_peered":"2024-03-11T22:08:33.041492+0000","last_unstale":"2024-03-11T22:10:09.274472+0000","last_undegraded":"2024-03-11T22:10:09.274472+0000","last_fullsized":"2024-03-11T22:10:09.274472+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:55.169587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:55.169587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:18.640477+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":102,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418701+0000","last_change":"2024-03-11T22:08:54.614766+0000","last_active":"2024-03-11T22:10:09.418701+0000","last_peered":"2024-03-11T22:10:09.418701+0000","last_clean":"2024-03-11T22:10:09.418701+0000","last_became_active":"2024-03-11T22:08:54.607153+0000","last_became_peered":"2024-03-11T22:08:54.607153+0000","last_unstale":"2024-03-11T22:10:09.418701+0000","last_undegraded":"2024-03-11T22:10:09.418701+0000","last_fullsized":"2024-03-11T22:10:09.418701+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178374+0000","last_change":"2024-03-11T22:09:48.721747+0000","last_active":"2024-03-11T22:10:09.178374+0000","last_peered":"2024-03-11T22:10:09.178374+0000","last_clean":"2024-03-11T22:10:09.178374+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:10:09.178374+0000","last_undegraded":"2024-03-11T22:10:09.178374+0000","last_fullsized":"2024-03-11T22:10:09.178374+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:48.721656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:48.721656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.253690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":61,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.273987+0000","last_change":"2024-03-11T22:10:06.292102+0000","last_active":"2024-03-11T22:10:09.273987+0000","last_peered":"2024-03-11T22:10:09.273987+0000","last_clean":"2024-03-11T22:10:09.273987+0000","last_became_active":"2024-03-11T22:08:39.079238+0000","last_became_peered":"2024-03-11T22:08:39.079238+0000","last_unstale":"2024-03-11T22:10:09.273987+0000","last_undegraded":"2024-03-11T22:10:09.273987+0000","last_fullsized":"2024-03-11T22:10:09.273987+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:06.291982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:06.291982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:09.555069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178701+0000","last_change":"2024-03-11T22:10:03.710406+0000","last_active":"2024-03-11T22:10:09.178701+0000","last_peered":"2024-03-11T22:10:09.178701+0000","last_clean":"2024-03-11T22:10:09.178701+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:10:09.178701+0000","last_undegraded":"2024-03-11T22:10:09.178701+0000","last_fullsized":"2024-03-11T22:10:09.178701+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:03.710319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.710319+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.710319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:24.719061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.274559+0000","last_change":"2024-03-11T22:09:42.079812+0000","last_active":"2024-03-11T22:10:09.274559+0000","last_peered":"2024-03-11T22:10:09.274559+0000","last_clean":"2024-03-11T22:10:09.274559+0000","last_became_active":"2024-03-11T22:08:33.035188+0000","last_became_peered":"2024-03-11T22:08:33.035188+0000","last_unstale":"2024-03-11T22:10:09.274559+0000","last_undegraded":"2024-03-11T22:10:09.274559+0000","last_fullsized":"2024-03-11T22:10:09.274559+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:42.079695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:42.079695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:45.595806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":100,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178773+0000","last_change":"2024-03-11T22:09:54.637633+0000","last_active":"2024-03-11T22:10:09.178773+0000","last_peered":"2024-03-11T22:10:09.178773+0000","last_clean":"2024-03-11T22:10:09.178773+0000","last_became_active":"2024-03-11T22:08:35.087046+0000","last_became_peered":"2024-03-11T22:08:35.087046+0000","last_unstale":"2024-03-11T22:10:09.178773+0000","last_undegraded":"2024-03-11T22:10:09.178773+0000","last_fullsized":"2024-03-11T22:10:09.178773+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:09:54.637462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:54.637462+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:00.008092+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":61,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178426+0000","last_change":"2024-03-11T22:09:57.233798+0000","last_active":"2024-03-11T22:10:09.178426+0000","last_peered":"2024-03-11T22:10:09.178426+0000","last_clean":"2024-03-11T22:10:09.178426+0000","last_became_active":"2024-03-11T22:08:39.085496+0000","last_became_peered":"2024-03-11T22:08:39.085496+0000","last_unstale":"2024-03-11T22:10:09.178426+0000","last_undegraded":"2024-03-11T22:10:09.178426+0000","last_fullsized":"2024-03-11T22:10:09.178426+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:57.233610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:57.233610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:19.937525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179257+0000","last_change":"2024-03-11T22:10:01.692571+0000","last_active":"2024-03-11T22:10:09.179257+0000","last_peered":"2024-03-11T22:10:09.179257+0000","last_clean":"2024-03-11T22:10:09.179257+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:10:09.179257+0000","last_undegraded":"2024-03-11T22:10:09.179257+0000","last_fullsized":"2024-03-11T22:10:09.179257+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:01.692464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:01.692464+0000","last_clean_scrub_stamp":"2024-03-11T22:10:01.692464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:27.908248+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484455+0000","last_change":"2024-03-11T22:08:33.038820+0000","last_active":"2024-03-11T22:10:09.484455+0000","last_peered":"2024-03-11T22:10:09.484455+0000","last_clean":"2024-03-11T22:10:09.484455+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:10:09.484455+0000","last_undegraded":"2024-03-11T22:10:09.484455+0000","last_fullsized":"2024-03-11T22:10:09.484455+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":93,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179786+0000","last_change":"2024-03-11T22:08:54.617268+0000","last_active":"2024-03-11T22:10:09.179786+0000","last_peered":"2024-03-11T22:10:09.179786+0000","last_clean":"2024-03-11T22:10:09.179786+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:10:09.179786+0000","last_undegraded":"2024-03-11T22:10:09.179786+0000","last_fullsized":"2024-03-11T22:10:09.179786+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276665+0000","last_change":"2024-03-11T22:08:39.084054+0000","last_active":"2024-03-11T22:10:09.276665+0000","last_peered":"2024-03-11T22:10:09.276665+0000","last_clean":"2024-03-11T22:10:09.276665+0000","last_became_active":"2024-03-11T22:08:39.083738+0000","last_became_peered":"2024-03-11T22:08:39.083738+0000","last_unstale":"2024-03-11T22:10:09.276665+0000","last_undegraded":"2024-03-11T22:10:09.276665+0000","last_fullsized":"2024-03-11T22:10:09.276665+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":85,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484235+0000","last_change":"2024-03-11T22:10:00.178117+0000","last_active":"2024-03-11T22:10:09.484235+0000","last_peered":"2024-03-11T22:10:09.484235+0000","last_clean":"2024-03-11T22:10:09.484235+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:10:09.484235+0000","last_undegraded":"2024-03-11T22:10:09.484235+0000","last_fullsized":"2024-03-11T22:10:09.484235+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:10:00.177978+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:18.053454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418415+0000","last_change":"2024-03-11T22:09:55.458484+0000","last_active":"2024-03-11T22:10:09.418415+0000","last_peered":"2024-03-11T22:10:09.418415+0000","last_clean":"2024-03-11T22:10:09.418415+0000","last_became_active":"2024-03-11T22:08:33.035469+0000","last_became_peered":"2024-03-11T22:08:33.035469+0000","last_unstale":"2024-03-11T22:10:09.418415+0000","last_undegraded":"2024-03-11T22:10:09.418415+0000","last_fullsized":"2024-03-11T22:10:09.418415+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:55.458385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:55.458385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:19.365475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":137,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.316826+0000","last_change":"2024-03-11T22:09:47.779316+0000","last_active":"2024-03-11T22:10:09.316826+0000","last_peered":"2024-03-11T22:10:09.316826+0000","last_clean":"2024-03-11T22:10:09.316826+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:10:09.316826+0000","last_undegraded":"2024-03-11T22:10:09.316826+0000","last_fullsized":"2024-03-11T22:10:09.316826+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-11T22:09:47.779209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:47.779209+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:10.865609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179690+0000","last_change":"2024-03-11T22:09:44.165077+0000","last_active":"2024-03-11T22:10:09.179690+0000","last_peered":"2024-03-11T22:10:09.179690+0000","last_clean":"2024-03-11T22:10:09.179690+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:10:09.179690+0000","last_undegraded":"2024-03-11T22:10:09.179690+0000","last_fullsized":"2024-03-11T22:10:09.179690+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:44.164939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:44.164939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:55.636705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":61,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.275298+0000","last_change":"2024-03-11T22:10:09.275337+0000","last_active":"2024-03-11T22:10:09.275298+0000","last_peered":"2024-03-11T22:10:09.275298+0000","last_clean":"2024-03-11T22:10:09.275298+0000","last_became_active":"2024-03-11T22:08:39.077566+0000","last_became_peered":"2024-03-11T22:08:39.077566+0000","last_unstale":"2024-03-11T22:10:09.275298+0000","last_undegraded":"2024-03-11T22:10:09.275298+0000","last_fullsized":"2024-03-11T22:10:09.275298+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:09.275205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:09.275205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:12.561365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.316188+0000","last_change":"2024-03-11T22:09:35.724312+0000","last_active":"2024-03-11T22:10:09.316188+0000","last_peered":"2024-03-11T22:10:09.316188+0000","last_clean":"2024-03-11T22:10:09.316188+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:10:09.316188+0000","last_undegraded":"2024-03-11T22:10:09.316188+0000","last_fullsized":"2024-03-11T22:10:09.316188+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:35.724224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:35.724224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:59.123602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":82,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178493+0000","last_change":"2024-03-11T22:10:00.733336+0000","last_active":"2024-03-11T22:10:09.178493+0000","last_peered":"2024-03-11T22:10:09.178493+0000","last_clean":"2024-03-11T22:10:09.178493+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:10:09.178493+0000","last_undegraded":"2024-03-11T22:10:09.178493+0000","last_fullsized":"2024-03-11T22:10:09.178493+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:10:00.733239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:12.509827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":96,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.317168+0000","last_change":"2024-03-11T22:10:02.604337+0000","last_active":"2024-03-11T22:10:09.317168+0000","last_peered":"2024-03-11T22:10:09.317168+0000","last_clean":"2024-03-11T22:10:09.317168+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:10:09.317168+0000","last_undegraded":"2024-03-11T22:10:09.317168+0000","last_fullsized":"2024-03-11T22:10:09.317168+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-03-11T22:10:02.604241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:02.604241+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:06.403890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178907+0000","last_change":"2024-03-11T22:08:37.076581+0000","last_active":"2024-03-11T22:10:09.178907+0000","last_peered":"2024-03-11T22:10:09.178907+0000","last_clean":"2024-03-11T22:10:09.178907+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:10:09.178907+0000","last_undegraded":"2024-03-11T22:10:09.178907+0000","last_fullsized":"2024-03-11T22:10:09.178907+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484680+0000","last_change":"2024-03-11T22:08:39.073180+0000","last_active":"2024-03-11T22:10:09.484680+0000","last_peered":"2024-03-11T22:10:09.484680+0000","last_clean":"2024-03-11T22:10:09.484680+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:10:09.484680+0000","last_undegraded":"2024-03-11T22:10:09.484680+0000","last_fullsized":"2024-03-11T22:10:09.484680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":328,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.275009+0000","last_change":"2024-03-11T22:09:59.243266+0000","last_active":"2024-03-11T22:10:09.275009+0000","last_peered":"2024-03-11T22:10:09.275009+0000","last_clean":"2024-03-11T22:10:09.275009+0000","last_became_active":"2024-03-11T22:08:22.015394+0000","last_became_peered":"2024-03-11T22:08:22.015394+0000","last_unstale":"2024-03-11T22:10:09.275009+0000","last_undegraded":"2024-03-11T22:10:09.275009+0000","last_fullsized":"2024-03-11T22:10:09.275009+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:09:59.243159+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:59.365375+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276548+0000","last_change":"2024-03-11T22:09:41.615925+0000","last_active":"2024-03-11T22:10:09.276548+0000","last_peered":"2024-03-11T22:10:09.276548+0000","last_clean":"2024-03-11T22:10:09.276548+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:10:09.276548+0000","last_undegraded":"2024-03-11T22:10:09.276548+0000","last_fullsized":"2024-03-11T22:10:09.276548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:41.615829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:41.615829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:04.514072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":103,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179133+0000","last_change":"2024-03-11T22:08:35.086266+0000","last_active":"2024-03-11T22:10:09.179133+0000","last_peered":"2024-03-11T22:10:09.179133+0000","last_clean":"2024-03-11T22:10:09.179133+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:10:09.179133+0000","last_undegraded":"2024-03-11T22:10:09.179133+0000","last_fullsized":"2024-03-11T22:10:09.179133+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.275175+0000","last_change":"2024-03-11T22:08:37.076167+0000","last_active":"2024-03-11T22:10:09.275175+0000","last_peered":"2024-03-11T22:10:09.275175+0000","last_clean":"2024-03-11T22:10:09.275175+0000","last_became_active":"2024-03-11T22:08:37.075876+0000","last_became_peered":"2024-03-11T22:08:37.075876+0000","last_unstale":"2024-03-11T22:10:09.275175+0000","last_undegraded":"2024-03-11T22:10:09.275175+0000","last_fullsized":"2024-03-11T22:10:09.275175+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":61,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.316707+0000","last_change":"2024-03-11T22:09:53.690828+0000","last_active":"2024-03-11T22:10:09.316707+0000","last_peered":"2024-03-11T22:10:09.316707+0000","last_clean":"2024-03-11T22:10:09.316707+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:10:09.316707+0000","last_undegraded":"2024-03-11T22:10:09.316707+0000","last_fullsized":"2024-03-11T22:10:09.316707+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:53.690738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:53.690738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:11.065631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179326+0000","last_change":"2024-03-11T22:08:33.035933+0000","last_active":"2024-03-11T22:10:09.179326+0000","last_peered":"2024-03-11T22:10:09.179326+0000","last_clean":"2024-03-11T22:10:09.179326+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:10:09.179326+0000","last_undegraded":"2024-03-11T22:10:09.179326+0000","last_fullsized":"2024-03-11T22:10:09.179326+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"76'5","reported_seq":167,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.274955+0000","last_change":"2024-03-11T22:08:36.060461+0000","last_active":"2024-03-11T22:10:09.274955+0000","last_peered":"2024-03-11T22:10:09.274955+0000","last_clean":"2024-03-11T22:10:09.274955+0000","last_became_active":"2024-03-11T22:08:32.033652+0000","last_became_peered":"2024-03-11T22:08:32.033652+0000","last_unstale":"2024-03-11T22:10:09.274955+0000","last_undegraded":"2024-03-11T22:10:09.274955+0000","last_fullsized":"2024-03-11T22:10:09.274955+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:31.015772+0000","last_clean_scrub_stamp":"2024-03-11T22:08:31.015772+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":85,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":87,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.180005+0000","last_change":"2024-03-11T22:09:51.213643+0000","last_active":"2024-03-11T22:10:09.180005+0000","last_peered":"2024-03-11T22:10:09.180005+0000","last_clean":"2024-03-11T22:10:09.180005+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:10:09.180005+0000","last_undegraded":"2024-03-11T22:10:09.180005+0000","last_fullsized":"2024-03-11T22:10:09.180005+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:09:51.213532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:51.213532+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:51.292156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484855+0000","last_change":"2024-03-11T22:08:37.082329+0000","last_active":"2024-03-11T22:10:09.484855+0000","last_peered":"2024-03-11T22:10:09.484855+0000","last_clean":"2024-03-11T22:10:09.484855+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:10:09.484855+0000","last_undegraded":"2024-03-11T22:10:09.484855+0000","last_fullsized":"2024-03-11T22:10:09.484855+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":61,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.316344+0000","last_change":"2024-03-11T22:09:54.681445+0000","last_active":"2024-03-11T22:10:09.316344+0000","last_peered":"2024-03-11T22:10:09.316344+0000","last_clean":"2024-03-11T22:10:09.316344+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:10:09.316344+0000","last_undegraded":"2024-03-11T22:10:09.316344+0000","last_fullsized":"2024-03-11T22:10:09.316344+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:54.681338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:09:54.681338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:22.214871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.316210+0000","last_change":"2024-03-11T22:08:33.298533+0000","last_active":"2024-03-11T22:10:09.316210+0000","last_peered":"2024-03-11T22:10:09.316210+0000","last_clean":"2024-03-11T22:10:09.316210+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:10:09.316210+0000","last_undegraded":"2024-03-11T22:10:09.316210+0000","last_fullsized":"2024-03-11T22:10:09.316210+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":97,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179653+0000","last_change":"2024-03-11T22:08:35.087309+0000","last_active":"2024-03-11T22:10:09.179653+0000","last_peered":"2024-03-11T22:10:09.179653+0000","last_clean":"2024-03-11T22:10:09.179653+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:10:09.179653+0000","last_undegraded":"2024-03-11T22:10:09.179653+0000","last_fullsized":"2024-03-11T22:10:09.179653+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178954+0000","last_change":"2024-03-11T22:08:37.074393+0000","last_active":"2024-03-11T22:10:09.178954+0000","last_peered":"2024-03-11T22:10:09.178954+0000","last_clean":"2024-03-11T22:10:09.178954+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:10:09.178954+0000","last_undegraded":"2024-03-11T22:10:09.178954+0000","last_fullsized":"2024-03-11T22:10:09.178954+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178544+0000","last_change":"2024-03-11T22:08:39.085179+0000","last_active":"2024-03-11T22:10:09.178544+0000","last_peered":"2024-03-11T22:10:09.178544+0000","last_clean":"2024-03-11T22:10:09.178544+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:10:09.178544+0000","last_undegraded":"2024-03-11T22:10:09.178544+0000","last_fullsized":"2024-03-11T22:10:09.178544+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.483965+0000","last_change":"2024-03-11T22:09:43.271410+0000","last_active":"2024-03-11T22:10:09.483965+0000","last_peered":"2024-03-11T22:10:09.483965+0000","last_clean":"2024-03-11T22:10:09.483965+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:10:09.483965+0000","last_undegraded":"2024-03-11T22:10:09.483965+0000","last_fullsized":"2024-03-11T22:10:09.483965+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:43.271304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:43.271304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:50.753986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":76,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.316773+0000","last_change":"2024-03-11T22:08:35.058060+0000","last_active":"2024-03-11T22:10:09.316773+0000","last_peered":"2024-03-11T22:10:09.316773+0000","last_clean":"2024-03-11T22:10:09.316773+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:10:09.316773+0000","last_undegraded":"2024-03-11T22:10:09.316773+0000","last_fullsized":"2024-03-11T22:10:09.316773+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":93,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179860+0000","last_change":"2024-03-11T22:09:48.187735+0000","last_active":"2024-03-11T22:10:09.179860+0000","last_peered":"2024-03-11T22:10:09.179860+0000","last_clean":"2024-03-11T22:10:09.179860+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:10:09.179860+0000","last_undegraded":"2024-03-11T22:10:09.179860+0000","last_fullsized":"2024-03-11T22:10:09.179860+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:09:48.187641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:48.187641+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:56.947907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":61,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.274705+0000","last_change":"2024-03-11T22:10:03.279201+0000","last_active":"2024-03-11T22:10:09.274705+0000","last_peered":"2024-03-11T22:10:09.274705+0000","last_clean":"2024-03-11T22:10:09.274705+0000","last_became_active":"2024-03-11T22:08:39.076114+0000","last_became_peered":"2024-03-11T22:08:39.076114+0000","last_unstale":"2024-03-11T22:10:09.274705+0000","last_undegraded":"2024-03-11T22:10:09.274705+0000","last_fullsized":"2024-03-11T22:10:09.274705+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.279112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:24.270748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":74,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.180046+0000","last_change":"2024-03-11T22:10:03.110027+0000","last_active":"2024-03-11T22:10:09.180046+0000","last_peered":"2024-03-11T22:10:09.180046+0000","last_clean":"2024-03-11T22:10:09.180046+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:10:09.180046+0000","last_undegraded":"2024-03-11T22:10:09.180046+0000","last_fullsized":"2024-03-11T22:10:09.180046+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:10:03.109931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.109931+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:28.184077+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":90,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418484+0000","last_change":"2024-03-11T22:09:53.414248+0000","last_active":"2024-03-11T22:10:09.418484+0000","last_peered":"2024-03-11T22:10:09.418484+0000","last_clean":"2024-03-11T22:10:09.418484+0000","last_became_active":"2024-03-11T22:08:35.053452+0000","last_became_peered":"2024-03-11T22:08:35.053452+0000","last_unstale":"2024-03-11T22:10:09.418484+0000","last_undegraded":"2024-03-11T22:10:09.418484+0000","last_fullsized":"2024-03-11T22:10:09.418484+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:09:53.414127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:53.414127+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:17.661472+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":85,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276316+0000","last_change":"2024-03-11T22:08:37.078918+0000","last_active":"2024-03-11T22:10:09.276316+0000","last_peered":"2024-03-11T22:10:09.276316+0000","last_clean":"2024-03-11T22:10:09.276316+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:10:09.276316+0000","last_undegraded":"2024-03-11T22:10:09.276316+0000","last_fullsized":"2024-03-11T22:10:09.276316+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418339+0000","last_change":"2024-03-11T22:08:39.078715+0000","last_active":"2024-03-11T22:10:09.418339+0000","last_peered":"2024-03-11T22:10:09.418339+0000","last_clean":"2024-03-11T22:10:09.418339+0000","last_became_active":"2024-03-11T22:08:39.078244+0000","last_became_peered":"2024-03-11T22:08:39.078244+0000","last_unstale":"2024-03-11T22:10:09.418339+0000","last_undegraded":"2024-03-11T22:10:09.418339+0000","last_fullsized":"2024-03-11T22:10:09.418339+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276964+0000","last_change":"2024-03-11T22:09:36.565005+0000","last_active":"2024-03-11T22:10:09.276964+0000","last_peered":"2024-03-11T22:10:09.276964+0000","last_clean":"2024-03-11T22:10:09.276964+0000","last_became_active":"2024-03-11T22:08:33.036469+0000","last_became_peered":"2024-03-11T22:08:33.036469+0000","last_unstale":"2024-03-11T22:10:09.276964+0000","last_undegraded":"2024-03-11T22:10:09.276964+0000","last_fullsized":"2024-03-11T22:10:09.276964+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:36.564912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:36.564912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:49.963865+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":93,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276180+0000","last_change":"2024-03-11T22:08:35.086576+0000","last_active":"2024-03-11T22:10:09.276180+0000","last_peered":"2024-03-11T22:10:09.276180+0000","last_clean":"2024-03-11T22:10:09.276180+0000","last_became_active":"2024-03-11T22:08:35.085979+0000","last_became_peered":"2024-03-11T22:08:35.085979+0000","last_unstale":"2024-03-11T22:10:09.276180+0000","last_undegraded":"2024-03-11T22:10:09.276180+0000","last_fullsized":"2024-03-11T22:10:09.276180+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418332+0000","last_change":"2024-03-11T22:09:54.458851+0000","last_active":"2024-03-11T22:10:09.418332+0000","last_peered":"2024-03-11T22:10:09.418332+0000","last_clean":"2024-03-11T22:10:09.418332+0000","last_became_active":"2024-03-11T22:08:37.076522+0000","last_became_peered":"2024-03-11T22:08:37.076522+0000","last_unstale":"2024-03-11T22:10:09.418332+0000","last_undegraded":"2024-03-11T22:10:09.418332+0000","last_fullsized":"2024-03-11T22:10:09.418332+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:54.458752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:09:54.458752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.358836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484474+0000","last_change":"2024-03-11T22:08:39.080281+0000","last_active":"2024-03-11T22:10:09.484474+0000","last_peered":"2024-03-11T22:10:09.484474+0000","last_clean":"2024-03-11T22:10:09.484474+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:10:09.484474+0000","last_undegraded":"2024-03-11T22:10:09.484474+0000","last_fullsized":"2024-03-11T22:10:09.484474+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":85,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.419169+0000","last_change":"2024-03-11T22:08:37.077452+0000","last_active":"2024-03-11T22:10:09.419169+0000","last_peered":"2024-03-11T22:10:09.419169+0000","last_clean":"2024-03-11T22:10:09.419169+0000","last_became_active":"2024-03-11T22:08:37.077142+0000","last_became_peered":"2024-03-11T22:08:37.077142+0000","last_unstale":"2024-03-11T22:10:09.419169+0000","last_undegraded":"2024-03-11T22:10:09.419169+0000","last_fullsized":"2024-03-11T22:10:09.419169+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276185+0000","last_change":"2024-03-11T22:08:33.037839+0000","last_active":"2024-03-11T22:10:09.276185+0000","last_peered":"2024-03-11T22:10:09.276185+0000","last_clean":"2024-03-11T22:10:09.276185+0000","last_became_active":"2024-03-11T22:08:33.037537+0000","last_became_peered":"2024-03-11T22:08:33.037537+0000","last_unstale":"2024-03-11T22:10:09.276185+0000","last_undegraded":"2024-03-11T22:10:09.276185+0000","last_fullsized":"2024-03-11T22:10:09.276185+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":94,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.616081+0000","last_change":"2024-03-11T22:10:09.616153+0000","last_active":"2024-03-11T22:10:09.616081+0000","last_peered":"2024-03-11T22:10:09.616081+0000","last_clean":"2024-03-11T22:10:09.616081+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:10:09.616081+0000","last_undegraded":"2024-03-11T22:10:09.616081+0000","last_fullsized":"2024-03-11T22:10:09.616081+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:10:09.615965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:09.615965+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:39.584538+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276283+0000","last_change":"2024-03-11T22:08:39.077317+0000","last_active":"2024-03-11T22:10:09.276283+0000","last_peered":"2024-03-11T22:10:09.276283+0000","last_clean":"2024-03-11T22:10:09.276283+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:10:09.276283+0000","last_undegraded":"2024-03-11T22:10:09.276283+0000","last_fullsized":"2024-03-11T22:10:09.276283+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.484819+0000","last_change":"2024-03-11T22:08:37.082228+0000","last_active":"2024-03-11T22:10:09.484819+0000","last_peered":"2024-03-11T22:10:09.484819+0000","last_clean":"2024-03-11T22:10:09.484819+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:10:09.484819+0000","last_undegraded":"2024-03-11T22:10:09.484819+0000","last_fullsized":"2024-03-11T22:10:09.484819+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418068+0000","last_change":"2024-03-11T22:08:33.300125+0000","last_active":"2024-03-11T22:10:09.418068+0000","last_peered":"2024-03-11T22:10:09.418068+0000","last_clean":"2024-03-11T22:10:09.418068+0000","last_became_active":"2024-03-11T22:08:33.299778+0000","last_became_peered":"2024-03-11T22:08:33.299778+0000","last_unstale":"2024-03-11T22:10:09.418068+0000","last_undegraded":"2024-03-11T22:10:09.418068+0000","last_fullsized":"2024-03-11T22:10:09.418068+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:08:32.020092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":88,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.419075+0000","last_change":"2024-03-11T22:09:40.448862+0000","last_active":"2024-03-11T22:10:09.419075+0000","last_peered":"2024-03-11T22:10:09.419075+0000","last_clean":"2024-03-11T22:10:09.419075+0000","last_became_active":"2024-03-11T22:08:35.053123+0000","last_became_peered":"2024-03-11T22:08:35.053123+0000","last_unstale":"2024-03-11T22:10:09.419075+0000","last_undegraded":"2024-03-11T22:10:09.419075+0000","last_fullsized":"2024-03-11T22:10:09.419075+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:09:40.448751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:40.448751+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:07.077279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":61,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179020+0000","last_change":"2024-03-11T22:10:04.693960+0000","last_active":"2024-03-11T22:10:09.179020+0000","last_peered":"2024-03-11T22:10:09.179020+0000","last_clean":"2024-03-11T22:10:09.179020+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:10:09.179020+0000","last_undegraded":"2024-03-11T22:10:09.179020+0000","last_fullsized":"2024-03-11T22:10:09.179020+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:04.693849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:04.693849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:24.922133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.316908+0000","last_change":"2024-03-11T22:08:37.078886+0000","last_active":"2024-03-11T22:10:09.316908+0000","last_peered":"2024-03-11T22:10:09.316908+0000","last_clean":"2024-03-11T22:10:09.316908+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:10:09.316908+0000","last_undegraded":"2024-03-11T22:10:09.316908+0000","last_fullsized":"2024-03-11T22:10:09.316908+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:08:36.051029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.179003+0000","last_change":"2024-03-11T22:09:46.542145+0000","last_active":"2024-03-11T22:10:09.179003+0000","last_peered":"2024-03-11T22:10:09.179003+0000","last_clean":"2024-03-11T22:10:09.179003+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:10:09.179003+0000","last_undegraded":"2024-03-11T22:10:09.179003+0000","last_fullsized":"2024-03-11T22:10:09.179003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:46.542054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:46.542054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:02.271294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":89,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418835+0000","last_change":"2024-03-11T22:08:35.055974+0000","last_active":"2024-03-11T22:10:09.418835+0000","last_peered":"2024-03-11T22:10:09.418835+0000","last_clean":"2024-03-11T22:10:09.418835+0000","last_became_active":"2024-03-11T22:08:35.055770+0000","last_became_peered":"2024-03-11T22:08:35.055770+0000","last_unstale":"2024-03-11T22:10:09.418835+0000","last_undegraded":"2024-03-11T22:10:09.418835+0000","last_fullsized":"2024-03-11T22:10:09.418835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.419185+0000","last_change":"2024-03-11T22:08:39.073197+0000","last_active":"2024-03-11T22:10:09.419185+0000","last_peered":"2024-03-11T22:10:09.419185+0000","last_clean":"2024-03-11T22:10:09.419185+0000","last_became_active":"2024-03-11T22:08:39.072422+0000","last_became_peered":"2024-03-11T22:08:39.072422+0000","last_unstale":"2024-03-11T22:10:09.419185+0000","last_undegraded":"2024-03-11T22:10:09.419185+0000","last_fullsized":"2024-03-11T22:10:09.419185+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:08:38.057455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":94,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.178670+0000","last_change":"2024-03-11T22:09:38.769937+0000","last_active":"2024-03-11T22:10:09.178670+0000","last_peered":"2024-03-11T22:10:09.178670+0000","last_clean":"2024-03-11T22:10:09.178670+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:10:09.178670+0000","last_undegraded":"2024-03-11T22:10:09.178670+0000","last_fullsized":"2024-03-11T22:10:09.178670+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_clean_scrub_stamp":"2024-03-11T22:09:38.769825+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:10:58.315026+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.276661+0000","last_change":"2024-03-11T22:09:58.618349+0000","last_active":"2024-03-11T22:10:09.276661+0000","last_peered":"2024-03-11T22:10:09.276661+0000","last_clean":"2024-03-11T22:10:09.276661+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:10:09.276661+0000","last_undegraded":"2024-03-11T22:10:09.276661+0000","last_fullsized":"2024-03-11T22:10:09.276661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:58.618258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:58.618258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:11:08.349075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":76,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-11T22:10:09.418731+0000","last_change":"2024-03-11T22:08:35.057039+0000","last_active":"2024-03-11T22:10:09.418731+0000","last_peered":"2024-03-11T22:10:09.418731+0000","last_clean":"2024-03-11T22:10:09.418731+0000","last_became_active":"2024-03-11T22:08:35.056698+0000","last_became_peered":"2024-03-11T22:08:35.056698+0000","last_unstale":"2024-03-11T22:10:09.418731+0000","last_undegraded":"2024-03-11T22:10:09.418731+0000","last_fullsized":"2024-03-11T22:10:09.418731+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:08:34.037605+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":85,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005080,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28324,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716828,"statfs":{"total":95995035648,"available":95966031872,"internally_reserved":0,"allocated":1335296,"data_stored":942628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332125,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28316,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716836,"statfs":{"total":95995035648,"available":95966040064,"internally_reserved":0,"allocated":1327104,"data_stored":950196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31948,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713204,"statfs":{"total":95995035648,"available":95962320896,"internally_reserved":0,"allocated":782336,"data_stored":360024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32480,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712672,"statfs":{"total":95995035648,"available":95961776128,"internally_reserved":0,"allocated":1331200,"data_stored":941960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":729088,"data_stored":359654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":770048,"data_stored":360334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920808,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31908,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713244,"statfs":{"total":95995035648,"available":95962361856,"internally_reserved":0,"allocated":745472,"data_stored":351801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659169,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":724992,"data_stored":351343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:10:14.182 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T22:10:14.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd dump 2024-03-11T22:10:14.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:14 smithi144 ceph-mon[33984]: 6.10 scrub starts 2024-03-11T22:10:14.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:14 smithi144 ceph-mon[33984]: 6.10 scrub ok 2024-03-11T22:10:14.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:14 smithi144 ceph-mon[33984]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:10:14.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:10:14.431 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:14 smithi045 ceph-mon[30178]: 6.10 scrub starts 2024-03-11T22:10:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:14 smithi045 ceph-mon[30178]: 6.10 scrub ok 2024-03-11T22:10:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:14 smithi045 ceph-mon[30178]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:10:14.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:10:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:14 smithi045 ceph-mon[39715]: 6.10 scrub starts 2024-03-11T22:10:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:14 smithi045 ceph-mon[39715]: 6.10 scrub ok 2024-03-11T22:10:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:14 smithi045 ceph-mon[39715]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:10:14.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:10:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:15 smithi045 ceph-mon[39715]: 4.18 scrub starts 2024-03-11T22:10:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:15 smithi045 ceph-mon[39715]: 4.18 scrub ok 2024-03-11T22:10:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:15 smithi045 ceph-mon[39715]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:10:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:15 smithi045 ceph-mon[39715]: 2.0 deep-scrub starts 2024-03-11T22:10:15.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:15 smithi045 ceph-mon[39715]: 2.0 deep-scrub ok 2024-03-11T22:10:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:15 smithi045 ceph-mon[30178]: 4.18 scrub starts 2024-03-11T22:10:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:15 smithi045 ceph-mon[30178]: 4.18 scrub ok 2024-03-11T22:10:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:15 smithi045 ceph-mon[30178]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:10:15.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:15 smithi045 ceph-mon[30178]: 2.0 deep-scrub starts 2024-03-11T22:10:15.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:15 smithi045 ceph-mon[30178]: 2.0 deep-scrub ok 2024-03-11T22:10:15.573 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:15 smithi144 ceph-mon[33984]: 4.18 scrub starts 2024-03-11T22:10:15.573 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:15 smithi144 ceph-mon[33984]: 4.18 scrub ok 2024-03-11T22:10:15.573 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:15 smithi144 ceph-mon[33984]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:10:15.573 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:15 smithi144 ceph-mon[33984]: 2.0 deep-scrub starts 2024-03-11T22:10:15.573 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:15 smithi144 ceph-mon[33984]: 2.0 deep-scrub ok 2024-03-11T22:10:15.839 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:10:15 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:10:16.198 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 systemd[1]: Starting Ceph grafana.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:10:16.397 INFO:teuthology.orchestra.run.smithi045.stdout:epoch 95 2024-03-11T22:10:16.397 INFO:teuthology.orchestra.run.smithi045.stdout:fsid f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:10:16.397 INFO:teuthology.orchestra.run.smithi045.stdout:created 2024-03-11T22:02:51.075940+0000 2024-03-11T22:10:16.397 INFO:teuthology.orchestra.run.smithi045.stdout:modified 2024-03-11T22:10:09.175391+0000 2024-03-11T22:10:16.398 INFO:teuthology.orchestra.run.smithi045.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-11T22:10:16.398 INFO:teuthology.orchestra.run.smithi045.stdout:crush_version 18 2024-03-11T22:10:16.398 INFO:teuthology.orchestra.run.smithi045.stdout:full_ratio 0.95 2024-03-11T22:10:16.398 INFO:teuthology.orchestra.run.smithi045.stdout:backfillfull_ratio 0.9 2024-03-11T22:10:16.398 INFO:teuthology.orchestra.run.smithi045.stdout:nearfull_ratio 0.85 2024-03-11T22:10:16.398 INFO:teuthology.orchestra.run.smithi045.stdout:require_min_compat_client luminous 2024-03-11T22:10:16.398 INFO:teuthology.orchestra.run.smithi045.stdout:min_compat_client luminous 2024-03-11T22:10:16.398 INFO:teuthology.orchestra.run.smithi045.stdout:require_osd_release squid 2024-03-11T22:10:16.398 INFO:teuthology.orchestra.run.smithi045.stdout:stretch_mode_enabled false 2024-03-11T22:10:16.398 INFO:teuthology.orchestra.run.smithi045.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-11T22:10:16.398 INFO:teuthology.orchestra.run.smithi045.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 71 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-11T22:10:16.398 INFO:teuthology.orchestra.run.smithi045.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 69 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-11T22:10:16.398 INFO:teuthology.orchestra.run.smithi045.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 73 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 75 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:max_osd 8 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 up in weight 1 up_from 8 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6802/2216353820,v1:172.21.15.45:6803/2216353820] [v2:172.21.15.45:6804/2216353820,v1:172.21.15.45:6805/2216353820] exists,up 55a4e633-f2a6-4d27-b10c-de1c0f87cf32 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 up in weight 1 up_from 13 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6810/1241486091,v1:172.21.15.45:6811/1241486091] [v2:172.21.15.45:6812/1241486091,v1:172.21.15.45:6813/1241486091] exists,up 60274eeb-591a-43fe-a579-f13d13631e14 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 up in weight 1 up_from 18 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518] [v2:172.21.15.45:6820/4251983518,v1:172.21.15.45:6821/4251983518] exists,up 15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 up in weight 1 up_from 26 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830] [v2:172.21.15.45:6828/3178755830,v1:172.21.15.45:6829/3178755830] exists,up e51a65f3-96f6-478c-8d26-df1ace4a5058 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 up in weight 1 up_from 33 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6800/912745159,v1:172.21.15.144:6801/912745159] [v2:172.21.15.144:6802/912745159,v1:172.21.15.144:6803/912745159] exists,up 32d0b71f-c696-4677-bdd3-b88a04c97604 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 up in weight 1 up_from 41 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6808/3162869174,v1:172.21.15.144:6809/3162869174] [v2:172.21.15.144:6810/3162869174,v1:172.21.15.144:6811/3162869174] exists,up f46b91af-d67d-404c-aad1-75348e35d1c4 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 up in weight 1 up_from 51 up_thru 71 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6816/3439446288,v1:172.21.15.144:6817/3439446288] [v2:172.21.15.144:6818/3439446288,v1:172.21.15.144:6819/3439446288] exists,up 970b128d-cd56-4dff-8345-3196b46c1257 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 up in weight 1 up_from 61 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6824/2371064429,v1:172.21.15.144:6825/2371064429] [v2:172.21.15.144:6826/2371064429,v1:172.21.15.144:6827/2371064429] exists,up 40100aaf-830d-4137-b037-5cf167cba308 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 4.9 [1,5] 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 4.b [1,2] 2024-03-11T22:10:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:pg_upmap_items 4.10 [1,2] 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/1901139892 expires 2024-03-12T22:09:14.021723+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/1110781721 expires 2024-03-12T22:03:52.557006+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6800/2299156110 expires 2024-03-12T22:03:25.087713+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/2331445880 expires 2024-03-12T22:03:25.087713+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6801/4256719031 expires 2024-03-12T22:03:52.557006+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6801/2299156110 expires 2024-03-12T22:03:25.087713+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6800/3882489981 expires 2024-03-12T22:09:14.021723+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/272906373 expires 2024-03-12T22:09:14.021723+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/11898002 expires 2024-03-12T22:09:14.021723+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/2777740355 expires 2024-03-12T22:03:25.087713+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/1903379086 expires 2024-03-12T22:09:14.021723+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/926341889 expires 2024-03-12T22:03:25.087713+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6800/4256719031 expires 2024-03-12T22:03:52.557006+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/2513993410 expires 2024-03-12T22:03:52.557006+0000 2024-03-11T22:10:16.400 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/2584267737 expires 2024-03-12T22:03:52.557006+0000 2024-03-11T22:10:16.401 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:0/3509006862 expires 2024-03-12T22:09:14.021723+0000 2024-03-11T22:10:16.401 INFO:teuthology.orchestra.run.smithi045.stdout:blocklist 172.21.15.45:6801/3882489981 expires 2024-03-12T22:09:14.021723+0000 2024-03-11T22:10:16.511 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-mon[33984]: 5.e scrub starts 2024-03-11T22:10:16.511 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-mon[33984]: 5.e scrub ok 2024-03-11T22:10:16.512 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-mon[33984]: 6.18 scrub starts 2024-03-11T22:10:16.512 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-mon[33984]: 6.18 scrub ok 2024-03-11T22:10:16.512 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-mon[33984]: 6.15 deep-scrub starts 2024-03-11T22:10:16.512 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-mon[33984]: 6.15 deep-scrub ok 2024-03-11T22:10:16.512 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-mon[33984]: 5.2 scrub starts 2024-03-11T22:10:16.512 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-mon[33984]: 5.2 scrub ok 2024-03-11T22:10:16.512 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 podman[67522]: 2024-03-11 22:10:16.192204572 +0000 UTC m=+0.017470177 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-11T22:10:16.512 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 podman[67522]: 2024-03-11 22:10:16.323076209 +0000 UTC m=+0.148341800 container create 5d44c69f500708ad3bf7fc5a1006d65329450fbd6c66623e2bd4d4e5e5f468f3 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, architecture=x86_64, release=239.1651231664, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc.) 2024-03-11T22:10:16.512 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 podman[67522]: 2024-03-11 22:10:16.517337179 +0000 UTC m=+0.342602788 container init 5d44c69f500708ad3bf7fc5a1006d65329450fbd6c66623e2bd4d4e5e5f468f3 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, com.redhat.component=ubi8-container, distribution-scope=public, io.openshift.expose-services=, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , description=Ceph Grafana Container, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-03-11T22:10:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[39715]: 5.e scrub starts 2024-03-11T22:10:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[39715]: 5.e scrub ok 2024-03-11T22:10:16.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[39715]: 6.18 scrub starts 2024-03-11T22:10:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[39715]: 6.18 scrub ok 2024-03-11T22:10:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[39715]: 6.15 deep-scrub starts 2024-03-11T22:10:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[39715]: 6.15 deep-scrub ok 2024-03-11T22:10:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[39715]: 5.2 scrub starts 2024-03-11T22:10:16.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[39715]: 5.2 scrub ok 2024-03-11T22:10:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[30178]: 5.e scrub starts 2024-03-11T22:10:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[30178]: 5.e scrub ok 2024-03-11T22:10:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[30178]: 6.18 scrub starts 2024-03-11T22:10:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[30178]: 6.18 scrub ok 2024-03-11T22:10:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[30178]: 6.15 deep-scrub starts 2024-03-11T22:10:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[30178]: 6.15 deep-scrub ok 2024-03-11T22:10:16.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[30178]: 5.2 scrub starts 2024-03-11T22:10:16.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:16 smithi045 ceph-mon[30178]: 5.2 scrub ok 2024-03-11T22:10:16.764 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 podman[67522]: 2024-03-11 22:10:16.523494421 +0000 UTC m=+0.348760015 container start 5d44c69f500708ad3bf7fc5a1006d65329450fbd6c66623e2bd4d4e5e5f468f3 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, vcs-type=git, name=ubi8, com.redhat.component=ubi8-container, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, maintainer=Paul Cuzner , io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., architecture=x86_64) 2024-03-11T22:10:16.764 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 bash[67522]: 5d44c69f500708ad3bf7fc5a1006d65329450fbd6c66623e2bd4d4e5e5f468f3 2024-03-11T22:10:16.764 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 systemd[1]: Started Ceph grafana.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:10:16.764 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.689300805Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-11T22:10:16.764 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.689526092Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-11T22:10:16.764 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.689545012Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-11T22:10:16.765 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.689552303Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-11T22:10:16.765 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.689558121Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-11T22:10:16.765 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.68956434Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-11T22:10:16.765 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.689570581Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-11T22:10:16.765 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.689576397Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-11T22:10:16.765 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.689582443Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-11T22:10:16.765 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.689588027Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-11T22:10:16.766 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.689593407Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-11T22:10:16.766 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.68959876Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-11T22:10:16.766 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.689604341Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-11T22:10:16.766 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=settings t=2024-03-11T22:10:16.689609953Z level=info msg="App mode production" 2024-03-11T22:10:16.766 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=sqlstore t=2024-03-11T22:10:16.689656421Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=sqlstore t=2024-03-11T22:10:16.689673595Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.690511341Z level=info msg="Starting DB migrations" 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.69060403Z level=info msg="Executing migration" id="create migration_log table" 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.691106769Z level=info msg="Executing migration" id="create user table" 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.691537439Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.692018015Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.692448379Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.693009716Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.693928574Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.694680592Z level=info msg="Executing migration" id="create user table v2" 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.695088084Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.695530127Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.695987653Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.696392745Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-11T22:10:16.767 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.696749493Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.69719427Z level=info msg="Executing migration" id="Update user table charset" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.697517888Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.697923134Z level=info msg="Executing migration" id="Add missing user data" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.698300116Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.698741225Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.699147619Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.699583926Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.700719174Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.701195936Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.701642063Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.702038996Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.702428982Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.702820366Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.703079241Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-11T22:10:16.768 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.703445284Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.703791827Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.704167733Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.704562534Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.705201933Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.705622827Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.706044653Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.706483642Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.706862336Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.707317593Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.707667997Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.708002207Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.708378587Z level=info msg="Executing migration" id="create star table" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.708740659Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-11T22:10:16.769 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.709185064Z level=info msg="Executing migration" id="create org table v1" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.709625873Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.71007455Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.710489903Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.710924286Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.711378738Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.711820912Z level=info msg="Executing migration" id="Update org table charset" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.712106596Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.712423073Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.712727844Z level=info msg="Executing migration" id="create dashboard table" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.713125456Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.71354472Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.714307692Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.714724706Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-11T22:10:16.770 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.71515919Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.715995836Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.71687813Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.717735014Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.718188555Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.719102146Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.719856858Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.720281851Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.720609828Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.72109189Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.721602613Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.722083189Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.722525988Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.723000469Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-11T22:10:16.771 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.723507375Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.723906119Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.7242273Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.724745007Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.725291092Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.726126751Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.726650534Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.72749474Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.728394787Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.729420366Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.729837349Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.730749938Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.731359331Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.731779871Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.73270633Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-11T22:10:16.772 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.733143441Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.736103606Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.738908758Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.741774249Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.742190355Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.745036098Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.747862517Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.750756279Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.753692366Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.756560419Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.757179496Z level=info msg="Executing migration" id="create data_source table" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.760119132Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.762921575Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.763390428Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-11T22:10:16.773 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.763893081Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-11T22:10:16.774 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.766793214Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-11T22:10:16.774 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.769863527Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-11T22:10:16.774 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.772454345Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-11T22:10:17.015 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.772927163Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-11T22:10:17.015 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.775551895Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-11T22:10:17.015 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.778090471Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-11T22:10:17.015 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.780690629Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-11T22:10:17.015 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.781234652Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-11T22:10:17.015 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.783949024Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-11T22:10:17.015 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.786417994Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-11T22:10:17.015 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.789024509Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-11T22:10:17.015 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.789378956Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-11T22:10:17.015 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.791902587Z level=info msg="Executing migration" id="Add uid column" 2024-03-11T22:10:17.015 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.792487445Z level=info msg="Executing migration" id="Update uid value" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.794508525Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.797109682Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.799802101Z level=info msg="Executing migration" id="create api_key table" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.802228067Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.804673094Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.807292519Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.807771136Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.810294091Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.811261939Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.812002185Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.813010916Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.813909807Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.814578184Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-11T22:10:17.016 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.81556397Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-11T22:10:17.017 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.816297699Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-11T22:10:17.017 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.817190061Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-11T22:10:17.017 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.817886562Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-11T22:10:17.017 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.818778906Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-11T22:10:17.017 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.819518033Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-11T22:10:17.017 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.820374338Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-11T22:10:17.017 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.821202166Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-11T22:10:17.017 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.82211887Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-11T22:10:17.017 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.822847279Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-11T22:10:17.017 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.823846235Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-11T22:10:17.017 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.824578594Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-11T22:10:17.017 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.825550271Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-11T22:10:17.017 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.826426078Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-11T22:10:17.019 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.827166715Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-11T22:10:17.019 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.828117697Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-11T22:10:17.019 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.828837617Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-11T22:10:17.019 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.829656911Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-11T22:10:17.019 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.830242083Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-11T22:10:17.019 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.830795345Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-11T22:10:17.019 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.831617421Z level=info msg="Executing migration" id="create quota table v1" 2024-03-11T22:10:17.019 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.832049267Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-11T22:10:17.019 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.832486516Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-11T22:10:17.019 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.833314956Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-11T22:10:17.019 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.833705619Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-11T22:10:17.019 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.834158218Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.835006Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.83817566Z level=info msg="Executing migration" id="create session table" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.840627906Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.843326516Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.846184697Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.846568842Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.849454891Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.852340442Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.855123837Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.855412001Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.858433795Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.858926873Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.861201191Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.863985225Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-11T22:10:17.020 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.864297987Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.86727793Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.868109093Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.870098109Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.872878211Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.873434188Z level=info msg="Executing migration" id="create alert table v1" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.876148666Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.878921644Z level=info msg="Executing migration" id="add index alert state" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.881738236Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.88216942Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.884714215Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.887410043Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.890129429Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.891506541Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.894127177Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-11T22:10:17.021 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.896847911Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.899541981Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.89989846Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.902786035Z level=info msg="Executing migration" id="Add column is_default" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.905365884Z level=info msg="Executing migration" id="Add column frequency" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.908067971Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.908714751Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.909708521Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.910344578Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.911181782Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.911466908Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.912441818Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.91315201Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.914107818Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-11T22:10:17.022 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.914590597Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.915393648Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.916045419Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.916705965Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.917492403Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.917922189Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.91831464Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.919129274Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.92006583Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.920809739Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.921230957Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.922192722Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.922933146Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.923907835Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-11T22:10:17.023 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.924399172Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.925276817Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.92612234Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.926987549Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.927288394Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.928150677Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.928558099Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.929221108Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.930233568Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.930835071Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.93176629Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.933513831Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.934473053Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.935064845Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-11T22:10:17.024 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.938058043Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-11T22:10:17.025 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.940681054Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-11T22:10:17.025 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.943306805Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-11T22:10:17.025 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.94406654Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-11T22:10:17.025 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.946746758Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-11T22:10:17.025 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.949699758Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-11T22:10:17.025 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.95210925Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-11T22:10:17.025 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.952434569Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-11T22:10:17.025 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.955255869Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-11T22:10:17.025 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.955754844Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-11T22:10:17.025 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.958239438Z level=info msg="Executing migration" id="Move region to single row" 2024-03-11T22:10:17.025 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.958693487Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-11T22:10:17.025 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.960873417Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-11T22:10:17.025 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.961335497Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-11T22:10:17.026 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.961800441Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-11T22:10:17.026 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.96466145Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-11T22:10:17.026 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.967369837Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-11T22:10:17.026 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.970175038Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-11T22:10:17.026 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.970671047Z level=info msg="Executing migration" id="create test_data table" 2024-03-11T22:10:17.026 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.973197466Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-11T22:10:17.026 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.976120617Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-11T22:10:17.026 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.978751545Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-11T22:10:17.026 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.979293302Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-11T22:10:17.026 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.981912746Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-11T22:10:17.026 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.984623582Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-11T22:10:17.026 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.987274079Z level=info msg="Executing migration" id="create team table" 2024-03-11T22:10:17.026 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.987742186Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.990496054Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.993095597Z level=info msg="Executing migration" id="create team member table" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.995646972Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.996278766Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:16.999157213Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.001787514Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.004499058Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.005234026Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.006103602Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.006907218Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.007844718Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.008402166Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.008836011Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.009741224Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-11T22:10:17.027 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.010616525Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-11T22:10:17.028 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.011659845Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-11T22:10:17.028 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.012525434Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-11T22:10:17.028 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.013261195Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-11T22:10:17.028 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.014130612Z level=info msg="Executing migration" id="create tag table" 2024-03-11T22:10:17.028 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.015063531Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-11T22:10:17.028 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.016135313Z level=info msg="Executing migration" id="create login attempt table" 2024-03-11T22:10:17.028 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.016841849Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-11T22:10:17.028 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.017797396Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-11T22:10:17.028 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.018527692Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-11T22:10:17.028 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.020388093Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-11T22:10:17.028 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.021268221Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-11T22:10:17.028 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.022121944Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-11T22:10:17.222 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6802/2216353820,v1:172.21.15.45:6803/2216353820] [v2:172.21.15.45:6804/2216353820,v1:172.21.15.45:6805/2216353820] exists,up 55a4e633-f2a6-4d27-b10c-de1c0f87cf32', 'osd.1 up in weight 1 up_from 13 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6810/1241486091,v1:172.21.15.45:6811/1241486091] [v2:172.21.15.45:6812/1241486091,v1:172.21.15.45:6813/1241486091] exists,up 60274eeb-591a-43fe-a579-f13d13631e14', 'osd.2 up in weight 1 up_from 18 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6818/4251983518,v1:172.21.15.45:6819/4251983518] [v2:172.21.15.45:6820/4251983518,v1:172.21.15.45:6821/4251983518] exists,up 15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b', 'osd.3 up in weight 1 up_from 26 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6826/3178755830,v1:172.21.15.45:6827/3178755830] [v2:172.21.15.45:6828/3178755830,v1:172.21.15.45:6829/3178755830] exists,up e51a65f3-96f6-478c-8d26-df1ace4a5058', 'osd.4 up in weight 1 up_from 33 up_thru 78 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6800/912745159,v1:172.21.15.144:6801/912745159] [v2:172.21.15.144:6802/912745159,v1:172.21.15.144:6803/912745159] exists,up 32d0b71f-c696-4677-bdd3-b88a04c97604', 'osd.5 up in weight 1 up_from 41 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6808/3162869174,v1:172.21.15.144:6809/3162869174] [v2:172.21.15.144:6810/3162869174,v1:172.21.15.144:6811/3162869174] exists,up f46b91af-d67d-404c-aad1-75348e35d1c4', 'osd.6 up in weight 1 up_from 51 up_thru 71 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6816/3439446288,v1:172.21.15.144:6817/3439446288] [v2:172.21.15.144:6818/3439446288,v1:172.21.15.144:6819/3439446288] exists,up 970b128d-cd56-4dff-8345-3196b46c1257', 'osd.7 up in weight 1 up_from 61 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6824/2371064429,v1:172.21.15.144:6825/2371064429] [v2:172.21.15.144:6826/2371064429,v1:172.21.15.144:6827/2371064429] exists,up 40100aaf-830d-4137-b037-5cf167cba308'] 2024-03-11T22:10:17.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon mon.a config show 2024-03-11T22:10:17.266 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:17.266 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-mon[33984]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:10:17.266 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2278787025' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T22:10:17.266 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-mon[33984]: 6.12 scrub starts 2024-03-11T22:10:17.266 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-mon[33984]: 6.12 scrub ok 2024-03-11T22:10:17.266 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-mon[33984]: 4.17 scrub starts 2024-03-11T22:10:17.266 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-mon[33984]: 4.17 scrub ok 2024-03-11T22:10:17.266 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:17.266 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:17.266 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:17.267 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:17.267 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.025201776Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-11T22:10:17.267 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.027877669Z level=info msg="Executing migration" id="create user auth table" 2024-03-11T22:10:17.267 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.030852191Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-11T22:10:17.267 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.033775114Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-11T22:10:17.267 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.036631159Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-11T22:10:17.267 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.039495168Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.042441765Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.045277522Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.048352764Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.051069945Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.054275776Z level=info msg="Executing migration" id="create server_lock table" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.057838897Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.060555188Z level=info msg="Executing migration" id="create user auth token table" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.063427369Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.066196502Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.0688306Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.071827919Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.074820048Z level=info msg="Executing migration" id="create cache_data table" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.077293719Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-11T22:10:17.268 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.080060742Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.082573175Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.085062755Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.087841722Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.088164017Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.088618448Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.091223501Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.091770488Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.092250099Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.092653142Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.093032758Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.093447873Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.093856351Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.094626619Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.095057249Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-11T22:10:17.269 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.095349423Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-11T22:10:17.270 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.095716188Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-11T22:10:17.270 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.096102026Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-11T22:10:17.270 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.09652536Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.096794786Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.097219094Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.097655043Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.098041006Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.098501268Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.099284188Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.099720001Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.100101866Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.103419835Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.10678839Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.107254541Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.107657776Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.108515373Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.108916797Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-11T22:10:17.271 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.109337088Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.109748122Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.110161314Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.110441914Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.111245394Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.112036658Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.112835859Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.113250254Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.11365802Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.114481635Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.115281826Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.115699613Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.11652804Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.117339142Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-11T22:10:17.272 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.117641066Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.118097846Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.120440449Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.120894274Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.121187611Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.122054777Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.124655201Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.12562416Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.126487564Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.127313395Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.127607675Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.12798552Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.12881944Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.129083425Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.129958586Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-11T22:10:17.273 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.13265391Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.135336927Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.138272465Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.141010864Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.143617049Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.145879274Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.148621787Z level=info msg="Executing migration" id="create alert_image table" 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.151144774Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.153910035Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.154334761Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.154771002Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.155615749Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.155689093Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.156567476Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-11T22:10:17.274 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.157306767Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-11T22:10:17.275 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.158120968Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-11T22:10:17.275 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.158234622Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-11T22:10:17.275 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.158351086Z level=info msg="alerts found to migrate" alerts=0 2024-03-11T22:10:17.275 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.158985234Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-11T22:10:17.275 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.159867295Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-11T22:10:17.275 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.16069594Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-11T22:10:17.275 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.161725871Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-11T22:10:17.275 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.162420872Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-11T22:10:17.275 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.163271443Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-11T22:10:17.275 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.164190165Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-11T22:10:17.275 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.165088937Z level=info msg="Executing migration" id="create data_keys table" 2024-03-11T22:10:17.275 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.165744346Z level=info msg="Executing migration" id="create secrets table" 2024-03-11T22:10:17.275 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.166753134Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-11T22:10:17.276 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.171164212Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-11T22:10:17.276 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.17208008Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-11T22:10:17.276 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.172949195Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-11T22:10:17.276 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.177155605Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.181832946Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.18267598Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.183680011Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.184428101Z level=info msg="Executing migration" id="create permission table" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.185282441Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.186251667Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.187174876Z level=info msg="Executing migration" id="create role table" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.187875479Z level=info msg="Executing migration" id="add column display_name" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.188817471Z level=info msg="Executing migration" id="add column group_name" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.191491842Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.19419137Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.195194544Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.195898857Z level=info msg="Executing migration" id="create team role table" 2024-03-11T22:10:17.277 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.196833128Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.197572124Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.198399795Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.199236022Z level=info msg="Executing migration" id="create user role table" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.200208818Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.200919094Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.201865962Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.202608016Z level=info msg="Executing migration" id="create builtin role table" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.203380169Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.204173445Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.205207185Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.206340396Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.207185583Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.208013957Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-11T22:10:17.278 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.208881656Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.209718039Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.210573295Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.211459099Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.212444073Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.213280753Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.214282794Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.215001188Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.216164308Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.216706706Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.217546454Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.217890568Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.218773386Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.219627374Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-11T22:10:17.279 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.220656694Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.22169043Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.222663644Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.22341468Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.224393999Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.227345176Z level=info msg="Executing migration" id="add correlation config column" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.229775656Z level=info msg="Executing migration" id="create entity_events table" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.232620693Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.235461622Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.235556926Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.238375941Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.238473649Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.24107625Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.243922618Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-11T22:10:17.280 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.246609175Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:10:17.281 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.24945132Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:10:17.281 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.252213046Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-11T22:10:17.281 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.254907334Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T22:10:17.281 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.257810034Z level=info msg="Executing migration" id="Drop public config table" 2024-03-11T22:10:17.281 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.260511044Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-11T22:10:17.281 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.263389682Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-11T22:10:17.281 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.266357654Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T22:10:17.281 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.26916515Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-11T22:10:17.281 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.271976558Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-11T22:10:17.281 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.277285816Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-11T22:10:17.498 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:17.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[30178]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:10:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2278787025' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T22:10:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[30178]: 6.12 scrub starts 2024-03-11T22:10:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[30178]: 6.12 scrub ok 2024-03-11T22:10:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[30178]: 4.17 scrub starts 2024-03-11T22:10:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[30178]: 4.17 scrub ok 2024-03-11T22:10:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:17.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:17.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[39715]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:10:17.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2278787025' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T22:10:17.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[39715]: 6.12 scrub starts 2024-03-11T22:10:17.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[39715]: 6.12 scrub ok 2024-03-11T22:10:17.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[39715]: 4.17 scrub starts 2024-03-11T22:10:17.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[39715]: 4.17 scrub ok 2024-03-11T22:10:17.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:17.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:17.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:17.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:17 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:17.589 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.27987415Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-11T22:10:17.590 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.282526791Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-11T22:10:17.590 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.285299216Z level=info msg="Executing migration" id="add share column" 2024-03-11T22:10:17.590 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.288079999Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-11T22:10:17.590 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.290852538Z level=info msg="Executing migration" id="create file table" 2024-03-11T22:10:17.590 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.293266325Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-11T22:10:17.590 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.295838801Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-11T22:10:17.590 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.298637523Z level=info msg="Executing migration" id="create file_meta table" 2024-03-11T22:10:17.590 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.301253399Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-11T22:10:17.590 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.303745569Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-11T22:10:17.590 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.306397611Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-11T22:10:17.590 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.309236977Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-11T22:10:17.590 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.311871493Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-11T22:10:17.590 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.31473306Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-11T22:10:17.591 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.317344845Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-11T22:10:17.591 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.320143251Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-11T22:10:17.591 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.322762459Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-11T22:10:17.591 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.323516669Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-11T22:10:17.591 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.324317935Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-11T22:10:17.591 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.324909571Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-11T22:10:17.591 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.325945391Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-11T22:10:17.591 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.32698356Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.327903541Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.337310351Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.338123956Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.338965448Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.342431453Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.343278066Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.344017683Z level=info msg="Executing migration" id="create folder table" 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.344890514Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.345668048Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.346527891Z level=info msg="Executing migration" id="Update folder title length" 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.347337244Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=migrator t=2024-03-11T22:10:17.34827407Z level=info msg="migrations completed" performed=485 skipped=0 duration=657.699196ms 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=sqlstore t=2024-03-11T22:10:17.348777425Z level=info msg="Created default organization" 2024-03-11T22:10:17.592 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=plugin.loader t=2024-03-11T22:10:17.403648052Z level=info msg="Plugin registered" pluginID=input 2024-03-11T22:10:17.593 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=plugin.loader t=2024-03-11T22:10:17.433001879Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-11T22:10:17.593 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=plugin.loader t=2024-03-11T22:10:17.43305996Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-11T22:10:17.593 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=secrets t=2024-03-11T22:10:17.433096031Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-11T22:10:17.593 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=query_data t=2024-03-11T22:10:17.435240181Z level=info msg="Query Service initialization" 2024-03-11T22:10:17.593 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=live.push_http t=2024-03-11T22:10:17.437632723Z level=info msg="Live Push Gateway initialization" 2024-03-11T22:10:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:18 smithi045 ceph-mon[39715]: 5.c deep-scrub starts 2024-03-11T22:10:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:18 smithi045 ceph-mon[39715]: 5.c deep-scrub ok 2024-03-11T22:10:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:18 smithi045 ceph-mon[39715]: Deploying daemon node-exporter.a on smithi045 2024-03-11T22:10:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:18 smithi045 ceph-mon[39715]: 4.b scrub starts 2024-03-11T22:10:18.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:18 smithi045 ceph-mon[39715]: 4.b scrub ok 2024-03-11T22:10:18.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:18 smithi045 ceph-mon[30178]: 5.c deep-scrub starts 2024-03-11T22:10:18.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:18 smithi045 ceph-mon[30178]: 5.c deep-scrub ok 2024-03-11T22:10:18.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:18 smithi045 ceph-mon[30178]: Deploying daemon node-exporter.a on smithi045 2024-03-11T22:10:18.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:18 smithi045 ceph-mon[30178]: 4.b scrub starts 2024-03-11T22:10:18.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:18 smithi045 ceph-mon[30178]: 4.b scrub ok 2024-03-11T22:10:18.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-mon[33984]: 5.c deep-scrub starts 2024-03-11T22:10:18.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-mon[33984]: 5.c deep-scrub ok 2024-03-11T22:10:18.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-mon[33984]: Deploying daemon node-exporter.a on smithi045 2024-03-11T22:10:18.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-mon[33984]: 4.b scrub starts 2024-03-11T22:10:18.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-mon[33984]: 4.b scrub ok 2024-03-11T22:10:18.340 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=infra.usagestats.collector t=2024-03-11T22:10:18.040444977Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-11T22:10:18.340 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=provisioning.datasources t=2024-03-11T22:10:18.041441631Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T22:10:18.340 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=provisioning.datasources t=2024-03-11T22:10:18.052472689Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-11T22:10:18.340 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=provisioning.alerting t=2024-03-11T22:10:18.063463874Z level=info msg="starting to provision alerting" 2024-03-11T22:10:18.340 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=provisioning.alerting t=2024-03-11T22:10:18.063482814Z level=info msg="finished to provision alerting" 2024-03-11T22:10:18.340 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=ngalert.state.manager t=2024-03-11T22:10:18.063556591Z level=info msg="Warming state cache for startup" 2024-03-11T22:10:18.340 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=grafanaStorageLogger t=2024-03-11T22:10:18.063675781Z level=info msg="storage starting" 2024-03-11T22:10:18.340 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=ngalert.state.manager t=2024-03-11T22:10:18.06377817Z level=info msg="State cache has been initialized" states=0 duration=221.342µs 2024-03-11T22:10:18.340 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=ticker t=2024-03-11T22:10:18.063882717Z level=info msg=starting first_tick=2024-03-11T22:10:20Z 2024-03-11T22:10:18.340 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=ngalert.multiorg.alertmanager t=2024-03-11T22:10:18.063907902Z level=info msg="starting MultiOrg Alertmanager" 2024-03-11T22:10:18.341 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=http.server t=2024-03-11T22:10:18.065724439Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T22:10:18.341 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=sqlstore.transactions t=2024-03-11T22:10:18.081891397Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T22:10:18.341 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=sqlstore.transactions t=2024-03-11T22:10:18.093534915Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T22:10:18.341 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:10:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=sqlstore.transactions t=2024-03-11T22:10:18.105062692Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T22:10:19.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:19 smithi045 ceph-mon[30178]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:19.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:19 smithi045 ceph-mon[30178]: 3.17 scrub starts 2024-03-11T22:10:19.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:19 smithi045 ceph-mon[30178]: 3.17 scrub ok 2024-03-11T22:10:19.298 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:19.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:19 smithi045 ceph-mon[39715]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:19.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:19 smithi045 ceph-mon[39715]: 3.17 scrub starts 2024-03-11T22:10:19.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:19 smithi045 ceph-mon[39715]: 3.17 scrub ok 2024-03-11T22:10:19.299 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:19.299 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:10:19.299 INFO:teuthology.orchestra.run.smithi045.stdout: "name": "mon.a", 2024-03-11T22:10:19.300 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster": "ceph", 2024-03-11T22:10:19.300 INFO:teuthology.orchestra.run.smithi045.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-11T22:10:19.300 INFO:teuthology.orchestra.run.smithi045.stdout: "admin_socket_mode": "", 2024-03-11T22:10:19.300 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-11T22:10:19.300 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_client_required": "cephx, none", 2024-03-11T22:10:19.300 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_cluster_required": "cephx", 2024-03-11T22:10:19.300 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_debug": "true", 2024-03-11T22:10:19.300 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-11T22:10:19.300 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-11T22:10:19.300 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_service_required": "cephx", 2024-03-11T22:10:19.300 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-11T22:10:19.300 INFO:teuthology.orchestra.run.smithi045.stdout: "auth_supported": "", 2024-03-11T22:10:19.300 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_aio": "true", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_aio_poll_ms": "250", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_aio_reap_max": "16", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_async_discard": "false", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_block_size": "4096", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_debug_aio": "false", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_debug_inflight_ios": "false", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_enable_discard": "false", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_flock_retry": "3", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_inject_crash": "0", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-11T22:10:19.301 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_ioring": "false", 2024-03-11T22:10:19.302 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_ioring_hipri": "false", 2024-03-11T22:10:19.302 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-11T22:10:19.302 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-11T22:10:19.302 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-11T22:10:19.302 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-11T22:10:19.302 INFO:teuthology.orchestra.run.smithi045.stdout: "bdev_type": "", 2024-03-11T22:10:19.302 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_alloc_size": "1048576", 2024-03-11T22:10:19.302 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_allocator": "hybrid", 2024-03-11T22:10:19.302 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_buffered_io": "true", 2024-03-11T22:10:19.302 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_check_for_zeros": "false", 2024-03-11T22:10:19.302 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-11T22:10:19.302 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_compact_log_sync": "false", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_max_log_runway": "4194304", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_max_prefetch": "1048576", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_min_flush_size": "524288", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_min_log_runway": "1048576", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_replay_recovery": "false", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluefs_sync_write": "false", 2024-03-11T22:10:19.303 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_allocation_from_file": "true", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_allocator": "hybrid", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_block_create": "true", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_block_db_create": "false", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_block_db_path": "", 2024-03-11T22:10:19.304 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_block_db_size": "0", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_block_path": "", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_block_preallocate_file": "false", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_block_size": "107374182400", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_block_wal_create": "false", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_block_wal_path": "", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_block_wal_size": "100663296", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_bluefs": "true", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-11T22:10:19.305 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-11T22:10:19.306 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-11T22:10:19.306 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_autotune": "true", 2024-03-11T22:10:19.306 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-11T22:10:19.306 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-11T22:10:19.306 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-11T22:10:19.306 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-11T22:10:19.306 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_size": "0", 2024-03-11T22:10:19.306 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-11T22:10:19.306 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-11T22:10:19.306 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-11T22:10:19.306 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-11T22:10:19.306 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cache_type": "2q", 2024-03-11T22:10:19.306 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_clone_cow": "true", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_compression_mode": "none", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_csum_type": "crc32c", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_freelist": "false", 2024-03-11T22:10:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_small_allocations": "0", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-11T22:10:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_default_buffered_read": "true", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_default_buffered_write": "false", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fail_eio": "false", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-11T22:10:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-11T22:10:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-11T22:10:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-11T22:10:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_on_mount": "false", 2024-03-11T22:10:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-11T22:10:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_on_umount": "false", 2024-03-11T22:10:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-11T22:10:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-11T22:10:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-11T22:10:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-11T22:10:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-11T22:10:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-11T22:10:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-11T22:10:19.310 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_ignore_data_csum": "false", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_log_op_age": "5.000000", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_max_alloc_size": "0", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_max_blob_size": "0", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_max_deferred_txc": "32", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_min_alloc_size": "0", 2024-03-11T22:10:19.311 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-11T22:10:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-11T22:10:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_nid_prealloc": "1024", 2024-03-11T22:10:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-11T22:10:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-11T22:10:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-11T22:10:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-11T22:10:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_retry_disk_reads": "3", 2024-03-11T22:10:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_rocksdb_cf": "true", 2024-03-11T22:10:19.312 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-11T22:10:19.312 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:19 smithi045 bash[79367]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T22:10:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-11T22:10:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-11T22:10:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-11T22:10:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-11T22:10:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-11T22:10:19.313 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_spdk_mem": "512", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_tracing": "false", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-11T22:10:19.314 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "bluestore_zero_block_detection": "false", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-11T22:10:19.315 INFO:teuthology.orchestra.run.smithi045.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "cephx_cluster_require_signatures": "false", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "cephx_cluster_require_version": "2", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "cephx_require_signatures": "false", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "cephx_require_version": "2", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "cephx_service_require_signatures": "false", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "cephx_service_require_version": "2", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "cephx_sign_messages": "true", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "chdir": "", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "client_acl_type": "", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "client_asio_thread_count": "2", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "client_cache_mid": "0.750000", 2024-03-11T22:10:19.316 INFO:teuthology.orchestra.run.smithi045.stdout: "client_cache_size": "16384", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_caps_release_delay": "5", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_check_pool_perm": "true", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_debug_force_sync_read": "false", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_debug_getattr_caps": "false", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_debug_inject_tick_delay": "0", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_die_on_failed_remount": "false", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_dirsize_rbytes": "true", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_force_lazyio": "false", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_fs": "", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_inject_release_failure": "false", 2024-03-11T22:10:19.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_max_inline_size": "4096", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_mds_namespace": "", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_metadata": "", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_mount_gid": "-1", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_mount_timeout": "300", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_mount_uid": "-1", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_mountpoint": "/", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_notify_timeout": "10", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_oc": "true", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_oc_max_dirty": "104857600", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_oc_max_objects": "1000", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_oc_size": "209715200", 2024-03-11T22:10:19.318 INFO:teuthology.orchestra.run.smithi045.stdout: "client_oc_target_dirty": "8388608", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_permissions": "true", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_quota": "true", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_quota_df": "true", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_readahead_max_bytes": "0", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_readahead_max_periods": "4", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_readahead_min": "131072", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_reconnect_stale": "false", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_shutdown_timeout": "30", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_snapdir": ".snap", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_tick_interval": "1", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_trace": "", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_try_dentry_invalidate": "false", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_use_faked_inos": "false", 2024-03-11T22:10:19.319 INFO:teuthology.orchestra.run.smithi045.stdout: "client_use_random_mds": "false", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "clog_to_graylog": "false", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "clog_to_graylog_port": "12201", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "clog_to_monitors": "default=true", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "clog_to_syslog": "false", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "clog_to_syslog_level": "info", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "-", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_network": "", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_network_interface": "", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "compressor_zlib_isal": "false", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "compressor_zlib_level": "5", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "compressor_zlib_winsize": "-15", 2024-03-11T22:10:19.320 INFO:teuthology.orchestra.run.smithi045.stdout: "compressor_zstd_level": "1", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "crimson_alien_op_num_threads": "6", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "crimson_seastar_cpu_cores": "", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location": "", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location_hook": "", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_location_hook_timeout": "10", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "daemonize": "false", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "daos_pool": "tank", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-11T22:10:19.321 INFO:teuthology.orchestra.run.smithi045.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_alienstore": "0/5", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_allow_any_pool_priority": "false", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_asok": "1/5", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_asok_assert_abort": "false", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_asserts_on_shutdown": "true", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_auth": "1/5", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_bdev": "1/3", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_bluefs": "1/5", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_bluestore": "1/5", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_buffer": "0/1", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_ceph_exporter": "1/5", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_cephfs_mirror": "0/5", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_cephsqlite": "0/5", 2024-03-11T22:10:19.322 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_client": "0/5", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_compressor": "1/5", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_context": "0/1", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_crimson_interrupt": "0/5", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_crush": "1/1", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_crypto": "1/5", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_cyanstore": "0/5", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_deliberately_leak_memory": "false", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_disable_randomized_ping": "false", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_dpdk": "1/5", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_eventtrace": "1/5", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_filer": "0/1", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_filestore": "1/3", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_finisher": "1/1", 2024-03-11T22:10:19.323 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_fuse": "1/5", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_heartbeat_testing_span": "0", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_heartbeatmap": "1/5", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_javaclient": "1/5", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_journal": "1/3", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_journaler": "0/5", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_kstore": "1/5", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_lockdep": "0/1", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_mclock": "1/5", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_mds": "1/5", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_mds_balancer": "1/5", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_mds_locker": "1/5", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_mds_log": "1/5", 2024-03-11T22:10:19.324 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_mds_log_expire": "1/5", 2024-03-11T22:10:19.325 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_mds_migrator": "1/5", 2024-03-11T22:10:19.325 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_memstore": "1/5", 2024-03-11T22:10:19.325 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_mgr": "2/5", 2024-03-11T22:10:19.325 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_mgrc": "1/5", 2024-03-11T22:10:19.325 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_mon": "20/20", 2024-03-11T22:10:19.325 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_monc": "0/10", 2024-03-11T22:10:19.325 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_ms": "1/1", 2024-03-11T22:10:19.325 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_none": "0/5", 2024-03-11T22:10:19.325 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_objclass": "0/5", 2024-03-11T22:10:19.325 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_objectcacher": "0/5", 2024-03-11T22:10:19.325 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_objecter": "0/1", 2024-03-11T22:10:19.325 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_optracker": "0/5", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_osd": "1/5", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_paxos": "20/20", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_perfcounter": "1/5", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_prioritycache": "1/5", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_rados": "0/5", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_rbd": "0/5", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_rbd_mirror": "0/5", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_rbd_pwl": "0/5", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_rbd_replay": "0/5", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_refs": "0/0", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_reserver": "1/1", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_rgw": "1/5", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_rgw_access": "1/5", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_rgw_datacache": "1/5", 2024-03-11T22:10:19.326 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_rgw_dbstore": "1/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_rgw_flight": "1/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_rgw_sync": "1/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_rocksdb": "4/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore": "0/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore_backref": "0/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore_cache": "0/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore_cleaner": "0/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore_device": "0/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore_epm": "0/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore_journal": "0/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore_lba": "0/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore_odata": "0/5", 2024-03-11T22:10:19.327 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore_omap": "0/5", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore_onode": "0/5", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore_t": "0/5", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_seastore_tm": "0/5", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_striper": "0/1", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_test": "0/5", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_throttle": "1/1", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_timer": "0/1", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_tp": "0/5", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "debug_trace": "1/5", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "device_failure_prediction_mode": "none", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-11T22:10:19.328 INFO:teuthology.orchestra.run.smithi045.stdout: "err_to_graylog": "false", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "err_to_journald": "false", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "err_to_stderr": "true", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "err_to_syslog": "false", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event_tracing": "false", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "exporter_addr": "0.0.0.0", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "exporter_http_port": "9926", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "exporter_prio_limit": "5", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "exporter_sort_metrics": "true", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "exporter_stats_period": "5", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "fake_statfs_for_testing": "0", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "fatal_signal_handlers": "true", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "filer_max_purge_ops": "10", 2024-03-11T22:10:19.329 INFO:teuthology.orchestra.run.smithi045.stdout: "filer_max_truncate_ops": "128", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_apply_finisher_threads": "1", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_blackhole": "false", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_btrfs_clone_range": "true", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_btrfs_snap": "true", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_caller_concurrency": "10", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_collect_device_partition_information": "true", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_commit_timeout": "600.000000", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_debug_inject_read_err": "false", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_debug_omap_check": "false", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_debug_verify_split": "false", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_dump_file": "", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-11T22:10:19.330 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_fadvise": "true", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_fail_eio": "true", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_fd_cache_shards": "16", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_fd_cache_size": "128", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_fiemap": "false", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_fiemap_threshold": "4096", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_inject_stall": "0", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_journal_parallel": "false", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_journal_trailing": "false", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_journal_writeahead": "false", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_kill_at": "0", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-11T22:10:19.331 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_inline_xattrs": "0", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_xattr_value_size": "0", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_merge_threshold": "-10", 2024-03-11T22:10:19.332 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_odsync_write": "false", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_omap_backend": "rocksdb", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_omap_backend_path": "", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_op_thread_timeout": "60", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_op_threads": "2", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_punch_hole": "false", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-11T22:10:19.333 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-11T22:10:19.334 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-11T22:10:19.334 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-11T22:10:19.334 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-11T22:10:19.334 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_queue_max_ops": "50", 2024-03-11T22:10:19.334 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-11T22:10:19.334 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_seek_data_hole": "false", 2024-03-11T22:10:19.334 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_sloppy_crc": "false", 2024-03-11T22:10:19.334 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-11T22:10:19.334 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_splice": "false", 2024-03-11T22:10:19.334 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_split_multiple": "2", 2024-03-11T22:10:19.334 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_split_rand_factor": "20", 2024-03-11T22:10:19.334 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_update_to": "1000", 2024-03-11T22:10:19.334 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_wbthrottle_enable": "true", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_xfs_extsize": "false", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "filestore_zfs_snap": "false", 2024-03-11T22:10:19.335 INFO:teuthology.orchestra.run.smithi045.stdout: "fio_dir": "/tmp/fio", 2024-03-11T22:10:19.336 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "f75aafa4-dff2-11ee-95c8-87774f69a715", 2024-03-11T22:10:19.336 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_allow_other": "true", 2024-03-11T22:10:19.336 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_atomic_o_trunc": "true", 2024-03-11T22:10:19.336 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_big_writes": "true", 2024-03-11T22:10:19.336 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_debug": "false", 2024-03-11T22:10:19.336 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_default_permissions": "false", 2024-03-11T22:10:19.336 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_disable_pagecache": "false", 2024-03-11T22:10:19.336 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_max_write": "0", 2024-03-11T22:10:19.336 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_multithreaded": "true", 2024-03-11T22:10:19.336 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_require_active_mds": "true", 2024-03-11T22:10:19.336 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_set_user_groups": "true", 2024-03-11T22:10:19.336 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_splice_move": "true", 2024-03-11T22:10:19.336 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_splice_read": "true", 2024-03-11T22:10:19.337 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_splice_write": "true", 2024-03-11T22:10:19.337 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-11T22:10:19.337 INFO:teuthology.orchestra.run.smithi045.stdout: "fuse_use_invalidate_cb": "true", 2024-03-11T22:10:19.337 INFO:teuthology.orchestra.run.smithi045.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-11T22:10:19.337 INFO:teuthology.orchestra.run.smithi045.stdout: "gss_target_name": "ceph", 2024-03-11T22:10:19.337 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_file": "", 2024-03-11T22:10:19.337 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_inject_failure": "0", 2024-03-11T22:10:19.337 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_interval": "5", 2024-03-11T22:10:19.337 INFO:teuthology.orchestra.run.smithi045.stdout: "host": "smithi045", 2024-03-11T22:10:19.337 INFO:teuthology.orchestra.run.smithi045.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-11T22:10:19.337 INFO:teuthology.orchestra.run.smithi045.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-11T22:10:19.337 INFO:teuthology.orchestra.run.smithi045.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-11T22:10:19.337 INFO:teuthology.orchestra.run.smithi045.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "inject_early_sigterm": "false", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "jaeger_agent_port": "6799", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "jaeger_tracing_enable": "false", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_aio": "true", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_align_min_size": "65536", 2024-03-11T22:10:19.338 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_block_align": "true", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_block_size": "4096", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_dio": "true", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_discard": "false", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_force_aio": "false", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_ignore_corruption": "false", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_max_write_bytes": "10485760", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_max_write_entries": "100", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_replay_from": "0", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_write_header_frequency": "0", 2024-03-11T22:10:19.339 INFO:teuthology.orchestra.run.smithi045.stdout: "journal_zero_on_create": "false", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "journaler_prefetch_periods": "10", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "journaler_prezero_periods": "5", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "journaler_write_head_interval": "15", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "key": "", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "keyfile": "", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "kstore_backend": "rocksdb", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "kstore_default_stripe_size": "65536", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "kstore_fsck_on_mount": "false", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "kstore_max_bytes": "67108864", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "kstore_max_ops": "512", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "kstore_nid_prealloc": "1024", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "kstore_onode_map_size": "1024", 2024-03-11T22:10:19.340 INFO:teuthology.orchestra.run.smithi045.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-11T22:10:19.341 INFO:teuthology.orchestra.run.smithi045.stdout: "kstore_sync_submit_transaction": "false", 2024-03-11T22:10:19.341 INFO:teuthology.orchestra.run.smithi045.stdout: "kstore_sync_transaction": "false", 2024-03-11T22:10:19.341 INFO:teuthology.orchestra.run.smithi045.stdout: "librados_thread_count": "2", 2024-03-11T22:10:19.341 INFO:teuthology.orchestra.run.smithi045.stdout: "lockdep": "false", 2024-03-11T22:10:19.341 INFO:teuthology.orchestra.run.smithi045.stdout: "lockdep_force_backtrace": "false", 2024-03-11T22:10:19.341 INFO:teuthology.orchestra.run.smithi045.stdout: "log_coarse_timestamps": "true", 2024-03-11T22:10:19.341 INFO:teuthology.orchestra.run.smithi045.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-11T22:10:19.341 INFO:teuthology.orchestra.run.smithi045.stdout: "log_flush_on_exit": "false", 2024-03-11T22:10:19.341 INFO:teuthology.orchestra.run.smithi045.stdout: "log_graylog_host": "127.0.0.1", 2024-03-11T22:10:19.341 INFO:teuthology.orchestra.run.smithi045.stdout: "log_graylog_port": "12201", 2024-03-11T22:10:19.341 INFO:teuthology.orchestra.run.smithi045.stdout: "log_max_new": "1000", 2024-03-11T22:10:19.341 INFO:teuthology.orchestra.run.smithi045.stdout: "log_max_recent": "10000", 2024-03-11T22:10:19.341 INFO:teuthology.orchestra.run.smithi045.stdout: "log_stderr_prefix": "", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "log_stop_at_utilization": "0.970000", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_file": "true", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_graylog": "false", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_journald": "false", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_stderr": "false", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "log_to_syslog": "false", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "max_rotating_auth_attempts": "10", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_action_on_write_error": "1", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_alternate_name_max": "8192", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_asio_thread_count": "2", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_export_pin": "true", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_fragment_dirs": "true", 2024-03-11T22:10:19.342 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_fragment_interval": "5", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_interval": "10", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_max": "-1", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_max_until": "-1", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_merge_size": "50", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_midchunk": "0.300000", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_min_start": "0.200000", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_minchunk": "0.001000", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_mode": "0", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_need_max": "1.200000", 2024-03-11T22:10:19.343 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_need_min": "0.800000", 2024-03-11T22:10:19.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_overload_epochs": "2", 2024-03-11T22:10:19.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-11T22:10:19.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-11T22:10:19.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_split_bits": "3", 2024-03-11T22:10:19.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-11T22:10:19.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_split_size": "10000", 2024-03-11T22:10:19.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-11T22:10:19.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_target_decay": "10.000000", 2024-03-11T22:10:19.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-11T22:10:19.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_beacon_grace": "15.000000", 2024-03-11T22:10:19.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_beacon_interval": "4.000000", 2024-03-11T22:10:19.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-11T22:10:19.344 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_cache_mid": "0.700000", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_cache_release_free_interval": "10", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_cache_reservation": "0.050000", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_cache_trim_interval": "1", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_cache_trim_threshold": "262144", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_client_prealloc_inos": "1000", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_connect_bootstrapping": "false", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_damage_table_max_entries": "10000", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-11T22:10:19.345 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_debug_auth_pins": "false", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_debug_frag": "false", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_debug_scatterstat": "false", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_debug_subtrees": "false", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_decay_halflife": "5.000000", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_default_dir_hash": "2", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_defer_session_stale": "true", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_deny_all_reconnect": "false", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_dir_keys_per_op": "16384", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_dir_max_commit_size": "10", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_dir_max_entries": "0", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_dir_prefetch": "true", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-11T22:10:19.346 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_dump_cache_on_map": "false", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_early_reply": "true", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_enable_op_tracker": "true", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_enforce_unique_name": "true", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_export_ephemeral_random": "true", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-11T22:10:19.347 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_health_summarize_threshold": "10", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_inject_health_dummy": "false", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_inject_migrator_session_race": "false", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_join_fs": "", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_journal_format": "1", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_kill_create_at": "0", 2024-03-11T22:10:19.348 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_kill_export_at": "0", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_kill_import_at": "0", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_kill_journal_at": "0", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_kill_journal_expire_at": "0", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_kill_journal_replay_at": "0", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_kill_link_at": "0", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_kill_mdstable_at": "0", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_kill_openc_at": "0", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_kill_rename_at": "0", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_kill_shutdown_at": "0", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log_event_large_threshold": "524288", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log_events_per_segment": "1024", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log_max_events": "-1", 2024-03-11T22:10:19.349 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log_max_segments": "128", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log_pause": "false", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log_segment_size": "0", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log_trim_threshold": "128", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_log_warn_factor": "2.000000", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_max_caps_per_client": "1048576", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_max_completed_flushes": "100000", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_max_completed_requests": "100000", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_max_export_size": "20971520", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_max_file_recover": "32", 2024-03-11T22:10:19.350 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_max_purge_files": "64", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_max_purge_ops": "8192", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_max_snaps_per_dir": "100", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_metrics_update_interval": "2", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_min_caps_per_client": "100", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_min_caps_working_set": "10000", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_numa_node": "-1", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_op_complaint_time": "30.000000", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_op_history_duration": "600", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_op_history_size": "20", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_op_history_slow_op_size": "20", 2024-03-11T22:10:19.351 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_op_log_threshold": "5", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_ping_grace": "15", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_ping_interval": "5", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_recall_max_caps": "30000", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_recall_warning_threshold": "262144", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_replay_interval": "1.000000", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-11T22:10:19.352 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_root_ino_gid": "0", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_root_ino_uid": "0", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_shutdown_check": "0", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_skip_ino": "0", 2024-03-11T22:10:19.353 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_snap_max_uid": "4294967294", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_snap_min_uid": "0", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_snap_rstat": "false", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_standby_replay_damaged": "false", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_symlink_recovery": "true", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_thrash_exports": "0", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_thrash_fragments": "0", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_tick_interval": "5.000000", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_valgrind_exit": "false", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_verify_backtrace": "1", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_verify_scatter": "false", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-11T22:10:19.354 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_wipe_sessions": "false", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mempool_debug": "false", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "memstore_device_bytes": "1073741824", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "memstore_page_set": "false", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "memstore_page_size": "65536", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_client_bytes": "134217728", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_client_messages": "512", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_disabled_modules": "", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_max_pg_num_change": "128", 2024-03-11T22:10:19.355 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_mds_bytes": "134217728", 2024-03-11T22:10:19.356 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_mds_messages": "128", 2024-03-11T22:10:19.356 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-11T22:10:19.356 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_mon_bytes": "134217728", 2024-03-11T22:10:19.356 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_mon_messages": "128", 2024-03-11T22:10:19.357 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_osd_bytes": "536870912", 2024-03-11T22:10:19.357 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_osd_messages": "8192", 2024-03-11T22:10:19.357 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_pool": "true", 2024-03-11T22:10:19.357 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-11T22:10:19.357 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_standby_modules": "true", 2024-03-11T22:10:19.357 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_stats_period": "5", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_stats_threshold": "5", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_tick_period": "2", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_allow_pool_delete": "true", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_allow_pool_size_one": "false", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_auth_validate_all_caps": "true", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_client_bytes": "104857600", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_client_directed_command_retry": "2", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-11T22:10:19.358 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_client_hunt_parallel": "3", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_client_log_interval": "1.000000", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_client_ping_interval": "10.000000", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_client_target_rank": "-1", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_cluster_log_file_level": "debug", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_cluster_log_to_file": "true", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-11T22:10:19.359 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_cluster_log_to_journald": "true", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_compact_on_bootstrap": "false", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_compact_on_start": "false", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_compact_on_trim": "true", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_cpu_threads": "4", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_crush_min_required_version": "hammer", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_daemon_bytes": "419430400", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-11T22:10:19.360 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_data_avail_crit": "5", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_data_avail_warn": "5", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_data_size_warn": "16106127360", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_debug_dump_json": "false", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_debug_dump_transactions": "false", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_debug_extra_checks": "false", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_debug_no_require_reef": "false", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_debug_no_require_squid": "false", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-11T22:10:19.361 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-11T22:10:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_down_mkfs_grace": "60", 2024-03-11T22:10:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_election_default_strategy": "1", 2024-03-11T22:10:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_election_timeout": "5.000000", 2024-03-11T22:10:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-11T22:10:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_elector_ping_divisor": "2", 2024-03-11T22:10:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-11T22:10:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_enable_op_tracker": "true", 2024-03-11T22:10:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_fake_pool_delete": "false", 2024-03-11T22:10:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_force_quorum_join": "false", 2024-03-11T22:10:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_globalid_prealloc": "10000", 2024-03-11T22:10:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_health_detail_to_clog": "true", 2024-03-11T22:10:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_health_log_update_period": "5", 2024-03-11T22:10:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_health_max_detail": "50", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_health_to_clog": "true", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_health_to_clog_interval": "600", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_host": "[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0]", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_host_override": "", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_initial_members": "", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_lease": "5.000000", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-11T22:10:19.363 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_log_full_interval": "50", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_log_max": "10000", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_log_max_summary": "50", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_max_log_epochs": "500", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_max_osd": "10000", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_max_pg_per_osd": "10000", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_max_pool_pg_num": "65536", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_mds_force_trim_to": "0", 2024-03-11T22:10:19.364 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_mds_skip_sanity": "false", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_memory_autotune": "true", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_memory_target": "2147483648", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_mgr_beacon_grace": "30", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_mgr_digest_period": "5", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_mgr_inactive_grace": "60", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_min_osdmap_epochs": "50", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_op_complaint_time": "30", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_op_history_duration": "600", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_op_history_size": "20", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_op_history_slow_op_size": "20", 2024-03-11T22:10:19.365 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_op_log_threshold": "5", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_auto_mark_in": "false", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_cache_size": "500", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_down_out_interval": "600", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-11T22:10:19.366 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-11T22:10:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_force_trim_to": "0", 2024-03-11T22:10:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-11T22:10:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-11T22:10:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-11T22:10:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-11T22:10:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-11T22:10:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-11T22:10:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-11T22:10:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-11T22:10:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_min_down_reporters": "2", 2024-03-11T22:10:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-11T22:10:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-11T22:10:19.367 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_report_timeout": "900", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-11T22:10:19.368 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_pg_stuck_threshold": "60", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_probe_timeout": "2.000000", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_reweight_max_change": "0.050000", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_reweight_max_osds": "4", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-11T22:10:19.369 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_scrub_interval": "86400", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_scrub_max_keys": "100", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_scrub_timeout": "300", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_session_timeout": "300", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_smart_report_timeout": "5", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_stat_smooth_intervals": "6", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_stretch_pool_min_size": "2", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_stretch_pool_size": "4", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_sync_debug": "false", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_sync_provider_kill_at": "0", 2024-03-11T22:10:19.370 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_sync_requester_kill_at": "0", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_sync_timeout": "60.000000", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_target_pg_per_osd": "100", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_tick_interval": "5", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_timecheck_interval": "300.000000", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_use_min_delay_socket": "false", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_older_version_delay": "604800", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-11T22:10:19.371 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_misplaced": "false", 2024-03-11T22:10:19.372 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-11T22:10:19.372 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_older_version": "true", 2024-03-11T22:10:19.372 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-11T22:10:19.372 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-11T22:10:19.372 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-11T22:10:19.372 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-11T22:10:19.372 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-11T22:10:19.372 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-11T22:10:19.372 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-11T22:10:19.372 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-11T22:10:19.372 INFO:teuthology.orchestra.run.smithi045.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-11T22:10:19.372 INFO:teuthology.orchestra.run.smithi045.stdout: "monmap": "", 2024-03-11T22:10:19.372 INFO:teuthology.orchestra.run.smithi045.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "motr_tracing_enabled": "false", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_op_threads": "3", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_cm": "false", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_device_name": "", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_dscp": "96", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_local_gid": "", 2024-03-11T22:10:19.373 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_port_num": "1", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_sl": "3", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_support_srq": "true", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_rdma_type": "ib", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_async_reap_threshold": "5", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_bind_before_connect": "false", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_bind_ipv4": "true", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_bind_ipv6": "false", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_bind_msgr1": "true", 2024-03-11T22:10:19.374 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_bind_msgr2": "true", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_bind_port_max": "7568", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_bind_port_min": "6800", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_bind_retry_count": "3", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_bind_retry_delay": "5", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_blackhole_client": "false", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_blackhole_mds": "false", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_blackhole_mgr": "false", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_blackhole_mon": "false", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_blackhole_osd": "false", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_client_mode": "crc secure", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_cluster_mode": "crc secure", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_cluster_type": "", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_compress_secure": "false", 2024-03-11T22:10:19.375 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_connection_idle_timeout": "900", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_connection_ready_timeout": "10", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_crc_data": "true", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_crc_header": "true", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_die_on_bad_msg": "false", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_die_on_bug": "true", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_die_on_old_message": "true", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_die_on_skipped_message": "false", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_coremask": "0xF", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_enable_tso": "true", 2024-03-11T22:10:19.376 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_hugepages": "", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_lro": "true", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_memory_channel": "4", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_pmd": "", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_port_id": "0", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_dump_on_send": "false", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_initial_backoff": "0.200000", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_inject_delay_max": "1.000000", 2024-03-11T22:10:19.377 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_inject_delay_type": "", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_inject_network_congestion": "0", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_inject_socket_failures": "0", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_learn_addr_from_peer": "true", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_max_accept_failures": "4", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_max_backoff": "15.000000", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_mon_client_mode": "secure crc", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_mon_service_mode": "secure crc", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_osd_compress_min_size": "1024", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_osd_compress_mode": "none", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-11T22:10:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_pq_min_cost": "65536", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_public_type": "", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_service_mode": "crc secure", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_tcp_listen_backlog": "512", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_tcp_nodelay": "true", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_tcp_rcvbuf": "0", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "ms_type": "async+posix", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "no_config_file": "false", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "objecter_completion_locks_per_session": "32", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "objecter_inflight_ops": "1024", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-11T22:10:19.379 INFO:teuthology.orchestra.run.smithi045.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "objecter_tick_interval": "5.000000", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "objecter_timeout": "10.000000", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "objectstore_blackhole": "false", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "openssl_engine_opts": "", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_agent_delay_time": "5.000000", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_agent_hist_halflife": "1000", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_agent_max_low_ops": "2", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_agent_max_ops": "4", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_agent_slop": "0.020000", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-11T22:10:19.380 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_asio_thread_count": "2", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_async_recovery_min_cost": "100", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_backfill_scan_max": "512", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_backfill_scan_min": "64", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_backoff_on_degraded": "false", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_backoff_on_peering": "false", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_backoff_on_unfound": "true", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_beacon_report_interval": "300", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_bench_duration": "30", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_bench_max_block_size": "67108864", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_blkin_trace_all": "false", 2024-03-11T22:10:19.381 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_check_for_log_corruption": "false", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_class_update_on_start": "true", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_client_message_cap": "256", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_client_message_size_cap": "524288000", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_client_op_priority": "63", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_client_watch_timeout": "30", 2024-03-11T22:10:19.382 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_command_max_records": "256", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_compact_on_start": "false", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_crush_update_on_start": "true", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_crush_update_weight_set": "true", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_feed_pullee": "-1", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-11T22:10:19.383 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_misdirected_ops": "false", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_no_acting_change": "false", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_no_purge_strays": "false", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_op_order": "false", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_shutdown": "false", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_trim_objects": "false", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-11T22:10:19.384 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_verify_snaps": "false", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_deep_scrub_keys": "1024", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_deep_scrub_stride": "524288", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_default_notify_timeout": "30", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_delete_sleep": "0.000000", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-11T22:10:19.385 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-11T22:10:19.386 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-11T22:10:19.386 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_discard_disconnected_ops": "true", 2024-03-11T22:10:19.386 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_enable_op_tracker": "true", 2024-03-11T22:10:19.386 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-11T22:10:19.386 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-11T22:10:19.386 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-11T22:10:19.386 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-11T22:10:19.386 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_fast_info": "true", 2024-03-11T22:10:19.386 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_fast_shutdown": "true", 2024-03-11T22:10:19.386 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-11T22:10:19.386 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-11T22:10:19.386 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-11T22:10:19.386 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-11T22:10:19.387 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-11T22:10:19.387 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_function_tracing": "false", 2024-03-11T22:10:19.387 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_heartbeat_grace": "20", 2024-03-11T22:10:19.387 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_heartbeat_interval": "6", 2024-03-11T22:10:19.387 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-11T22:10:19.387 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_heartbeat_min_peers": "10", 2024-03-11T22:10:19.387 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_heartbeat_min_size": "2000", 2024-03-11T22:10:19.387 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_heartbeat_stale": "600", 2024-03-11T22:10:19.387 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-11T22:10:19.387 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_hit_set_max_size": "100000", 2024-03-11T22:10:19.387 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_hit_set_min_size": "1000", 2024-03-11T22:10:19.387 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_journal_size": "5120", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_kill_backfill_at": "0", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_map_cache_size": "50", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_map_dedup": "true", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_map_message_max": "40", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_map_share_max_epochs": "40", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_attr_name_len": "100", 2024-03-11T22:10:19.388 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_attr_size": "0", 2024-03-11T22:10:19.389 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_backfills": "1", 2024-03-11T22:10:19.389 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_markdown_count": "5", 2024-03-11T22:10:19.389 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_markdown_period": "600", 2024-03-11T22:10:19.389 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_object_name_len": "2048", 2024-03-11T22:10:19.389 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_object_namespace_len": "256", 2024-03-11T22:10:19.389 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_object_size": "134217728", 2024-03-11T22:10:19.389 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-11T22:10:19.389 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-11T22:10:19.389 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_pg_blocked_by": "16", 2024-03-11T22:10:19.389 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_pg_log_entries": "10000", 2024-03-11T22:10:19.389 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-11T22:10:19.389 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_pgls": "1024", 2024-03-11T22:10:19.389 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_push_cost": "8388608", 2024-03-11T22:10:19.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_push_objects": "10", 2024-03-11T22:10:19.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_scrubs": "3", 2024-03-11T22:10:19.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-11T22:10:19.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_trimming_pgs": "2", 2024-03-11T22:10:19.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_write_op_reply_len": "64", 2024-03-11T22:10:19.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_max_write_size": "90", 2024-03-11T22:10:19.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-11T22:10:19.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-11T22:10:19.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-11T22:10:19.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-11T22:10:19.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-11T22:10:19.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-11T22:10:19.390 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_profile": "balanced", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-11T22:10:19.391 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_memory_base": "805306368", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_memory_cache_min": "134217728", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_memory_target": "4294967296", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_memory_target_autotune": "false", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_min_pg_log_entries": "250", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_min_recovery_priority": "0", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mon_report_interval": "5", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_num_cache_shards": "32", 2024-03-11T22:10:19.392 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_num_op_tracker_shard": "32", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_numa_auto_affinity": "true", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_numa_node": "-1", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_numa_prefer_iface": "true", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_objecter_finishers": "1", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_objectstore": "bluestore", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_objectstore_fuse": "false", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_objectstore_tracing": "false", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_complaint_time": "30.000000", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_history_duration": "600", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_history_size": "20", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_history_slow_op_size": "20", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-11T22:10:19.393 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_log_threshold": "5", 2024-03-11T22:10:19.394 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_num_shards": "0", 2024-03-11T22:10:19.394 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_num_shards_hdd": "5", 2024-03-11T22:10:19.394 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_num_shards_ssd": "8", 2024-03-11T22:10:19.394 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-11T22:10:19.394 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-11T22:10:19.394 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-11T22:10:19.394 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-11T22:10:19.394 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_pq_min_cost": "65536", 2024-03-11T22:10:19.394 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-11T22:10:19.394 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_queue_cut_off": "high", 2024-03-11T22:10:19.394 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-11T22:10:19.394 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_op_thread_timeout": "15", 2024-03-11T22:10:19.394 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_open_classes_on_start": "true", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_os_flags": "0", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_peering_op_priority": "255", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pg_delete_cost": "1048576", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pg_delete_priority": "5", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pg_log_trim_max": "10000", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pg_log_trim_min": "100", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-11T22:10:19.395 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_crimson": "false", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-11T22:10:19.396 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_flags": "0", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_min_size": "0", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_pg_num": "32", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_pgp_num": "0", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_read_ratio": "70", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_size": "3", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_default_type": "replicated", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_push_per_object_cost": "1000", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-11T22:10:19.397 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recover_clone_overlap": "true", 2024-03-11T22:10:19.398 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-11T22:10:19.398 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_cost": "20971520", 2024-03-11T22:10:19.398 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-11T22:10:19.398 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_max_active": "0", 2024-03-11T22:10:19.398 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-11T22:10:19.398 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-11T22:10:19.398 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-11T22:10:19.398 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-11T22:10:19.398 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_max_single_start": "1", 2024-03-11T22:10:19.398 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_op_priority": "3", 2024-03-11T22:10:19.398 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-11T22:10:19.398 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_priority": "5", 2024-03-11T22:10:19.398 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_sleep": "0.000000", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_repair_during_recovery": "false", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_requested_scrub_priority": "120", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_auto_repair": "false", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_begin_hour": "0", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_begin_week_day": "0", 2024-03-11T22:10:19.399 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_chunk_max": "25", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_chunk_min": "5", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_cost": "52428800", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_during_recovery": "false", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_end_hour": "0", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_end_week_day": "0", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_event_cost": "4096", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_invalid_stats": "true", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_max_preemptions": "5", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-11T22:10:19.400 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_priority": "5", 2024-03-11T22:10:19.401 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-11T22:10:19.401 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_sleep": "0.000000", 2024-03-11T22:10:19.401 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-11T22:10:19.401 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-11T22:10:19.401 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-11T22:10:19.401 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-11T22:10:19.401 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_skip_data_digest": "false", 2024-03-11T22:10:19.401 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_smart_report_timeout": "5", 2024-03-11T22:10:19.401 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_snap_trim_cost": "1048576", 2024-03-11T22:10:19.401 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_snap_trim_priority": "5", 2024-03-11T22:10:19.401 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_target_transaction_size": "30", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-11T22:10:19.402 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_tracing": "false", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_use_stale_snap": "false", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "osdc_blkin_trace_all": "false", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "paxos_kill_at": "0", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "paxos_max_join_drift": "10", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "paxos_min": "500", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "paxos_min_wait": "0.050000", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "paxos_propose_interval": "1.000000", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "paxos_service_trim_max": "500", 2024-03-11T22:10:19.403 INFO:teuthology.orchestra.run.smithi045.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-11T22:10:19.404 INFO:teuthology.orchestra.run.smithi045.stdout: "paxos_service_trim_min": "10", 2024-03-11T22:10:19.404 INFO:teuthology.orchestra.run.smithi045.stdout: "paxos_stash_full_interval": "25", 2024-03-11T22:10:19.404 INFO:teuthology.orchestra.run.smithi045.stdout: "paxos_trim_max": "500", 2024-03-11T22:10:19.404 INFO:teuthology.orchestra.run.smithi045.stdout: "paxos_trim_min": "250", 2024-03-11T22:10:19.404 INFO:teuthology.orchestra.run.smithi045.stdout: "perf": "true", 2024-03-11T22:10:19.404 INFO:teuthology.orchestra.run.smithi045.stdout: "pid_file": "", 2024-03-11T22:10:19.404 INFO:teuthology.orchestra.run.smithi045.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-11T22:10:19.404 INFO:teuthology.orchestra.run.smithi045.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-11T22:10:19.404 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "-", 2024-03-11T22:10:19.404 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrv": "", 2024-03-11T22:10:19.404 INFO:teuthology.orchestra.run.smithi045.stdout: "public_bind_addr": "-", 2024-03-11T22:10:19.404 INFO:teuthology.orchestra.run.smithi045.stdout: "public_network": "", 2024-03-11T22:10:19.404 INFO:teuthology.orchestra.run.smithi045.stdout: "public_network_interface": "", 2024-03-11T22:10:19.405 INFO:teuthology.orchestra.run.smithi045.stdout: "qat_compressor_enabled": "false", 2024-03-11T22:10:19.405 INFO:teuthology.orchestra.run.smithi045.stdout: "qat_compressor_session_max_number": "256", 2024-03-11T22:10:19.405 INFO:teuthology.orchestra.run.smithi045.stdout: "rados_mon_op_timeout": "0", 2024-03-11T22:10:19.405 INFO:teuthology.orchestra.run.smithi045.stdout: "rados_osd_op_timeout": "0", 2024-03-11T22:10:19.405 INFO:teuthology.orchestra.run.smithi045.stdout: "rados_tracing": "false", 2024-03-11T22:10:19.405 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_atime_update_interval": "60", 2024-03-11T22:10:19.405 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-11T22:10:19.405 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_balance_parent_reads": "false", 2024-03-11T22:10:19.405 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_balance_snap_reads": "false", 2024-03-11T22:10:19.405 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_blkin_trace_all": "false", 2024-03-11T22:10:19.405 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-11T22:10:19.405 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-11T22:10:19.405 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_cache": "true", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_cache_policy": "writearound", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_cache_size": "33554432", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_clone_copy_on_read": "false", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_compression_hint": "none", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_concurrent_management_ops": "10", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_default_clone_format": "auto", 2024-03-11T22:10:19.406 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_default_data_pool": "", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_default_features": "61", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_default_format": "2", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_default_map_options": "", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_default_order": "22", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_default_pool": "rbd", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_default_stripe_count": "0", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_default_stripe_unit": "0", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_enable_alloc_hint": "true", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-11T22:10:19.407 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_io_scheduler": "simple", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_journal_order": "24", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_journal_pool": "", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_journal_splay_width": "4", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_localize_parent_reads": "false", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_localize_snap_reads": "false", 2024-03-11T22:10:19.408 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-11T22:10:19.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-11T22:10:19.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-11T22:10:19.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-11T22:10:19.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-11T22:10:19.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-11T22:10:19.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-11T22:10:19.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-11T22:10:19.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-11T22:10:19.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-11T22:10:19.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-11T22:10:19.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-11T22:10:19.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-11T22:10:19.409 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-11T22:10:19.410 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_mtime_update_interval": "60", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_non_blocking_aio": "true", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_op_thread_timeout": "60", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_op_threads": "1", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_parent_cache_enabled": "false", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_plugins": "", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_bps_burst": "0", 2024-03-11T22:10:19.411 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_bps_limit": "0", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_exclude_ops": "0", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_iops_burst": "0", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_iops_limit": "0", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-11T22:10:19.412 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_read_from_replica_policy": "default", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_skip_partial_discard": "true", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_tracing": "false", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_validate_names": "true", 2024-03-11T22:10:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd_validate_pool": "true", 2024-03-11T22:10:19.414 INFO:teuthology.orchestra.run.smithi045.stdout: "restapi_base_url": "", 2024-03-11T22:10:19.414 INFO:teuthology.orchestra.run.smithi045.stdout: "restapi_log_level": "", 2024-03-11T22:10:19.414 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_acl_grants_max_num": "100", 2024-03-11T22:10:19.414 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_admin_entry": "admin", 2024-03-11T22:10:19.414 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-11T22:10:19.414 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_backend_store": "rados", 2024-03-11T22:10:19.414 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_barbican_url": "", 2024-03-11T22:10:19.414 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_beast_enable_async": "true", 2024-03-11T22:10:19.414 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_bucket_counters_cache": "false", 2024-03-11T22:10:19.414 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-11T22:10:19.414 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-11T22:10:19.414 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-11T22:10:19.414 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_cache_enabled": "true", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_cache_expiry_interval": "900", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_cache_lru_size": "10000", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_config_store": "rados", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_content_length_compat": "false", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_copy_obj_progress": "true", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_cors_rules_max_num": "100", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_cross_domain_policy": "", 2024-03-11T22:10:19.415 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_kmip_addr": "", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_kmip_password": "", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_kmip_username": "", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_require_ssl": "true", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-11T22:10:19.416 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_vault_addr": "", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_vault_auth": "token", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_vault_namespace": "", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_vault_prefix": "", 2024-03-11T22:10:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_vault_token_file": "", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_curl_buffersize": "524288", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_curl_low_speed_time": "300", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-11T22:10:19.418 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_d4n_port": "6379", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_data_log_changes_size": "1000", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_data_log_num_shards": "128", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_data_log_window": "30", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-11T22:10:19.419 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-11T22:10:19.420 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dns_name": "", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dns_s3website_name": "", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_dynamic_resharding": "true", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_enable_gc_threads": "true", 2024-03-11T22:10:19.421 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_enable_lc_threads": "true", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_enable_ops_log": "false", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_enable_quota_threads": "true", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_enable_static_website": "false", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_enable_usage_log": "false", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_enforce_swift_acls": "true", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_exit_timeout_secs": "120", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_expose_bucket": "false", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_extended_http_attrs": "", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_filter": "none", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_frontends": "beast port=7480", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_gc_max_deferred": "50", 2024-03-11T22:10:19.422 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_gc_max_objs": "32", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_gc_processor_period": "3600", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_init_timeout": "300", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_kafka_connection_idle": "30", 2024-03-11T22:10:19.423 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_admin_domain": "", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_admin_password": "", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_admin_password_path": "", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_admin_project": "", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_admin_tenant": "", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_admin_token": "", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_admin_token_path": "", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_admin_user": "", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_api_version": "2", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_barbican_domain": "", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_barbican_password": "", 2024-03-11T22:10:19.424 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_barbican_project": "", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_barbican_user": "", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_url": "", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_lc_debug_interval": "-1", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_lc_lock_max_time": "90", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_lc_max_objs": "32", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_lc_max_rules": "1000", 2024-03-11T22:10:19.425 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_lc_max_worker": "3", 2024-03-11T22:10:19.426 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-11T22:10:19.426 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_lc_thread_delay": "0", 2024-03-11T22:10:19.426 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-11T22:10:19.426 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_ldap_dnattr": "uid", 2024-03-11T22:10:19.426 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-11T22:10:19.426 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_ldap_searchfilter": "", 2024-03-11T22:10:19.426 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-11T22:10:19.426 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-11T22:10:19.426 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-11T22:10:19.426 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-11T22:10:19.426 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-11T22:10:19.426 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_log_http_headers": "", 2024-03-11T22:10:19.426 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-11T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-11T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_log_object_name_utc": "false", 2024-03-11T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-11T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-11T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_max_attr_name_len": "0", 2024-03-11T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_max_attr_size": "0", 2024-03-11T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-11T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_max_chunk_size": "4194304", 2024-03-11T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-11T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-11T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-11T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_max_listing_results": "1000", 2024-03-11T22:10:19.427 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_max_notify_retries": "10", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_max_put_param_size": "1048576", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_max_put_size": "5368709120", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_max_slo_entries": "1000", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_md_log_max_shards": "64", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_mp_lock_max_time": "600", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-11T22:10:19.428 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_nfs_max_gc": "300", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_num_async_rados_threads": "32", 2024-03-11T22:10:19.429 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_num_control_oids": "8", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_numa_node": "-1", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_objexp_chunk_size": "100", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_objexp_gc_interval": "600", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_op_thread_timeout": "600", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_op_tracing": "false", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_opa_token": "", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_opa_url": "", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_opa_verify_ssl": "true", 2024-03-11T22:10:19.430 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_ops_log_rados": "false", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_ops_log_socket_path": "", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_period_push_interval": "2.000000", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_posix_cache_lanes": "3", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-11T22:10:19.431 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_posix_cache_partitions": "3", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_print_continue": "true", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_rados_tracing": "false", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_realm": "", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_realm_id": "", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_region": "", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-11T22:10:19.432 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_request_uri": "", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_reshard_batch_size": "64", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_reshard_max_aio": "128", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_reshard_num_logs": "16", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_reshard_thread_interval": "600", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_resolve_cname": "false", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_run_sync_thread": "true", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-11T22:10:19.433 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_scheduler_type": "throttler", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_script_uri": "", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_service_provider_name": "", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sts_client_id": "", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sts_client_secret": "", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sts_entry": "sts", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sts_key": "sts", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sts_min_session_duration": "900", 2024-03-11T22:10:19.434 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sts_token_introspection_url": "", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_swift_account_in_url": "false", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_swift_auth_entry": "auth", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_swift_auth_url": "", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_swift_custom_header": "", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_swift_need_stats": "true", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_swift_tenant_name": "", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_swift_token_expiration": "86400", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_swift_url": "", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_swift_url_prefix": "swift", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-11T22:10:19.435 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sync_lease_period": "120", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_thread_pool_size": "512", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-11T22:10:19.436 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_torrent_comment": "", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_torrent_createby": "", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_torrent_encoding": "", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_torrent_flag": "false", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_torrent_origin": "", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_torrent_tracker": "", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_trust_forwarded_https": "false", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_usage_max_shards": "32", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_usage_max_user_shards": "1", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_use_opa_authz": "false", 2024-03-11T22:10:19.437 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_user_counters_cache": "false", 2024-03-11T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-11T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-11T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-11T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_user_max_buckets": "1000", 2024-03-11T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_user_policies_max_num": "100", 2024-03-11T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-11T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-11T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-11T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-11T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_user_unique_email": "true", 2024-03-11T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_verify_ssl": "true", 2024-03-11T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-11T22:10:19.438 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_zone": "", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_zone_id": "", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_zonegroup": "", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_zonegroup_id": "", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rgwlc_auto_session_clear": "true", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_block_size": "4096", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_cache_size": "536870912", 2024-03-11T22:10:19.439 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_index_type": "binary_search", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_partition_filters": "false", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_perf": "false", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-11T22:10:19.440 INFO:teuthology.orchestra.run.smithi045.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "run_dir": "/var/run/ceph", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_block_create": "true", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_cache_lru_size": "67108864", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_default_max_object_size": "16777216", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_device_size": "53687091200", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_journal_batch_capacity": "16", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-11T22:10:19.441 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_main_device_type": "SSD", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "seastore_segment_size": "67108864", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "set_keepcaps": "false", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "setgroup": "ceph", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "setuser": "ceph", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "setuser_match_path": "", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "thp": "false", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "threadpool_default_timeout": "60", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-11T22:10:19.442 INFO:teuthology.orchestra.run.smithi045.stdout: "throttler_perf_counter": "true" 2024-03-11T22:10:19.443 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:10:19.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:19 smithi144 ceph-mon[33984]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:19.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:19 smithi144 ceph-mon[33984]: 3.17 scrub starts 2024-03-11T22:10:19.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:19 smithi144 ceph-mon[33984]: 3.17 scrub ok 2024-03-11T22:10:19.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:20.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:20 smithi045 ceph-mon[39715]: 4.1b deep-scrub starts 2024-03-11T22:10:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:20 smithi045 ceph-mon[39715]: 4.1b deep-scrub ok 2024-03-11T22:10:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:20 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:10:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:20 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:10:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:20 smithi045 ceph-mon[30178]: 4.1b deep-scrub starts 2024-03-11T22:10:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:20 smithi045 ceph-mon[30178]: 4.1b deep-scrub ok 2024-03-11T22:10:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:20 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:10:20.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:20 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:10:20.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:20 smithi144 ceph-mon[33984]: 4.1b deep-scrub starts 2024-03-11T22:10:20.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:20 smithi144 ceph-mon[33984]: 4.1b deep-scrub ok 2024-03-11T22:10:20.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:20 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:10:20.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:20 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:10:21.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-11T22:10:21.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:21 smithi045 ceph-mon[39715]: 6.7 scrub starts 2024-03-11T22:10:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:21 smithi045 ceph-mon[39715]: 6.7 scrub ok 2024-03-11T22:10:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:21 smithi045 ceph-mon[39715]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-11T22:10:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:21 smithi045 ceph-mon[39715]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T22:10:21.526 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:21 smithi045 bash[79367]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T22:10:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:21 smithi045 ceph-mon[30178]: 6.7 scrub starts 2024-03-11T22:10:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:21 smithi045 ceph-mon[30178]: 6.7 scrub ok 2024-03-11T22:10:21.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:21 smithi045 ceph-mon[30178]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-11T22:10:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:21 smithi045 ceph-mon[30178]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T22:10:21.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:21 smithi144 ceph-mon[33984]: 6.7 scrub starts 2024-03-11T22:10:21.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:21 smithi144 ceph-mon[33984]: 6.7 scrub ok 2024-03-11T22:10:21.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:21 smithi144 ceph-mon[33984]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-11T22:10:21.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:21 smithi144 ceph-mon[33984]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T22:10:21.857 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:21 smithi045 bash[79367]: Writing manifest to image destination 2024-03-11T22:10:21.857 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:21 smithi045 podman[79367]: 2024-03-11 22:10:21.764272081 +0000 UTC m=+2.990282236 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T22:10:21.857 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:21 smithi045 podman[79367]: 2024-03-11 22:10:21.863036894 +0000 UTC m=+3.089047076 container create c47febbb96576283bc98c962f252f8126eb910c4af365179ef9610fc3558ce8f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:10:22.258 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:22.402 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-mon[39715]: 5.f scrub starts 2024-03-11T22:10:22.403 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-mon[39715]: 5.f scrub ok 2024-03-11T22:10:22.403 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-mon[30178]: 5.f scrub starts 2024-03-11T22:10:22.403 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-mon[30178]: 5.f scrub ok 2024-03-11T22:10:22.403 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 podman[79367]: 2024-03-11 22:10:22.148967928 +0000 UTC m=+3.374978077 container init c47febbb96576283bc98c962f252f8126eb910c4af365179ef9610fc3558ce8f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:10:22.403 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 podman[79367]: 2024-03-11 22:10:22.154250264 +0000 UTC m=+3.380260400 container start c47febbb96576283bc98c962f252f8126eb910c4af365179ef9610fc3558ce8f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:10:22.404 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.158Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T22:10:22.404 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.158Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T22:10:22.404 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.158Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T22:10:22.404 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.158Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T22:10:22.404 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.158Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T22:10:22.404 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.158Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T22:10:22.404 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T22:10:22.404 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T22:10:22.404 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T22:10:22.404 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T22:10:22.404 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T22:10:22.404 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T22:10:22.404 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T22:10:22.405 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T22:10:22.406 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T22:10:22.406 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T22:10:22.406 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T22:10:22.406 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T22:10:22.406 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T22:10:22.406 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T22:10:22.406 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T22:10:22.406 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T22:10:22.406 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T22:10:22.406 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T22:10:22.407 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T22:10:22.407 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T22:10:22.407 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T22:10:22.407 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T22:10:22.407 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T22:10:22.407 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T22:10:22.408 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T22:10:22.408 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T22:10:22.408 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T22:10:22.408 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T22:10:22.408 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T22:10:22.408 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T22:10:22.409 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T22:10:22.409 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T22:10:22.409 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T22:10:22.409 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a[79543]: ts=2024-03-11T22:10:22.159Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T22:10:22.409 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 bash[79367]: c47febbb96576283bc98c962f252f8126eb910c4af365179ef9610fc3558ce8f 2024-03-11T22:10:22.409 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:10:22 smithi045 systemd[1]: Started Ceph node-exporter.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:10:22.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:22 smithi144 ceph-mon[33984]: 5.f scrub starts 2024-03-11T22:10:22.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:22 smithi144 ceph-mon[33984]: 5.f scrub ok 2024-03-11T22:10:22.722 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:10:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:10:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:23 smithi045 ceph-mon[39715]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:23 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:23 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:23 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:23.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:23 smithi045 ceph-mon[39715]: Deploying daemon node-exporter.b on smithi144 2024-03-11T22:10:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:23 smithi045 ceph-mon[30178]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:23 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:23 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:23.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:23 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:23.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:23 smithi045 ceph-mon[30178]: Deploying daemon node-exporter.b on smithi144 2024-03-11T22:10:23.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:23 smithi144 ceph-mon[33984]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:23.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:23 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:23.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:23 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:23.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:23 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:23.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:23 smithi144 ceph-mon[33984]: Deploying daemon node-exporter.b on smithi144 2024-03-11T22:10:23.590 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:23 smithi144 bash[67813]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-11T22:10:24.307 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a: {} 2024-03-11T22:10:24.307 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b: {} 2024-03-11T22:10:24.307 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c: {} 2024-03-11T22:10:24.307 INFO:teuthology.orchestra.run.smithi045.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T22:10:24.307 INFO:teuthology.orchestra.run.smithi045.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T22:10:24.307 INFO:teuthology.orchestra.run.smithi045.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T22:10:24.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:24 smithi144 ceph-mon[33984]: 6.d deep-scrub starts 2024-03-11T22:10:24.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:24 smithi144 ceph-mon[33984]: 6.d deep-scrub ok 2024-03-11T22:10:24.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:24 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2831382511' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:10:24.340 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:24 smithi144 bash[67813]: Getting image source signatures 2024-03-11T22:10:24.340 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:24 smithi144 bash[67813]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-11T22:10:24.340 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:24 smithi144 bash[67813]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T22:10:24.340 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:24 smithi144 bash[67813]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T22:10:24.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:24 smithi045 ceph-mon[30178]: 6.d deep-scrub starts 2024-03-11T22:10:24.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:24 smithi045 ceph-mon[30178]: 6.d deep-scrub ok 2024-03-11T22:10:24.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:24 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2831382511' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:10:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:24 smithi045 ceph-mon[39715]: 6.d deep-scrub starts 2024-03-11T22:10:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:24 smithi045 ceph-mon[39715]: 6.d deep-scrub ok 2024-03-11T22:10:24.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:24 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2831382511' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:10:24.939 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-11T22:10:24.950 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T22:10:24.966 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T22:10:24.967 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-11T22:10:24.967 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-11T22:10:24.967 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-11T22:10:24.968 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:10:24.968 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:10:24.968 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-11T22:10:24.968 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.000000 2024-03-11T22:10:24.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 7 0 2024-03-11T22:10:24.968 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-11T22:10:24.969 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:24.969 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-11T22:10:24.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:10:24.969 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-11T22:10:24.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T22:10:24.970 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-11T22:10:24.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:10:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:25.002 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:25.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:25.103 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:25.126 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:25.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:25.228 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:25.252 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:25.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:25.284 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:25.289 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:25.293 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:25.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:25.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:25.379 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:25.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:25.480 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:25.521 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:25.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:25.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:25 smithi144 ceph-mon[33984]: 6.e scrub starts 2024-03-11T22:10:25.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:25 smithi144 ceph-mon[33984]: 6.e scrub ok 2024-03-11T22:10:25.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:25 smithi144 ceph-mon[33984]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:25.622 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:25.635 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:25 smithi045 ceph-mon[39715]: 6.e scrub starts 2024-03-11T22:10:25.635 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:25 smithi045 ceph-mon[39715]: 6.e scrub ok 2024-03-11T22:10:25.635 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:25 smithi045 ceph-mon[39715]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:25.635 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:25 smithi045 ceph-mon[30178]: 6.e scrub starts 2024-03-11T22:10:25.635 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:25 smithi045 ceph-mon[30178]: 6.e scrub ok 2024-03-11T22:10:25.635 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:25 smithi045 ceph-mon[30178]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:25.647 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:25.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:25.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:25.773 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:25.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:25.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:25.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:25.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:25.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:25.946 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:10:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:10:25.947 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:25 smithi144 bash[67813]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T22:10:25.947 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:25 smithi144 bash[67813]: Writing manifest to image destination 2024-03-11T22:10:26.001 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:26.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:26.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:26.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:26.127 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:26.151 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:26.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:26.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:26.276 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:26.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:26.298 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:26.377 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:26.402 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:26.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:26.502 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:26.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:26.527 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:26.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:26.561 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-mon[33984]: 5.1b deep-scrub starts 2024-03-11T22:10:26.561 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-mon[33984]: 5.1b deep-scrub ok 2024-03-11T22:10:26.561 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 podman[67813]: 2024-03-11 22:10:26.208071303 +0000 UTC m=+2.803916353 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T22:10:26.561 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 podman[67813]: 2024-03-11 22:10:26.314860687 +0000 UTC m=+2.910705743 container create d0124b7a789784acc3c81361e9711f53310fc3c1996761a5620d4ecd9a1ea3dd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:10:26.561 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 podman[67813]: 2024-03-11 22:10:26.567145443 +0000 UTC m=+3.162990502 container init d0124b7a789784acc3c81361e9711f53310fc3c1996761a5620d4ecd9a1ea3dd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:10:26.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:26 smithi045 ceph-mon[30178]: 5.1b deep-scrub starts 2024-03-11T22:10:26.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:26 smithi045 ceph-mon[30178]: 5.1b deep-scrub ok 2024-03-11T22:10:26.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:26 smithi045 ceph-mon[39715]: 5.1b deep-scrub starts 2024-03-11T22:10:26.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:26 smithi045 ceph-mon[39715]: 5.1b deep-scrub ok 2024-03-11T22:10:26.629 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:26.652 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:26.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:26.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:26.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:26.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:26.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:26.840 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 podman[67813]: 2024-03-11 22:10:26.572358224 +0000 UTC m=+3.168203273 container start d0124b7a789784acc3c81361e9711f53310fc3c1996761a5620d4ecd9a1ea3dd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:10:26.840 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.576Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T22:10:26.840 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.576Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T22:10:26.840 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.576Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T22:10:26.840 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.576Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T22:10:26.840 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.576Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T22:10:26.840 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.576Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T22:10:26.840 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T22:10:26.840 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T22:10:26.841 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T22:10:26.842 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T22:10:26.843 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T22:10:26.843 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T22:10:26.843 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T22:10:26.843 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T22:10:26.843 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T22:10:26.843 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T22:10:26.843 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T22:10:26.843 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T22:10:26.843 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T22:10:26.843 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T22:10:26.844 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T22:10:26.844 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T22:10:26.844 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T22:10:26.844 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T22:10:26.844 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T22:10:26.844 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T22:10:26.844 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b[67932]: ts=2024-03-11T22:10:26.577Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T22:10:26.844 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 bash[67813]: d0124b7a789784acc3c81361e9711f53310fc3c1996761a5620d4ecd9a1ea3dd 2024-03-11T22:10:26.844 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:10:26 smithi144 systemd[1]: Started Ceph node-exporter.b for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:10:26.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:26.904 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:26.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:26.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:27.006 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:27.029 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:27.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:27.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:27.132 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:27.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:27.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:27.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:27.258 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:27.283 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:27.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:27.385 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:27.409 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:27.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:27.511 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:27.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[39715]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[39715]: 6.a scrub starts 2024-03-11T22:10:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[39715]: 6.a scrub ok 2024-03-11T22:10:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:10:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[30178]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[30178]: 6.a scrub starts 2024-03-11T22:10:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[30178]: 6.a scrub ok 2024-03-11T22:10:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:27 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:10:27.535 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:27.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:27.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:27.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:27 smithi144 ceph-mon[33984]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:27 smithi144 ceph-mon[33984]: 6.a scrub starts 2024-03-11T22:10:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:27 smithi144 ceph-mon[33984]: 6.a scrub ok 2024-03-11T22:10:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:27 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:27 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:27 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:27 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:27 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:27 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:10:27.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:27.662 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:27.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:27.763 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:27.788 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:27.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:27.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:27.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:27.914 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:27.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:27.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:28.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:28.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:28.040 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:28.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:28.141 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:28.166 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:28.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:28.268 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:28.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:28.294 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:28.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:28.395 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:28.420 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:28.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:28.521 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:28.545 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:28.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:28.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:28.647 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:28.673 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:28.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:28.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:28.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:28.799 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:28.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:28.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:28.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:28.914 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:28.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:28.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:29.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:29.052 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:29.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:29.153 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:29.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:29.178 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:29.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:29.281 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:29.305 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:29.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:29.406 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:29.430 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:29.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:29.532 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:29.556 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:29.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:29.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:29 smithi144 ceph-mon[33984]: 3.19 scrub starts 2024-03-11T22:10:29.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:29 smithi144 ceph-mon[33984]: 3.19 scrub ok 2024-03-11T22:10:29.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:29 smithi144 ceph-mon[33984]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:29.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:29 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:29.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:29 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:10:29.658 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:29.681 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:29.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:29.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:29.698 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:29 smithi045 ceph-mon[30178]: 3.19 scrub starts 2024-03-11T22:10:29.698 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:29 smithi045 ceph-mon[30178]: 3.19 scrub ok 2024-03-11T22:10:29.698 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:29 smithi045 ceph-mon[30178]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:29.698 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:29 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:29.699 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:29 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:10:29.699 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:29 smithi045 ceph-mon[39715]: 3.19 scrub starts 2024-03-11T22:10:29.699 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:29 smithi045 ceph-mon[39715]: 3.19 scrub ok 2024-03-11T22:10:29.699 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:29 smithi045 ceph-mon[39715]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:29.699 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:29 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:29.699 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:29 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:10:29.783 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:29.807 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:29.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:29.908 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:29.933 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:29.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:30.034 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:30.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:30.059 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:30.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:30.160 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:30.185 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:30.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:30.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:30.287 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:30.312 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:30.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:30.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:30.437 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:30.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:30.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:30.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:30.564 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:30.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:30.666 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:30.691 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:30.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:30.793 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:30.817 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:30.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:30.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:30.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:30.943 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:30.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:31.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:31.067 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:31.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:31.124 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:10:31.124 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:10:31.124 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-03-11T22:10:31.124 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:10:31.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:31.193 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:31.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:31.294 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:31.317 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:31.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:31.419 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:31.442 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:31.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:31.443 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:31 smithi045 ceph-mon[39715]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:31.444 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:31 smithi045 ceph-mon[30178]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:31.544 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:31.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:31.569 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:31.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:31.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:31 smithi144 ceph-mon[33984]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:31.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:31.695 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:31.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:31.796 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:31.822 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:31.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:31.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:31.923 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:31.947 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:31.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:32.049 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:32.073 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:32.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:32.174 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:32.197 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:32.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:32.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:32.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:32.324 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:32.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:32.401 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:10:32.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:32.449 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:32.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:32.551 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:32.575 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:32.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:32.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:32.677 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:32.700 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:32.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:32.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:32.750 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:10:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:10:32.751 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:32 smithi045 ceph-mon[39715]: 4.3 scrub starts 2024-03-11T22:10:32.751 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:32 smithi045 ceph-mon[39715]: 4.3 scrub ok 2024-03-11T22:10:32.751 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:32 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:32.751 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:32 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:32.751 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:32 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/551109168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:10:32.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:32 smithi045 ceph-mon[30178]: 4.3 scrub starts 2024-03-11T22:10:32.751 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:32 smithi045 ceph-mon[30178]: 4.3 scrub ok 2024-03-11T22:10:32.752 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:32 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:32.752 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:32 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:32.752 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:32 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/551109168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:10:32.802 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:32.827 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:32.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:32.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:32 smithi144 ceph-mon[33984]: 4.3 scrub starts 2024-03-11T22:10:32.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:32 smithi144 ceph-mon[33984]: 4.3 scrub ok 2024-03-11T22:10:32.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:32 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:32.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:32 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:32.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:32 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/551109168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:10:32.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:32.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:32.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:33.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:33.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:33.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:33.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:33.180 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:33.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:33.205 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:33.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:33.306 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:33.332 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:33.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:33.433 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:33.458 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:33.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:33.560 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:33.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:33.585 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:33.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:33.679 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[39715]: 6.b deep-scrub starts 2024-03-11T22:10:33.679 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[39715]: 6.b deep-scrub ok 2024-03-11T22:10:33.679 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[39715]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-03-11T22:10:33.679 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[39715]: 5.7 scrub starts 2024-03-11T22:10:33.680 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[39715]: 5.7 scrub ok 2024-03-11T22:10:33.680 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[39715]: 4.1e scrub starts 2024-03-11T22:10:33.680 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[39715]: 4.1e scrub ok 2024-03-11T22:10:33.680 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:10:33.680 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/551109168' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:10:33.680 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[39715]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T22:10:33.680 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[39715]: 4.0 scrub starts 2024-03-11T22:10:33.680 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[30178]: 6.b deep-scrub starts 2024-03-11T22:10:33.680 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[30178]: 6.b deep-scrub ok 2024-03-11T22:10:33.680 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[30178]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-03-11T22:10:33.680 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[30178]: 5.7 scrub starts 2024-03-11T22:10:33.680 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[30178]: 5.7 scrub ok 2024-03-11T22:10:33.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[30178]: 4.1e scrub starts 2024-03-11T22:10:33.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[30178]: 4.1e scrub ok 2024-03-11T22:10:33.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:10:33.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/551109168' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:10:33.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[30178]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T22:10:33.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:33 smithi045 ceph-mon[30178]: 4.0 scrub starts 2024-03-11T22:10:33.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:33.709 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:33.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:33.811 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:33.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:33.837 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:33.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:33.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:33 smithi144 ceph-mon[33984]: 6.b deep-scrub starts 2024-03-11T22:10:33.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:33 smithi144 ceph-mon[33984]: 6.b deep-scrub ok 2024-03-11T22:10:33.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:33 smithi144 ceph-mon[33984]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-03-11T22:10:33.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:33 smithi144 ceph-mon[33984]: 5.7 scrub starts 2024-03-11T22:10:33.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:33 smithi144 ceph-mon[33984]: 5.7 scrub ok 2024-03-11T22:10:33.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:33 smithi144 ceph-mon[33984]: 4.1e scrub starts 2024-03-11T22:10:33.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:33 smithi144 ceph-mon[33984]: 4.1e scrub ok 2024-03-11T22:10:33.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:33 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:10:33.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:33 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/551109168' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:10:33.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:33 smithi144 ceph-mon[33984]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T22:10:33.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:33 smithi144 ceph-mon[33984]: 4.0 scrub starts 2024-03-11T22:10:33.938 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:33.961 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:33.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:33.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:34.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T22:10:34.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:34.111 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:34.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:34.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:34.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:34.238 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:34.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:34.340 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:34.369 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:34.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:34.470 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:34.498 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:34.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:34.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:34.548 INFO:teuthology.orchestra.run.smithi045.stderr:set osd.7 primary-affinity to 0 (802) 2024-03-11T22:10:34.600 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:34.626 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:34.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:34.728 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:34.754 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:34.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:34.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:34 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3859020743' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-03-11T22:10:34.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:34 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-03-11T22:10:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:34 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3859020743' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-03-11T22:10:34.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:34 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-03-11T22:10:34.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:34 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3859020743' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-03-11T22:10:34.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:34 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-03-11T22:10:34.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:34.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:34.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:34.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:34.982 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:35.006 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:35.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:35.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:35.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:35.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:35.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:35.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:35.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:35.256 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:35.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:35.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:35.358 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:35.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:35.387 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:35.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:35.410 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'unique_pool_0' created 2024-03-11T22:10:35.471 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:35 smithi045 ceph-mon[39715]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:35.471 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:35 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-03-11T22:10:35.471 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:35 smithi045 ceph-mon[39715]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T22:10:35.471 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:35 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3444594348' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-11T22:10:35.472 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:35 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3444594348' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-11T22:10:35.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:35 smithi045 ceph-mon[30178]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:35.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:35 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-03-11T22:10:35.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:35 smithi045 ceph-mon[30178]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T22:10:35.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:35 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3444594348' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-11T22:10:35.472 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:35 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3444594348' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-11T22:10:35.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:35.499 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:10:35.511 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:35.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:35.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:35.519 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:10:35.530 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:10:35.539 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:10:35.560 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:10:35.581 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:10:35.601 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:10:35.613 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:35.622 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:10:35.639 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:35.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:35.649 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:10:35.650 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:10:35.650 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:10:35.650 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:10:35.650 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:10:35.650 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:10:35.650 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:10:35.650 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:10:35.650 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:10:35.675 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:35 smithi144 ceph-mon[33984]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:10:35.675 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:35 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-03-11T22:10:35.675 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:35 smithi144 ceph-mon[33984]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T22:10:35.675 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:35 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3444594348' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-11T22:10:35.675 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:35 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3444594348' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-11T22:10:35.741 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:35.764 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:35.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:35.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:35.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:35.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:35.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:35.943 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:10:35 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:10:35.991 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:36.015 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:36.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:36.053 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-11T22:10:36.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:36.139 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:36.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:36.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:36.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:36.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:36.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:36.367 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:36.392 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:36.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:36.493 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:36.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:36.517 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:36.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:36.618 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:36.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:36.642 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:36.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:36.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:36.767 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:36.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:36.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:36 smithi045 ceph-mon[39715]: osdmap e99: 8 total, 8 up, 8 in 2024-03-11T22:10:36.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:36 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3154036706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:10:36.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:36 smithi045 ceph-mon[39715]: osdmap e100: 8 total, 8 up, 8 in 2024-03-11T22:10:36.772 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:36 smithi045 ceph-mon[30178]: osdmap e99: 8 total, 8 up, 8 in 2024-03-11T22:10:36.772 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:36 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3154036706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:10:36.772 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:36 smithi045 ceph-mon[30178]: osdmap e100: 8 total, 8 up, 8 in 2024-03-11T22:10:36.834 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:36.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:36 smithi144 ceph-mon[33984]: osdmap e99: 8 total, 8 up, 8 in 2024-03-11T22:10:36.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:36 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3154036706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:10:36.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:36 smithi144 ceph-mon[33984]: osdmap e100: 8 total, 8 up, 8 in 2024-03-11T22:10:36.859 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:36.859 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:36.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:36.894 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:36.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:36.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:36.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:37.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:37.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:37.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:37.146 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:37.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:37.247 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:37.272 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:37.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:37.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:37.374 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:37.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:37.399 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:37.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:10:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:10:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[30178]: pgmap v57: 148 pgs: 16 unknown, 114 active+clean, 18 peering; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:10:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[30178]: Health check failed: Reduced data availability: 13 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-03-11T22:10:37.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[30178]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:10:37.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[30178]: osdmap e101: 8 total, 8 up, 8 in 2024-03-11T22:10:37.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:37.524 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:37.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:37.626 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:37.651 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:37.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:37.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:37.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:10:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:10:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[39715]: pgmap v57: 148 pgs: 16 unknown, 114 active+clean, 18 peering; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:10:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[39715]: Health check failed: Reduced data availability: 13 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-03-11T22:10:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[39715]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:10:37.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:37 smithi045 ceph-mon[39715]: osdmap e101: 8 total, 8 up, 8 in 2024-03-11T22:10:37.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:37.777 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:37.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:37.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:37 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:10:37.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:37 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:10:37.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:37.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:37 smithi144 ceph-mon[33984]: pgmap v57: 148 pgs: 16 unknown, 114 active+clean, 18 peering; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:10:37.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:37 smithi144 ceph-mon[33984]: Health check failed: Reduced data availability: 13 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-03-11T22:10:37.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:37 smithi144 ceph-mon[33984]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:10:37.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:37 smithi144 ceph-mon[33984]: osdmap e101: 8 total, 8 up, 8 in 2024-03-11T22:10:37.878 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:37.902 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:37.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:37.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:38.007 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:38.031 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:38.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:38.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:38.133 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:38.156 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:38.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:38.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:38.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:38.281 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:38.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:38.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:38.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:38.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:38.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:38.508 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:38.534 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:38.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:38.636 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:38.660 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:38.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:38.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:38.762 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:38.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:38.787 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:38.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:38.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:10:38.888 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:38.911 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:38.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:39.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:39.038 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:39.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:39.140 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:39.164 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:39.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:39.266 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:39.289 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:39.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:39.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:39.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:39.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:39.416 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:39.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:39.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:39.541 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:39.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:39.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:39.643 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:39.668 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:39.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:39.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:39.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:39.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:39 smithi045 ceph-mon[39715]: pgmap v60: 148 pgs: 16 unknown, 114 active+clean, 18 peering; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:10:39.775 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:39 smithi045 ceph-mon[30178]: pgmap v60: 148 pgs: 16 unknown, 114 active+clean, 18 peering; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:10:39.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:39.794 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:39.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:39.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:39 smithi144 ceph-mon[33984]: pgmap v60: 148 pgs: 16 unknown, 114 active+clean, 18 peering; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:10:39.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:39.918 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:39.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:39.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:40.020 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:40.048 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:40.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:40.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:40.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:40.174 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:40.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:40.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:40.245 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:10:40.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T22:10:40.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:40.311 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:40.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:40.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:40.413 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:40.442 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:40.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:40.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:40.567 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:40.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:40.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:40.669 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:40.697 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:40.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:40.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:40.766 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:40.799 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:40.826 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:40.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:40.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:40.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:40.952 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:40.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:41.053 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:41.081 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:41.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:41.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:41.183 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:41.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:41.211 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:41.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:41.312 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:41.337 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:41.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:41.427 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-11T22:10:41.440 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:41.466 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:41.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:41.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:41 smithi045 ceph-mon[39715]: pgmap v61: 148 pgs: 16 unknown, 114 active+clean, 18 peering; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:10:41.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:41 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3530315966' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:10:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:41 smithi045 ceph-mon[30178]: pgmap v61: 148 pgs: 16 unknown, 114 active+clean, 18 peering; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:10:41.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:41 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3530315966' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:10:41.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:41.591 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:41.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:41.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:41.693 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:41.720 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:41.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:41.822 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:41.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:41 smithi144 ceph-mon[33984]: pgmap v61: 148 pgs: 16 unknown, 114 active+clean, 18 peering; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-03-11T22:10:41.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:41 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3530315966' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:10:41.845 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:41.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:41.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:41.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:41.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:41.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:41.982 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:42.072 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:42.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:42.101 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:42.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:42.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:42.225 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:42.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:42.327 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:42.351 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:42.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:42.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:42.453 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:42.479 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:42.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:42.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:42.581 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:42.608 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:42.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:42.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:42.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:42.735 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:42.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:42.761 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:10:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:10:42.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:42 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3530315966' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:10:42.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:42 smithi045 ceph-mon[39715]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T22:10:42.761 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:42 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3530315966' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:10:42.762 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:42 smithi045 ceph-mon[30178]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T22:10:42.836 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:42.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:42 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3530315966' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:10:42.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:42 smithi144 ceph-mon[33984]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T22:10:42.864 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:42.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:42.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:42.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:42.991 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:42.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:43.092 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:43.116 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:43.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:43.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:43.242 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:43.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:43.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:43.344 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:43.372 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:43.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:43.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:43.474 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:43.502 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:43.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:43.546 DEBUG:teuthology.orchestra.run.smithi045:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-11T22:10:43.549 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T22:10:43.549 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T22:10:43.605 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:43.616 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:hints = 1 2024-03-11T22:10:43.616 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T22:10:43.616 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Object prefix: benchmark_data_smithi045_83007 2024-03-11T22:10:43.616 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:10:43.616 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 0 0 0 0 0 0 - 0 2024-03-11T22:10:43.641 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:43.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:43.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:43.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:43.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:43 smithi045 ceph-mon[39715]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:10:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:43 smithi045 ceph-mon[39715]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 16 pgs peering) 2024-03-11T22:10:43.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:43 smithi045 ceph-mon[39715]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:10:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:43 smithi045 ceph-mon[30178]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:10:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:43 smithi045 ceph-mon[30178]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 16 pgs peering) 2024-03-11T22:10:43.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:43 smithi045 ceph-mon[30178]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:10:43.795 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:43.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:43.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:43 smithi144 ceph-mon[33984]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:10:43.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:43 smithi144 ceph-mon[33984]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 16 pgs peering) 2024-03-11T22:10:43.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:43 smithi144 ceph-mon[33984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:10:43.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:43.938 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:43.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:43.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:44.040 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:44.072 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:44.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:44.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:44.174 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:44.205 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:44.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:44.307 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:44.337 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:44.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:44.438 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:44.468 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:44.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:44.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:44.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:44.616 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 1 15 2944 2929 183.022 183.062 0.00396469 0.00544175 2024-03-11T22:10:44.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:44.620 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:44.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:44.721 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:44.753 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:44.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:44.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:44.897 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:44.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:44.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:44.999 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:45.033 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:45.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:45.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:45.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:45.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:45.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:45.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:45.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:10:45.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:10:45.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[39715]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 668 B/s rd, 0 op/s 2024-03-11T22:10:45.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:45.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:10:45.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[39715]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T22:10:45.290 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/872678107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:10:45.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:10:45.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:10:45.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[30178]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 668 B/s rd, 0 op/s 2024-03-11T22:10:45.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:45.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:10:45.290 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[30178]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T22:10:45.291 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:45 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/872678107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:10:45.309 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:45.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:45.410 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:45.438 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:10:45.440 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:45.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:45.455 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:10:45.497 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:10:45.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:45.547 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:10:45.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:45 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:10:45.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:45 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:10:45.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:45 smithi144 ceph-mon[33984]: pgmap v64: 148 pgs: 148 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 668 B/s rd, 0 op/s 2024-03-11T22:10:45.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:45.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:10:45.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:45 smithi144 ceph-mon[33984]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T22:10:45.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:45 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/872678107' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:10:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:45.591 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:45.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:45.594 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:10:45.616 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 2 15 5845 5830 182.16 181.312 0.00259724 0.00547891 2024-03-11T22:10:45.637 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:10:45.665 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:10:45.665 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:10:45.665 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-03-11T22:10:45.665 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-03-11T22:10:45.665 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:10:45.681 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:10:45.693 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:45.723 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:45.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:45.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:45.728 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:10:45.771 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:10:45.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:10:45.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:10:45.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:10:45.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:10:45.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:10:45.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:10:45.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:10:45.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:10:45.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:45.865 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:45.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:45.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:46.004 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:46.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:46.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:46.089 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:10:45 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:10:46.109 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:46.139 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:46.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:46.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:46.285 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:46.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:46.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:46.387 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:46.419 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:46.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:46.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:46.503 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:46 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3607268527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:10:46.504 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:46 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:10:46.504 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:46 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/872678107' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:10:46.504 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:46 smithi045 ceph-mon[39715]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T22:10:46.504 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:46 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3607268527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:10:46.504 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:46 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:10:46.504 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:46 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/872678107' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:10:46.504 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:46 smithi045 ceph-mon[30178]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T22:10:46.522 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:46.552 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:46.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:46.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:46 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3607268527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:10:46.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:46 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:10:46.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:46 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/872678107' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:10:46.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:46 smithi144 ceph-mon[33984]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T22:10:46.616 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 3 16 8510 8494 176.935 166.5 0.00315898 0.00564053 2024-03-11T22:10:46.654 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:46.683 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:46.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:46.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:46.785 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:46.816 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:46.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:46.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:46.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:46.963 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:46.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:46.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:47.065 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:47.098 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:47.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:47.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:47.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:47.243 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:47.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:47.344 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:47.374 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:47.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:47.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:47.476 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:47.506 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:47.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:47.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:47 smithi045 ceph-mon[39715]: pgmap v67: 148 pgs: 148 active+clean; 276 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 46 MiB/s wr, 736 op/s 2024-03-11T22:10:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:47.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:47 smithi045 ceph-mon[30178]: pgmap v67: 148 pgs: 148 active+clean; 276 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 46 MiB/s wr, 736 op/s 2024-03-11T22:10:47.608 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:47.616 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 4 16 10975 10959 171.214 154.062 0.0105455 0.0058303 2024-03-11T22:10:47.638 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:47.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:47.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:47.739 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:47.769 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:47.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:47.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:47.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:47.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:47 smithi144 ceph-mon[33984]: pgmap v67: 148 pgs: 148 active+clean; 276 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 46 MiB/s wr, 736 op/s 2024-03-11T22:10:47.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:47.915 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:47.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:48.017 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:48.046 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:48.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:48.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:48.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:48.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:48.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:48.285 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:48.318 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:48.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:48.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:48.420 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:48.452 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:48.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:48.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:48.596 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:48.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:48.616 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 5 16 13412 13396 167.431 152.312 0.00295244 0.00596304 2024-03-11T22:10:48.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:48.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:48.736 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:48.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:48.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:48.884 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:48.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:48.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:48.986 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:49.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:49.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:49.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:49.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:49.167 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:49.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:49.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:49.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:49.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:49.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:10:49.409 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:49.440 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:49.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:49.542 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:49.573 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:49.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:49.617 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 6 16 15696 15680 163.312 142.75 0.00908253 0.00611357 2024-03-11T22:10:49.675 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:49.707 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:49.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:49.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:49 smithi045 ceph-mon[39715]: pgmap v68: 148 pgs: 148 active+clean; 276 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 42 MiB/s wr, 665 op/s 2024-03-11T22:10:49.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:49 smithi045 ceph-mon[30178]: pgmap v68: 148 pgs: 148 active+clean; 276 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 42 MiB/s wr, 665 op/s 2024-03-11T22:10:49.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:49.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:49 smithi144 ceph-mon[33984]: pgmap v68: 148 pgs: 148 active+clean; 276 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 772 B/s rd, 42 MiB/s wr, 665 op/s 2024-03-11T22:10:49.852 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:49.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:49.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:49.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:49.994 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:49.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:50.096 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:50.135 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:50.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:50.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:50.237 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:50.269 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:50.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:50.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:50.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:50.415 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:50.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:50.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T22:10:50.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:50.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:50.584 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:50.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:50.618 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 7 16 18105 18089 161.462 150.562 0.00917748 0.00618687 2024-03-11T22:10:50.685 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:50.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:50.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:50.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:50.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:50.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:50.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:50.975 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:51.008 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:51.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:51.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:51.110 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:51.141 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:51.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:51.242 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:51.269 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:51.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:51.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:51.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:51.422 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:51.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:51.443 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:51.524 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:51.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:51 smithi045 ceph-mon[39715]: pgmap v69: 148 pgs: 148 active+clean; 276 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 551 op/s 2024-03-11T22:10:51.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:51 smithi045 ceph-mon[30178]: pgmap v69: 148 pgs: 148 active+clean; 276 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 551 op/s 2024-03-11T22:10:51.554 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:51.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:51.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:51.619 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 8 16 20162 20146 157.334 128.562 0.00622382 0.00634833 2024-03-11T22:10:51.655 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:51.687 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:51.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:51.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:51.789 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:51.818 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:51.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:51.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:51 smithi144 ceph-mon[33984]: pgmap v69: 148 pgs: 148 active+clean; 276 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 551 op/s 2024-03-11T22:10:51.851 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:10:51.920 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:51.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:51.969 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:51.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:51.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:10:52.071 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:52.155 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:52.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:52.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:52.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:52.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:52.420 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:52.448 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:52.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:52.473 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T22:10:52.550 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:52.581 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:52.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:52.620 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 9 16 22050 22034 152.955 118 0.00322894 0.00653117 2024-03-11T22:10:52.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:52.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:52.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:52.775 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:10:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:10:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:10:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:52 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4242168489' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:10:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:52 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:52.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:52 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:52 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4242168489' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:10:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:52 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:52.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:52 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:52.832 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:52.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:52 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4242168489' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:10:52.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:52 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:52.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:52 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:52.861 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:52.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:52.963 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:53.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:53.006 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:53.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:53.107 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:53.135 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:53.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:53.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:53.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:53.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:53.288 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:53.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:53.389 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:53.414 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:53.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:53.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:53.515 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:53.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:53.545 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:53.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:53.619 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 10 16 23893 23877 149.178 115.188 0.00490597 0.00666984 2024-03-11T22:10:53.646 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:53.676 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:53.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:53.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[39715]: pgmap v70: 148 pgs: 148 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-11T22:10:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4242168489' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:10:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[39715]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T22:10:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:10:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:10:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:10:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:10:53.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[30178]: pgmap v70: 148 pgs: 148 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-11T22:10:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4242168489' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:10:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[30178]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T22:10:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:10:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:10:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:10:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:10:53.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:53 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:53.778 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:53.803 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:53.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:53.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:53 smithi144 ceph-mon[33984]: pgmap v70: 148 pgs: 148 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-11T22:10:53.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:53 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4242168489' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:10:53.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:53 smithi144 ceph-mon[33984]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T22:10:53.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:53 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:10:53.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:53 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:10:53.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:53 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:10:53.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:53 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:10:53.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:53 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:10:53.905 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:53.931 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:53.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:53.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:54.033 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:54.065 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:54.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:54.167 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:54.199 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:54.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:54.301 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:54.332 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:54.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:54.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:54.434 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:54.463 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:54.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:54.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:54.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:54.609 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:54.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:54.619 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 11 16 25779 25763 146.332 117.875 0.00316066 0.00682742 2024-03-11T22:10:54.710 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:54.734 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:54.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:54.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:54.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:54 smithi045 ceph-mon[39715]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T22:10:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:54 smithi045 ceph-mon[39715]: Reconfiguring daemon alertmanager.a on smithi045 2024-03-11T22:10:54.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:54 smithi045 ceph-mon[39715]: osdmap e106: 8 total, 8 up, 8 in 2024-03-11T22:10:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:54 smithi045 ceph-mon[30178]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T22:10:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:54 smithi045 ceph-mon[30178]: Reconfiguring daemon alertmanager.a on smithi045 2024-03-11T22:10:54.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:54 smithi045 ceph-mon[30178]: osdmap e106: 8 total, 8 up, 8 in 2024-03-11T22:10:54.837 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:54.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:54 smithi144 ceph-mon[33984]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T22:10:54.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:54 smithi144 ceph-mon[33984]: Reconfiguring daemon alertmanager.a on smithi045 2024-03-11T22:10:54.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:54 smithi144 ceph-mon[33984]: osdmap e106: 8 total, 8 up, 8 in 2024-03-11T22:10:54.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:54.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:54.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:54.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:55.002 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:55.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:55.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:55.135 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:55.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:55.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:55.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:55.276 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:55.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:55.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:55.379 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:55.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:55.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:55.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:55.543 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:55.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:55.620 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 12 16 27415 27399 142.659 102.25 0.0053706 0.00700411 2024-03-11T22:10:55.645 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:55.677 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:55.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:55.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:55 smithi045 ceph-mon[39715]: pgmap v73: 148 pgs: 148 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T22:10:55.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:55 smithi045 ceph-mon[30178]: pgmap v73: 148 pgs: 148 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T22:10:55.779 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:55.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:55.803 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:55.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:55.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:55 smithi144 ceph-mon[33984]: pgmap v73: 148 pgs: 148 active+clean; 1.0 GiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T22:10:55.840 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:10:55 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:10:55.905 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:55.936 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:55.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:55.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:56.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:56.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:56.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:56.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:56.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:56.213 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:56.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:56.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:56.343 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:56.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:56.445 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:56.477 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:56.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:56.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:56.579 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:56.610 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:56.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:56.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:56.620 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 13 16 28796 28780 138.325 86.3125 0.00924305 0.0072246 2024-03-11T22:10:56.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:56.750 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:56.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:56.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:56.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:56.875 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:56.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:56.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:57.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:57.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:57.160 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:57.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:57.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:57.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:57.304 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:57.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:57.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:57.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:57.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:57 smithi144 ceph-mon[33984]: pgmap v74: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 172 MiB/s wr, 2.75k op/s 2024-03-11T22:10:57.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:57 smithi144 ceph-mon[33984]: osdmap e107: 8 total, 8 up, 8 in 2024-03-11T22:10:57.406 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:57.439 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:57.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:57.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:57.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:57 smithi045 ceph-mon[30178]: pgmap v74: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 172 MiB/s wr, 2.75k op/s 2024-03-11T22:10:57.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:57 smithi045 ceph-mon[30178]: osdmap e107: 8 total, 8 up, 8 in 2024-03-11T22:10:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:10:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:57 smithi045 ceph-mon[39715]: pgmap v74: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 172 MiB/s wr, 2.75k op/s 2024-03-11T22:10:57.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:57 smithi045 ceph-mon[39715]: osdmap e107: 8 total, 8 up, 8 in 2024-03-11T22:10:57.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:57.575 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:57.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:57.620 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 14 16 30356 30340 135.409 97.5 0.0603704 0.0073536 2024-03-11T22:10:57.676 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:57.707 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:57.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:57.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:57.809 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:10:57.840 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:57.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:57.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:10:57.942 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:57.980 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:57.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:58.082 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:58.105 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:58.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:58.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:58.207 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:10:58.236 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:58.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:58.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:10:58.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:58.379 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:58.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:58.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:10:58.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:58.521 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:58.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:58.620 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 15 15 32232 32217 134.202 117.312 0.015464 0.00744518 2024-03-11T22:10:58.622 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:58.654 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:58.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:58.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:58.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:58.798 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:58.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:58.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:58.900 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:58.930 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:58.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:58.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:59.032 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:10:59.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:59.057 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:59.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:10:59.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:10:59.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:59.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:59.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:10:59.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:59.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:10:59 smithi144 ceph-mon[33984]: pgmap v76: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T22:10:59.340 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:59.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:59.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:59.361 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:10:59 smithi045 ceph-mon[39715]: pgmap v76: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T22:10:59.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:10:59 smithi045 ceph-mon[30178]: pgmap v76: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T22:10:59.442 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:10:59.472 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:10:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:59.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:10:59.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:10:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:59.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:59.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:10:59.620 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 16 16 34110 34094 133.146 117.312 0.0104749 0.00750453 2024-03-11T22:10:59.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:10:59.755 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:59.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:59.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:10:59.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:10:59.903 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:10:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:10:59.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:00.005 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:00.036 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:00.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:00.138 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:00.171 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:00.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:00.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:00.273 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:00.304 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:00.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:00.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:00.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:00.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:00.437 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:00.437 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:00.452 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:00.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:00.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T22:11:00.555 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:00.584 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:00.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:00.622 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 17 16 35999 35983 132.258 118.062 0.00625308 0.00755542 2024-03-11T22:11:00.686 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:00.713 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:00.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:00.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:00.815 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:00.845 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:00.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:00.947 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:00.978 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:00.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:00.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:01.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:01.120 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:01.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:01.222 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:01.253 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:01.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:01.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:01.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:01 smithi144 ceph-mon[33984]: pgmap v77: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 674 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T22:11:01.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:01 smithi144 ceph-mon[33984]: osdmap e108: 8 total, 8 up, 8 in 2024-03-11T22:11:01.356 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:01.384 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:01.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:01.416 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:01 smithi045 ceph-mon[39715]: pgmap v77: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 674 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T22:11:01.416 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:01 smithi045 ceph-mon[39715]: osdmap e108: 8 total, 8 up, 8 in 2024-03-11T22:11:01.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:01 smithi045 ceph-mon[30178]: pgmap v77: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 674 B/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-11T22:11:01.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:01 smithi045 ceph-mon[30178]: osdmap e108: 8 total, 8 up, 8 in 2024-03-11T22:11:01.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:01.527 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:01.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:01.620 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 18 16 37739 37723 130.952 108.75 0.00311953 0.00763086 2024-03-11T22:11:01.629 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:01.662 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:01.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:01.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:01.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:01.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:01.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:01.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:01.883 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:11:01.883 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-03-11T22:11:01.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-03-11T22:11:01.883 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:11:01.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.883 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:26430 7.6 7:64ef84b0:::benchmark_data_smithi045_83007_object26429:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.883 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.968685+0000", 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9153040170000004, 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16196454099999999, 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 26430 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.968685+0000", 2024-03-11T22:11:01.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.968685+0000", 2024-03-11T22:11:01.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.968686+0000", 2024-03-11T22:11:01.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.189e-06 2024-03-11T22:11:01.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.968722+0000", 2024-03-11T22:11:01.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5363e-05 2024-03-11T22:11:01.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.968732+0000", 2024-03-11T22:11:01.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0688e-05 2024-03-11T22:11:01.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.968740+0000", 2024-03-11T22:11:01.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5469999999999999e-06 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.968846+0000", 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000105874 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.969026+0000", 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017975700000000001 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:11:01.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.969165+0000", 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013904600000000001 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972601+0000", 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0034361489999999999 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972644+0000", 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3597999999999998e-05 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.130550+0000", 2024-03-11T22:11:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.157905501 2024-03-11T22:11:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.130605+0000", 2024-03-11T22:11:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4679000000000003e-05 2024-03-11T22:11:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.130650+0000", 2024-03-11T22:11:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.515e-05 2024-03-11T22:11:01.889 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.890 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.890 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.890 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:26434 7.4 7:2b0c49ce:::benchmark_data_smithi045_83007_object26433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.890 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.969288+0000", 2024-03-11T22:11:01.890 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9147011220000003, 2024-03-11T22:11:01.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.157050998, 2024-03-11T22:11:01.890 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.890 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.890 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.890 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.890 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 26434 2024-03-11T22:11:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.969288+0000", 2024-03-11T22:11:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.969288+0000", 2024-03-11T22:11:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.969289+0000", 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0500000000000002e-07 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.969310+0000", 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.126e-05 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.969320+0000", 2024-03-11T22:11:01.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8910000000000007e-06 2024-03-11T22:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.969327+0000", 2024-03-11T22:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3289999999999996e-06 2024-03-11T22:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.969400+0000", 2024-03-11T22:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2253000000000001e-05 2024-03-11T22:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.969526+0000", 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012628699999999999 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.970382+0000", 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00085626300000000001 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126175+0000", 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.155792981 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126223+0000", 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7589999999999997e-05 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126238+0000", 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4973999999999999e-05 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126285+0000", 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7678e-05 2024-03-11T22:11:01.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126339+0000", 2024-03-11T22:11:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3687000000000003e-05 2024-03-11T22:11:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.896 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:26439 7.f 7:fce4f53b:::benchmark_data_smithi045_83007_object26438:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.897 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.971795+0000", 2024-03-11T22:11:01.897 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9121944390000003, 2024-03-11T22:11:01.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15714998799999999, 2024-03-11T22:11:01.897 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.897 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.897 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.897 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.897 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 26439 2024-03-11T22:11:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.971795+0000", 2024-03-11T22:11:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.971795+0000", 2024-03-11T22:11:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.971796+0000", 2024-03-11T22:11:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1400000000000001e-06 2024-03-11T22:11:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.971817+0000", 2024-03-11T22:11:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1049000000000001e-05 2024-03-11T22:11:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.971829+0000", 2024-03-11T22:11:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1875000000000001e-05 2024-03-11T22:11:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.971866+0000", 2024-03-11T22:11:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6816000000000001e-05 2024-03-11T22:11:01.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972108+0000", 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000242444 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972234+0000", 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000125709 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972369+0000", 2024-03-11T22:11:01.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000135053 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974602+0000", 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0022333919999999998 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974661+0000", 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9104999999999997e-05 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128830+0000", 2024-03-11T22:11:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15416827599999999 2024-03-11T22:11:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128902+0000", 2024-03-11T22:11:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2700999999999994e-05 2024-03-11T22:11:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128945+0000", 2024-03-11T22:11:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2428000000000001e-05 2024-03-11T22:11:01.903 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:26441 7.4 7:2d21c005:::benchmark_data_smithi045_83007_object26440:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.972044+0000", 2024-03-11T22:11:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9119450049999998, 2024-03-11T22:11:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.154702323, 2024-03-11T22:11:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.904 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 26441 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972044+0000", 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972044+0000", 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972045+0000", 2024-03-11T22:11:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9400000000000005e-07 2024-03-11T22:11:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972072+0000", 2024-03-11T22:11:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6635000000000002e-05 2024-03-11T22:11:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972083+0000", 2024-03-11T22:11:01.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1313e-05 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972090+0000", 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.61e-06 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972865+0000", 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00077423399999999995 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973010+0000", 2024-03-11T22:11:01.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014484 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973991+0000", 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00098134900000000007 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126347+0000", 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.152356357 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126380+0000", 2024-03-11T22:11:01.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3117999999999997e-05 2024-03-11T22:11:01.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126667+0000", 2024-03-11T22:11:01.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028695000000000001 2024-03-11T22:11:01.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126712+0000", 2024-03-11T22:11:01.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4659000000000001e-05 2024-03-11T22:11:01.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126747+0000", 2024-03-11T22:11:01.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4564000000000003e-05 2024-03-11T22:11:01.910 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.910 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.910 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.910 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.14790.0:26438 7.7 e106/99 7:e97d8559:::benchmark_data_smithi045_83007_object26437:head v 106'1627, mlcod=106'1624)", 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.972120+0000", 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9118690860000003, 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15686543999999999, 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972120+0000", 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972120+0000", 2024-03-11T22:11:01.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972121+0000", 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0100000000000005e-07 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972152+0000", 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1278000000000001e-05 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972161+0000", 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7530000000000007e-06 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972167+0000", 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0499999999999997e-06 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972300+0000", 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013329600000000001 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972320+0000", 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9602999999999998e-05 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128953+0000", 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15663285699999999 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128986+0000", 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3002e-05 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.914 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:26434 7.4 e106/99 ondisk, result = 0)", 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.972693+0000", 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9112960760000002, 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15348999799999999, 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972693+0000", 2024-03-11T22:11:01.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972693+0000", 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972694+0000", 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0299999999999999e-07 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972698+0000", 2024-03-11T22:11:01.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1570000000000002e-06 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972707+0000", 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3330000000000003e-06 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972713+0000", 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1199999999999999e-06 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126159+0000", 2024-03-11T22:11:01.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.153446048 2024-03-11T22:11:01.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126168+0000", 2024-03-11T22:11:01.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7199999999999995e-06 2024-03-11T22:11:01.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126183+0000", 2024-03-11T22:11:01.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5017e-05 2024-03-11T22:11:01.918 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.918 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:26443 7.6 7:6653a772:::benchmark_data_smithi045_83007_object26442:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.972795+0000", 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9111938630000003, 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.160837336, 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 26443 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.919 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972795+0000", 2024-03-11T22:11:01.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972795+0000", 2024-03-11T22:11:01.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972796+0000", 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7700000000000003e-07 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972816+0000", 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0106999999999999e-05 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972827+0000", 2024-03-11T22:11:01.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1090000000000001e-05 2024-03-11T22:11:01.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.972955+0000", 2024-03-11T22:11:01.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012780300000000001 2024-03-11T22:11:01.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973429+0000", 2024-03-11T22:11:01.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00047381000000000002 2024-03-11T22:11:01.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973537+0000", 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010768900000000001 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973669+0000", 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000132287 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.130730+0000", 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.157060795 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.133222+0000", 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024919740000000001 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.133271+0000", 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9157999999999999e-05 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.133322+0000", 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0640999999999998e-05 2024-03-11T22:11:01.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.133633+0000", 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000311105 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:26434 7.4 e106/99 ondisk, result = 0)", 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.973220+0000", 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9107695009999999, 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.153011657, 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.925 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973220+0000", 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973220+0000", 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973220+0000", 2024-03-11T22:11:01.927 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:01.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8999999999999996e-07 2024-03-11T22:11:01.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973224+0000", 2024-03-11T22:11:01.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0389999999999998e-06 2024-03-11T22:11:01.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973236+0000", 2024-03-11T22:11:01.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1284000000000001e-05 2024-03-11T22:11:01.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973242+0000", 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6549999999999998e-06 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126210+0000", 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15296816799999999 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126218+0000", 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1439999999999997e-06 2024-03-11T22:11:01.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126231+0000", 2024-03-11T22:11:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3777e-05 2024-03-11T22:11:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:26444 7.6 7:6c585ecd:::benchmark_data_smithi045_83007_object26443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.973294+0000", 2024-03-11T22:11:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9106951859999999, 2024-03-11T22:11:01.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16184172599999999, 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 26444 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973294+0000", 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973294+0000", 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973295+0000", 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4000000000000001e-07 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973313+0000", 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8637000000000001e-05 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973323+0000", 2024-03-11T22:11:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6330000000000008e-06 2024-03-11T22:11:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.973330+0000", 2024-03-11T22:11:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8000000000000001e-06 2024-03-11T22:11:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974656+0000", 2024-03-11T22:11:01.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001326001 2024-03-11T22:11:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974922+0000", 2024-03-11T22:11:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00026565600000000002 2024-03-11T22:11:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:11:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.127454+0000", 2024-03-11T22:11:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15253276099999999 2024-03-11T22:11:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.133233+0000", 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0057789829999999997 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.133702+0000", 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00046904200000000003 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.135033+0000", 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001330456 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.135091+0000", 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7794999999999998e-05 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.135136+0000", 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5222000000000002e-05 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.936 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:26446 7.f 7:f607f6ed:::benchmark_data_smithi045_83007_object26445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.974568+0000", 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9094214640000002, 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16482787500000001, 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 26446 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974568+0000", 2024-03-11T22:11:01.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974568+0000", 2024-03-11T22:11:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974569+0000", 2024-03-11T22:11:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.15e-06 2024-03-11T22:11:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974665+0000", 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5860999999999996e-05 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974675+0000", 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0583e-05 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974683+0000", 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6159999999999996e-06 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974768+0000", 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5048000000000003e-05 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974887+0000", 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119072 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974994+0000", 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000106875 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.129023+0000", 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15402893400000001 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.129323+0000", 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00030026599999999999 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.139308+0000", 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0099852580000000003 2024-03-11T22:11:01.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.139358+0000", 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9440000000000001e-05 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.139396+0000", 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7771999999999997e-05 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:26447 7.6 7:64033a17:::benchmark_data_smithi045_83007_object26446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.974701+0000", 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9092878459999998, 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16441954, 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 26447 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974701+0000", 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974701+0000", 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974702+0000", 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1600000000000001e-07 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974723+0000", 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0472999999999999e-05 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974731+0000", 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8820000000000007e-06 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.974738+0000", 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5540000000000003e-06 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.130757+0000", 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.156018874 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.130917+0000", 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000159989 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.131761+0000", 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00084370599999999995 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.133654+0000", 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001893329 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.135369+0000", 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001714925 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.139021+0000", 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003652569 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.139080+0000", 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8652e-05 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.139121+0000", 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0871000000000002e-05 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.948 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:26443 7.6 e106/99 ondisk, result = 0)", 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.975105+0000", 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9088837700000001, 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.158549511, 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975105+0000", 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975105+0000", 2024-03-11T22:11:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975106+0000", 2024-03-11T22:11:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8400000000000004e-07 2024-03-11T22:11:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975111+0000", 2024-03-11T22:11:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.566e-06 2024-03-11T22:11:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.125727+0000", 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.150616732 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126699+0000", 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000971057 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.133258+0000", 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065594970000000001 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.133266+0000", 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2220000000000002e-06 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.133655+0000", 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00038875299999999998 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.952 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:26443 7.6 e106/99 ondisk, result = 0)", 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.975114+0000", 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9088755700000002, 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.158120546, 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975114+0000", 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975114+0000", 2024-03-11T22:11:01.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975114+0000", 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4099999999999998e-07 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975118+0000", 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8e-06 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126101+0000", 2024-03-11T22:11:01.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15098298199999999 2024-03-11T22:11:01.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126111+0000", 2024-03-11T22:11:01.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.747e-06 2024-03-11T22:11:01.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.133208+0000", 2024-03-11T22:11:01.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.007096738 2024-03-11T22:11:01.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.133216+0000", 2024-03-11T22:11:01.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2190000000000004e-06 2024-03-11T22:11:01.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.133234+0000", 2024-03-11T22:11:01.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8419000000000001e-05 2024-03-11T22:11:01.957 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.957 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.957 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.957 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:26441 7.4 e106/99 ondisk, result = 0)", 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.975495+0000", 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.9084944650000004, 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.150891939, 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975495+0000", 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975495+0000", 2024-03-11T22:11:01.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975495+0000", 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8400000000000004e-07 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975500+0000", 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6090000000000003e-06 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126225+0000", 2024-03-11T22:11:01.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15072508300000001 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126233+0000", 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3170000000000002e-06 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126370+0000", 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013611 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126376+0000", 2024-03-11T22:11:01.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.161e-06 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126387+0000", 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0975000000000001e-05 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:27618 7.9 7:92784e39:::benchmark_data_smithi045_83007_object27617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.715510+0000", 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.1684788140000002, 2024-03-11T22:11:01.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.136508251, 2024-03-11T22:11:01.962 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.962 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.962 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.962 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.962 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.962 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27618 2024-03-11T22:11:01.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.962 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715510+0000", 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715510+0000", 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715516+0000", 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9170000000000003e-06 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715542+0000", 2024-03-11T22:11:01.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5760000000000001e-05 2024-03-11T22:11:01.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715554+0000", 2024-03-11T22:11:01.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1460999999999999e-05 2024-03-11T22:11:01.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715561+0000", 2024-03-11T22:11:01.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4170000000000003e-06 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715877+0000", 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000315798 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716003+0000", 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000126446 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-03-11T22:11:01.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716156+0000", 2024-03-11T22:11:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015320000000000001 2024-03-11T22:11:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.851870+0000", 2024-03-11T22:11:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13571324500000001 2024-03-11T22:11:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.851882+0000", 2024-03-11T22:11:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2731999999999999e-05 2024-03-11T22:11:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.851924+0000", 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1978999999999999e-05 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.851978+0000", 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3411000000000001e-05 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852019+0000", 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0884999999999999e-05 2024-03-11T22:11:01.967 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:27619 7.9 7:93a16506:::benchmark_data_smithi045_83007_object27618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.715581+0000", 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.1684081309999996, 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14090144499999999, 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27619 2024-03-11T22:11:01.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715581+0000", 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715581+0000", 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715582+0000", 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3499999999999996e-07 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715603+0000", 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.086e-05 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715610+0000", 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5669999999999997e-06 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715615+0000", 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2730000000000002e-06 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.850445+0000", 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.134829848 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.850560+0000", 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114831 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.850685+0000", 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000125318 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.853311+0000", 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026252620000000002 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.854760+0000", 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014492649999999999 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.856376+0000", 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00161586 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.856432+0000", 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6515999999999999e-05 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.856483+0000", 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0309999999999998e-05 2024-03-11T22:11:01.973 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.974 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.974 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.974 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:27620 7.f 7:f7308ece:::benchmark_data_smithi045_83007_object27619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.974 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.715624+0000", 2024-03-11T22:11:01.974 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.1683653969999996, 2024-03-11T22:11:01.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.135500763, 2024-03-11T22:11:01.974 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.974 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.974 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.974 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.974 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27620 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715624+0000", 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715624+0000", 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715624+0000", 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6100000000000001e-07 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715655+0000", 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0244e-05 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715663+0000", 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2749999999999995e-06 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715669+0000", 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8640000000000001e-06 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715723+0000", 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4245e-05 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715849+0000", 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012563900000000001 2024-03-11T22:11:01.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715978+0000", 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000129691 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849157+0000", 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.133178451 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849171+0000", 2024-03-11T22:11:01.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4265e-05 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.851019+0000", 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0018482049999999999 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.851076+0000", 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6549999999999999e-05 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.851125+0000", 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8872999999999998e-05 2024-03-11T22:11:01.979 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.980 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.980 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.980 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:27621 7.6 7:6bec1564:::benchmark_data_smithi045_83007_object27620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.980 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.715686+0000", 2024-03-11T22:11:01.980 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.1683034489999997, 2024-03-11T22:11:01.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13578126600000001, 2024-03-11T22:11:01.980 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.980 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.980 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.980 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.980 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27621 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715686+0000", 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715686+0000", 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715686+0000", 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1899999999999998e-07 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715707+0000", 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.084e-05 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715714+0000", 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8709999999999999e-06 2024-03-11T22:11:01.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715720+0000", 2024-03-11T22:11:01.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2079999999999997e-06 2024-03-11T22:11:01.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715772+0000", 2024-03-11T22:11:01.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1892999999999999e-05 2024-03-11T22:11:01.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715901+0000", 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012919200000000001 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716074+0000", 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017301799999999999 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849216+0000", 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13314130900000001 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849266+0000", 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0800000000000002e-05 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.851372+0000", 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021057910000000001 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.851427+0000", 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4849000000000003e-05 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.851467+0000", 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0076000000000001e-05 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:27625 7.9 7:93bc967b:::benchmark_data_smithi045_83007_object27624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.716869+0000", 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.1671205640000002, 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13977463700000001, 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.987 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27625 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716869+0000", 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716869+0000", 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716870+0000", 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0470000000000001e-06 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716896+0000", 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5806e-05 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849067+0000", 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13217130099999999 2024-03-11T22:11:01.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849077+0000", 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7319999999999993e-06 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852054+0000", 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029774139999999998 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852187+0000", 2024-03-11T22:11:01.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000132562 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852338+0000", 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015133 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.853929+0000", 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001590707 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.856519+0000", 2024-03-11T22:11:01.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025908820000000001 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.856552+0000", 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2098000000000003e-05 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.856602+0000", 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0436999999999998e-05 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.856643+0000", 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1321e-05 2024-03-11T22:11:01.992 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:28068 7.6 7:68ea7166:::benchmark_data_smithi045_83007_object28067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:56.070713+0000", 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.8132761330000005, 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.140084563, 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 28068 2024-03-11T22:11:01.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.070713+0000", 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.070713+0000", 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:01.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.070714+0000", 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2100000000000001e-06 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.070734+0000", 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9749999999999999e-05 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.070745+0000", 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0521e-05 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.070752+0000", 2024-03-11T22:11:01.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0040000000000002e-06 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.205221+0000", 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13446923499999999 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.205383+0000", 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016239499999999999 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.205542+0000", 2024-03-11T22:11:01.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015863699999999999 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.209024+0000", 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0034823229999999998 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.209037+0000", 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2532e-05 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.210695+0000", 2024-03-11T22:11:01.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016587259999999999 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.210748+0000", 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2553999999999999e-05 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.210798+0000", 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9675999999999997e-05 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:01.998 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:01.999 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:11:01.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:01.999 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:02.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:02.100 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:02.131 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:02.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:02.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:02.233 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:02.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:02.266 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:02.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:02.367 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:02.399 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:02.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:02.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:02.543 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:02.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:02.620 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 19 16 39469 39453 129.75 108.125 0.00667442 0.00770187 2024-03-11T22:11:02.645 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:02.676 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:02.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:02.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:02.778 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:02.808 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:02.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:02.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:02.910 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:02.942 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:02.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:02.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:03.026 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:11:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:11:03.044 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:03.073 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:03.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:03.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:03.175 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:03.208 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:03.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:03.310 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:03.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:03 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:11:03.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:03 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:11:03.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:03 smithi144 ceph-mon[33984]: pgmap v79: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.28k op/s 2024-03-11T22:11:03.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:03 smithi144 ceph-mon[33984]: osdmap e109: 8 total, 8 up, 8 in 2024-03-11T22:11:03.341 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:03.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:03.368 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:03 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:11:03.369 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:03 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:11:03.369 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:03 smithi045 ceph-mon[39715]: pgmap v79: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.28k op/s 2024-03-11T22:11:03.369 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:03 smithi045 ceph-mon[39715]: osdmap e109: 8 total, 8 up, 8 in 2024-03-11T22:11:03.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:03 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:11:03.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:03 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:11:03.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:03 smithi045 ceph-mon[30178]: pgmap v79: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 142 MiB/s wr, 2.28k op/s 2024-03-11T22:11:03.369 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:03 smithi045 ceph-mon[30178]: osdmap e109: 8 total, 8 up, 8 in 2024-03-11T22:11:03.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:03.489 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:03.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:03.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:03.591 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:03.621 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:11:03.627103+0000 min lat: 0.00145914 max lat: 0.165961 avg lat: 0.00775487 2024-03-11T22:11:03.621 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:11:03.621 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 20 16 41260 41244 128.857 111.938 0.00801458 0.00775487 2024-03-11T22:11:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:03.622 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:03.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:03.723 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:03.752 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:03.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:03.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:03.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:03.893 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:03.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:03.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:04.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:04.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:04.109 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:11:04.145 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:04.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:04.174 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:04.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:04.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:04.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:04.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:04.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:04 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3866768341' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:11:04.411 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:04 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3866768341' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:11:04.411 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:04 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3866768341' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:11:04.415 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:04.446 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:04.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:04.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:04.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:04.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:04.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:04.621 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 21 16 42691 42675 126.98 89.4375 0.0103411 0.0078699 2024-03-11T22:11:04.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:04.735 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:04.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:04.837 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:04.880 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:04.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:04.982 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:05.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:05.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:05.124 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:05.165 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:05.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:05.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:05.267 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:05.301 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:05.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:05.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:05.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:05 smithi144 ceph-mon[33984]: pgmap v81: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 640 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T22:11:05.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:05 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:11:05.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:05 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3866768341' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:11:05.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:05 smithi144 ceph-mon[33984]: osdmap e110: 8 total, 8 up, 8 in 2024-03-11T22:11:05.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:05 smithi144 ceph-mon[33984]: 3.d scrub starts 2024-03-11T22:11:05.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:05 smithi144 ceph-mon[33984]: 3.d scrub ok 2024-03-11T22:11:05.403 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:05.437 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:05.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:05 smithi045 ceph-mon[39715]: pgmap v81: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 640 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T22:11:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:05 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:11:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:05 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3866768341' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:11:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:05 smithi045 ceph-mon[39715]: osdmap e110: 8 total, 8 up, 8 in 2024-03-11T22:11:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:05 smithi045 ceph-mon[39715]: 3.d scrub starts 2024-03-11T22:11:05.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:05 smithi045 ceph-mon[39715]: 3.d scrub ok 2024-03-11T22:11:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:05 smithi045 ceph-mon[30178]: pgmap v81: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 640 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T22:11:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:05 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:11:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:05 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3866768341' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:11:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:05 smithi045 ceph-mon[30178]: osdmap e110: 8 total, 8 up, 8 in 2024-03-11T22:11:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:05 smithi045 ceph-mon[30178]: 3.d scrub starts 2024-03-11T22:11:05.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:05 smithi045 ceph-mon[30178]: 3.d scrub ok 2024-03-11T22:11:05.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:05.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:05.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:05.621 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 22 16 44293 44277 125.759 100.125 0.00690358 0.00794548 2024-03-11T22:11:05.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:05.717 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:05.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:05.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:05.819 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:05.855 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:05.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:05.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:05.957 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:05.992 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:05.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:06.088 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:11:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:11:06.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:06.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:06.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:06.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:06.234 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:06.263 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:06.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:06.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:06.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:06 smithi144 ceph-mon[33984]: 5.11 scrub starts 2024-03-11T22:11:06.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:06 smithi144 ceph-mon[33984]: 5.11 scrub ok 2024-03-11T22:11:06.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:06 smithi144 ceph-mon[33984]: osdmap e111: 8 total, 8 up, 8 in 2024-03-11T22:11:06.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:06.413 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:06.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:06.514 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:06 smithi045 ceph-mon[30178]: 5.11 scrub starts 2024-03-11T22:11:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:06 smithi045 ceph-mon[30178]: 5.11 scrub ok 2024-03-11T22:11:06.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:06 smithi045 ceph-mon[30178]: osdmap e111: 8 total, 8 up, 8 in 2024-03-11T22:11:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:06 smithi045 ceph-mon[39715]: 5.11 scrub starts 2024-03-11T22:11:06.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:06 smithi045 ceph-mon[39715]: 5.11 scrub ok 2024-03-11T22:11:06.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:06 smithi045 ceph-mon[39715]: osdmap e111: 8 total, 8 up, 8 in 2024-03-11T22:11:06.527 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:06 smithi045 systemd[1]: Stopping Ceph alertmanager.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:11:06.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:06.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:06.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:06.621 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 23 16 45939 45923 124.764 102.875 0.00850116 0.00800914 2024-03-11T22:11:06.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:06.714 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:06.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:06.816 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:06.848 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:06.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:06.950 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:06.994 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:06.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:07.093 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:07.096 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:07.123 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:07.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:07.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:07.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:07.256 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:07.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:07.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:07.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:07 smithi045 ceph-mon[30178]: pgmap v83: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 177 MiB/s wr, 2.84k op/s 2024-03-11T22:11:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:07 smithi045 ceph-mon[30178]: 6.14 scrub starts 2024-03-11T22:11:07.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:07 smithi045 ceph-mon[30178]: 6.14 scrub ok 2024-03-11T22:11:07.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:07.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:07 smithi045 ceph-mon[39715]: pgmap v83: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 177 MiB/s wr, 2.84k op/s 2024-03-11T22:11:07.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:07 smithi045 ceph-mon[39715]: 6.14 scrub starts 2024-03-11T22:11:07.276 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:07 smithi045 ceph-mon[39715]: 6.14 scrub ok 2024-03-11T22:11:07.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:07.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:07 smithi144 ceph-mon[33984]: pgmap v83: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.7 KiB/s rd, 177 MiB/s wr, 2.84k op/s 2024-03-11T22:11:07.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:07 smithi144 ceph-mon[33984]: 6.14 scrub starts 2024-03-11T22:11:07.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:07 smithi144 ceph-mon[33984]: 6.14 scrub ok 2024-03-11T22:11:07.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:07.399 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:07.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:07.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:07.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:07.538 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:07.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:07.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:07.566 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:07.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T22:11:07.622 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 24 15 47491 47476 123.609 97.0625 0.00748072 0.00808534 2024-03-11T22:11:07.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:07.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:07.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:07.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:07.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:07.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:07.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:07.996 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:08.024 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:08.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:08.114 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:07 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[77897]: ts=2024-03-11T22:11:07.961Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:11:08.114 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:08 smithi045 ceph-mon[39715]: 4.6 scrub starts 2024-03-11T22:11:08.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:08 smithi045 ceph-mon[30178]: 4.6 scrub starts 2024-03-11T22:11:08.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:08 smithi045 ceph-mon[30178]: 4.6 scrub ok 2024-03-11T22:11:08.125 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:08.154 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:08.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:08.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:08.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:08.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:08.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:08.399 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:08.429 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:08.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:08.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:08.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:08 smithi045 ceph-mon[39715]: 4.6 scrub ok 2024-03-11T22:11:08.531 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:08.569 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:08.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:08.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:08 smithi144 ceph-mon[33984]: 4.6 scrub starts 2024-03-11T22:11:08.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:08 smithi144 ceph-mon[33984]: 4.6 scrub ok 2024-03-11T22:11:08.621 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 25 15 49013 48998 122.469 95.125 0.0202024 0.00815995 2024-03-11T22:11:08.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:08.712 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:08.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:08.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:08.855 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:08.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:08.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:08.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:08.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:08.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:09.071 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:11:09.100 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:09.129 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:09.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:09.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:09.231 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:09.261 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:09.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:09.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:09.270 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:09 smithi045 ceph-mon[39715]: 4.0 scrub starts 2024-03-11T22:11:09.270 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:09 smithi045 ceph-mon[39715]: 4.0 scrub ok 2024-03-11T22:11:09.271 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:09 smithi045 ceph-mon[39715]: pgmap v85: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-11T22:11:09.271 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:09 smithi045 ceph-mon[30178]: 4.0 scrub starts 2024-03-11T22:11:09.271 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:09 smithi045 ceph-mon[30178]: 4.0 scrub ok 2024-03-11T22:11:09.271 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:09 smithi045 ceph-mon[30178]: pgmap v85: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-11T22:11:09.363 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:09.393 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:09.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:09.495 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:09.526 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:09.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:09.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:09 smithi144 ceph-mon[33984]: 4.0 scrub starts 2024-03-11T22:11:09.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:09 smithi144 ceph-mon[33984]: 4.0 scrub ok 2024-03-11T22:11:09.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:09 smithi144 ceph-mon[33984]: pgmap v85: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-11T22:11:09.622 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 26 16 50536 50520 121.413 95.125 0.0077872 0.00823082 2024-03-11T22:11:09.629 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:09.678 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:09.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:09.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:09.780 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:09.810 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:09.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:09.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:09.860 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:09 smithi045 podman[85717]: 2024-03-11 22:11:09.538901757 +0000 UTC m=+1.962168715 container died 92b3721cb5b78104493f3eabcf0691942a890dfc48da95f86ba3f45f430d247b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:11:09.912 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:09.941 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:09.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:09.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:10.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:10.093 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:10.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:10.120 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:10 smithi045 podman[85717]: 2024-03-11 22:11:10.078979352 +0000 UTC m=+2.502246316 container cleanup 92b3721cb5b78104493f3eabcf0691942a890dfc48da95f86ba3f45f430d247b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:11:10.121 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:10 smithi045 bash[85717]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T22:11:10.194 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:10.226 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:10.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:10.328 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:10.357 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:10.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:10.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:10.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:10 smithi045 ceph-mon[39715]: 4.2 scrub starts 2024-03-11T22:11:10.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:10 smithi045 ceph-mon[39715]: 4.2 scrub ok 2024-03-11T22:11:10.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:10 smithi045 ceph-mon[39715]: 6.c scrub starts 2024-03-11T22:11:10.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:10 smithi045 ceph-mon[39715]: 6.c scrub ok 2024-03-11T22:11:10.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:10 smithi045 ceph-mon[30178]: 4.2 scrub starts 2024-03-11T22:11:10.394 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:10 smithi045 ceph-mon[30178]: 4.2 scrub ok 2024-03-11T22:11:10.394 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:10 smithi045 ceph-mon[30178]: 6.c scrub starts 2024-03-11T22:11:10.394 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:10 smithi045 ceph-mon[30178]: 6.c scrub ok 2024-03-11T22:11:10.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:10.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:10.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:10 smithi144 ceph-mon[33984]: 4.2 scrub starts 2024-03-11T22:11:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:10 smithi144 ceph-mon[33984]: 4.2 scrub ok 2024-03-11T22:11:10.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:10 smithi144 ceph-mon[33984]: 6.c scrub starts 2024-03-11T22:11:10.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:10 smithi144 ceph-mon[33984]: 6.c scrub ok 2024-03-11T22:11:10.605 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:10.625 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 27 16 52096 52080 120.514 97.5 0.00693468 0.00829195 2024-03-11T22:11:10.634 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:10.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:10.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:10.736 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:10.769 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:10.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:10.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:10.775 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:10 smithi045 podman[85822]: 2024-03-11 22:11:10.398638801 +0000 UTC m=+2.389719354 container remove 92b3721cb5b78104493f3eabcf0691942a890dfc48da95f86ba3f45f430d247b (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:11:10.775 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:10 smithi045 podman[85822]: 2024-03-11 22:11:10.50807096 +0000 UTC m=+2.499151508 volume remove 9aa1c240518f01ef6b3f68c9e0296fafdb30983cfa3ff6bb840254192c93724d 2024-03-11T22:11:10.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:10.917 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:10.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:11.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:11.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:11.071 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:11.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:11.173 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:11.203 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:11.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:11.304 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:11.333 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:11.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:11.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:11 smithi045 ceph-mon[39715]: 3.18 scrub starts 2024-03-11T22:11:11.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:11 smithi045 ceph-mon[39715]: 3.18 scrub ok 2024-03-11T22:11:11.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:11 smithi045 ceph-mon[39715]: pgmap v86: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 641 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-11T22:11:11.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:11 smithi045 ceph-mon[39715]: 4.a scrub starts 2024-03-11T22:11:11.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:11 smithi045 ceph-mon[39715]: 4.a scrub ok 2024-03-11T22:11:11.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:11 smithi045 ceph-mon[39715]: osdmap e112: 8 total, 8 up, 8 in 2024-03-11T22:11:11.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:11 smithi045 ceph-mon[30178]: 3.18 scrub starts 2024-03-11T22:11:11.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:11 smithi045 ceph-mon[30178]: 3.18 scrub ok 2024-03-11T22:11:11.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:11 smithi045 ceph-mon[30178]: pgmap v86: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 641 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-11T22:11:11.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:11 smithi045 ceph-mon[30178]: 4.a scrub starts 2024-03-11T22:11:11.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:11 smithi045 ceph-mon[30178]: 4.a scrub ok 2024-03-11T22:11:11.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:11 smithi045 ceph-mon[30178]: osdmap e112: 8 total, 8 up, 8 in 2024-03-11T22:11:11.435 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:11.450 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:11:11.450 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:11:11.450 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:11:11.451 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:11:11.451 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:11:11.461 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:11:11.463 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:11.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:11.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:11.504 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:11:11.549 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:11:11.555 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:11:11.555 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:11:11.556 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:11:11.556 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:11:11.556 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:11:11.556 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:11:11.556 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:11:11.556 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:11:11.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:11 smithi144 ceph-mon[33984]: 3.18 scrub starts 2024-03-11T22:11:11.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:11 smithi144 ceph-mon[33984]: 3.18 scrub ok 2024-03-11T22:11:11.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:11 smithi144 ceph-mon[33984]: pgmap v86: 148 pgs: 148 active+clean; 2.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 641 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-11T22:11:11.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:11 smithi144 ceph-mon[33984]: 4.a scrub starts 2024-03-11T22:11:11.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:11 smithi144 ceph-mon[33984]: 4.a scrub ok 2024-03-11T22:11:11.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:11 smithi144 ceph-mon[33984]: osdmap e112: 8 total, 8 up, 8 in 2024-03-11T22:11:11.616 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:11.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:11.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:11.625 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 28 16 53440 53424 119.21 84 0.00934272 0.00838298 2024-03-11T22:11:11.718 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:11.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:11.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:11.862 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:11.897 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:11.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:11.998 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:12.028 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:12.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:12.131 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:12.160 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:12.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:12.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:12.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:12.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:12.409 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:12.452 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:12.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:12.502 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[39715]: 6.0 scrub starts 2024-03-11T22:11:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[39715]: 6.0 scrub ok 2024-03-11T22:11:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1798025826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:11:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[39715]: 4.1f scrub starts 2024-03-11T22:11:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[39715]: 4.1f scrub ok 2024-03-11T22:11:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[39715]: 5.d scrub starts 2024-03-11T22:11:12.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[39715]: 5.d scrub ok 2024-03-11T22:11:12.527 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:12 smithi045 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T22:11:12.527 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:12 smithi045 systemd[1]: Stopped Ceph alertmanager.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:11:12.527 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:12 smithi045 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.266s CPU time. 2024-03-11T22:11:12.527 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:12 smithi045 systemd[1]: Starting Ceph alertmanager.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:11:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[30178]: 6.0 scrub starts 2024-03-11T22:11:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[30178]: 6.0 scrub ok 2024-03-11T22:11:12.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1798025826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:11:12.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[30178]: 4.1f scrub starts 2024-03-11T22:11:12.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[30178]: 4.1f scrub ok 2024-03-11T22:11:12.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[30178]: 5.d scrub starts 2024-03-11T22:11:12.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-mon[30178]: 5.d scrub ok 2024-03-11T22:11:12.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:12.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:12 smithi144 ceph-mon[33984]: 6.0 scrub starts 2024-03-11T22:11:12.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:12 smithi144 ceph-mon[33984]: 6.0 scrub ok 2024-03-11T22:11:12.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:12 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1798025826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:11:12.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:12 smithi144 ceph-mon[33984]: 4.1f scrub starts 2024-03-11T22:11:12.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:12 smithi144 ceph-mon[33984]: 4.1f scrub ok 2024-03-11T22:11:12.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:12 smithi144 ceph-mon[33984]: 5.d scrub starts 2024-03-11T22:11:12.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:12 smithi144 ceph-mon[33984]: 5.d scrub ok 2024-03-11T22:11:12.606 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:12.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:12.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:12.626 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 29 16 54890 54874 118.22 90.625 0.00574407 0.00845333 2024-03-11T22:11:12.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:12.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:12.751 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:12.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:12.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:12.893 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:12.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:12.995 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:13.025 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:11:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:11:13.026 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:13.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:13.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:13.128 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:13.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:13.159 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:13.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:13.259 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:13.290 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:13.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:13.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:13.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:13.426 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:13.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:13.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[30178]: 5.1d scrub starts 2024-03-11T22:11:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[30178]: 5.1d scrub ok 2024-03-11T22:11:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[30178]: pgmap v88: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 128 MiB/s wr, 2.04k op/s 2024-03-11T22:11:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[30178]: 4.8 scrub starts 2024-03-11T22:11:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[30178]: 4.8 scrub ok 2024-03-11T22:11:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[30178]: 4.c scrub starts 2024-03-11T22:11:13.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[30178]: 4.c scrub ok 2024-03-11T22:11:13.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[30178]: 5.8 scrub starts 2024-03-11T22:11:13.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[30178]: 5.8 scrub ok 2024-03-11T22:11:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[39715]: 5.1d scrub starts 2024-03-11T22:11:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[39715]: 5.1d scrub ok 2024-03-11T22:11:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[39715]: pgmap v88: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 128 MiB/s wr, 2.04k op/s 2024-03-11T22:11:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[39715]: 4.8 scrub starts 2024-03-11T22:11:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[39715]: 4.8 scrub ok 2024-03-11T22:11:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[39715]: 4.c scrub starts 2024-03-11T22:11:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[39715]: 4.c scrub ok 2024-03-11T22:11:13.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[39715]: 5.8 scrub starts 2024-03-11T22:11:13.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:13 smithi045 ceph-mon[39715]: 5.8 scrub ok 2024-03-11T22:11:13.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:13.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:13.567 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:13.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:13.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:13 smithi144 ceph-mon[33984]: 5.1d scrub starts 2024-03-11T22:11:13.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:13 smithi144 ceph-mon[33984]: 5.1d scrub ok 2024-03-11T22:11:13.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:13 smithi144 ceph-mon[33984]: pgmap v88: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 128 MiB/s wr, 2.04k op/s 2024-03-11T22:11:13.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:13 smithi144 ceph-mon[33984]: 4.8 scrub starts 2024-03-11T22:11:13.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:13 smithi144 ceph-mon[33984]: 4.8 scrub ok 2024-03-11T22:11:13.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:13 smithi144 ceph-mon[33984]: 4.c scrub starts 2024-03-11T22:11:13.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:13 smithi144 ceph-mon[33984]: 4.c scrub ok 2024-03-11T22:11:13.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:13 smithi144 ceph-mon[33984]: 5.8 scrub starts 2024-03-11T22:11:13.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:13 smithi144 ceph-mon[33984]: 5.8 scrub ok 2024-03-11T22:11:13.626 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 30 16 56370 56354 117.363 92.5 0.00596 0.00851568 2024-03-11T22:11:13.668 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:13.700 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:13.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:13.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:13.802 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:13.832 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:13.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:13.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:13.934 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:13.963 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:13.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:13.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:14.066 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:14.093 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:14.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:14.195 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:14.225 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:14.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:14.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:14.327 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:14.352 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:14.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:14.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:14.453 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:14.484 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:14.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:14.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:14 smithi045 ceph-mon[39715]: 6.1f scrub starts 2024-03-11T22:11:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:14 smithi045 ceph-mon[39715]: 6.1f scrub ok 2024-03-11T22:11:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:14 smithi045 ceph-mon[39715]: 3.1d scrub starts 2024-03-11T22:11:14.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:14 smithi045 ceph-mon[39715]: 3.1d scrub ok 2024-03-11T22:11:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:14 smithi045 ceph-mon[30178]: 6.1f scrub starts 2024-03-11T22:11:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:14 smithi045 ceph-mon[30178]: 6.1f scrub ok 2024-03-11T22:11:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:14 smithi045 ceph-mon[30178]: 3.1d scrub starts 2024-03-11T22:11:14.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:14 smithi045 ceph-mon[30178]: 3.1d scrub ok 2024-03-11T22:11:14.586 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:14.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:14 smithi144 ceph-mon[33984]: 6.1f scrub starts 2024-03-11T22:11:14.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:14 smithi144 ceph-mon[33984]: 6.1f scrub ok 2024-03-11T22:11:14.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:14 smithi144 ceph-mon[33984]: 3.1d scrub starts 2024-03-11T22:11:14.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:14 smithi144 ceph-mon[33984]: 3.1d scrub ok 2024-03-11T22:11:14.617 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:14.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:14.627 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 31 16 57786 57770 116.432 88.5 0.0120408 0.00858315 2024-03-11T22:11:14.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:14.786 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:14.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:14.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:14.926 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:14.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:15.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:15.070 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:15.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:15.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:15.125 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:11:15.172 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:15.203 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:15.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[30178]: 6.16 scrub starts 2024-03-11T22:11:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[30178]: 6.16 scrub ok 2024-03-11T22:11:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[30178]: 5.12 scrub starts 2024-03-11T22:11:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[30178]: 5.12 scrub ok 2024-03-11T22:11:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[30178]: pgmap v89: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 978 op/s 2024-03-11T22:11:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[30178]: 3.a scrub starts 2024-03-11T22:11:15.276 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[30178]: 3.a scrub ok 2024-03-11T22:11:15.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[30178]: 4.1c scrub starts 2024-03-11T22:11:15.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[30178]: 4.1c scrub ok 2024-03-11T22:11:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[39715]: 6.16 scrub starts 2024-03-11T22:11:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[39715]: 6.16 scrub ok 2024-03-11T22:11:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[39715]: 5.12 scrub starts 2024-03-11T22:11:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[39715]: 5.12 scrub ok 2024-03-11T22:11:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[39715]: pgmap v89: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 978 op/s 2024-03-11T22:11:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[39715]: 3.a scrub starts 2024-03-11T22:11:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[39715]: 3.a scrub ok 2024-03-11T22:11:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[39715]: 4.1c scrub starts 2024-03-11T22:11:15.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:15 smithi045 ceph-mon[39715]: 4.1c scrub ok 2024-03-11T22:11:15.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:15.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:15.362 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:15.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:15.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:15.510 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:15.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:15.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:15 smithi144 ceph-mon[33984]: 6.16 scrub starts 2024-03-11T22:11:15.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:15 smithi144 ceph-mon[33984]: 6.16 scrub ok 2024-03-11T22:11:15.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:15 smithi144 ceph-mon[33984]: 5.12 scrub starts 2024-03-11T22:11:15.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:15 smithi144 ceph-mon[33984]: 5.12 scrub ok 2024-03-11T22:11:15.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:15 smithi144 ceph-mon[33984]: pgmap v89: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 61 MiB/s wr, 978 op/s 2024-03-11T22:11:15.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:15.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:15 smithi144 ceph-mon[33984]: 3.a scrub starts 2024-03-11T22:11:15.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:15 smithi144 ceph-mon[33984]: 3.a scrub ok 2024-03-11T22:11:15.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:15 smithi144 ceph-mon[33984]: 4.1c scrub starts 2024-03-11T22:11:15.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:15 smithi144 ceph-mon[33984]: 4.1c scrub ok 2024-03-11T22:11:15.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:15.627 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 32 16 59131 59115 115.42 84.0625 0.0121511 0.00865769 2024-03-11T22:11:15.669 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:15.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:15.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:15.771 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:15.814 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:15.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:15.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:15.916 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:15.947 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:15.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:16.049 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:16.080 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:16.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:16.089 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:11:15 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:11:16.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:16.232 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:16.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:16.333 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:16.361 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:16.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:16.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:16.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:16 smithi045 ceph-mon[30178]: 5.5 scrub starts 2024-03-11T22:11:16.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:16 smithi045 ceph-mon[30178]: 5.5 scrub ok 2024-03-11T22:11:16.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:16 smithi045 ceph-mon[30178]: 5.1f scrub starts 2024-03-11T22:11:16.421 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:16 smithi045 ceph-mon[30178]: 5.1f scrub ok 2024-03-11T22:11:16.421 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:16 smithi045 ceph-mon[39715]: 5.5 scrub starts 2024-03-11T22:11:16.421 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:16 smithi045 ceph-mon[39715]: 5.5 scrub ok 2024-03-11T22:11:16.421 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:16 smithi045 ceph-mon[39715]: 5.1f scrub starts 2024-03-11T22:11:16.421 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:16 smithi045 ceph-mon[39715]: 5.1f scrub ok 2024-03-11T22:11:16.422 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:16 smithi045 podman[86677]: 2024-03-11 22:11:16.205388273 +0000 UTC m=+0.590699256 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-11T22:11:16.422 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:16 smithi045 podman[86677]: 2024-03-11 22:11:16.308755137 +0000 UTC m=+0.694066128 volume create 9636ff0db1f0237a3ca036f8ef5164c6d30fd6f6fb34452095f79747947ec1e1 2024-03-11T22:11:16.422 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:16 smithi045 podman[86677]: 2024-03-11 22:11:16.426123855 +0000 UTC m=+0.811434885 container create 668cd5003e5013eb7fc070440b520ba2a37bc2ebbd7501b99a35c6d69fd99bb0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:11:16.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:16.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:16.510 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:16.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:16.589 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:16 smithi144 ceph-mon[33984]: 5.5 scrub starts 2024-03-11T22:11:16.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:16 smithi144 ceph-mon[33984]: 5.5 scrub ok 2024-03-11T22:11:16.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:16 smithi144 ceph-mon[33984]: 5.1f scrub starts 2024-03-11T22:11:16.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:16 smithi144 ceph-mon[33984]: 5.1f scrub ok 2024-03-11T22:11:16.610 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:16.627 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 33 16 60533 60517 114.578 87.625 0.00494861 0.00872249 2024-03-11T22:11:16.639 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:16.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:16.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:16.741 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:16.770 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:16.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:16.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:16.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:16.915 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:16.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:17.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:17.065 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:17.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:17.089 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:16 smithi045 podman[86677]: 2024-03-11 22:11:16.931987584 +0000 UTC m=+1.317298584 container init 668cd5003e5013eb7fc070440b520ba2a37bc2ebbd7501b99a35c6d69fd99bb0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:11:17.089 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:16 smithi045 podman[86677]: 2024-03-11 22:11:16.943289893 +0000 UTC m=+1.328600884 container start 668cd5003e5013eb7fc070440b520ba2a37bc2ebbd7501b99a35c6d69fd99bb0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:11:17.089 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:16 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[86801]: ts=2024-03-11T22:11:16.966Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T22:11:17.089 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:16 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[86801]: ts=2024-03-11T22:11:16.966Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T22:11:17.089 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:16 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[86801]: ts=2024-03-11T22:11:16.970Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.45 port=9094 2024-03-11T22:11:17.089 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:16 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[86801]: ts=2024-03-11T22:11:16.972Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T22:11:17.089 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:17 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[86801]: ts=2024-03-11T22:11:17.019Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T22:11:17.090 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:17 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[86801]: ts=2024-03-11T22:11:17.020Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T22:11:17.090 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:17 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[86801]: ts=2024-03-11T22:11:17.024Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T22:11:17.090 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:17 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[86801]: ts=2024-03-11T22:11:17.024Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T22:11:17.167 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:17.193 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:17.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:17.294 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:17.321 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:17.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:17.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:17.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:17 smithi144 ceph-mon[33984]: 3.4 scrub starts 2024-03-11T22:11:17.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:17 smithi144 ceph-mon[33984]: 3.4 scrub ok 2024-03-11T22:11:17.339 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:17 smithi144 ceph-mon[33984]: pgmap v90: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-11T22:11:17.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:17.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:17 smithi045 ceph-mon[39715]: 3.4 scrub starts 2024-03-11T22:11:17.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:17 smithi045 ceph-mon[39715]: 3.4 scrub ok 2024-03-11T22:11:17.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:17 smithi045 ceph-mon[39715]: pgmap v90: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-11T22:11:17.380 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:17 smithi045 bash[86677]: 668cd5003e5013eb7fc070440b520ba2a37bc2ebbd7501b99a35c6d69fd99bb0 2024-03-11T22:11:17.380 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:17 smithi045 systemd[1]: Started Ceph alertmanager.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:11:17.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:17.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:17 smithi045 ceph-mon[30178]: 3.4 scrub starts 2024-03-11T22:11:17.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:17 smithi045 ceph-mon[30178]: 3.4 scrub ok 2024-03-11T22:11:17.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:17 smithi045 ceph-mon[30178]: pgmap v90: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-11T22:11:17.422 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:17.448 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:17.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:17.497 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:11:17.497 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:11:17.550 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:17.579 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:17.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:17.627 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 34 16 61675 61659 113.307 71.375 0.0122871 0.00881893 2024-03-11T22:11:17.682 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:17.724 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:17.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:17.826 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:17.854 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:17.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:17.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:17.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:17.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:17.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:18.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:18.138 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:18.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:18.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:18.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:18.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:18.287 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:18.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:18.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:18.429 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:18.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:18.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:18.516 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T22:11:18.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[39715]: 6.1b scrub starts 2024-03-11T22:11:18.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[39715]: 6.1b scrub ok 2024-03-11T22:11:18.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:18.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:18.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[39715]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T22:11:18.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[39715]: 4.d scrub starts 2024-03-11T22:11:18.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[39715]: 4.d scrub ok 2024-03-11T22:11:18.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[39715]: 2.1 scrub starts 2024-03-11T22:11:18.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[39715]: 2.1 scrub ok 2024-03-11T22:11:18.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[39715]: Reconfiguring daemon prometheus.a on smithi144 2024-03-11T22:11:18.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3294378058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:11:18.518 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:11:18.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[30178]: 6.1b scrub starts 2024-03-11T22:11:18.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[30178]: 6.1b scrub ok 2024-03-11T22:11:18.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:18.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:18.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[30178]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T22:11:18.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[30178]: 4.d scrub starts 2024-03-11T22:11:18.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[30178]: 4.d scrub ok 2024-03-11T22:11:18.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[30178]: 2.1 scrub starts 2024-03-11T22:11:18.519 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[30178]: 2.1 scrub ok 2024-03-11T22:11:18.519 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[30178]: Reconfiguring daemon prometheus.a on smithi144 2024-03-11T22:11:18.519 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3294378058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:11:18.519 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:11:18.531 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:18.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:18.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:18.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:18.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:18 smithi144 ceph-mon[33984]: 6.1b scrub starts 2024-03-11T22:11:18.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:18 smithi144 ceph-mon[33984]: 6.1b scrub ok 2024-03-11T22:11:18.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:18 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:18.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:18 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:18.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:18 smithi144 ceph-mon[33984]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T22:11:18.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:18 smithi144 ceph-mon[33984]: 4.d scrub starts 2024-03-11T22:11:18.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:18 smithi144 ceph-mon[33984]: 4.d scrub ok 2024-03-11T22:11:18.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:18 smithi144 ceph-mon[33984]: 2.1 scrub starts 2024-03-11T22:11:18.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:18 smithi144 ceph-mon[33984]: 2.1 scrub ok 2024-03-11T22:11:18.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:18 smithi144 ceph-mon[33984]: Reconfiguring daemon prometheus.a on smithi144 2024-03-11T22:11:18.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:18 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3294378058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:11:18.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:18 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:11:18.627 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 35 16 62623 62607 111.763 59.25 0.00442363 0.00893897 2024-03-11T22:11:18.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:18.706 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:18.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:18.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:18.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:18.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:18.945 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:18.978 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:18.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:19.080 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:19.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:19.104 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:19.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:19.205 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:19.228 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:18 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[86801]: ts=2024-03-11T22:11:18.975Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.002790242s 2024-03-11T22:11:19.248 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:19.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:19.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:19.350 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:19.373 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:19.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:19.475 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:19.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:19.507 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:19.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[39715]: 6.11 scrub starts 2024-03-11T22:11:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[39715]: 6.11 scrub ok 2024-03-11T22:11:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[39715]: pgmap v91: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-11T22:11:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[39715]: 3.5 scrub starts 2024-03-11T22:11:19.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[39715]: 3.5 scrub ok 2024-03-11T22:11:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:11:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[39715]: osdmap e113: 8 total, 8 up, 8 in 2024-03-11T22:11:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[39715]: 5.6 scrub starts 2024-03-11T22:11:19.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[39715]: 5.6 scrub ok 2024-03-11T22:11:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[30178]: 6.11 scrub starts 2024-03-11T22:11:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[30178]: 6.11 scrub ok 2024-03-11T22:11:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[30178]: pgmap v91: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-11T22:11:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[30178]: 3.5 scrub starts 2024-03-11T22:11:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[30178]: 3.5 scrub ok 2024-03-11T22:11:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:11:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[30178]: osdmap e113: 8 total, 8 up, 8 in 2024-03-11T22:11:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[30178]: 5.6 scrub starts 2024-03-11T22:11:19.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:19 smithi045 ceph-mon[30178]: 5.6 scrub ok 2024-03-11T22:11:19.554 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:19 smithi144 ceph-mon[33984]: 6.11 scrub starts 2024-03-11T22:11:19.554 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:19 smithi144 ceph-mon[33984]: 6.11 scrub ok 2024-03-11T22:11:19.555 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:19 smithi144 ceph-mon[33984]: pgmap v91: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-11T22:11:19.555 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:19 smithi144 ceph-mon[33984]: 3.5 scrub starts 2024-03-11T22:11:19.555 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:19 smithi144 ceph-mon[33984]: 3.5 scrub ok 2024-03-11T22:11:19.555 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:19 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:11:19.555 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:19 smithi144 ceph-mon[33984]: osdmap e113: 8 total, 8 up, 8 in 2024-03-11T22:11:19.555 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:19 smithi144 ceph-mon[33984]: 5.6 scrub starts 2024-03-11T22:11:19.555 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:19 smithi144 ceph-mon[33984]: 5.6 scrub ok 2024-03-11T22:11:19.609 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:19.627 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 36 16 63721 63705 110.565 68.625 0.00607681 0.00903814 2024-03-11T22:11:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:19.641 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:19.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:19.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:19.766 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:19.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:19.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:19.908 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:19.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:19.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:20.009 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:20.041 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:20.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:20.143 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:20.173 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:20.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:20.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:20.274 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:20.283 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:20.301 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:20.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:20.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:20 smithi144 ceph-mon[33984]: 5.19 scrub starts 2024-03-11T22:11:20.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:20 smithi144 ceph-mon[33984]: 5.19 scrub ok 2024-03-11T22:11:20.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:20 smithi144 ceph-mon[33984]: 4.13 scrub starts 2024-03-11T22:11:20.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:20 smithi144 ceph-mon[33984]: 4.13 scrub ok 2024-03-11T22:11:20.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:20 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:11:20.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:20 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:11:20.403 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:20.430 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:20.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:20 smithi045 ceph-mon[39715]: 5.19 scrub starts 2024-03-11T22:11:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:20 smithi045 ceph-mon[39715]: 5.19 scrub ok 2024-03-11T22:11:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:20 smithi045 ceph-mon[39715]: 4.13 scrub starts 2024-03-11T22:11:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:20 smithi045 ceph-mon[39715]: 4.13 scrub ok 2024-03-11T22:11:20.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:20 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:11:20.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:20 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:11:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:20 smithi045 ceph-mon[30178]: 5.19 scrub starts 2024-03-11T22:11:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:20 smithi045 ceph-mon[30178]: 5.19 scrub ok 2024-03-11T22:11:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:20 smithi045 ceph-mon[30178]: 4.13 scrub starts 2024-03-11T22:11:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:20 smithi045 ceph-mon[30178]: 4.13 scrub ok 2024-03-11T22:11:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:20 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:11:20.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:20 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:11:20.533 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:20.563 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:20.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:20.628 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 37 16 65006 64990 109.747 80.3125 0.00312465 0.00910152 2024-03-11T22:11:20.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:20.697 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:20.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:20.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:20.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:20.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:20.840 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:20.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:20.941 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:20.972 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:20.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:21.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:21.118 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:21.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:21.219 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:21.248 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:21.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:21.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:21.387 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:21.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:21.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[39715]: 6.6 scrub starts 2024-03-11T22:11:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[39715]: 6.6 scrub ok 2024-03-11T22:11:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[39715]: 3.0 scrub starts 2024-03-11T22:11:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[39715]: 3.0 scrub ok 2024-03-11T22:11:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[39715]: pgmap v93: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 569 B/s rd, 50 MiB/s wr, 799 op/s 2024-03-11T22:11:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:11:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:11:21.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[39715]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T22:11:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[30178]: 6.6 scrub starts 2024-03-11T22:11:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[30178]: 6.6 scrub ok 2024-03-11T22:11:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[30178]: 3.0 scrub starts 2024-03-11T22:11:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[30178]: 3.0 scrub ok 2024-03-11T22:11:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[30178]: pgmap v93: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 569 B/s rd, 50 MiB/s wr, 799 op/s 2024-03-11T22:11:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:11:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:11:21.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:21 smithi045 ceph-mon[30178]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T22:11:21.535 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:21.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:21.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:21.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:21 smithi144 ceph-mon[33984]: 6.6 scrub starts 2024-03-11T22:11:21.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:21 smithi144 ceph-mon[33984]: 6.6 scrub ok 2024-03-11T22:11:21.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:21 smithi144 ceph-mon[33984]: 3.0 scrub starts 2024-03-11T22:11:21.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:21 smithi144 ceph-mon[33984]: 3.0 scrub ok 2024-03-11T22:11:21.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:21 smithi144 ceph-mon[33984]: pgmap v93: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 569 B/s rd, 50 MiB/s wr, 799 op/s 2024-03-11T22:11:21.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:21 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:11:21.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:21 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:11:21.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:21 smithi144 ceph-mon[33984]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T22:11:21.628 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 38 16 66133 66117 108.713 70.4375 0.00347233 0.00918918 2024-03-11T22:11:21.637 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:21.662 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:21.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:21.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:21.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:21.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:21.807 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:21.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:21.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:21.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:21.951 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:21.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:22.052 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:22.090 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:21 smithi144 systemd[1]: Stopping Ceph prometheus.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:11:22.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:22.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:22.197 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:22.229 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:22.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:22.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:22.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:22.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:22.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:22.474 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-mon[33984]: 4.19 scrub starts 2024-03-11T22:11:22.475 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-mon[33984]: 4.19 scrub ok 2024-03-11T22:11:22.475 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-mon[33984]: 4.9 scrub starts 2024-03-11T22:11:22.475 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-mon[33984]: 4.9 scrub ok 2024-03-11T22:11:22.475 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:11:22.158Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:11:22.475 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:11:22.158Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T22:11:22.475 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:11:22.158Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T22:11:22.475 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:11:22.158Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T22:11:22.475 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:11:22.158Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T22:11:22.475 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:11:22.158Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T22:11:22.475 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:11:22.158Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T22:11:22.476 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:11:22.158Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T22:11:22.476 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:11:22.158Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T22:11:22.476 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:11:22.159Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T22:11:22.476 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:11:22.159Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T22:11:22.476 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[65528]: ts=2024-03-11T22:11:22.159Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T22:11:22.476 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 podman[72648]: 2024-03-11 22:11:22.168946036 +0000 UTC m=+0.159473443 container died 23c49ad0fa001d3e32ddf4328aaa972fcc8dc58290f3f1382ea76376ac108b95 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:11:22.486 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:22.514 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:22.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:22.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:22 smithi045 ceph-mon[39715]: 4.19 scrub starts 2024-03-11T22:11:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:22 smithi045 ceph-mon[39715]: 4.19 scrub ok 2024-03-11T22:11:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:22 smithi045 ceph-mon[39715]: 4.9 scrub starts 2024-03-11T22:11:22.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:22 smithi045 ceph-mon[39715]: 4.9 scrub ok 2024-03-11T22:11:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:22 smithi045 ceph-mon[30178]: 4.19 scrub starts 2024-03-11T22:11:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:22 smithi045 ceph-mon[30178]: 4.19 scrub ok 2024-03-11T22:11:22.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:22 smithi045 ceph-mon[30178]: 4.9 scrub starts 2024-03-11T22:11:22.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:22 smithi045 ceph-mon[30178]: 4.9 scrub ok 2024-03-11T22:11:22.616 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:22.628 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 39 16 67311 67295 107.813 73.625 0.0101712 0.00926958 2024-03-11T22:11:22.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:22.643 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:22.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:22.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:22.783 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 podman[72648]: 2024-03-11 22:11:22.587379729 +0000 UTC m=+0.577907139 container cleanup 23c49ad0fa001d3e32ddf4328aaa972fcc8dc58290f3f1382ea76376ac108b95 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:11:22.783 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 bash[72648]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T22:11:22.785 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:22.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:22.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:22.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:22.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:23.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:23.067 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:23.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:23.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:23.074 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:22 smithi144 podman[72667]: 2024-03-11 22:11:22.789056478 +0000 UTC m=+0.620858975 container remove 23c49ad0fa001d3e32ddf4328aaa972fcc8dc58290f3f1382ea76376ac108b95 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:11:23.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T22:11:23.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:23.216 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:23.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:23.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:23.340 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-11T22:11:23.340 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 systemd[1]: Stopped Ceph prometheus.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:11:23.340 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@prometheus.a.service: Consumed 1.389s CPU time. 2024-03-11T22:11:23.340 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 systemd[1]: Starting Ceph prometheus.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:11:23.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-mon[33984]: 6.8 scrub starts 2024-03-11T22:11:23.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-mon[33984]: 6.8 scrub ok 2024-03-11T22:11:23.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-mon[33984]: 4.11 scrub starts 2024-03-11T22:11:23.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-mon[33984]: 4.11 scrub ok 2024-03-11T22:11:23.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-mon[33984]: 6.1c scrub starts 2024-03-11T22:11:23.340 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-mon[33984]: 6.1c scrub ok 2024-03-11T22:11:23.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-mon[33984]: pgmap v95: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-11T22:11:23.371 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:23.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:23.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:23.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:23.519 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:23.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[30178]: 6.8 scrub starts 2024-03-11T22:11:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[30178]: 6.8 scrub ok 2024-03-11T22:11:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[30178]: 4.11 scrub starts 2024-03-11T22:11:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[30178]: 4.11 scrub ok 2024-03-11T22:11:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[30178]: 6.1c scrub starts 2024-03-11T22:11:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[30178]: 6.1c scrub ok 2024-03-11T22:11:23.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[30178]: pgmap v95: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-11T22:11:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[39715]: 6.8 scrub starts 2024-03-11T22:11:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[39715]: 6.8 scrub ok 2024-03-11T22:11:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[39715]: 4.11 scrub starts 2024-03-11T22:11:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[39715]: 4.11 scrub ok 2024-03-11T22:11:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[39715]: 6.1c scrub starts 2024-03-11T22:11:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[39715]: 6.1c scrub ok 2024-03-11T22:11:23.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-mon[39715]: pgmap v95: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-11T22:11:23.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:23.629 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:11:23.634158+0000 min lat: 0.00145914 max lat: 0.165961 avg lat: 0.00934163 2024-03-11T22:11:23.629 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:11:23.629 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 40 16 68503 68487 106.979 74.5 0.00968267 0.00934163 2024-03-11T22:11:23.654 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 podman[72814]: 2024-03-11 22:11:23.255520621 +0000 UTC m=+0.027303174 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-11T22:11:23.654 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 podman[72814]: 2024-03-11 22:11:23.407380606 +0000 UTC m=+0.179163138 container create 06382fce5ba2561ae5c5da17a2755856508b886dde7c480b11295a0f12eba4f8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:11:23.662 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T22:11:23.678 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:23.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:23.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:23.780 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:23.834 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:23.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:23.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:23.991 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:23.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:23.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:24.064 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:24.090 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 podman[72814]: 2024-03-11 22:11:23.66006482 +0000 UTC m=+0.431847369 container init 06382fce5ba2561ae5c5da17a2755856508b886dde7c480b11295a0f12eba4f8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:11:24.091 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 podman[72814]: 2024-03-11 22:11:23.666742205 +0000 UTC m=+0.438524739 container start 06382fce5ba2561ae5c5da17a2755856508b886dde7c480b11295a0f12eba4f8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:11:24.091 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.702Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T22:11:24.091 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.702Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T22:11:24.091 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.702Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi144 (none))" 2024-03-11T22:11:24.091 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.702Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T22:11:24.091 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.702Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T22:11:24.091 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.704Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T22:11:24.091 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.705Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T22:11:24.091 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.706Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T22:11:24.091 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.706Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T22:11:24.091 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.710Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T22:11:24.091 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.710Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.49µs 2024-03-11T22:11:24.091 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.710Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T22:11:24.092 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.711Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-11T22:11:24.092 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.711Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-11T22:11:24.092 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.711Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=27.769µs wal_replay_duration=785.793µs wbl_replay_duration=98ns total_replay_duration=841.149µs 2024-03-11T22:11:24.092 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.712Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T22:11:24.092 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.712Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T22:11:24.092 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.712Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T22:11:24.092 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.741Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=28.30114ms db_storage=737ns remote_storage=1.215µs web_handler=329ns query_engine=521ns scrape=476.032µs scrape_sd=74.041µs notify=16.18µs notify_sd=9.532µs rules=27.504888ms tracing=8.337µs 2024-03-11T22:11:24.092 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.741Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T22:11:24.092 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:11:23.741Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T22:11:24.092 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 bash[72814]: 06382fce5ba2561ae5c5da17a2755856508b886dde7c480b11295a0f12eba4f8 2024-03-11T22:11:24.092 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:11:23 smithi144 systemd[1]: Started Ceph prometheus.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:11:24.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:24.141 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:23] ENGINE Bus STOPPING 2024-03-11T22:11:24.142 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:11:24.142 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:23] ENGINE Bus STOPPED 2024-03-11T22:11:24.142 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:23 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:23] ENGINE Bus STARTING 2024-03-11T22:11:24.142 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE Serving on http://:::9283 2024-03-11T22:11:24.142 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE Bus STARTED 2024-03-11T22:11:24.142 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE Bus STOPPING 2024-03-11T22:11:24.142 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:11:24.142 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE Bus STOPPED 2024-03-11T22:11:24.142 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE Bus STARTING 2024-03-11T22:11:24.142 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE Serving on http://:::9283 2024-03-11T22:11:24.142 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE Bus STARTED 2024-03-11T22:11:24.142 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE Bus STOPPING 2024-03-11T22:11:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:24.177 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:24.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:24.278 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:24.311 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:24.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:24.409 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: 3.1e scrub starts 2024-03-11T22:11:24.409 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: 3.1e scrub ok 2024-03-11T22:11:24.409 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: 3.9 scrub starts 2024-03-11T22:11:24.409 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: 3.9 scrub ok 2024-03-11T22:11:24.409 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.409 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.409 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:11:24.410 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:11:24.410 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.410 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:11:24.410 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi144.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:11:24.410 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.410 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:11:24.410 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi144.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:11:24.410 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.411 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:11:24.411 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:11:24.411 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE Bus STOPPED 2024-03-11T22:11:24.411 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE Bus STARTING 2024-03-11T22:11:24.411 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE Serving on http://:::9283 2024-03-11T22:11:24.411 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:11:24] ENGINE Bus STARTED 2024-03-11T22:11:24.411 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: 3.1e scrub starts 2024-03-11T22:11:24.411 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: 3.1e scrub ok 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: 3.9 scrub starts 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: 3.9 scrub ok 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi144.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi144.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.412 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:11:24.413 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:24.442 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:24.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:24.543 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:24.575 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:24.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:24.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:24.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: 3.1e scrub starts 2024-03-11T22:11:24.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: 3.1e scrub ok 2024-03-11T22:11:24.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: 3.9 scrub starts 2024-03-11T22:11:24.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: 3.9 scrub ok 2024-03-11T22:11:24.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:11:24.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:11:24.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:11:24.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi144.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:11:24.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:11:24.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi144.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:11:24.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:24.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:11:24.629 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 41 16 69549 69533 105.962 65.375 0.0125405 0.00943123 2024-03-11T22:11:24.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:24.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:24.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:24.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:24.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:24.863 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:24.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:24.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:25.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:25.109 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:25.139 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:25.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:25.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:25.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:25.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:25.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:25.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:25.384 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:25.417 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:25.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:25.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:25.514 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[30178]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:11:25.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[30178]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:11:25.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[30178]: 4.7 scrub starts 2024-03-11T22:11:25.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[30178]: 4.7 scrub ok 2024-03-11T22:11:25.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[30178]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:11:25.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[30178]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi144.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:11:25.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[30178]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:11:25.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[30178]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi144.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:11:25.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[30178]: pgmap v96: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 733 op/s 2024-03-11T22:11:25.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[30178]: 5.4 scrub starts 2024-03-11T22:11:25.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[30178]: 5.4 scrub ok 2024-03-11T22:11:25.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[30178]: 5.2 scrub starts 2024-03-11T22:11:25.515 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[30178]: 5.2 scrub ok 2024-03-11T22:11:25.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[39715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:11:25.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[39715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:11:25.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[39715]: 4.7 scrub starts 2024-03-11T22:11:25.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[39715]: 4.7 scrub ok 2024-03-11T22:11:25.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[39715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:11:25.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[39715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi144.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:11:25.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[39715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:11:25.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[39715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi144.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:11:25.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[39715]: pgmap v96: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 733 op/s 2024-03-11T22:11:25.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[39715]: 5.4 scrub starts 2024-03-11T22:11:25.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[39715]: 5.4 scrub ok 2024-03-11T22:11:25.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[39715]: 5.2 scrub starts 2024-03-11T22:11:25.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:25 smithi045 ceph-mon[39715]: 5.2 scrub ok 2024-03-11T22:11:25.519 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:25.550 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:25.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:25.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-mon[33984]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:11:25.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-mon[33984]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:11:25.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-mon[33984]: 4.7 scrub starts 2024-03-11T22:11:25.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-mon[33984]: 4.7 scrub ok 2024-03-11T22:11:25.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-mon[33984]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:11:25.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-mon[33984]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi144.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:11:25.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-mon[33984]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:11:25.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-mon[33984]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi144.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:11:25.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-mon[33984]: pgmap v96: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 733 op/s 2024-03-11T22:11:25.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-mon[33984]: 5.4 scrub starts 2024-03-11T22:11:25.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-mon[33984]: 5.4 scrub ok 2024-03-11T22:11:25.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-mon[33984]: 5.2 scrub starts 2024-03-11T22:11:25.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-mon[33984]: 5.2 scrub ok 2024-03-11T22:11:25.629 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 42 16 70807 70791 105.311 78.625 0.00778477 0.00948976 2024-03-11T22:11:25.652 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:25.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:25.683 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:25.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:25.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:25.831 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:25.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:25.932 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:25.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:25.969 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:25.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:26.070 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:26.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:11:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:11:26.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:26.133 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:26.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:26.234 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:26.265 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:26.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:26.343 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:26.367 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:26.397 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:26.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:26.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:26.499 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[30178]: 4.4 scrub starts 2024-03-11T22:11:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[30178]: 4.4 scrub ok 2024-03-11T22:11:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[30178]: 3.7 scrub starts 2024-03-11T22:11:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[30178]: 3.7 scrub ok 2024-03-11T22:11:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[30178]: 5.3 scrub starts 2024-03-11T22:11:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[30178]: 5.3 scrub ok 2024-03-11T22:11:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[30178]: 6.12 scrub starts 2024-03-11T22:11:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[30178]: 6.12 scrub ok 2024-03-11T22:11:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[30178]: 5.9 scrub starts 2024-03-11T22:11:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[30178]: 5.9 scrub ok 2024-03-11T22:11:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[39715]: 4.4 scrub starts 2024-03-11T22:11:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[39715]: 4.4 scrub ok 2024-03-11T22:11:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[39715]: 3.7 scrub starts 2024-03-11T22:11:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[39715]: 3.7 scrub ok 2024-03-11T22:11:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[39715]: 5.3 scrub starts 2024-03-11T22:11:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[39715]: 5.3 scrub ok 2024-03-11T22:11:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[39715]: 6.12 scrub starts 2024-03-11T22:11:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[39715]: 6.12 scrub ok 2024-03-11T22:11:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[39715]: 5.9 scrub starts 2024-03-11T22:11:26.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-mon[39715]: 5.9 scrub ok 2024-03-11T22:11:26.540 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:26.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:26.572 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:26 smithi144 ceph-mon[33984]: 4.4 scrub starts 2024-03-11T22:11:26.572 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:26 smithi144 ceph-mon[33984]: 4.4 scrub ok 2024-03-11T22:11:26.572 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:26 smithi144 ceph-mon[33984]: 3.7 scrub starts 2024-03-11T22:11:26.572 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:26 smithi144 ceph-mon[33984]: 3.7 scrub ok 2024-03-11T22:11:26.572 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:26 smithi144 ceph-mon[33984]: 5.3 scrub starts 2024-03-11T22:11:26.572 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:26 smithi144 ceph-mon[33984]: 5.3 scrub ok 2024-03-11T22:11:26.572 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:26 smithi144 ceph-mon[33984]: 6.12 scrub starts 2024-03-11T22:11:26.572 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:26 smithi144 ceph-mon[33984]: 6.12 scrub ok 2024-03-11T22:11:26.572 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:26 smithi144 ceph-mon[33984]: 5.9 scrub starts 2024-03-11T22:11:26.572 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:26 smithi144 ceph-mon[33984]: 5.9 scrub ok 2024-03-11T22:11:26.630 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 43 16 72024 72008 104.631 76.0625 0.0104734 0.00954268 2024-03-11T22:11:26.642 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:26.671 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:26.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:26.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:26.813 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:26.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:26.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:26.915 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:27.000 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:27.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:27.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:27.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:27.145 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:27.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:27.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:27.248 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:27.270 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:11:26 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[86801]: ts=2024-03-11T22:11:26.980Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.007860176s 2024-03-11T22:11:27.278 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:27.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:27.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:27.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:27.419 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:27.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:27.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:27.521 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:27.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:27 smithi045 ceph-mon[30178]: 6.4 scrub starts 2024-03-11T22:11:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:27 smithi045 ceph-mon[30178]: 6.4 scrub ok 2024-03-11T22:11:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:27 smithi045 ceph-mon[30178]: 4.10 scrub starts 2024-03-11T22:11:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:27 smithi045 ceph-mon[30178]: 4.10 scrub ok 2024-03-11T22:11:27.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:27 smithi045 ceph-mon[30178]: pgmap v97: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T22:11:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:27 smithi045 ceph-mon[39715]: 6.4 scrub starts 2024-03-11T22:11:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:27 smithi045 ceph-mon[39715]: 6.4 scrub ok 2024-03-11T22:11:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:27 smithi045 ceph-mon[39715]: 4.10 scrub starts 2024-03-11T22:11:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:27 smithi045 ceph-mon[39715]: 4.10 scrub ok 2024-03-11T22:11:27.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:27 smithi045 ceph-mon[39715]: pgmap v97: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T22:11:27.550 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:27.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:27.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:27 smithi144 ceph-mon[33984]: 6.4 scrub starts 2024-03-11T22:11:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:27 smithi144 ceph-mon[33984]: 6.4 scrub ok 2024-03-11T22:11:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:27 smithi144 ceph-mon[33984]: 4.10 scrub starts 2024-03-11T22:11:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:27 smithi144 ceph-mon[33984]: 4.10 scrub ok 2024-03-11T22:11:27.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:27 smithi144 ceph-mon[33984]: pgmap v97: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T22:11:27.630 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 44 16 73197 73181 103.919 73.3125 0.0214141 0.00961732 2024-03-11T22:11:27.652 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:27.665 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:27.679 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:27.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:27.781 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:27.817 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:27.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:27.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:27.919 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:27.945 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:27.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:28.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:28.088 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:28.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:28.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:28.234 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:28.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:28.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:28.337 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:28.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:28 smithi144 ceph-mon[33984]: 4.12 scrub starts 2024-03-11T22:11:28.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:28 smithi144 ceph-mon[33984]: 4.12 scrub ok 2024-03-11T22:11:28.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:28 smithi144 ceph-mon[33984]: 5.1c scrub starts 2024-03-11T22:11:28.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:28 smithi144 ceph-mon[33984]: 5.1c scrub ok 2024-03-11T22:11:28.365 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:28.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:28.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:28.467 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:28.497 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:28.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:28 smithi045 ceph-mon[30178]: 4.12 scrub starts 2024-03-11T22:11:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:28 smithi045 ceph-mon[30178]: 4.12 scrub ok 2024-03-11T22:11:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:28 smithi045 ceph-mon[30178]: 5.1c scrub starts 2024-03-11T22:11:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:28 smithi045 ceph-mon[30178]: 5.1c scrub ok 2024-03-11T22:11:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:28 smithi045 ceph-mon[39715]: 4.12 scrub starts 2024-03-11T22:11:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:28 smithi045 ceph-mon[39715]: 4.12 scrub ok 2024-03-11T22:11:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:28 smithi045 ceph-mon[39715]: 5.1c scrub starts 2024-03-11T22:11:28.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:28 smithi045 ceph-mon[39715]: 5.1c scrub ok 2024-03-11T22:11:28.599 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:28.630 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 45 16 74363 74347 103.229 72.875 0.0081059 0.00968211 2024-03-11T22:11:28.637 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:28.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:28.741 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:28.778 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:28.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:28.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:28.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:28.922 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:28.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:29.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:29.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:29.072 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:29.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:29.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:29.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:29.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:29.331 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:29.361 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:29.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:29.461 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:29 smithi045 ceph-mon[30178]: pgmap v98: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T22:11:29.461 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:29 smithi045 ceph-mon[30178]: 5.b scrub starts 2024-03-11T22:11:29.461 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:29 smithi045 ceph-mon[30178]: 5.b scrub ok 2024-03-11T22:11:29.461 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:29 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:29.461 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:29 smithi045 ceph-mon[39715]: pgmap v98: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T22:11:29.462 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:29 smithi045 ceph-mon[39715]: 5.b scrub starts 2024-03-11T22:11:29.462 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:29 smithi045 ceph-mon[39715]: 5.b scrub ok 2024-03-11T22:11:29.462 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:29 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:29.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:29.509 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:29.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:29.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:29.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:29 smithi144 ceph-mon[33984]: pgmap v98: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T22:11:29.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:29 smithi144 ceph-mon[33984]: 5.b scrub starts 2024-03-11T22:11:29.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:29 smithi144 ceph-mon[33984]: 5.b scrub ok 2024-03-11T22:11:29.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:29 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:29.610 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:29.630 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 46 16 75543 75527 102.588 73.75 0.0219448 0.0097421 2024-03-11T22:11:29.639 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:29.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:29.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:29.741 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:29.768 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:29.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:29.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:29.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:29.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:29.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:30.014 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:30.043 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:30.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:30.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:30.141 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:11:30.142 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:11:30.142 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:11:30.142 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:11:30.142 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:11:30.142 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:11:30.142 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:11:30.142 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:11:30.145 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:30.145 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:11:30.145 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:11:30.146 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:11:30.146 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:11:30.146 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:11:30.146 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:11:30.146 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:11:30.146 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:11:30.174 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:30.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:30.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:30.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:30.318 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:30.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:30.419 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:30.449 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:30.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:30.511 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[39715]: 6.1 scrub starts 2024-03-11T22:11:30.511 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[39715]: 6.1 scrub ok 2024-03-11T22:11:30.511 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:11:30.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:11:30.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[39715]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T22:11:30.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2802256381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:11:30.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:30.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:30.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[30178]: 6.1 scrub starts 2024-03-11T22:11:30.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[30178]: 6.1 scrub ok 2024-03-11T22:11:30.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:11:30.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:11:30.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[30178]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T22:11:30.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2802256381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:11:30.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:30.513 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:30.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:30.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:30 smithi144 ceph-mon[33984]: 6.1 scrub starts 2024-03-11T22:11:30.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:30 smithi144 ceph-mon[33984]: 6.1 scrub ok 2024-03-11T22:11:30.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:30 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:11:30.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:30 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:11:30.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:30 smithi144 ceph-mon[33984]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T22:11:30.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:30 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2802256381' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:11:30.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:30.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:30.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:30.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:30.630 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 47 16 76679 76663 101.916 71 0.0124122 0.00980661 2024-03-11T22:11:30.695 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:30.718 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:30.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:30.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:30.819 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:30.849 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:30.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:30.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:30.950 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:30.978 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:30.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:30.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:31.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:31.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:31.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:31.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:31.273 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:31.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:31.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:31.375 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:31.407 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:31.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:31.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:31.546 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:31.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:31.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:31.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:31 smithi144 ceph-mon[33984]: pgmap v100: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 520 B/s rd, 37 MiB/s wr, 589 op/s 2024-03-11T22:11:31.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:31 smithi144 ceph-mon[33984]: 5.15 scrub starts 2024-03-11T22:11:31.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:31 smithi144 ceph-mon[33984]: 5.15 scrub ok 2024-03-11T22:11:31.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:31 smithi144 ceph-mon[33984]: 6.1a scrub starts 2024-03-11T22:11:31.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:31 smithi144 ceph-mon[33984]: 6.1a scrub ok 2024-03-11T22:11:31.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:31 smithi144 ceph-mon[33984]: 4.5 scrub starts 2024-03-11T22:11:31.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:31 smithi144 ceph-mon[33984]: 4.5 scrub ok 2024-03-11T22:11:31.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:31 smithi144 ceph-mon[33984]: osdmap e116: 8 total, 8 up, 8 in 2024-03-11T22:11:31.630 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 48 16 77794 77778 101.245 69.6875 0.0121009 0.00987194 2024-03-11T22:11:31.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:31.691 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:31.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:31.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[39715]: pgmap v100: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 520 B/s rd, 37 MiB/s wr, 589 op/s 2024-03-11T22:11:31.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[39715]: 5.15 scrub starts 2024-03-11T22:11:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[39715]: 5.15 scrub ok 2024-03-11T22:11:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[39715]: 6.1a scrub starts 2024-03-11T22:11:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[39715]: 6.1a scrub ok 2024-03-11T22:11:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[39715]: 4.5 scrub starts 2024-03-11T22:11:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[39715]: 4.5 scrub ok 2024-03-11T22:11:31.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[39715]: osdmap e116: 8 total, 8 up, 8 in 2024-03-11T22:11:31.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[30178]: pgmap v100: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 520 B/s rd, 37 MiB/s wr, 589 op/s 2024-03-11T22:11:31.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[30178]: 5.15 scrub starts 2024-03-11T22:11:31.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[30178]: 5.15 scrub ok 2024-03-11T22:11:31.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[30178]: 6.1a scrub starts 2024-03-11T22:11:31.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[30178]: 6.1a scrub ok 2024-03-11T22:11:31.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[30178]: 4.5 scrub starts 2024-03-11T22:11:31.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[30178]: 4.5 scrub ok 2024-03-11T22:11:31.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:31 smithi045 ceph-mon[30178]: osdmap e116: 8 total, 8 up, 8 in 2024-03-11T22:11:31.792 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:31.824 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:31.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:31.926 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:31.956 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:31.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:32.057 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:32.085 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:32.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:32.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:32.187 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:32.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:32.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:32.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:32.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:32.379 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:32.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:32.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[30178]: 6.1d scrub starts 2024-03-11T22:11:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[30178]: 6.1d scrub ok 2024-03-11T22:11:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[30178]: 5.17 scrub starts 2024-03-11T22:11:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[30178]: 5.17 scrub ok 2024-03-11T22:11:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[30178]: 6.17 scrub starts 2024-03-11T22:11:32.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[30178]: 6.17 scrub ok 2024-03-11T22:11:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[30178]: 3.10 scrub starts 2024-03-11T22:11:32.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[30178]: 3.10 scrub ok 2024-03-11T22:11:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[39715]: 6.1d scrub starts 2024-03-11T22:11:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[39715]: 6.1d scrub ok 2024-03-11T22:11:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[39715]: 5.17 scrub starts 2024-03-11T22:11:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[39715]: 5.17 scrub ok 2024-03-11T22:11:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[39715]: 6.17 scrub starts 2024-03-11T22:11:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[39715]: 6.17 scrub ok 2024-03-11T22:11:32.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[39715]: 3.10 scrub starts 2024-03-11T22:11:32.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-mon[39715]: 3.10 scrub ok 2024-03-11T22:11:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:32.530 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:32.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:32.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:32 smithi144 ceph-mon[33984]: 6.1d scrub starts 2024-03-11T22:11:32.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:32 smithi144 ceph-mon[33984]: 6.1d scrub ok 2024-03-11T22:11:32.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:32 smithi144 ceph-mon[33984]: 5.17 scrub starts 2024-03-11T22:11:32.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:32 smithi144 ceph-mon[33984]: 5.17 scrub ok 2024-03-11T22:11:32.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:32 smithi144 ceph-mon[33984]: 6.17 scrub starts 2024-03-11T22:11:32.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:32 smithi144 ceph-mon[33984]: 6.17 scrub ok 2024-03-11T22:11:32.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:32 smithi144 ceph-mon[33984]: 3.10 scrub starts 2024-03-11T22:11:32.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:32 smithi144 ceph-mon[33984]: 3.10 scrub ok 2024-03-11T22:11:32.630 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 49 16 78886 78870 100.571 68.25 0.00888065 0.00993832 2024-03-11T22:11:32.632 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:32.663 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:32.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:32.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:32.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:32.821 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:32.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:32.912 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:11:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:11:32.922 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:32.951 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:32.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:33.053 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:33.082 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:33.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:33.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:33.132 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:11:33.132 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:11:33.132 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-03-11T22:11:33.132 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-03-11T22:11:33.132 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:11:33.184 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:33.214 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:33.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:33.316 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:33.345 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:33.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:33.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:33.346 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:11:33.447 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:33.477 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:33.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:33.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[39715]: pgmap v102: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-11T22:11:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[39715]: 3.16 scrub starts 2024-03-11T22:11:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[39715]: 3.16 scrub ok 2024-03-11T22:11:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[39715]: 6.f scrub starts 2024-03-11T22:11:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[39715]: 6.f scrub ok 2024-03-11T22:11:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4109270346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:11:33.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:11:33.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[30178]: pgmap v102: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-11T22:11:33.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[30178]: 3.16 scrub starts 2024-03-11T22:11:33.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[30178]: 3.16 scrub ok 2024-03-11T22:11:33.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[30178]: 6.f scrub starts 2024-03-11T22:11:33.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[30178]: 6.f scrub ok 2024-03-11T22:11:33.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4109270346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:11:33.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:33 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:11:33.579 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:33.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:33 smithi144 ceph-mon[33984]: pgmap v102: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-11T22:11:33.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:33 smithi144 ceph-mon[33984]: 3.16 scrub starts 2024-03-11T22:11:33.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:33 smithi144 ceph-mon[33984]: 3.16 scrub ok 2024-03-11T22:11:33.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:33 smithi144 ceph-mon[33984]: 6.f scrub starts 2024-03-11T22:11:33.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:33 smithi144 ceph-mon[33984]: 6.f scrub ok 2024-03-11T22:11:33.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:33 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4109270346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:11:33.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:33 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:11:33.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:33.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:33.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:33.630 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 50 16 79962 79946 99.9048 67.25 0.0249388 0.0100041 2024-03-11T22:11:33.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:33.769 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:33.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:33.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:33.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:11:33.873 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:33.902 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:33.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:33.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:34.005 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:34.032 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:34.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:34.134 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:34.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:34.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:34.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:34.334 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:34.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:34.436 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:34.484 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:34.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:34.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:34.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:34.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:34 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:11:34.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:34 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:11:34.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:34 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:11:34.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:34 smithi144 ceph-mon[33984]: osdmap e117: 8 total, 8 up, 8 in 2024-03-11T22:11:34.630 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:34.631 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 51 16 80981 80965 99.1947 63.6875 0.0137448 0.0100763 2024-03-11T22:11:34.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:34.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:34.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:34 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:11:34.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:34 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:11:34.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:34 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:11:34.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:34 smithi045 ceph-mon[39715]: osdmap e117: 8 total, 8 up, 8 in 2024-03-11T22:11:34.687 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:34 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:11:34.687 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:34 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:11:34.688 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:34 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:11:34.688 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:34 smithi045 ceph-mon[30178]: osdmap e117: 8 total, 8 up, 8 in 2024-03-11T22:11:34.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:34.769 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:34.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:34.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:34.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:34.917 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:34.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:35.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:35.060 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:35.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:35.162 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:35.192 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:35.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:35.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T22:11:35.294 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:35.323 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:35.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:35.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:35.374 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T22:11:35.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:35.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:35.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:35.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[30178]: 3.2 deep-scrub starts 2024-03-11T22:11:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[30178]: 3.2 deep-scrub ok 2024-03-11T22:11:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[30178]: pgmap v104: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 988 op/s 2024-03-11T22:11:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[30178]: 1.0 scrub starts 2024-03-11T22:11:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[30178]: 1.0 scrub ok 2024-03-11T22:11:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[30178]: 6.3 scrub starts 2024-03-11T22:11:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[30178]: 6.3 scrub ok 2024-03-11T22:11:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3532741226' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:11:35.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:11:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[39715]: 3.2 deep-scrub starts 2024-03-11T22:11:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[39715]: 3.2 deep-scrub ok 2024-03-11T22:11:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[39715]: pgmap v104: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 988 op/s 2024-03-11T22:11:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[39715]: 1.0 scrub starts 2024-03-11T22:11:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[39715]: 1.0 scrub ok 2024-03-11T22:11:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[39715]: 6.3 scrub starts 2024-03-11T22:11:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[39715]: 6.3 scrub ok 2024-03-11T22:11:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3532741226' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:11:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:35 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:11:35.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:35.631 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 52 16 82032 82016 98.5504 65.6875 0.0258613 0.0101419 2024-03-11T22:11:35.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:35.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:35.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:35.741 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:35.744 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:35 smithi144 ceph-mon[33984]: 3.2 deep-scrub starts 2024-03-11T22:11:35.744 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:35 smithi144 ceph-mon[33984]: 3.2 deep-scrub ok 2024-03-11T22:11:35.744 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:35 smithi144 ceph-mon[33984]: pgmap v104: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 988 op/s 2024-03-11T22:11:35.744 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:35 smithi144 ceph-mon[33984]: 1.0 scrub starts 2024-03-11T22:11:35.745 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:35 smithi144 ceph-mon[33984]: 1.0 scrub ok 2024-03-11T22:11:35.745 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:35 smithi144 ceph-mon[33984]: 6.3 scrub starts 2024-03-11T22:11:35.745 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:35 smithi144 ceph-mon[33984]: 6.3 scrub ok 2024-03-11T22:11:35.745 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:35 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3532741226' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:11:35.745 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:35 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:11:35.778 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:35.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:35.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:35.880 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:35.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:35.907 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:35.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:36.008 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:36.042 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:36.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:36.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:36.083 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:11:35 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:11:36.144 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:36.182 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:36.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:36.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:36.337 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:36.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:36.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:36.439 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:36.472 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:36.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:36.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:11:36.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:36.627 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:36.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:36.632 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 53 16 83010 82994 97.8442 61.125 0.0138279 0.0102143 2024-03-11T22:11:36.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[39715]: 2.0 scrub starts 2024-03-11T22:11:36.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[39715]: 2.0 scrub ok 2024-03-11T22:11:36.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[39715]: 6.2 scrub starts 2024-03-11T22:11:36.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[39715]: 6.2 scrub ok 2024-03-11T22:11:36.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:11:36.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[39715]: osdmap e118: 8 total, 8 up, 8 in 2024-03-11T22:11:36.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[39715]: 4.16 scrub starts 2024-03-11T22:11:36.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[39715]: 4.16 scrub ok 2024-03-11T22:11:36.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[39715]: 6.5 scrub starts 2024-03-11T22:11:36.645 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[39715]: 6.5 scrub ok 2024-03-11T22:11:36.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[30178]: 2.0 scrub starts 2024-03-11T22:11:36.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[30178]: 2.0 scrub ok 2024-03-11T22:11:36.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[30178]: 6.2 scrub starts 2024-03-11T22:11:36.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[30178]: 6.2 scrub ok 2024-03-11T22:11:36.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:11:36.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[30178]: osdmap e118: 8 total, 8 up, 8 in 2024-03-11T22:11:36.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[30178]: 4.16 scrub starts 2024-03-11T22:11:36.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[30178]: 4.16 scrub ok 2024-03-11T22:11:36.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[30178]: 6.5 scrub starts 2024-03-11T22:11:36.646 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:36 smithi045 ceph-mon[30178]: 6.5 scrub ok 2024-03-11T22:11:36.728 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:36.755 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:36.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:36.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:36.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:36 smithi144 ceph-mon[33984]: 2.0 scrub starts 2024-03-11T22:11:36.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:36 smithi144 ceph-mon[33984]: 2.0 scrub ok 2024-03-11T22:11:36.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:36 smithi144 ceph-mon[33984]: 6.2 scrub starts 2024-03-11T22:11:36.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:36 smithi144 ceph-mon[33984]: 6.2 scrub ok 2024-03-11T22:11:36.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:36 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:11:36.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:36 smithi144 ceph-mon[33984]: osdmap e118: 8 total, 8 up, 8 in 2024-03-11T22:11:36.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:36 smithi144 ceph-mon[33984]: 4.16 scrub starts 2024-03-11T22:11:36.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:36 smithi144 ceph-mon[33984]: 4.16 scrub ok 2024-03-11T22:11:36.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:36 smithi144 ceph-mon[33984]: 6.5 scrub starts 2024-03-11T22:11:36.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:36 smithi144 ceph-mon[33984]: 6.5 scrub ok 2024-03-11T22:11:36.858 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:36.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:36.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:36.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:37.006 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:37.043 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:37.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:37.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:37.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T22:11:37.156 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:37.160 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:37.223 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:37.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:37.324 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:37.355 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:37.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:37.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:37.457 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:37.486 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:37.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:37.588 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:37.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:37.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:37 smithi045 ceph-mon[30178]: pgmap v106: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-11T22:11:37.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:37 smithi045 ceph-mon[30178]: 5.a deep-scrub starts 2024-03-11T22:11:37.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:37 smithi045 ceph-mon[30178]: 5.a deep-scrub ok 2024-03-11T22:11:37.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:37 smithi045 ceph-mon[30178]: 6.15 scrub starts 2024-03-11T22:11:37.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:37 smithi045 ceph-mon[30178]: 6.15 scrub ok 2024-03-11T22:11:37.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:37.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:37 smithi045 ceph-mon[39715]: pgmap v106: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-11T22:11:37.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:37 smithi045 ceph-mon[39715]: 5.a deep-scrub starts 2024-03-11T22:11:37.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:37 smithi045 ceph-mon[39715]: 5.a deep-scrub ok 2024-03-11T22:11:37.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:37 smithi045 ceph-mon[39715]: 6.15 scrub starts 2024-03-11T22:11:37.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:37 smithi045 ceph-mon[39715]: 6.15 scrub ok 2024-03-11T22:11:37.631 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 54 16 84012 83996 97.1915 62.625 0.0123322 0.0102833 2024-03-11T22:11:37.637 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:37.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:37.739 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:37.771 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:37.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:37.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:37.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:37 smithi144 ceph-mon[33984]: pgmap v106: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-11T22:11:37.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:37 smithi144 ceph-mon[33984]: 5.a deep-scrub starts 2024-03-11T22:11:37.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:37 smithi144 ceph-mon[33984]: 5.a deep-scrub ok 2024-03-11T22:11:37.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:37 smithi144 ceph-mon[33984]: 6.15 scrub starts 2024-03-11T22:11:37.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:37 smithi144 ceph-mon[33984]: 6.15 scrub ok 2024-03-11T22:11:37.873 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:37.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:37.923 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:37.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:38.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:38.065 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:38.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:38.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:38.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:38.210 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:38.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:38.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:38.312 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:38.344 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:38.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:38.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:38.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:38.485 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:38.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:38.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:38 smithi045 ceph-mon[30178]: 5.13 deep-scrub starts 2024-03-11T22:11:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:38 smithi045 ceph-mon[30178]: 5.13 deep-scrub ok 2024-03-11T22:11:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:38 smithi045 ceph-mon[30178]: 5.1e scrub starts 2024-03-11T22:11:38.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:38 smithi045 ceph-mon[30178]: 5.1e scrub ok 2024-03-11T22:11:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:38 smithi045 ceph-mon[39715]: 5.13 deep-scrub starts 2024-03-11T22:11:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:38 smithi045 ceph-mon[39715]: 5.13 deep-scrub ok 2024-03-11T22:11:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:38 smithi045 ceph-mon[39715]: 5.1e scrub starts 2024-03-11T22:11:38.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:38 smithi045 ceph-mon[39715]: 5.1e scrub ok 2024-03-11T22:11:38.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:38.631 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:38.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:38.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:38.632 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 55 16 85091 85075 96.65 67.4375 0.0244219 0.0103417 2024-03-11T22:11:38.733 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:38.764 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:38.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:38.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:38.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:38 smithi144 ceph-mon[33984]: 5.13 deep-scrub starts 2024-03-11T22:11:38.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:38 smithi144 ceph-mon[33984]: 5.13 deep-scrub ok 2024-03-11T22:11:38.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:38 smithi144 ceph-mon[33984]: 5.1e scrub starts 2024-03-11T22:11:38.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:38 smithi144 ceph-mon[33984]: 5.1e scrub ok 2024-03-11T22:11:38.867 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:38.894 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:38.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:38.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:39.033 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:39.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:39.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:39.135 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:39.163 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:39.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:39.219 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:39.265 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:39.293 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:39.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:39.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:39.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:39.444 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:39.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:39.546 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:39.574 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:39.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:39.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:39.632 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 56 16 86221 86205 96.1852 70.625 0.00958705 0.0103919 2024-03-11T22:11:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:39 smithi045 ceph-mon[30178]: 3.15 scrub starts 2024-03-11T22:11:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:39 smithi045 ceph-mon[30178]: 3.15 scrub ok 2024-03-11T22:11:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:39 smithi045 ceph-mon[30178]: 4.15 scrub starts 2024-03-11T22:11:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:39 smithi045 ceph-mon[30178]: 4.15 scrub ok 2024-03-11T22:11:39.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:39 smithi045 ceph-mon[30178]: pgmap v107: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 731 B/s rd, 49 MiB/s wr, 778 op/s 2024-03-11T22:11:39.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:39 smithi045 ceph-mon[39715]: 3.15 scrub starts 2024-03-11T22:11:39.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:39 smithi045 ceph-mon[39715]: 3.15 scrub ok 2024-03-11T22:11:39.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:39 smithi045 ceph-mon[39715]: 4.15 scrub starts 2024-03-11T22:11:39.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:39 smithi045 ceph-mon[39715]: 4.15 scrub ok 2024-03-11T22:11:39.666 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:39 smithi045 ceph-mon[39715]: pgmap v107: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 731 B/s rd, 49 MiB/s wr, 778 op/s 2024-03-11T22:11:39.675 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:39.702 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:39.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:39.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:39.804 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:39.833 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:39.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:39.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:39 smithi144 ceph-mon[33984]: 3.15 scrub starts 2024-03-11T22:11:39.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:39 smithi144 ceph-mon[33984]: 3.15 scrub ok 2024-03-11T22:11:39.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:39 smithi144 ceph-mon[33984]: 4.15 scrub starts 2024-03-11T22:11:39.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:39 smithi144 ceph-mon[33984]: 4.15 scrub ok 2024-03-11T22:11:39.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:39 smithi144 ceph-mon[33984]: pgmap v107: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 731 B/s rd, 49 MiB/s wr, 778 op/s 2024-03-11T22:11:39.935 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:39.966 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:39.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:40.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:40.125 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:40.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:40.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:40.290 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:40.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:40.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:40.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:40.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:40.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:40.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:40.548 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:40.579 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:40.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:40.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:40.632 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 57 16 87289 87273 95.6688 66.75 0.0125416 0.0104476 2024-03-11T22:11:40.680 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:40.710 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:40.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:40.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:40 smithi045 ceph-mon[30178]: 3.13 scrub starts 2024-03-11T22:11:40.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:40 smithi045 ceph-mon[30178]: 3.13 scrub ok 2024-03-11T22:11:40.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:40 smithi045 ceph-mon[39715]: 3.13 scrub starts 2024-03-11T22:11:40.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:40 smithi045 ceph-mon[39715]: 3.13 scrub ok 2024-03-11T22:11:40.812 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:40.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:40 smithi144 ceph-mon[33984]: 3.13 scrub starts 2024-03-11T22:11:40.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:40 smithi144 ceph-mon[33984]: 3.13 scrub ok 2024-03-11T22:11:40.842 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:40.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:40.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:40.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:40.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:40.988 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:40.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:41.089 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:41.117 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:41.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:41.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:41.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:41.267 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:41.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:41.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:41.411 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:41.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:41.513 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:41.541 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:41.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:41.632 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 58 16 88311 88295 95.1206 63.875 0.00758597 0.0105077 2024-03-11T22:11:41.642 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:41.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:41.673 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:41.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:41.774 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:41 smithi045 ceph-mon[30178]: 3.b scrub starts 2024-03-11T22:11:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:41 smithi045 ceph-mon[30178]: pgmap v108: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 719 op/s 2024-03-11T22:11:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:41 smithi045 ceph-mon[30178]: 3.b scrub ok 2024-03-11T22:11:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:41 smithi045 ceph-mon[30178]: 5.1 scrub starts 2024-03-11T22:11:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:41 smithi045 ceph-mon[30178]: 5.1 scrub ok 2024-03-11T22:11:41.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:41 smithi045 ceph-mon[30178]: 7.5 scrub starts 2024-03-11T22:11:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:41 smithi045 ceph-mon[39715]: 3.b scrub starts 2024-03-11T22:11:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:41 smithi045 ceph-mon[39715]: pgmap v108: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 719 op/s 2024-03-11T22:11:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:41 smithi045 ceph-mon[39715]: 3.b scrub ok 2024-03-11T22:11:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:41 smithi045 ceph-mon[39715]: 5.1 scrub starts 2024-03-11T22:11:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:41 smithi045 ceph-mon[39715]: 5.1 scrub ok 2024-03-11T22:11:41.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:41 smithi045 ceph-mon[39715]: 7.5 scrub starts 2024-03-11T22:11:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:41.806 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:41.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:41.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:41 smithi144 ceph-mon[33984]: 3.b scrub starts 2024-03-11T22:11:41.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:41 smithi144 ceph-mon[33984]: pgmap v108: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 719 op/s 2024-03-11T22:11:41.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:41 smithi144 ceph-mon[33984]: 3.b scrub ok 2024-03-11T22:11:41.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:41 smithi144 ceph-mon[33984]: 5.1 scrub starts 2024-03-11T22:11:41.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:41 smithi144 ceph-mon[33984]: 5.1 scrub ok 2024-03-11T22:11:41.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:41 smithi144 ceph-mon[33984]: 7.5 scrub starts 2024-03-11T22:11:41.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:41.954 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:41.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:42.055 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:42.085 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:42.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:42.187 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:42.217 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:42.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:42.319 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:42.353 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:42.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:42.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:42.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:42.496 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:42.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:42.598 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:42.629 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:42.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:42.632 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 59 16 89401 89385 94.663 68.125 0.00835696 0.0105589 2024-03-11T22:11:42.730 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:42.759 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:42.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:42.772 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:11:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:11:42.773 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:42 smithi045 ceph-mon[39715]: 5.16 scrub starts 2024-03-11T22:11:42.773 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:42 smithi045 ceph-mon[39715]: 5.16 scrub ok 2024-03-11T22:11:42.773 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:42 smithi045 ceph-mon[30178]: 5.16 scrub starts 2024-03-11T22:11:42.773 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:42 smithi045 ceph-mon[30178]: 5.16 scrub ok 2024-03-11T22:11:42.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:42 smithi144 ceph-mon[33984]: 5.16 scrub starts 2024-03-11T22:11:42.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:42 smithi144 ceph-mon[33984]: 5.16 scrub ok 2024-03-11T22:11:42.861 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:42.893 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:42.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:42.989 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:42.994 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:43.021 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:43.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:43.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:43.180 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:43.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:43.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:43.332 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:43.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:43.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:43.434 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:43.462 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:43.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:43.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:43.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:43.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:43.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:43.632 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:11:43.637705+0000 min lat: 0.00145914 max lat: 0.165961 avg lat: 0.0106198 2024-03-11T22:11:43.632 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:11:43.632 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 60 16 90392 90376 94.1176 61.9375 0.0128044 0.0106198 2024-03-11T22:11:43.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:43.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[39715]: 4.f scrub starts 2024-03-11T22:11:43.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[39715]: 4.f scrub ok 2024-03-11T22:11:43.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[39715]: pgmap v109: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T22:11:43.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[39715]: 2.2 scrub starts 2024-03-11T22:11:43.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[39715]: 2.2 scrub ok 2024-03-11T22:11:43.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[39715]: 5.10 deep-scrub starts 2024-03-11T22:11:43.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[39715]: 5.10 deep-scrub ok 2024-03-11T22:11:43.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[39715]: 7.5 scrub ok 2024-03-11T22:11:43.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[30178]: 4.f scrub starts 2024-03-11T22:11:43.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[30178]: 4.f scrub ok 2024-03-11T22:11:43.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[30178]: pgmap v109: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T22:11:43.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[30178]: 2.2 scrub starts 2024-03-11T22:11:43.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[30178]: 2.2 scrub ok 2024-03-11T22:11:43.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[30178]: 5.10 deep-scrub starts 2024-03-11T22:11:43.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[30178]: 5.10 deep-scrub ok 2024-03-11T22:11:43.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:43 smithi045 ceph-mon[30178]: 7.5 scrub ok 2024-03-11T22:11:43.764 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:43.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:43.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:43.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:43 smithi144 ceph-mon[33984]: 4.f scrub starts 2024-03-11T22:11:43.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:43 smithi144 ceph-mon[33984]: 4.f scrub ok 2024-03-11T22:11:43.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:43 smithi144 ceph-mon[33984]: pgmap v109: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T22:11:43.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:43 smithi144 ceph-mon[33984]: 2.2 scrub starts 2024-03-11T22:11:43.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:43 smithi144 ceph-mon[33984]: 2.2 scrub ok 2024-03-11T22:11:43.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:43 smithi144 ceph-mon[33984]: 5.10 deep-scrub starts 2024-03-11T22:11:43.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:43 smithi144 ceph-mon[33984]: 5.10 deep-scrub ok 2024-03-11T22:11:43.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:43 smithi144 ceph-mon[33984]: 7.5 scrub ok 2024-03-11T22:11:43.861 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:11:43.861 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-03-11T22:11:43.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-03-11T22:11:43.861 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:11:43.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.861 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-03-11T22:11:43.861 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:06:14.230307+0000", 2024-03-11T22:11:43.861 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 329.63185723200002, 2024-03-11T22:11:43.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.86020543199999999, 2024-03-11T22:11:43.861 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.861 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.861 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14158", 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3509006862", 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 1 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230307+0000", 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230307+0000", 2024-03-11T22:11:43.862 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230309+0000", 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.714e-06 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230315+0000", 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.353e-06 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230326+0000", 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0332e-05 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230336+0000", 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0094e-05 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230394+0000", 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.842e-05 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-03-11T22:11:43.864 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230403+0000", 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4230000000000005e-06 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.089270+0000", 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.85886745099999995 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.089427+0000", 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000157069 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,2", 2024-03-11T22:11:43.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.089631+0000", 2024-03-11T22:11:43.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000203649 2024-03-11T22:11:43.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.090353+0000", 2024-03-11T22:11:43.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00072196000000000001 2024-03-11T22:11:43.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.090390+0000", 2024-03-11T22:11:43.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7076999999999998e-05 2024-03-11T22:11:43.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.090410+0000", 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0588e-05 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.090463+0000", 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2340999999999999e-05 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.090513+0000", 2024-03-11T22:11:43.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9960999999999998e-05 2024-03-11T22:11:43.868 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.868 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.868 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.868 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24326.0:4 3.0 3:00000000::::head [pgnls start_epoch 68] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e68)", 2024-03-11T22:11:43.868 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:08:33.043301+0000", 2024-03-11T22:11:43.868 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 190.818863642, 2024-03-11T22:11:43.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25786676200000003, 2024-03-11T22:11:43.868 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.868 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:11:43.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24326", 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2228576527", 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 4 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043301+0000", 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043301+0000", 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043302+0000", 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1209999999999999e-06 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043306+0000", 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3769999999999998e-06 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043314+0000", 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0700000000000007e-06 2024-03-11T22:11:43.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043324+0000", 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5529999999999998e-06 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043371+0000", 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7068999999999999e-05 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043380+0000", 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5229999999999995e-06 2024-03-11T22:11:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.300840+0000", 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.257460668 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.300879+0000", 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8714000000000002e-05 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.301168+0000", 2024-03-11T22:11:43.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028866699999999999 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24403.0:109 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e98)", 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:34.741322+0000", 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 69.120842155999995, 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.67899541299999999, 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24403", 2024-03-11T22:11:43.873 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.144:0/840743586", 2024-03-11T22:11:43.874 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 109 2024-03-11T22:11:43.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.874 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.874 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:43.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741322+0000", 2024-03-11T22:11:43.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741322+0000", 2024-03-11T22:11:43.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741324+0000", 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6279999999999999e-06 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741331+0000", 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8449999999999997e-06 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741340+0000", 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6190000000000006e-06 2024-03-11T22:11:43.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741351+0000", 2024-03-11T22:11:43.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0372e-05 2024-03-11T22:11:43.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741431+0000", 2024-03-11T22:11:43.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0368e-05 2024-03-11T22:11:43.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-03-11T22:11:43.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741444+0000", 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2971e-05 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:35.420168+0000", 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.67872427000000002 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:35.420267+0000", 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8834999999999998e-05 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:35.420318+0000", 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0504999999999998e-05 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:8690 7.b 7:d2b82346:::benchmark_data_smithi045_83007_object8689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:46.691231+0000", 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 57.170933157, 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.043376505000000003, 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.879 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:43.880 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:43.880 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 8690 2024-03-11T22:11:43.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.880 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.691231+0000", 2024-03-11T22:11:43.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.691231+0000", 2024-03-11T22:11:43.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.691233+0000", 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7239999999999999e-06 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.691259+0000", 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6387e-05 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.691277+0000", 2024-03-11T22:11:43.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7672e-05 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.691287+0000", 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5889999999999992e-06 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.692051+0000", 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00076423400000000003 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.692177+0000", 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000126303 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.692307+0000", 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012935 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.693563+0000", 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001256046 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.695016+0000", 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001453343 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.734229+0000", 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.039213099000000001 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.734550+0000", 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00032104200000000001 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.734608+0000", 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7716000000000001e-05 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:8693 7.b 7:d0ab9a08:::benchmark_data_smithi045_83007_object8692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:46.693325+0000", 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 57.168839613999999, 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.041459397000000002, 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.885 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 8693 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.693325+0000", 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.693325+0000", 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.693327+0000", 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.711e-06 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.693351+0000", 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4663000000000001e-05 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.693367+0000", 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6084000000000001e-05 2024-03-11T22:11:43.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.693376+0000", 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3769999999999996e-06 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.693670+0000", 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00029450100000000002 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.693809+0000", 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013924799999999999 2024-03-11T22:11:43.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.694034+0000", 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000224751 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.695362+0000", 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00132825 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.695741+0000", 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000378649 2024-03-11T22:11:43.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.734662+0000", 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038920578999999997 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.734747+0000", 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5519999999999994e-05 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.734784+0000", 2024-03-11T22:11:43.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7064000000000003e-05 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:8706 7.b 7:d9da3bf0:::benchmark_data_smithi045_83007_object8705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:46.699593+0000", 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 57.162571315000001, 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.035364686999999999, 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:43.891 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 8706 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.699593+0000", 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.699593+0000", 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.699595+0000", 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.598e-06 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.699617+0000", 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1778999999999999e-05 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.699631+0000", 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.486e-05 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.699640+0000", 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9109999999999999e-06 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.699705+0000", 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4806000000000003e-05 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.699845+0000", 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014007399999999999 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.699991+0000", 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000146117 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.700857+0000", 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00086529300000000005 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.701192+0000", 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00033550399999999998 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.734841+0000", 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.033649309000000002 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.734924+0000", 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2979999999999995e-05 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.734958+0000", 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3456e-05 2024-03-11T22:11:43.896 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:8716 7.b 7:df73487c:::benchmark_data_smithi045_83007_object8715:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:46.704727+0000", 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 57.157437178999999, 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.035031362000000003, 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:43.897 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 8716 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.704727+0000", 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.704727+0000", 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.704781+0000", 2024-03-11T22:11:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4165000000000001e-05 2024-03-11T22:11:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.704804+0000", 2024-03-11T22:11:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2802999999999998e-05 2024-03-11T22:11:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.704823+0000", 2024-03-11T22:11:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8736000000000002e-05 2024-03-11T22:11:43.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.704843+0000", 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0259e-05 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.704910+0000", 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6743999999999997e-05 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.705046+0000", 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013608899999999999 2024-03-11T22:11:43.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:11:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.705302+0000", 2024-03-11T22:11:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00025609299999999999 2024-03-11T22:11:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.705952+0000", 2024-03-11T22:11:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00064953999999999999 2024-03-11T22:11:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.706198+0000", 2024-03-11T22:11:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024670600000000002 2024-03-11T22:11:43.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.739638+0000", 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.033440016000000003 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.739711+0000", 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2739999999999998e-05 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:46.739759+0000", 2024-03-11T22:11:43.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7471e-05 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:27006 7.b 7:df2c124f:::benchmark_data_smithi045_83007_object27005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.373880+0000", 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 48.488284792999998, 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.066318402999999998, 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:43.904 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27006 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.373880+0000", 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.373880+0000", 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.373881+0000", 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4219999999999999e-06 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.373905+0000", 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4260000000000002e-05 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.373918+0000", 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2201000000000001e-05 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.373926+0000", 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3990000000000004e-06 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.373986+0000", 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0035999999999999e-05 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.374111+0000", 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012477000000000001 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.374236+0000", 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000125616 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.439163+0000", 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.064927000999999998 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.439219+0000", 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5689999999999997e-05 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.440039+0000", 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00081950499999999999 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.440146+0000", 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000107664 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.440198+0000", 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1839000000000003e-05 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.909 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:27006 7.b e106/99 ondisk, result = 0)", 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.375147+0000", 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 48.487017194000003, 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.065059611000000003, 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.375147+0000", 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.375147+0000", 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.375148+0000", 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2300000000000001e-07 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.375151+0000", 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2289999999999999e-06 2024-03-11T22:11:43.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.437460+0000", 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.062308808 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.437471+0000", 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0771000000000001e-05 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.440024+0000", 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025535340000000001 2024-03-11T22:11:43.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.440033+0000", 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7420000000000003e-06 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.440207+0000", 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017400400000000001 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.913 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:27006 7.b e106/99 ondisk, result = 0)", 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.375256+0000", 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 48.486908925999998, 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.063970874999999996, 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.375256+0000", 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.375256+0000", 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.375256+0000", 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2699999999999999e-07 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.375259+0000", 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.757e-06 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.437111+0000", 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.061851735999999997 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.437131+0000", 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0466e-05 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.439205+0000", 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020741710000000001 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.439214+0000", 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3229999999999998e-06 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.439226+0000", 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2894999999999999e-05 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:28072 7.b 7:d44f7efc:::benchmark_data_smithi045_83007_object28071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:11:43.917 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:56.072552+0000", 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 47.789612630999997, 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.052469421000000002, 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 28072 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.072552+0000", 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.072552+0000", 2024-03-11T22:11:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.072554+0000", 2024-03-11T22:11:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8560000000000001e-06 2024-03-11T22:11:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.072581+0000", 2024-03-11T22:11:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7455000000000001e-05 2024-03-11T22:11:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.072604+0000", 2024-03-11T22:11:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.279e-05 2024-03-11T22:11:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.072615+0000", 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1238e-05 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.072686+0000", 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0742999999999997e-05 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.072826+0000", 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000139858 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.073008+0000", 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018195799999999999 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074647+0000", 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001639625 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074659+0000", 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1950000000000001e-05 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.075865+0000", 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001205713 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.076058+0000", 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019326900000000001 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.125021+0000", 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.048962965999999997 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.924 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.14790.0:28069 7.4 e107/99 7:23fe5b83:::benchmark_data_smithi045_83007_object28068:head v 107'1743, mlcod=106'1741)", 2024-03-11T22:11:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:56.074581+0000", 2024-03-11T22:11:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 47.787583388000002, 2024-03-11T22:11:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.050437128999999997, 2024-03-11T22:11:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074581+0000", 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074581+0000", 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074581+0000", 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8299999999999998e-07 2024-03-11T22:11:43.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074604+0000", 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2668e-05 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074623+0000", 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9046999999999999e-05 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074631+0000", 2024-03-11T22:11:43.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5820000000000004e-06 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074836+0000", 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0002052 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074856+0000", 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0499000000000001e-05 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.124965+0000", 2024-03-11T22:11:43.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.050108584999999997 2024-03-11T22:11:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.125018+0000", 2024-03-11T22:11:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3264999999999999e-05 2024-03-11T22:11:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.929 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:28072 7.b e106/99 ondisk, result = 0)", 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:56.074720+0000", 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 47.787444041000001, 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.050313434999999997, 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074720+0000", 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074720+0000", 2024-03-11T22:11:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074721+0000", 2024-03-11T22:11:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7100000000000002e-07 2024-03-11T22:11:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074728+0000", 2024-03-11T22:11:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2579999999999998e-06 2024-03-11T22:11:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074738+0000", 2024-03-11T22:11:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3589999999999997e-06 2024-03-11T22:11:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.074745+0000", 2024-03-11T22:11:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3209999999999999e-06 2024-03-11T22:11:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.075828+0000", 2024-03-11T22:11:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001082895 2024-03-11T22:11:43.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.075859+0000", 2024-03-11T22:11:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1242999999999999e-05 2024-03-11T22:11:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.125034+0000", 2024-03-11T22:11:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.049174787999999997 2024-03-11T22:11:43.935 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.14790.0:28071 7.9 e107/99 7:9a2c2ee1:::benchmark_data_smithi045_83007_object28070:head v 107'1789, mlcod=106'1787)", 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:56.077562+0000", 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 47.784602847000002, 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.049098916999999999, 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.077562+0000", 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.077562+0000", 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.077563+0000", 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9600000000000008e-07 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.077590+0000", 2024-03-11T22:11:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6917e-05 2024-03-11T22:11:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.124442+0000", 2024-03-11T22:11:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.046852257000000001 2024-03-11T22:11:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.124462+0000", 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.065e-05 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.124984+0000", 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00052144799999999999 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.125046+0000", 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1945000000000004e-05 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.126614+0000", 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015681530000000001 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.126661+0000", 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6551000000000002e-05 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:28078 7.b 7:d636cc2a:::benchmark_data_smithi045_83007_object28077:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:56.079705+0000", 2024-03-11T22:11:43.940 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 47.782459811999999, 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.047585799999999998, 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 28078 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.079705+0000", 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.079705+0000", 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.079707+0000", 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0999999999999998e-06 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.079734+0000", 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7042e-05 2024-03-11T22:11:43.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.124642+0000", 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.044908605999999997 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.124655+0000", 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2819999999999999e-05 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.125339+0000", 2024-03-11T22:11:43.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00068361099999999998 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.125483+0000", 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000143957 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.125642+0000", 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015911300000000001 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.126981+0000", 2024-03-11T22:11:43.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013391480000000001 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.127180+0000", 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019941699999999999 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.127235+0000", 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4852000000000003e-05 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.127267+0000", 2024-03-11T22:11:43.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1860000000000003e-05 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.127290+0000", 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3274000000000001e-05 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:30354 7.b 7:db380bff:::benchmark_data_smithi045_83007_object30353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:57.573207+0000", 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 46.288957555000003, 2024-03-11T22:11:43.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.056048911999999999, 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 30354 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.573207+0000", 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.573207+0000", 2024-03-11T22:11:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.573208+0000", 2024-03-11T22:11:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1710000000000001e-06 2024-03-11T22:11:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.573226+0000", 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8168999999999998e-05 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.573236+0000", 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9580000000000002e-06 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.573243+0000", 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7560000000000002e-06 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.573289+0000", 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6010999999999997e-05 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.624131+0000", 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.050841586000000001 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.624325+0000", 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019482599999999999 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.625913+0000", 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001587897 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.625986+0000", 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2793999999999995e-05 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.629076+0000", 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003090247 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.629211+0000", 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000134678 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:57.629256+0000", 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4818999999999999e-05 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:41626 7.b 7:d90bee90:::benchmark_data_smithi045_83007_object41625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:11:43.952 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:11:03.868731+0000", 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 39.993433826999997, 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.112628617, 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 41626 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868731+0000", 2024-03-11T22:11:43.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868731+0000", 2024-03-11T22:11:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868732+0000", 2024-03-11T22:11:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.793e-06 2024-03-11T22:11:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868758+0000", 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5933000000000001e-05 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868778+0000", 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9499000000000001e-05 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868787+0000", 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4630000000000007e-06 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868863+0000", 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5364999999999994e-05 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868957+0000", 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4256e-05 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.869046+0000", 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8976999999999999e-05 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872439+0000", 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033925940000000001 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.876086+0000", 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003647127 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.981251+0000", 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105165353 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.981310+0000", 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9333000000000003e-05 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.981359+0000", 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8924000000000001e-05 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.958 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:41631 7.b 7:d85749de:::benchmark_data_smithi045_83007_object41630:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:11:03.872638+0000", 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 39.989526443000003, 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.113385656, 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 41631 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872638+0000", 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872638+0000", 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872640+0000", 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.556e-06 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872665+0000", 2024-03-11T22:11:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5123999999999999e-05 2024-03-11T22:11:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872680+0000", 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5719999999999999e-05 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872689+0000", 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.123e-06 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872753+0000", 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4731000000000001e-05 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872926+0000", 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017301600000000001 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.874603+0000", 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016767819999999999 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.876181+0000", 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015781809999999999 2024-03-11T22:11:43.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.877689+0000", 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001507944 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.985762+0000", 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10807291400000001 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.985940+0000", 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017804499999999999 2024-03-11T22:11:43.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.986024+0000", 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.352e-05 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:41652 7.b 7:d3bc2d9e:::benchmark_data_smithi045_83007_object41651:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:11:03.891081+0000", 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 39.971083176999997, 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.095143848000000003, 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.965 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 41652 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.891081+0000", 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.891081+0000", 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.891083+0000", 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.553e-06 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.891106+0000", 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3569000000000001e-05 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.891123+0000", 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6568000000000001e-05 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.891132+0000", 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2420000000000006e-06 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.891287+0000", 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015488700000000001 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.891449+0000", 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016202400000000001 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.891590+0000", 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014115699999999999 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.893167+0000", 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001576462 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.893182+0000", 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5345e-05 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.986075+0000", 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.092893166999999999 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.986186+0000", 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011092 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.986225+0000", 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8954e-05 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:41669 7.b 7:d954276e:::benchmark_data_smithi045_83007_object41668:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:11:03.904764+0000", 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 39.957400016000001, 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.084250971999999993, 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:11:43.971 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 41669 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.904764+0000", 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.904764+0000", 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.904767+0000", 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.153e-06 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.904788+0000", 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1531000000000001e-05 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.904805+0000", 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7096e-05 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:11:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.904814+0000", 2024-03-11T22:11:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4379999999999995e-06 2024-03-11T22:11:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:11:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.904887+0000", 2024-03-11T22:11:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3311000000000003e-05 2024-03-11T22:11:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:11:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.905029+0000", 2024-03-11T22:11:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014164000000000001 2024-03-11T22:11:43.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.905175+0000", 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000146188 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.906393+0000", 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001218221 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.973118+0000", 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.066724576999999993 2024-03-11T22:11:43.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:11:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.988852+0000", 2024-03-11T22:11:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015733912999999999 2024-03-11T22:11:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:11:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.988949+0000", 2024-03-11T22:11:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7275000000000006e-05 2024-03-11T22:11:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:11:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:11:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:11:43.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.989015+0000", 2024-03-11T22:11:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6629000000000002e-05 2024-03-11T22:11:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:11:43.977 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:11:43.977 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:11:43.977 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:43.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:44.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:44.119 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:44.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:44.220 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:44.247 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:44.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:44.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:44.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:44.392 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:44.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:44.493 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:44.523 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:44.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:44.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:44.625 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:44.632 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 61 16 91274 91258 93.4784 55.125 0.0246451 0.0106922 2024-03-11T22:11:44.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:44.655 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:44.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:44.756 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:44.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:44 smithi045 ceph-mon[39715]: 5.18 scrub starts 2024-03-11T22:11:44.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:44 smithi045 ceph-mon[39715]: 5.18 scrub ok 2024-03-11T22:11:44.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:44 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/74265626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:11:44.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:44 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:44.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:44 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:44 smithi045 ceph-mon[30178]: 5.18 scrub starts 2024-03-11T22:11:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:44 smithi045 ceph-mon[30178]: 5.18 scrub ok 2024-03-11T22:11:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:44 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/74265626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:11:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:44 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:44.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:44 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:44.786 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:44.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:44.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:44 smithi144 ceph-mon[33984]: 5.18 scrub starts 2024-03-11T22:11:44.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:44 smithi144 ceph-mon[33984]: 5.18 scrub ok 2024-03-11T22:11:44.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:44 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/74265626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:11:44.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:44 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:44.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:44 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:44.853 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:11:44.853 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:11:44.853 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:11:44.853 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:11:44.853 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:11:44.853 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:11:44.853 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:11:44.853 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:11:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:11:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:11:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:11:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:11:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:11:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:11:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:11:44.854 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:11:44.888 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:44.918 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:44.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:44.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:45.021 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:45.052 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:45.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:45.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:45.154 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:45.187 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:45.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:45.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:45.331 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:45.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:45.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:45.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:45.477 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:45.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:45.578 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:45.607 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:45.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:45.632 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 62 16 92262 92246 92.9666 61.75 0.019527 0.0107513 2024-03-11T22:11:45.709 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:45.737 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:45.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:45.753 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:45 smithi144 ceph-mon[33984]: 5.14 scrub starts 2024-03-11T22:11:45.753 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:45 smithi144 ceph-mon[33984]: pgmap v110: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T22:11:45.753 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:45 smithi144 ceph-mon[33984]: 5.14 scrub ok 2024-03-11T22:11:45.753 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:45 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:11:45.753 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:45 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:11:45.753 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:45 smithi144 ceph-mon[33984]: 7.e scrub starts 2024-03-11T22:11:45.753 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:45 smithi144 ceph-mon[33984]: osdmap e119: 8 total, 8 up, 8 in 2024-03-11T22:11:45.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[39715]: 5.14 scrub starts 2024-03-11T22:11:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[39715]: pgmap v110: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T22:11:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[39715]: 5.14 scrub ok 2024-03-11T22:11:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:11:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:11:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[39715]: 7.e scrub starts 2024-03-11T22:11:45.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[39715]: osdmap e119: 8 total, 8 up, 8 in 2024-03-11T22:11:45.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[30178]: 5.14 scrub starts 2024-03-11T22:11:45.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[30178]: pgmap v110: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T22:11:45.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[30178]: 5.14 scrub ok 2024-03-11T22:11:45.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:11:45.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:11:45.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[30178]: 7.e scrub starts 2024-03-11T22:11:45.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:45 smithi045 ceph-mon[30178]: osdmap e119: 8 total, 8 up, 8 in 2024-03-11T22:11:45.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:45.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:45.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:45.983 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:46.011 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:46.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:46.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:46.090 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:11:45 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:11:46.112 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:46.142 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:46.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:46.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:46.298 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:46.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:46.399 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:46.429 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:46.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:46.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:46 smithi045 ceph-mon[39715]: 4.1a scrub starts 2024-03-11T22:11:46.526 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:46 smithi045 ceph-mon[39715]: 4.1a scrub ok 2024-03-11T22:11:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:46 smithi045 ceph-mon[39715]: 3.1f scrub starts 2024-03-11T22:11:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:46 smithi045 ceph-mon[39715]: 3.1f scrub ok 2024-03-11T22:11:46.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:46 smithi045 ceph-mon[39715]: osdmap e120: 8 total, 8 up, 8 in 2024-03-11T22:11:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:46 smithi045 ceph-mon[30178]: 4.1a scrub starts 2024-03-11T22:11:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:46 smithi045 ceph-mon[30178]: 4.1a scrub ok 2024-03-11T22:11:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:46 smithi045 ceph-mon[30178]: 3.1f scrub starts 2024-03-11T22:11:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:46 smithi045 ceph-mon[30178]: 3.1f scrub ok 2024-03-11T22:11:46.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:46 smithi045 ceph-mon[30178]: osdmap e120: 8 total, 8 up, 8 in 2024-03-11T22:11:46.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:46.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:46.575 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:46.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:46.632 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 63 15 93209 93194 92.4315 59.25 0.0202048 0.0108136 2024-03-11T22:11:46.676 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:46.730 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:46.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:46.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:46.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:46.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:46 smithi144 ceph-mon[33984]: 4.1a scrub starts 2024-03-11T22:11:46.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:46 smithi144 ceph-mon[33984]: 4.1a scrub ok 2024-03-11T22:11:46.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:46 smithi144 ceph-mon[33984]: 3.1f scrub starts 2024-03-11T22:11:46.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:46 smithi144 ceph-mon[33984]: 3.1f scrub ok 2024-03-11T22:11:46.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:46 smithi144 ceph-mon[33984]: osdmap e120: 8 total, 8 up, 8 in 2024-03-11T22:11:46.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:46.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:46.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:46.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:47.014 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:47.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:47.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:47.159 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:47.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:47.260 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:47.288 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:47.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:47.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:47.390 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:47.420 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:47.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:47.456 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:11:47.522 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:47.570 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:47.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:47 smithi144 ceph-mon[33984]: 6.13 scrub starts 2024-03-11T22:11:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:47 smithi144 ceph-mon[33984]: 6.13 scrub ok 2024-03-11T22:11:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:47 smithi144 ceph-mon[33984]: pgmap v113: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T22:11:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:47 smithi144 ceph-mon[33984]: 6.7 scrub starts 2024-03-11T22:11:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:47 smithi144 ceph-mon[33984]: 6.7 scrub ok 2024-03-11T22:11:47.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:47 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2424957156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:11:47.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:47.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[39715]: 6.13 scrub starts 2024-03-11T22:11:47.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[39715]: 6.13 scrub ok 2024-03-11T22:11:47.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[39715]: pgmap v113: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T22:11:47.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[39715]: 6.7 scrub starts 2024-03-11T22:11:47.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[39715]: 6.7 scrub ok 2024-03-11T22:11:47.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2424957156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:11:47.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:47.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[30178]: 6.13 scrub starts 2024-03-11T22:11:47.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[30178]: 6.13 scrub ok 2024-03-11T22:11:47.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[30178]: pgmap v113: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T22:11:47.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[30178]: 6.7 scrub starts 2024-03-11T22:11:47.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[30178]: 6.7 scrub ok 2024-03-11T22:11:47.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2424957156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:11:47.633 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 64 16 93953 93937 91.7129 46.4375 0.00852495 0.0108986 2024-03-11T22:11:47.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:47.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:47.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:47.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:47.817 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:47.846 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:47.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:47.948 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:47.978 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:47.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:47.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:48.080 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:48.108 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:48.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:48.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:48.210 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:48.240 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:48.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:48.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:48.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:48.392 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:48.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:48.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:48.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:48.535 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:48.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:48.633 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 65 15 94876 94861 91.1903 57.75 0.00908692 0.0109584 2024-03-11T22:11:48.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:48.673 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:48.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:48.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:48.713 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[30178]: 7.e scrub ok 2024-03-11T22:11:48.714 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[30178]: 7.b deep-scrub starts 2024-03-11T22:11:48.714 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:11:48.714 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:11:48.714 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[30178]: 6.19 scrub starts 2024-03-11T22:11:48.714 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[30178]: 6.19 scrub ok 2024-03-11T22:11:48.714 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:11:48.714 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2424957156' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:11:48.714 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[30178]: osdmap e121: 8 total, 8 up, 8 in 2024-03-11T22:11:48.714 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[39715]: 7.e scrub ok 2024-03-11T22:11:48.715 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[39715]: 7.b deep-scrub starts 2024-03-11T22:11:48.715 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:11:48.715 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:11:48.715 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[39715]: 6.19 scrub starts 2024-03-11T22:11:48.715 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[39715]: 6.19 scrub ok 2024-03-11T22:11:48.715 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:11:48.715 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2424957156' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:11:48.715 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:48 smithi045 ceph-mon[39715]: osdmap e121: 8 total, 8 up, 8 in 2024-03-11T22:11:48.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:48.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:48 smithi144 ceph-mon[33984]: 7.e scrub ok 2024-03-11T22:11:48.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:48 smithi144 ceph-mon[33984]: 7.b deep-scrub starts 2024-03-11T22:11:48.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:48 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:11:48.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:48 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:11:48.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:48 smithi144 ceph-mon[33984]: 6.19 scrub starts 2024-03-11T22:11:48.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:48 smithi144 ceph-mon[33984]: 6.19 scrub ok 2024-03-11T22:11:48.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:48 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:11:48.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:48 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2424957156' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:11:48.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:48 smithi144 ceph-mon[33984]: osdmap e121: 8 total, 8 up, 8 in 2024-03-11T22:11:48.843 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:48.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:48.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:48.945 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:48.976 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:48.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:49.073 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T22:11:49.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:49.109 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:49.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:49.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:49.212 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:49.240 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:49.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:49.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:11:49.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:49.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:49.427 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:49.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:49.528 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:49.556 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:49.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:49.634 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 66 16 95749 95733 90.6336 54.5 0.0046796 0.0110271 2024-03-11T22:11:49.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:49.705 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:49 smithi045 ceph-mon[30178]: pgmap v115: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 829 op/s 2024-03-11T22:11:49.705 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:49 smithi045 ceph-mon[39715]: pgmap v115: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 829 op/s 2024-03-11T22:11:49.708 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:49.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:49.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:49.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:49 smithi144 ceph-mon[33984]: pgmap v115: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 829 op/s 2024-03-11T22:11:49.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:49.861 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:49.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:49.962 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:49.991 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:49.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:50.093 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:50.123 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:50.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:50.133 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:50.224 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:50.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:50.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:50.372 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:50.401 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:50.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:50.503 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:50.536 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:50.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:50.633 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 67 16 96707 96691 90.1745 59.875 0.00752767 0.0110831 2024-03-11T22:11:50.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:50.658 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:50.686 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:50.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:50.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:11:50.789 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:50.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:50.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:50.975 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:51.017 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:51.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:51.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:51.119 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:51.147 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:51.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:51.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:51.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:51.298 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:51.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:51.399 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:51.431 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:51.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:51.533 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:51.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:51.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:51.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:51.633 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 68 16 97599 97583 89.6683 55.75 0.0102825 0.0111468 2024-03-11T22:11:51.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:51.706 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[30178]: pgmap v116: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 872 op/s 2024-03-11T22:11:51.706 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[30178]: osdmap e122: 8 total, 8 up, 8 in 2024-03-11T22:11:51.706 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:51.707 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:51.707 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:11:51.707 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:11:51.707 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:51.707 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[39715]: pgmap v116: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 872 op/s 2024-03-11T22:11:51.707 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[39715]: osdmap e122: 8 total, 8 up, 8 in 2024-03-11T22:11:51.707 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:51.707 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:51.707 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:11:51.707 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:11:51.707 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:51 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:51.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:51.710 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:51.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:51.811 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:51.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:51 smithi144 ceph-mon[33984]: pgmap v116: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 872 op/s 2024-03-11T22:11:51.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:51 smithi144 ceph-mon[33984]: osdmap e122: 8 total, 8 up, 8 in 2024-03-11T22:11:51.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:51 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:51.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:51 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:51.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:51 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:11:51.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:51 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:11:51.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:51 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:51.860 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:51.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:51.961 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:51.991 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:51.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:51.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:52.093 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:52.123 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:52.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:52.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:52.276 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:52.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:52.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:52.378 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:52.406 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:52.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:52.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:52.508 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:52.541 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:52.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:52.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:52.634 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 69 16 98566 98550 89.2446 60.4375 0.00858986 0.0112001 2024-03-11T22:11:52.642 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:52.675 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:52.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:52.777 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:52.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:52.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:52.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:52.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:52.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:52.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:53.026 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:11:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:11:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:11:53.045 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:53.091 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:53.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:53.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:53.193 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:53.223 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:53.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:53.325 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:53.368 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:53.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:53.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:53.470 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:53.504 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:53.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:53.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:53.634 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 70 16 99394 99378 88.709 51.75 0.00895392 0.0112632 2024-03-11T22:11:53.643 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:53.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:53.745 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:53.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:53 smithi045 ceph-mon[30178]: pgmap v118: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 46 MiB/s wr, 744 op/s 2024-03-11T22:11:53.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:53 smithi045 ceph-mon[39715]: pgmap v118: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 46 MiB/s wr, 744 op/s 2024-03-11T22:11:53.778 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:53.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:53.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:53.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:53 smithi144 ceph-mon[33984]: pgmap v118: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 46 MiB/s wr, 744 op/s 2024-03-11T22:11:53.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:53.918 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:53.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:54.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:54.064 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:54.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:54.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:54.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:54.214 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:54.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:54.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:54.316 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:54.348 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:54.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:54.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:54.450 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:54.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:54.482 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:54.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:54.583 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:54.613 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:54.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:54.634 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 71 16 100364 100348 88.3135 60.625 0.00788913 0.0113185 2024-03-11T22:11:54.714 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:54.743 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:54.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:54.845 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:54.874 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:54.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:54.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:54.884 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:11:54.976 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:55.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:55.119 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:55.151 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:55.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:55.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:55.302 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:55.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:55.403 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:55.455 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:55.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:55.523 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:55 smithi045 ceph-mon[39715]: pgmap v119: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 35 MiB/s wr, 558 op/s 2024-03-11T22:11:55.523 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:55 smithi045 ceph-mon[30178]: pgmap v119: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 35 MiB/s wr, 558 op/s 2024-03-11T22:11:55.524 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:11:55.524 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:11:55.556 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:55.584 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:55.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:55.634 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 72 15 101280 101265 87.8829 57.3125 0.00416874 0.0113734 2024-03-11T22:11:55.686 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:55.717 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:55.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:55.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:55.765 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:55 smithi144 ceph-mon[33984]: pgmap v119: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 35 MiB/s wr, 558 op/s 2024-03-11T22:11:55.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:55.861 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:55.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:55.963 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:55.992 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:55.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:56.090 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:11:55 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:11:56.096 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:56.129 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:56.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:56.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:56.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:56.276 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:56.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:56.378 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:56.418 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:56.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T22:11:56.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:56 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3478905195' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:11:56.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:56 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:11:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:56 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3478905195' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:11:56.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:56 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:11:56.566 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:56.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:56.634 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 73 16 102132 102116 87.4076 53.1875 0.0506117 0.0114349 2024-03-11T22:11:56.668 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:56.699 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:56.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:56.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:56.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:56.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:56.826 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:56.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:56.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:56 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3478905195' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:11:56.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:56 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:11:56.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:56.957 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:56.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:57.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:57.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:57.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:57.197 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:57.227 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:57.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:57.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:57.372 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:57.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:57.472 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:57.502 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:57.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:57.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:57.604 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:57.634 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 74 16 103022 103006 86.9781 55.625 0.00674484 0.0114918 2024-03-11T22:11:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:57.635 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:57.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:57.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:57.775 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:57.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:57 smithi045 ceph-mon[39715]: pgmap v120: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-11T22:11:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:57 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:11:57.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:57 smithi045 ceph-mon[39715]: osdmap e123: 8 total, 8 up, 8 in 2024-03-11T22:11:57.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:57.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:57 smithi045 ceph-mon[30178]: pgmap v120: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-11T22:11:57.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:57 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:11:57.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:57 smithi045 ceph-mon[30178]: osdmap e123: 8 total, 8 up, 8 in 2024-03-11T22:11:57.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:11:57.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:57 smithi144 ceph-mon[33984]: pgmap v120: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-11T22:11:57.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:57 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:11:57.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:57 smithi144 ceph-mon[33984]: osdmap e123: 8 total, 8 up, 8 in 2024-03-11T22:11:57.878 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:57.908 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:57.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:57.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:58.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:58.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:58.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:58.153 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:58.181 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:58.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:58.282 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:58.314 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:58.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:58.415 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:11:58.445 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:58.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:58.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:11:58.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:11:58.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:58.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:58.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:11:58.634 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 75 15 103972 103957 86.6109 59.4375 0.00989494 0.0115401 2024-03-11T22:11:58.696 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:11:58.723 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:58.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:58.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:11:58.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:58 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:11:58.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:58 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:11:58.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:58 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:11:58.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:58 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:11:58.806 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:11:58.825 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:58.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:58 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:11:58.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:58 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:11:58.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:58.854 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:58.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:58.876 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:11:58.916 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:11:58.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:58.962 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:11:59.000 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:59.001 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:11:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:59.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:59.041 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:11:59.086 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:11:59.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:11:59.125 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:11:59.138 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:11:59.139 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:11:59.139 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:11:59.139 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:11:59.139 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:11:59.139 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:11:59.139 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:11:59.139 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:11:59.142 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:59.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:11:59.244 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:11:59.270 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:59.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:11:59.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:11:59.411 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:59.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:11:59.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:59.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:59.634 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 76 16 104789 104773 86.1423 51 0.018624 0.0116038 2024-03-11T22:11:59.649 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:11:59.680 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:11:59.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:59.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:11:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:59 smithi045 ceph-mon[39715]: pgmap v122: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-11T22:11:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:59 smithi045 ceph-mon[39715]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T22:11:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:59 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2048741745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:11:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:59 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:59.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:11:59 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:59.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:59 smithi045 ceph-mon[30178]: pgmap v122: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-11T22:11:59.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:59 smithi045 ceph-mon[30178]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T22:11:59.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:59 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2048741745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:11:59.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:59 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:59.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:11:59 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:59.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:59.817 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:59.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:11:59.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:59 smithi144 ceph-mon[33984]: pgmap v122: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-11T22:11:59.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:59 smithi144 ceph-mon[33984]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T22:11:59.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:59 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2048741745' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:11:59.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:59 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:11:59.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:11:59 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:11:59.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:11:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:11:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:11:59.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:00.065 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:00.095 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:00.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:00.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:00.241 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:00.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:00.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:00.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:00.388 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:00.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:00.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:00.490 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:00.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T22:12:00.520 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:00.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:00.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:00.622 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:00.634 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 77 16 105741 105725 85.7963 59.5 0.00712005 0.0116495 2024-03-11T22:12:00.688 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:00.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:00.789 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:00.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:00.819 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:00.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:00.920 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:00.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:00.951 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:00.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:00.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T22:12:01.051 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:01.079 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:01.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:01.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:01.181 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:01.211 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:01.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:01.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:01.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:01.368 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:01.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:01.469 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:01.501 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:01.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:01.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:01.635 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 78 16 106661 106645 85.4329 57.5 0.0389061 0.0116998 2024-03-11T22:12:01.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:01.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:01.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:01.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:01 smithi045 ceph-mon[30178]: pgmap v124: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 615 op/s 2024-03-11T22:12:01.651 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:01 smithi045 ceph-mon[39715]: pgmap v124: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 615 op/s 2024-03-11T22:12:01.741 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:01.772 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:01.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:01.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:01.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:01 smithi144 ceph-mon[33984]: pgmap v124: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 615 op/s 2024-03-11T22:12:01.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:01.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:01.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:01.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:02.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:02.065 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:02.102 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:02.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:02.204 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:02.235 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:02.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:02.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:02.337 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:02.365 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:02.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:02.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:02.514 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:02.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:02.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:02.616 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:02.635 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 79 16 107494 107478 85.0105 52.0625 0.0086225 0.0117556 2024-03-11T22:12:02.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:02.641 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:02.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:02.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:02.776 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:12:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:12:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:12:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:02.786 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:02.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:02.887 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:02.918 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:02.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:02.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:03.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:03.065 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:03.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:03.122 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:03.168 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:03.197 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:03.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:03.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:03.362 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:03.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:03.463 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:03.492 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:03.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:03.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:03.508 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:12:03.594 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:03.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:03.625 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:03.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:03.635 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:12:03.640667+0000 min lat: 0.00145914 max lat: 0.165961 avg lat: 0.011807 2024-03-11T22:12:03.635 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:12:03.636 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 80 16 108406 108390 84.6603 57 0.0164853 0.011807 2024-03-11T22:12:03.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:12:03.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:03 smithi045 ceph-mon[39715]: pgmap v125: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-11T22:12:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:03 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3410138072' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:12:03.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:03 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:12:03.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:03 smithi045 ceph-mon[30178]: pgmap v125: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-11T22:12:03.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:03 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3410138072' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:12:03.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:03 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:12:03.799 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:03.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:03.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:03.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:03 smithi144 ceph-mon[33984]: pgmap v125: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-11T22:12:03.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:03 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3410138072' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:12:03.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:03 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:12:03.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:03.946 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:03.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:04.047 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:04.078 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:04.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:04.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:04.180 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:04.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:04.210 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:04.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:04.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:04.347 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:04.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:04.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:04.485 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:04.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:04.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:04.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:04.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:04.634 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:04.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:04.636 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 81 16 109413 109397 84.3921 62.9375 0.0264965 0.011845 2024-03-11T22:12:04.736 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:04.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:04.769 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:04.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:04 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:12:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:04 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:12:04.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:04 smithi045 ceph-mon[39715]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T22:12:04.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:04 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:12:04.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:04 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:12:04.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:04 smithi045 ceph-mon[30178]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T22:12:04.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:04 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:12:04.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:04 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:12:04.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:04 smithi144 ceph-mon[33984]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T22:12:04.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:04.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:04.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:05.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:05.056 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:05.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:05.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:05.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:05.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:05.304 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:05.332 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:05.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:05.434 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:05.461 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:05.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:05.563 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:05.592 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:05.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:05.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:05.635 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 82 16 110484 110468 84.1792 66.9375 0.00963169 0.0118748 2024-03-11T22:12:05.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:05.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:05.739 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:05.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:05.840 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:05.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:05 smithi144 ceph-mon[33984]: pgmap v127: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 672 B/s rd, 36 MiB/s wr, 579 op/s 2024-03-11T22:12:05.842 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:12:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:12:05.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:05.875 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:05.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:05.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:06.024 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:05 smithi045 ceph-mon[39715]: pgmap v127: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 672 B/s rd, 36 MiB/s wr, 579 op/s 2024-03-11T22:12:06.025 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:05 smithi045 ceph-mon[30178]: pgmap v127: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 672 B/s rd, 36 MiB/s wr, 579 op/s 2024-03-11T22:12:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:06.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:06.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:06.127 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:06.159 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:06.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:06.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:06.260 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:06.287 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:06.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:06.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:06.389 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:06.415 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:06.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:06.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:06.517 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:06.545 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:06.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:06.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:06.636 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 83 16 111502 111486 83.9315 63.625 0.00819908 0.0119095 2024-03-11T22:12:06.647 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:06.673 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:06.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:06.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:06.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:06.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:06.822 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:06.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:06.927 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:06.971 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:06.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:06.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:06.978 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:12:06.978 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:12:06.978 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-03-11T22:12:06.978 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-03-11T22:12:06.978 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:12:07.073 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:07.099 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:07.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:07.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:07.201 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:07.228 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:07.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:07.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:07.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:07.378 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:07.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:07.479 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:07.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:07.511 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:07.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:07.555 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T22:12:07.612 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:07.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:07.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[39715]: pgmap v128: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T22:12:07.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:12:07.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:12:07.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[39715]: osdmap e126: 8 total, 8 up, 8 in 2024-03-11T22:12:07.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2564587501' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:12:07.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:12:07.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:07.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[30178]: pgmap v128: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T22:12:07.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:12:07.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:12:07.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[30178]: osdmap e126: 8 total, 8 up, 8 in 2024-03-11T22:12:07.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2564587501' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:12:07.627 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:07 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:12:07.636 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 84 16 112430 112414 83.6228 58 0.0155557 0.0119535 2024-03-11T22:12:07.643 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:07.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:07.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:07.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:07.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:07.786 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:07.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:07.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:07.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:07 smithi144 ceph-mon[33984]: pgmap v128: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T22:12:07.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:07 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:12:07.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:07 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:12:07.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:07 smithi144 ceph-mon[33984]: osdmap e126: 8 total, 8 up, 8 in 2024-03-11T22:12:07.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:07 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2564587501' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:12:07.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:07 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:12:07.882 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:07.887 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:07.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:07.920 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:07.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:08.021 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:08.050 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:08.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:08.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:08.193 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:08.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:08.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:08.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:08.351 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:08.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:08.369 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T22:12:08.453 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:08.482 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:08.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:08.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:08.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:08 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:12:08.584 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:08.611 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:08.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:08.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:08.636 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 85 16 113451 113435 83.3897 63.8125 0.0115316 0.011986 2024-03-11T22:12:08.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:08.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:08.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:08.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:08 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:12:08.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:08 smithi144 ceph-mon[33984]: osdmap e127: 8 total, 8 up, 8 in 2024-03-11T22:12:08.871 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:08.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:08 smithi045 ceph-mon[30178]: osdmap e127: 8 total, 8 up, 8 in 2024-03-11T22:12:08.878 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:08 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:12:08.878 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:08 smithi045 ceph-mon[39715]: osdmap e127: 8 total, 8 up, 8 in 2024-03-11T22:12:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:08.899 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:08.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:09.000 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:09.030 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:09.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:09.131 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:09.161 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:09.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:09.262 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:09.291 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:09.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:09.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:09.393 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:09.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:09.426 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:09.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:09.527 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:09.558 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:09.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:09.636 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 86 16 114507 114491 83.1874 66 0.00656454 0.0120158 2024-03-11T22:12:09.660 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:09.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:12:09.686 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:09.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:09.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:09.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:09 smithi045 ceph-mon[39715]: pgmap v131: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 783 op/s 2024-03-11T22:12:09.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:09 smithi045 ceph-mon[30178]: pgmap v131: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 783 op/s 2024-03-11T22:12:09.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:09.840 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:09 smithi144 ceph-mon[33984]: pgmap v131: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 783 op/s 2024-03-11T22:12:09.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:09.841 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:09.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:09.942 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:09.974 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:09.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:09.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:10.076 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:10.104 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:10.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:10.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:10.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:10.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:10.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:10.350 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:10.382 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:10.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:10.483 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:10.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:10.515 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:10.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:10.616 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:10.636 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 87 16 115498 115482 82.9431 61.9375 0.0114987 0.0120518 2024-03-11T22:12:10.647 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:10.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:10.748 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:10.777 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:10.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:10.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T22:12:10.866 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:10.878 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:10.960 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:10.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:10.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:11.062 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:11.092 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:11.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:11.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:11.242 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:11.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:11.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:11.385 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:11.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:11.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:11.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:11.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:11.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:11.630 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:11.636 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 88 16 116448 116432 82.6749 59.375 0.0347161 0.0120902 2024-03-11T22:12:11.659 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:11.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:11.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:11.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:11 smithi045 ceph-mon[30178]: pgmap v132: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 775 B/s rd, 48 MiB/s wr, 761 op/s 2024-03-11T22:12:11.757 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:11 smithi045 ceph-mon[39715]: pgmap v132: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 775 B/s rd, 48 MiB/s wr, 761 op/s 2024-03-11T22:12:11.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:11.793 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:12:11.832 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:11.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:11.842 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:12:11.893 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:12:11.934 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:11.943 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:12:11.979 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:11.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:11.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:11.982 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:12:12.025 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:12:12.065 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:12:12.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:12.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:11 smithi144 ceph-mon[33984]: pgmap v132: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 775 B/s rd, 48 MiB/s wr, 761 op/s 2024-03-11T22:12:12.109 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:12:12.113 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:12:12.113 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:12:12.113 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:12:12.114 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:12:12.114 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:12:12.114 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:12:12.114 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:12:12.114 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:12:12.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:12.128 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:12.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:12.229 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:12.263 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:12.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:12.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:12.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:12.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:12.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:12.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:12.507 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:12.535 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:12.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:12.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:12.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:12.639 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 89 16 117448 117432 82.4482 62.5 0.00836124 0.0121238 2024-03-11T22:12:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:12.682 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:12.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:12.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:12.828 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:12.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:12.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:12.890 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:12:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:12:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:12:12.891 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:12 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1145744845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:12:12.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:12 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1145744845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:12:12.929 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:12.959 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:12.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:13.061 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:13.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:12 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1145744845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:12:13.092 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:13.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:13.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:13.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:13.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:13.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:13.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:13.339 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:13.370 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:13.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:13.380 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:13.472 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:13.501 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:13.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:13.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:13.603 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:13.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:13.633 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:13.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:13.637 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Total time run: 90.0149 2024-03-11T22:12:13.637 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Total writes made: 118468 2024-03-11T22:12:13.637 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Write size: 65536 2024-03-11T22:12:13.637 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Object size: 65536 2024-03-11T22:12:13.637 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Bandwidth (MB/sec): 82.2559 2024-03-11T22:12:13.637 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev Bandwidth: 30.5886 2024-03-11T22:12:13.637 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max bandwidth (MB/sec): 183.062 2024-03-11T22:12:13.638 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min bandwidth (MB/sec): 46.4375 2024-03-11T22:12:13.638 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Average IOPS: 1316 2024-03-11T22:12:13.638 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev IOPS: 489.417 2024-03-11T22:12:13.638 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max IOPS: 2929 2024-03-11T22:12:13.638 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min IOPS: 743 2024-03-11T22:12:13.638 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Average Latency(s): 0.0121527 2024-03-11T22:12:13.638 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev Latency(s): 0.00971116 2024-03-11T22:12:13.638 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max latency(s): 0.165961 2024-03-11T22:12:13.638 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min latency(s): 0.00145914 2024-03-11T22:12:13.646 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T22:12:13.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:13 smithi045 ceph-mon[30178]: pgmap v133: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T22:12:13.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:13 smithi045 ceph-mon[39715]: pgmap v133: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T22:12:13.733 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:13.785 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:13.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:13.801 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:13.886 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:13.917 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:13.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:14.018 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:14.048 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:14.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:14.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:13 smithi144 ceph-mon[33984]: pgmap v133: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-11T22:12:14.151 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:14.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:14.181 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:14.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:14.283 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:14.312 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:14.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:14.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:14.414 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:14.443 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:14.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:14.545 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:14.573 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:14.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:14.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:14.675 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:14.703 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:14.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:14.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:14.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:14.845 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:14.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:14.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:14.985 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:14.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:15.027 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:12:15.027 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:12:15.086 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:15.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:12:15.119 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:15.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:15.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:15.221 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:15.252 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:15.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:15.354 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:15.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:15.384 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:15.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:15.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:12:15.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:15.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:15.534 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:15.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:15.635 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:15.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:15.663 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:15.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:15.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:15.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:15 smithi045 ceph-mon[30178]: pgmap v134: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 632 op/s 2024-03-11T22:12:15.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:15 smithi045 ceph-mon[39715]: pgmap v134: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 632 op/s 2024-03-11T22:12:15.815 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:15.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:15.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:15.916 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:15.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:15.950 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:15.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:16.051 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:16.080 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:16.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:16.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:15 smithi144 ceph-mon[33984]: pgmap v134: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 632 op/s 2024-03-11T22:12:16.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:12:15 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:12:16.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:16.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:16.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:16.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:16.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:16.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:16.372 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:16.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:16.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:16.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:16.517 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:16.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:16.618 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:16.644 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:16.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:16.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:16.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:16.787 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:16.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:16.888 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:16.931 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:16.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:17.033 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:17.060 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:17.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:17.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:17.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:17.107 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:17 smithi045 ceph-mon[30178]: pgmap v135: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 1.06k op/s 2024-03-11T22:12:17.161 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:17.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:17.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:17.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:17.203 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:12:17.203 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:12:17.301 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:17.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:17.328 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:17.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:17.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:17.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:17.431 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:17 smithi045 ceph-mon[39715]: pgmap v135: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 1.06k op/s 2024-03-11T22:12:17.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:17.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:17.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:17.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:17.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:17.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:17 smithi144 ceph-mon[33984]: pgmap v135: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 1.06k op/s 2024-03-11T22:12:17.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:17.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:17.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:17.717 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:17.742 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:17.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:17.844 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:17.870 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:17.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:17.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:17.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:18.015 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:18.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:18.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:18.117 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:18.133 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:12:18.166 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:18.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:18.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:18.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:18.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:18.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:18.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:18.409 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:18.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:18.440 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:18.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:18 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2456459068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:12:18.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:18 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:12:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:18 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2456459068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:12:18.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:18 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:12:18.541 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:18.571 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:18.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:18.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:18 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2456459068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:12:18.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:18 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:12:18.672 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:18.701 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:18.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:18.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:18.804 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:18.830 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:18.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:18.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:18.932 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:18.972 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:18.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:19.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:19.112 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:19.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:19.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:19.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T22:12:19.214 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:19.243 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:19.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:19.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:19.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:19 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:12:19.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:19 smithi045 ceph-mon[30178]: pgmap v136: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 958 op/s 2024-03-11T22:12:19.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:19 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:12:19.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:19 smithi045 ceph-mon[30178]: osdmap e128: 8 total, 8 up, 8 in 2024-03-11T22:12:19.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:19 smithi045 ceph-mon[30178]: 3.14 scrub starts 2024-03-11T22:12:19.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:19 smithi045 ceph-mon[30178]: 3.14 scrub ok 2024-03-11T22:12:19.382 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:19 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:12:19.382 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:19 smithi045 ceph-mon[39715]: pgmap v136: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 958 op/s 2024-03-11T22:12:19.382 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:19 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:12:19.382 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:19 smithi045 ceph-mon[39715]: osdmap e128: 8 total, 8 up, 8 in 2024-03-11T22:12:19.382 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:19 smithi045 ceph-mon[39715]: 3.14 scrub starts 2024-03-11T22:12:19.382 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:19 smithi045 ceph-mon[39715]: 3.14 scrub ok 2024-03-11T22:12:19.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:19.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:19.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:19.494 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:19.525 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:19.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:19.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:19.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:19 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:12:19.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:19 smithi144 ceph-mon[33984]: pgmap v136: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 958 op/s 2024-03-11T22:12:19.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:19 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:12:19.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:19 smithi144 ceph-mon[33984]: osdmap e128: 8 total, 8 up, 8 in 2024-03-11T22:12:19.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:19 smithi144 ceph-mon[33984]: 3.14 scrub starts 2024-03-11T22:12:19.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:19 smithi144 ceph-mon[33984]: 3.14 scrub ok 2024-03-11T22:12:19.627 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:19.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:19.774 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:19.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:19.807 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:19.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:19.908 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:19.936 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:19.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:20.038 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:20.067 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:20.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:20.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:20.180 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:20.243 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:20.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:20.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:20 smithi045 ceph-mon[30178]: 5.c scrub starts 2024-03-11T22:12:20.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:20 smithi045 ceph-mon[30178]: 5.c scrub ok 2024-03-11T22:12:20.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:20 smithi045 ceph-mon[30178]: 5.1b scrub starts 2024-03-11T22:12:20.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:20 smithi045 ceph-mon[30178]: 5.1b scrub ok 2024-03-11T22:12:20.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:20 smithi045 ceph-mon[39715]: 5.c scrub starts 2024-03-11T22:12:20.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:20 smithi045 ceph-mon[39715]: 5.c scrub ok 2024-03-11T22:12:20.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:20 smithi045 ceph-mon[39715]: 5.1b scrub starts 2024-03-11T22:12:20.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:20 smithi045 ceph-mon[39715]: 5.1b scrub ok 2024-03-11T22:12:20.344 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:20.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:20.374 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:20.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:20.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:20.522 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:20.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:20.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:20 smithi144 ceph-mon[33984]: 5.c scrub starts 2024-03-11T22:12:20.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:20 smithi144 ceph-mon[33984]: 5.c scrub ok 2024-03-11T22:12:20.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:20 smithi144 ceph-mon[33984]: 5.1b scrub starts 2024-03-11T22:12:20.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:20 smithi144 ceph-mon[33984]: 5.1b scrub ok 2024-03-11T22:12:20.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:20.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:20.672 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:20.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:20.773 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:20.803 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:20.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:20.824 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:20.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:20.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:20.951 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:20.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:21.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:21.092 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:21.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:21.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:21.146 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T22:12:21.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:21.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[39715]: 3.1 scrub starts 2024-03-11T22:12:21.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[39715]: 3.1 scrub ok 2024-03-11T22:12:21.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[39715]: pgmap v138: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 1.01k op/s 2024-03-11T22:12:21.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[39715]: 4.14 scrub starts 2024-03-11T22:12:21.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[39715]: 4.14 scrub ok 2024-03-11T22:12:21.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[39715]: 5.f scrub starts 2024-03-11T22:12:21.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[39715]: 5.f scrub ok 2024-03-11T22:12:21.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2379970517' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:12:21.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:12:21.230 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[30178]: 3.1 scrub starts 2024-03-11T22:12:21.230 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[30178]: 3.1 scrub ok 2024-03-11T22:12:21.230 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[30178]: pgmap v138: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 1.01k op/s 2024-03-11T22:12:21.230 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[30178]: 4.14 scrub starts 2024-03-11T22:12:21.230 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[30178]: 4.14 scrub ok 2024-03-11T22:12:21.230 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[30178]: 5.f scrub starts 2024-03-11T22:12:21.230 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[30178]: 5.f scrub ok 2024-03-11T22:12:21.230 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2379970517' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:12:21.230 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:21 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:12:21.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:21.247 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:21.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:21.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:21.395 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:21.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:21.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:21.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:21.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:21.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:21.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:21 smithi144 ceph-mon[33984]: 3.1 scrub starts 2024-03-11T22:12:21.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:21 smithi144 ceph-mon[33984]: 3.1 scrub ok 2024-03-11T22:12:21.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:21 smithi144 ceph-mon[33984]: pgmap v138: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 51 MiB/s wr, 1.01k op/s 2024-03-11T22:12:21.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:21 smithi144 ceph-mon[33984]: 4.14 scrub starts 2024-03-11T22:12:21.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:21 smithi144 ceph-mon[33984]: 4.14 scrub ok 2024-03-11T22:12:21.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:21 smithi144 ceph-mon[33984]: 5.f scrub starts 2024-03-11T22:12:21.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:21 smithi144 ceph-mon[33984]: 5.f scrub ok 2024-03-11T22:12:21.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:21 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2379970517' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:12:21.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:21 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:12:21.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:21.674 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:21.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:21.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:21.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:21.815 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:21.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:21.917 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:21.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:12:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:21.945 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:21.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:22.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:22.093 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:22.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:22.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:22.241 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:22.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:22.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:22 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:12:22.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:22 smithi045 ceph-mon[39715]: osdmap e129: 8 total, 8 up, 8 in 2024-03-11T22:12:22.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:22 smithi045 ceph-mon[39715]: 4.17 scrub starts 2024-03-11T22:12:22.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:22 smithi045 ceph-mon[39715]: 4.17 scrub ok 2024-03-11T22:12:22.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:22 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:12:22.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:22 smithi045 ceph-mon[30178]: osdmap e129: 8 total, 8 up, 8 in 2024-03-11T22:12:22.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:22 smithi045 ceph-mon[30178]: 4.17 scrub starts 2024-03-11T22:12:22.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:22 smithi045 ceph-mon[30178]: 4.17 scrub ok 2024-03-11T22:12:22.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:22.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T22:12:22.406 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:22.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:22.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:22.508 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:22.538 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:22.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:22.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:22 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:12:22.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:22 smithi144 ceph-mon[33984]: osdmap e129: 8 total, 8 up, 8 in 2024-03-11T22:12:22.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:22 smithi144 ceph-mon[33984]: 4.17 scrub starts 2024-03-11T22:12:22.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:22 smithi144 ceph-mon[33984]: 4.17 scrub ok 2024-03-11T22:12:22.640 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:22.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:22.668 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:22.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:22.769 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:22.800 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:22.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:22.846 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:12:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:12:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:12:22.901 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:22.931 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:22.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:23.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:23.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:23.076 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:23.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:23.177 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:23.206 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:23.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:23.308 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:23.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:23.336 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:23.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:23.437 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:23.468 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:23.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:23.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:23.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:23 smithi045 ceph-mon[39715]: pgmap v140: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 1.25k op/s 2024-03-11T22:12:23.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:23 smithi045 ceph-mon[30178]: pgmap v140: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 1.25k op/s 2024-03-11T22:12:23.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:23.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:23 smithi144 ceph-mon[33984]: pgmap v140: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 1.25k op/s 2024-03-11T22:12:23.612 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:23.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:23.714 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:23.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:23.783 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:23.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:23.883 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:23.912 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:23.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:24.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:24.061 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:24.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:24.162 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:24.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:24.201 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:24.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:24.302 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:24.332 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:24.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:24.433 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:24.457 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:24.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:24.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:24 smithi045 ceph-mon[39715]: 7.8 scrub starts 2024-03-11T22:12:24.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:24 smithi045 ceph-mon[39715]: 3.1b scrub starts 2024-03-11T22:12:24.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:24 smithi045 ceph-mon[39715]: 3.1b scrub ok 2024-03-11T22:12:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:24 smithi045 ceph-mon[39715]: 7.b scrub starts 2024-03-11T22:12:24.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:24 smithi045 ceph-mon[39715]: 7.0 scrub starts 2024-03-11T22:12:24.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:24 smithi045 ceph-mon[30178]: 7.8 scrub starts 2024-03-11T22:12:24.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:24 smithi045 ceph-mon[30178]: 3.1b scrub starts 2024-03-11T22:12:24.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:24 smithi045 ceph-mon[30178]: 3.1b scrub ok 2024-03-11T22:12:24.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:24 smithi045 ceph-mon[30178]: 7.b scrub starts 2024-03-11T22:12:24.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:24 smithi045 ceph-mon[30178]: 7.0 scrub starts 2024-03-11T22:12:24.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:24.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:24 smithi144 ceph-mon[33984]: 7.8 scrub starts 2024-03-11T22:12:24.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:24 smithi144 ceph-mon[33984]: 3.1b scrub starts 2024-03-11T22:12:24.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:24 smithi144 ceph-mon[33984]: 3.1b scrub ok 2024-03-11T22:12:24.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:24 smithi144 ceph-mon[33984]: 7.b scrub starts 2024-03-11T22:12:24.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:24 smithi144 ceph-mon[33984]: 7.0 scrub starts 2024-03-11T22:12:24.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:24.605 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:24.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:24.707 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:24.741 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:24.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:24.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:24.843 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:24.874 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:24.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:24.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:24.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:25.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:25.108 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:25.144 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:25.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:25.245 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:25.273 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:25.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:25.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:25.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:25.405 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[39715]: 4.e scrub starts 2024-03-11T22:12:25.405 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[39715]: 4.e scrub ok 2024-03-11T22:12:25.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[39715]: pgmap v141: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 617 op/s 2024-03-11T22:12:25.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[39715]: 3.8 scrub starts 2024-03-11T22:12:25.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[39715]: 3.8 scrub ok 2024-03-11T22:12:25.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[39715]: 3.1c scrub starts 2024-03-11T22:12:25.406 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[39715]: 3.1c scrub ok 2024-03-11T22:12:25.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[30178]: 4.e scrub starts 2024-03-11T22:12:25.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[30178]: 4.e scrub ok 2024-03-11T22:12:25.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[30178]: pgmap v141: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 617 op/s 2024-03-11T22:12:25.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[30178]: 3.8 scrub starts 2024-03-11T22:12:25.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[30178]: 3.8 scrub ok 2024-03-11T22:12:25.406 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[30178]: 3.1c scrub starts 2024-03-11T22:12:25.407 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:25 smithi045 ceph-mon[30178]: 3.1c scrub ok 2024-03-11T22:12:25.417 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:25.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:25.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:25.509 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:12:25.509 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:12:25.509 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:12:25.520 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:25.527 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:25.544 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:12:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:25.578 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:25.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:25.583 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:12:25.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:25 smithi144 ceph-mon[33984]: 4.e scrub starts 2024-03-11T22:12:25.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:25 smithi144 ceph-mon[33984]: 4.e scrub ok 2024-03-11T22:12:25.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:25 smithi144 ceph-mon[33984]: pgmap v141: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 617 op/s 2024-03-11T22:12:25.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:25 smithi144 ceph-mon[33984]: 3.8 scrub starts 2024-03-11T22:12:25.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:25 smithi144 ceph-mon[33984]: 3.8 scrub ok 2024-03-11T22:12:25.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:25 smithi144 ceph-mon[33984]: 3.1c scrub starts 2024-03-11T22:12:25.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:25 smithi144 ceph-mon[33984]: 3.1c scrub ok 2024-03-11T22:12:25.627 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:12:25.672 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:12:25.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:25.720 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:12:25.730 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:12:25.731 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:12:25.731 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:12:25.731 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:12:25.731 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:12:25.731 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:12:25.731 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:12:25.731 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:12:25.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:25.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:25.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:25.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:25.842 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:12:25.842 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:12:25.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.842 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:130338 7.3 7:c0ffd7ee:::benchmark_data_smithi045_83007_object11867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-11T22:12:25.842 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:25.841623+0000", 2024-03-11T22:12:25.842 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0030122819999999998, 2024-03-11T22:12:25.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003042473, 2024-03-11T22:12:25.842 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:25.842 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:12:25.842 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:25.842 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130338 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.841623+0000", 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.841623+0000", 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.844 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.844 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:25.844 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.841624+0000", 2024-03-11T22:12:25.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1850000000000001e-06 2024-03-11T22:12:25.844 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.844 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.844 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:25.844 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.841629+0000", 2024-03-11T22:12:25.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9189999999999998e-06 2024-03-11T22:12:25.844 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.844 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.844 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:25.844 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.841642+0000", 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2782999999999999e-05 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.841651+0000", 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0189999999999995e-06 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.843691+0000", 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002040324 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:25.846 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.843816+0000", 2024-03-11T22:12:25.846 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012526900000000001 2024-03-11T22:12:25.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,4", 2024-03-11T22:12:25.846 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.844666+0000", 2024-03-11T22:12:25.846 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000849535 2024-03-11T22:12:25.846 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.846 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:25.846 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.14790.0:130335 7.1 e129/99 7:8e4b5cc3:::benchmark_data_smithi045_83007_object11864:head v 129'8122, mlcod=129'8119)", 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:25.842373+0000", 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.002261903, 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023858130000000001, 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.842373+0000", 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:25.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.842373+0000", 2024-03-11T22:12:25.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:25.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.842374+0000", 2024-03-11T22:12:25.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1500000000000004e-07 2024-03-11T22:12:25.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:25.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.842380+0000", 2024-03-11T22:12:25.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5030000000000002e-06 2024-03-11T22:12:25.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:25.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.842393+0000", 2024-03-11T22:12:25.849 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2602000000000001e-05 2024-03-11T22:12:25.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:25.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.842400+0000", 2024-03-11T22:12:25.849 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6540000000000002e-06 2024-03-11T22:12:25.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:25.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.843129+0000", 2024-03-11T22:12:25.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00072888100000000002 2024-03-11T22:12:25.853 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.853 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:25.853 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.843146+0000", 2024-03-11T22:12:25.853 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.791e-05 2024-03-11T22:12:25.853 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.853 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:25.853 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.853 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.853 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.14790.0:130336 7.1 e129/99)", 2024-03-11T22:12:25.854 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:25.842412+0000", 2024-03-11T22:12:25.854 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0022229419999999999, 2024-03-11T22:12:25.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024120320000000002, 2024-03-11T22:12:25.854 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:25.854 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:12:25.854 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:25.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.854 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:25.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.842412+0000", 2024-03-11T22:12:25.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.842412+0000", 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.842413+0000", 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3000000000000002e-07 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.842416+0000", 2024-03-11T22:12:25.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6459999999999999e-06 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.842422+0000", 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7339999999999996e-06 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.842426+0000", 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3989999999999997e-06 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:130341 7.a 7:58ba5c2f:::benchmark_data_smithi045_83007_object11870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:25.844318+0000", 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.000317103, 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00057232300000000002, 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130341 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.857 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:25.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.844318+0000", 2024-03-11T22:12:25.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:25.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.844318+0000", 2024-03-11T22:12:25.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:25.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.844319+0000", 2024-03-11T22:12:25.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.446e-06 2024-03-11T22:12:25.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.859 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:25.859 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.844324+0000", 2024-03-11T22:12:25.859 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9259999999999999e-06 2024-03-11T22:12:25.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.859 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:25.859 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.844337+0000", 2024-03-11T22:12:25.859 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2499e-05 2024-03-11T22:12:25.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.859 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:25.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.844346+0000", 2024-03-11T22:12:25.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9560000000000003e-06 2024-03-11T22:12:25.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.860 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:25.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.844468+0000", 2024-03-11T22:12:25.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000121648 2024-03-11T22:12:25.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.861 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:25.861 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.844577+0000", 2024-03-11T22:12:25.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000109471 2024-03-11T22:12:25.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.861 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:12:25.861 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.844687+0000", 2024-03-11T22:12:25.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110289 2024-03-11T22:12:25.867 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.867 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:130327 7.3 7:cef679e3:::benchmark_data_smithi045_83007_object11856:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:25.836098+0000", 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0085370410000000004, 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0088749060000000001, 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130327 2024-03-11T22:12:25.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.836098+0000", 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.836098+0000", 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.836100+0000", 2024-03-11T22:12:25.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6330000000000001e-06 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.836104+0000", 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.673e-06 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.836114+0000", 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0009e-05 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:25.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.836121+0000", 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5300000000000002e-06 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837178+0000", 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001057468 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837441+0000", 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000262594 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,4", 2024-03-11T22:12:25.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837569+0000", 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000127508 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.843591+0000", 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060224359999999999 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.843649+0000", 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8242999999999998e-05 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:130333 7.a 7:52dd11cf:::benchmark_data_smithi045_83007_object11862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:25.837056+0000", 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0075793249999999996, 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0080047009999999995, 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130333 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.873 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837056+0000", 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837056+0000", 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837062+0000", 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4250000000000003e-06 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:25.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837087+0000", 2024-03-11T22:12:25.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4824000000000002e-05 2024-03-11T22:12:25.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:25.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837104+0000", 2024-03-11T22:12:25.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6918999999999998e-05 2024-03-11T22:12:25.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:25.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837112+0000", 2024-03-11T22:12:25.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5909999999999998e-06 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.840292+0000", 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031808130000000002 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.840419+0000", 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012617600000000001 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.840542+0000", 2024-03-11T22:12:25.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012342599999999999 2024-03-11T22:12:25.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:25.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.844036+0000", 2024-03-11T22:12:25.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0034937789999999998 2024-03-11T22:12:25.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:12:25.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.844444+0000", 2024-03-11T22:12:25.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000408155 2024-03-11T22:12:25.878 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.878 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:25.878 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.14790.0:130334 7.1 e129/99 7:8e418d57:::benchmark_data_smithi045_83007_object11863:head v 129'8121, mlcod=129'8119)", 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:25.837584+0000", 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0070509170000000003, 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0075629019999999998, 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837584+0000", 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837584+0000", 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837585+0000", 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9999999999999998e-07 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837590+0000", 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3410000000000002e-06 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837599+0000", 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5599999999999994e-06 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.837605+0000", 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2589999999999997e-06 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.840466+0000", 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028606809999999999 2024-03-11T22:12:25.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.840487+0000", 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1021000000000001e-05 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:130337 7.3 7:c307e627:::benchmark_data_smithi045_83007_object11866:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:25.838449+0000", 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0061864169999999996, 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0067636140000000003, 2024-03-11T22:12:25.882 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:25.883 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:12:25.883 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:25.883 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:25.883 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:25.883 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 130337 2024-03-11T22:12:25.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.883 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:25.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.838449+0000", 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.838449+0000", 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.838450+0000", 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0979999999999999e-06 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.838455+0000", 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8919999999999999e-06 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.838466+0000", 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1257000000000001e-05 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.838474+0000", 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2509999999999994e-06 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.841466+0000", 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029913779999999998 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.841556+0000", 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0676999999999999e-05 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,4", 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:25.841651+0000", 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4939999999999996e-05 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.886 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:25.887 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.887 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:25.887 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:12:25.887 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 8 2024-03-11T22:12:25.887 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:12:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:25.888 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:25.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:25.989 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:26.018 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:26.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:26.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:26.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:12:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:12:26.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:26.159 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:26.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:26.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:26.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:26.293 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:26.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:26.394 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:26.431 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:26.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:26.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:26 smithi045 ceph-mon[30178]: 3.19 scrub starts 2024-03-11T22:12:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:26 smithi045 ceph-mon[30178]: 3.19 scrub ok 2024-03-11T22:12:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:26 smithi045 ceph-mon[30178]: 7.8 scrub ok 2024-03-11T22:12:26.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:26 smithi045 ceph-mon[30178]: 7.0 scrub ok 2024-03-11T22:12:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:26 smithi045 ceph-mon[30178]: 7.b scrub ok 2024-03-11T22:12:26.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:26 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3595973481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:12:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:26 smithi045 ceph-mon[39715]: 3.19 scrub starts 2024-03-11T22:12:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:26 smithi045 ceph-mon[39715]: 3.19 scrub ok 2024-03-11T22:12:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:26 smithi045 ceph-mon[39715]: 7.8 scrub ok 2024-03-11T22:12:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:26 smithi045 ceph-mon[39715]: 7.0 scrub ok 2024-03-11T22:12:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:26 smithi045 ceph-mon[39715]: 7.b scrub ok 2024-03-11T22:12:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:26 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3595973481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:12:26.534 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:26.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:26.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:26.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:26.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:26 smithi144 ceph-mon[33984]: 3.19 scrub starts 2024-03-11T22:12:26.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:26 smithi144 ceph-mon[33984]: 3.19 scrub ok 2024-03-11T22:12:26.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:26 smithi144 ceph-mon[33984]: 7.8 scrub ok 2024-03-11T22:12:26.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:26 smithi144 ceph-mon[33984]: 7.0 scrub ok 2024-03-11T22:12:26.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:26 smithi144 ceph-mon[33984]: 7.b scrub ok 2024-03-11T22:12:26.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:26 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3595973481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:12:26.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:26.724 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:26.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:26.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:26.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:26.869 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:26.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:26.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:27.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:27.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:27.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:27.112 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:27.144 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:27.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:27.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:27.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:27.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:27.273 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:27.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:27.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:27.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:27 smithi045 ceph-mon[39715]: pgmap v142: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-11T22:12:27.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:27.277 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:27 smithi045 ceph-mon[30178]: pgmap v142: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-11T22:12:27.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:27.410 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:27.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:27.514 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:27.541 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:27.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:27.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:27.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:27 smithi144 ceph-mon[33984]: pgmap v142: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-11T22:12:27.644 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:27.672 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:27.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:27.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:27.746 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:27.774 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:27.800 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:27.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:27.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:27.902 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:27.932 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:27.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:28.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:28.062 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:28.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:28.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:28.204 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:28.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:28.306 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:28.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:28.336 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:28.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:28.437 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:28.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:28.467 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:28.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:28 smithi045 ceph-mon[39715]: 5.7 scrub starts 2024-03-11T22:12:28.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:28 smithi045 ceph-mon[39715]: 5.7 scrub ok 2024-03-11T22:12:28.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:28 smithi045 ceph-mon[30178]: 5.7 scrub starts 2024-03-11T22:12:28.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:28 smithi045 ceph-mon[30178]: 5.7 scrub ok 2024-03-11T22:12:28.568 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:28.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:28 smithi144 ceph-mon[33984]: 5.7 scrub starts 2024-03-11T22:12:28.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:28 smithi144 ceph-mon[33984]: 5.7 scrub ok 2024-03-11T22:12:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:28.600 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:28.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:28.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T22:12:28.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:28.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:28.760 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:28.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:28.861 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:28.889 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:28.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:28.991 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:29.021 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:29.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:29.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:29.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:29.180 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:29.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:29.281 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:29.311 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:29.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:29.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:29.413 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:29.443 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:29.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:29.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:29 smithi045 ceph-mon[39715]: 4.1c scrub starts 2024-03-11T22:12:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:29 smithi045 ceph-mon[39715]: 4.1c scrub ok 2024-03-11T22:12:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:29 smithi045 ceph-mon[39715]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 968 op/s 2024-03-11T22:12:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:29 smithi045 ceph-mon[39715]: 4.1d scrub starts 2024-03-11T22:12:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:29 smithi045 ceph-mon[39715]: 4.1d scrub ok 2024-03-11T22:12:29.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:29 smithi045 ceph-mon[39715]: 7.7 scrub starts 2024-03-11T22:12:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:29 smithi045 ceph-mon[30178]: 4.1c scrub starts 2024-03-11T22:12:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:29 smithi045 ceph-mon[30178]: 4.1c scrub ok 2024-03-11T22:12:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:29 smithi045 ceph-mon[30178]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 968 op/s 2024-03-11T22:12:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:29 smithi045 ceph-mon[30178]: 4.1d scrub starts 2024-03-11T22:12:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:29 smithi045 ceph-mon[30178]: 4.1d scrub ok 2024-03-11T22:12:29.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:29 smithi045 ceph-mon[30178]: 7.7 scrub starts 2024-03-11T22:12:29.546 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:29.575 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:29.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:29.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:29.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:29 smithi144 ceph-mon[33984]: 4.1c scrub starts 2024-03-11T22:12:29.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:29 smithi144 ceph-mon[33984]: 4.1c scrub ok 2024-03-11T22:12:29.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:29 smithi144 ceph-mon[33984]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 968 op/s 2024-03-11T22:12:29.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:29 smithi144 ceph-mon[33984]: 4.1d scrub starts 2024-03-11T22:12:29.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:29 smithi144 ceph-mon[33984]: 4.1d scrub ok 2024-03-11T22:12:29.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:29 smithi144 ceph-mon[33984]: 7.7 scrub starts 2024-03-11T22:12:29.677 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:29.719 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:29.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:29.821 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:29.851 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:29.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:29.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:29.953 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:29.985 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:29.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:30.087 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:30.122 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:30.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:30.224 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:30.279 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:30.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:30.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:30.330 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:12:30.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:30 smithi045 ceph-mon[30178]: 6.9 scrub starts 2024-03-11T22:12:30.377 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:30 smithi045 ceph-mon[30178]: 6.9 scrub ok 2024-03-11T22:12:30.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:30 smithi045 ceph-mon[30178]: 3.e scrub starts 2024-03-11T22:12:30.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:30 smithi045 ceph-mon[30178]: 3.e scrub ok 2024-03-11T22:12:30.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:12:30.378 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:30 smithi045 ceph-mon[30178]: 7.c scrub starts 2024-03-11T22:12:30.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:30 smithi045 ceph-mon[39715]: 6.9 scrub starts 2024-03-11T22:12:30.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:30 smithi045 ceph-mon[39715]: 6.9 scrub ok 2024-03-11T22:12:30.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:30 smithi045 ceph-mon[39715]: 3.e scrub starts 2024-03-11T22:12:30.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:30 smithi045 ceph-mon[39715]: 3.e scrub ok 2024-03-11T22:12:30.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:12:30.378 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:30 smithi045 ceph-mon[39715]: 7.c scrub starts 2024-03-11T22:12:30.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:30.430 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:30.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:30.534 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:30.564 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:30.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:30.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:30 smithi144 ceph-mon[33984]: 6.9 scrub starts 2024-03-11T22:12:30.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:30 smithi144 ceph-mon[33984]: 6.9 scrub ok 2024-03-11T22:12:30.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:30 smithi144 ceph-mon[33984]: 3.e scrub starts 2024-03-11T22:12:30.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:30 smithi144 ceph-mon[33984]: 3.e scrub ok 2024-03-11T22:12:30.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:12:30.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:30 smithi144 ceph-mon[33984]: 7.c scrub starts 2024-03-11T22:12:30.669 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:30.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:30.704 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:30.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:30.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:30.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:30.856 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:30.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:30.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:30.999 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:31.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:31.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:31.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:31.142 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:31.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:31.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:31.244 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:31.274 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:31.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:31.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:31.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:31.418 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:31.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:31.519 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:31.527 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[30178]: 3.12 scrub starts 2024-03-11T22:12:31.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[30178]: 3.12 scrub ok 2024-03-11T22:12:31.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:12:31.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:12:31.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[30178]: pgmap v144: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 966 op/s 2024-03-11T22:12:31.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[30178]: osdmap e130: 8 total, 8 up, 8 in 2024-03-11T22:12:31.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[30178]: 7.7 scrub ok 2024-03-11T22:12:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[39715]: 3.12 scrub starts 2024-03-11T22:12:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[39715]: 3.12 scrub ok 2024-03-11T22:12:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:12:31.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:12:31.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[39715]: pgmap v144: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 966 op/s 2024-03-11T22:12:31.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[39715]: osdmap e130: 8 total, 8 up, 8 in 2024-03-11T22:12:31.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:31 smithi045 ceph-mon[39715]: 7.7 scrub ok 2024-03-11T22:12:31.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:31.549 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:31.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:31.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:31 smithi144 ceph-mon[33984]: 3.12 scrub starts 2024-03-11T22:12:31.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:31 smithi144 ceph-mon[33984]: 3.12 scrub ok 2024-03-11T22:12:31.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:31 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:12:31.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:31 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:12:31.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:31 smithi144 ceph-mon[33984]: pgmap v144: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 966 op/s 2024-03-11T22:12:31.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:31 smithi144 ceph-mon[33984]: osdmap e130: 8 total, 8 up, 8 in 2024-03-11T22:12:31.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:31 smithi144 ceph-mon[33984]: 7.7 scrub ok 2024-03-11T22:12:31.650 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:31.678 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:31.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:31.780 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:31.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:31.804 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:31.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:31.905 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:31.935 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:31.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:32.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:32.066 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:32.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:32.167 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:32.198 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:32.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:32.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:32.300 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:32.330 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:32.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:32.432 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:32.462 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:32.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:32.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:32.499 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:32 smithi045 ceph-mon[30178]: 5.1d scrub starts 2024-03-11T22:12:32.499 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:32 smithi045 ceph-mon[30178]: 5.1d scrub ok 2024-03-11T22:12:32.499 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:32 smithi045 ceph-mon[30178]: 5.0 scrub starts 2024-03-11T22:12:32.499 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:32 smithi045 ceph-mon[30178]: 5.0 scrub ok 2024-03-11T22:12:32.499 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:32 smithi045 ceph-mon[30178]: 7.c scrub ok 2024-03-11T22:12:32.500 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:32 smithi045 ceph-mon[39715]: 5.1d scrub starts 2024-03-11T22:12:32.500 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:32 smithi045 ceph-mon[39715]: 5.1d scrub ok 2024-03-11T22:12:32.500 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:32 smithi045 ceph-mon[39715]: 5.0 scrub starts 2024-03-11T22:12:32.500 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:32 smithi045 ceph-mon[39715]: 5.0 scrub ok 2024-03-11T22:12:32.500 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:32 smithi045 ceph-mon[39715]: 7.c scrub ok 2024-03-11T22:12:32.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:32.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:32 smithi144 ceph-mon[33984]: 5.1d scrub starts 2024-03-11T22:12:32.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:32 smithi144 ceph-mon[33984]: 5.1d scrub ok 2024-03-11T22:12:32.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:32 smithi144 ceph-mon[33984]: 5.0 scrub starts 2024-03-11T22:12:32.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:32 smithi144 ceph-mon[33984]: 5.0 scrub ok 2024-03-11T22:12:32.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:32 smithi144 ceph-mon[33984]: 7.c scrub ok 2024-03-11T22:12:32.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:32.607 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:32.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:32.708 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:32.736 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:32.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:32.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:32.777 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:12:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:12:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:12:32.806 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:32.838 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:32.864 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:32.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:32.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:32.887 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:32.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:33.038 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:33.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:33.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:33.140 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:33.181 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:33.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:33.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:33.266 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T22:12:33.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:33.326 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:33.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:33.423 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[39715]: 5.6 scrub starts 2024-03-11T22:12:33.423 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[39715]: 5.6 scrub ok 2024-03-11T22:12:33.423 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[39715]: 6.1e scrub starts 2024-03-11T22:12:33.423 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[39715]: 6.1e scrub ok 2024-03-11T22:12:33.423 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[39715]: pgmap v146: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 993 op/s 2024-03-11T22:12:33.424 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[39715]: 7.2 scrub starts 2024-03-11T22:12:33.424 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/649023917' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:12:33.424 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:12:33.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[30178]: 5.6 scrub starts 2024-03-11T22:12:33.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[30178]: 5.6 scrub ok 2024-03-11T22:12:33.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[30178]: 6.1e scrub starts 2024-03-11T22:12:33.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[30178]: 6.1e scrub ok 2024-03-11T22:12:33.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[30178]: pgmap v146: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 993 op/s 2024-03-11T22:12:33.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[30178]: 7.2 scrub starts 2024-03-11T22:12:33.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/649023917' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:12:33.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:33 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:12:33.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:33.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:33.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:33.575 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:33.726 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:33 smithi144 ceph-mon[33984]: 5.6 scrub starts 2024-03-11T22:12:33.726 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:33 smithi144 ceph-mon[33984]: 5.6 scrub ok 2024-03-11T22:12:33.727 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:33 smithi144 ceph-mon[33984]: 6.1e scrub starts 2024-03-11T22:12:33.727 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:33 smithi144 ceph-mon[33984]: 6.1e scrub ok 2024-03-11T22:12:33.727 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:33 smithi144 ceph-mon[33984]: pgmap v146: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 993 op/s 2024-03-11T22:12:33.727 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:33 smithi144 ceph-mon[33984]: 7.2 scrub starts 2024-03-11T22:12:33.727 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:33 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/649023917' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:12:33.727 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:33 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:12:33.754 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:33.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:33.856 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:33.886 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:33.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:33.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:33.988 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:34.017 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:34.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:34.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:34.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:34.163 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:34.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:34.265 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:34.295 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:34.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:34.397 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:34.426 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:34.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:34.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:34 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:12:34.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:34 smithi045 ceph-mon[39715]: osdmap e131: 8 total, 8 up, 8 in 2024-03-11T22:12:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:34 smithi045 ceph-mon[39715]: 7.3 scrub starts 2024-03-11T22:12:34.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:34 smithi045 ceph-mon[39715]: 7.d scrub starts 2024-03-11T22:12:34.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:34 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:12:34.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:34 smithi045 ceph-mon[30178]: osdmap e131: 8 total, 8 up, 8 in 2024-03-11T22:12:34.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:34 smithi045 ceph-mon[30178]: 7.3 scrub starts 2024-03-11T22:12:34.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:34 smithi045 ceph-mon[30178]: 7.d scrub starts 2024-03-11T22:12:34.528 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:34.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:34.558 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:34.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:34.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:34 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:12:34.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:34 smithi144 ceph-mon[33984]: osdmap e131: 8 total, 8 up, 8 in 2024-03-11T22:12:34.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:34 smithi144 ceph-mon[33984]: 7.3 scrub starts 2024-03-11T22:12:34.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:34 smithi144 ceph-mon[33984]: 7.d scrub starts 2024-03-11T22:12:34.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:34.698 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:34.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:34.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:34.800 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:34.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:34.828 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:34.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:34.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:34.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:34.974 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:34.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:35.075 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:35.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:35.105 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:35.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:35.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:35.253 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:35.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:35.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:35.355 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:35.381 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:35.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:35.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:35.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T22:12:35.482 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:35.511 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:35.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:35.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:35 smithi045 ceph-mon[39715]: 5.4 scrub starts 2024-03-11T22:12:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:35 smithi045 ceph-mon[39715]: 5.4 scrub ok 2024-03-11T22:12:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:35 smithi045 ceph-mon[39715]: pgmap v148: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 650 op/s 2024-03-11T22:12:35.527 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:35 smithi045 ceph-mon[39715]: 3.c scrub starts 2024-03-11T22:12:35.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:35 smithi045 ceph-mon[39715]: 3.c scrub ok 2024-03-11T22:12:35.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:35 smithi045 ceph-mon[30178]: 5.4 scrub starts 2024-03-11T22:12:35.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:35 smithi045 ceph-mon[30178]: 5.4 scrub ok 2024-03-11T22:12:35.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:35 smithi045 ceph-mon[30178]: pgmap v148: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 650 op/s 2024-03-11T22:12:35.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:35 smithi045 ceph-mon[30178]: 3.c scrub starts 2024-03-11T22:12:35.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:35 smithi045 ceph-mon[30178]: 3.c scrub ok 2024-03-11T22:12:35.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:35 smithi144 ceph-mon[33984]: 5.4 scrub starts 2024-03-11T22:12:35.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:35 smithi144 ceph-mon[33984]: 5.4 scrub ok 2024-03-11T22:12:35.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:35 smithi144 ceph-mon[33984]: pgmap v148: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 650 op/s 2024-03-11T22:12:35.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:35 smithi144 ceph-mon[33984]: 3.c scrub starts 2024-03-11T22:12:35.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:35 smithi144 ceph-mon[33984]: 3.c scrub ok 2024-03-11T22:12:35.613 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:35.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:35.660 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:35.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:35.761 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:35.813 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:35.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:35.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:35.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:35.982 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:35.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:36.083 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:36.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:12:35 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:12:36.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:36.138 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:36.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:36.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:36.305 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:36.306 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:12:36.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:36.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:36.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[30178]: 6.8 scrub starts 2024-03-11T22:12:36.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[30178]: 6.8 scrub ok 2024-03-11T22:12:36.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[30178]: 7.2 scrub ok 2024-03-11T22:12:36.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[30178]: 7.a scrub starts 2024-03-11T22:12:36.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[30178]: 7.d scrub ok 2024-03-11T22:12:36.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4251957242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:12:36.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:12:36.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[30178]: 5.1f scrub starts 2024-03-11T22:12:36.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[30178]: 5.1f scrub ok 2024-03-11T22:12:36.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[30178]: 7.3 scrub ok 2024-03-11T22:12:36.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[39715]: 6.8 scrub starts 2024-03-11T22:12:36.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[39715]: 6.8 scrub ok 2024-03-11T22:12:36.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[39715]: 7.2 scrub ok 2024-03-11T22:12:36.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[39715]: 7.a scrub starts 2024-03-11T22:12:36.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[39715]: 7.d scrub ok 2024-03-11T22:12:36.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4251957242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:12:36.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:12:36.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[39715]: 5.1f scrub starts 2024-03-11T22:12:36.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[39715]: 5.1f scrub ok 2024-03-11T22:12:36.400 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:36 smithi045 ceph-mon[39715]: 7.3 scrub ok 2024-03-11T22:12:36.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:36.453 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:36.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:36.557 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:36.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:36 smithi144 ceph-mon[33984]: 6.8 scrub starts 2024-03-11T22:12:36.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:36 smithi144 ceph-mon[33984]: 6.8 scrub ok 2024-03-11T22:12:36.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:36 smithi144 ceph-mon[33984]: 7.2 scrub ok 2024-03-11T22:12:36.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:36 smithi144 ceph-mon[33984]: 7.a scrub starts 2024-03-11T22:12:36.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:36 smithi144 ceph-mon[33984]: 7.d scrub ok 2024-03-11T22:12:36.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:36 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4251957242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:12:36.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:36 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:12:36.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:36 smithi144 ceph-mon[33984]: 5.1f scrub starts 2024-03-11T22:12:36.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:36 smithi144 ceph-mon[33984]: 5.1f scrub ok 2024-03-11T22:12:36.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:36 smithi144 ceph-mon[33984]: 7.3 scrub ok 2024-03-11T22:12:36.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:36.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:36.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:36.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:36.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:36.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:36.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:36.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:36.857 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:36.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:36.958 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:36.989 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:36.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:37.091 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:37.120 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:37.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:37.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:37.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:37.269 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:37.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:37.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:37.395 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:37.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:37.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:37.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:37.539 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:37.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:37.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:37.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:37 smithi144 ceph-mon[33984]: pgmap v149: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T22:12:37.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:37 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:12:37.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:37 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:12:37.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:37 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:12:37.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:37 smithi144 ceph-mon[33984]: osdmap e132: 8 total, 8 up, 8 in 2024-03-11T22:12:37.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:37 smithi144 ceph-mon[33984]: 6.b scrub starts 2024-03-11T22:12:37.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:37 smithi144 ceph-mon[33984]: 6.b scrub ok 2024-03-11T22:12:37.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:37 smithi144 ceph-mon[33984]: 4.a scrub starts 2024-03-11T22:12:37.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:37 smithi144 ceph-mon[33984]: 4.a scrub ok 2024-03-11T22:12:37.641 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:37.668 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:37.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:37.770 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[39715]: pgmap v149: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T22:12:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:12:37.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:12:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:12:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[39715]: osdmap e132: 8 total, 8 up, 8 in 2024-03-11T22:12:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[39715]: 6.b scrub starts 2024-03-11T22:12:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[39715]: 6.b scrub ok 2024-03-11T22:12:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[39715]: 4.a scrub starts 2024-03-11T22:12:37.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[39715]: 4.a scrub ok 2024-03-11T22:12:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[30178]: pgmap v149: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T22:12:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:12:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:12:37.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:12:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[30178]: osdmap e132: 8 total, 8 up, 8 in 2024-03-11T22:12:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[30178]: 6.b scrub starts 2024-03-11T22:12:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[30178]: 6.b scrub ok 2024-03-11T22:12:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[30178]: 4.a scrub starts 2024-03-11T22:12:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:37 smithi045 ceph-mon[30178]: 4.a scrub ok 2024-03-11T22:12:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:37.817 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:37.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:37.918 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:37.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:37.948 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:37.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:38.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:38.088 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:38.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:38.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:38.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:38.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:38.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:38.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:38.375 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:38.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:38.476 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:38.513 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:38.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:38.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:38 smithi144 ceph-mon[33984]: 7.a scrub ok 2024-03-11T22:12:38.614 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:38.663 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:38.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:38.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:38.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:38 smithi045 ceph-mon[39715]: 7.a scrub ok 2024-03-11T22:12:38.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:38 smithi045 ceph-mon[30178]: 7.a scrub ok 2024-03-11T22:12:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:38.795 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:38.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:38.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:38.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:38.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:38.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:39.043 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:39.073 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:39.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:39.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:39.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:39.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:39.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:39.243 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:12:39.290 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:12:39.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:39.328 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:12:39.363 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:39.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:39.371 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:12:39.412 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:12:39.448 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:12:39.466 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:39.492 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:12:39.496 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:39.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:39.536 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:12:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:39 smithi045 ceph-mon[39715]: pgmap v151: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T22:12:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:39 smithi045 ceph-mon[39715]: 6.a scrub starts 2024-03-11T22:12:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:39 smithi045 ceph-mon[39715]: 6.a scrub ok 2024-03-11T22:12:39.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:39 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3322928939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:12:39.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:39 smithi045 ceph-mon[30178]: pgmap v151: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T22:12:39.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:39 smithi045 ceph-mon[30178]: 6.a scrub starts 2024-03-11T22:12:39.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:39 smithi045 ceph-mon[30178]: 6.a scrub ok 2024-03-11T22:12:39.543 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:39 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3322928939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:12:39.547 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:12:39.547 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:12:39.547 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:12:39.547 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:12:39.547 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:12:39.547 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:12:39.547 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:12:39.547 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:12:39.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:39 smithi144 ceph-mon[33984]: pgmap v151: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T22:12:39.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:39 smithi144 ceph-mon[33984]: 6.a scrub starts 2024-03-11T22:12:39.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:39 smithi144 ceph-mon[33984]: 6.a scrub ok 2024-03-11T22:12:39.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:39 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3322928939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:12:39.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:39.666 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:39.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:39.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:39.769 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:39.798 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:39.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:39.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:39.900 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:39.932 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:39.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:40.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:40.091 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:40.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:40.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:40.193 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:40.222 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:40.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:40.325 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:40.356 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:40.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:40.426 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:40.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:40.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:40.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:40 smithi144 ceph-mon[33984]: 5.8 deep-scrub starts 2024-03-11T22:12:40.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:40 smithi144 ceph-mon[33984]: 5.8 deep-scrub ok 2024-03-11T22:12:40.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:40 smithi144 ceph-mon[33984]: 4.2 scrub starts 2024-03-11T22:12:40.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:40 smithi144 ceph-mon[33984]: 4.2 scrub ok 2024-03-11T22:12:40.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:40 smithi144 ceph-mon[33984]: 4.0 scrub starts 2024-03-11T22:12:40.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:40 smithi144 ceph-mon[33984]: 4.0 scrub ok 2024-03-11T22:12:40.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:40.650 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:40.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:40.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:40.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:40 smithi045 ceph-mon[30178]: 5.8 deep-scrub starts 2024-03-11T22:12:40.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:40 smithi045 ceph-mon[30178]: 5.8 deep-scrub ok 2024-03-11T22:12:40.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:40 smithi045 ceph-mon[30178]: 4.2 scrub starts 2024-03-11T22:12:40.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:40 smithi045 ceph-mon[30178]: 4.2 scrub ok 2024-03-11T22:12:40.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:40 smithi045 ceph-mon[30178]: 4.0 scrub starts 2024-03-11T22:12:40.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:40 smithi045 ceph-mon[30178]: 4.0 scrub ok 2024-03-11T22:12:40.671 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:40 smithi045 ceph-mon[39715]: 5.8 deep-scrub starts 2024-03-11T22:12:40.672 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:40 smithi045 ceph-mon[39715]: 5.8 deep-scrub ok 2024-03-11T22:12:40.672 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:40 smithi045 ceph-mon[39715]: 4.2 scrub starts 2024-03-11T22:12:40.672 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:40 smithi045 ceph-mon[39715]: 4.2 scrub ok 2024-03-11T22:12:40.672 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:40 smithi045 ceph-mon[39715]: 4.0 scrub starts 2024-03-11T22:12:40.672 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:40 smithi045 ceph-mon[39715]: 4.0 scrub ok 2024-03-11T22:12:40.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:40.800 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:40.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:40.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:40.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:40.945 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:40.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:40.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:41.047 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:41.078 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:41.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:41.180 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:41.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:41.211 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:41.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:41.312 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:41.341 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:41.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:41.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:41 smithi045 ceph-mon[30178]: pgmap v152: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 613 op/s 2024-03-11T22:12:41.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:41 smithi045 ceph-mon[30178]: 5.2 deep-scrub starts 2024-03-11T22:12:41.364 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:41 smithi045 ceph-mon[30178]: 5.2 deep-scrub ok 2024-03-11T22:12:41.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:41 smithi045 ceph-mon[39715]: pgmap v152: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 613 op/s 2024-03-11T22:12:41.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:41 smithi045 ceph-mon[39715]: 5.2 deep-scrub starts 2024-03-11T22:12:41.364 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:41 smithi045 ceph-mon[39715]: 5.2 deep-scrub ok 2024-03-11T22:12:41.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:41.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:41.490 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:41.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:41.583 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:12:41.583 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:12:41.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:41 smithi144 ceph-mon[33984]: pgmap v152: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 613 op/s 2024-03-11T22:12:41.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:41 smithi144 ceph-mon[33984]: 5.2 deep-scrub starts 2024-03-11T22:12:41.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:41 smithi144 ceph-mon[33984]: 5.2 deep-scrub ok 2024-03-11T22:12:41.591 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:41.592 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:12:41.620 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:41.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:41.722 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:41.749 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:41.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:41.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:41.851 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:41.879 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:41.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:41.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:41.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:42.025 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:42.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:42.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:42.127 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:42.156 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:42.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:42.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:42.303 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:42.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:42.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:42.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:12:42.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:42.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:42.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:42.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:42 smithi045 ceph-mon[39715]: 6.14 scrub starts 2024-03-11T22:12:42.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:42 smithi045 ceph-mon[39715]: 6.14 scrub ok 2024-03-11T22:12:42.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:42 smithi045 ceph-mon[30178]: 6.14 scrub starts 2024-03-11T22:12:42.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:42 smithi045 ceph-mon[30178]: 6.14 scrub ok 2024-03-11T22:12:42.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:42 smithi144 ceph-mon[33984]: 6.14 scrub starts 2024-03-11T22:12:42.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:42 smithi144 ceph-mon[33984]: 6.14 scrub ok 2024-03-11T22:12:42.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:42.595 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:42.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:42.696 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:42.724 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:42.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:42.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:42.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:42.872 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:42.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:42.973 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:43.017 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:43.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:43.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:43.027 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:12:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:12:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:12:43.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:43.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:43.159 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:43.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:43.260 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:43.291 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:43.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:43.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:43.432 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:43.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:43.533 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:43.563 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:43.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:43.579 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:43.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:43 smithi144 ceph-mon[33984]: 5.12 scrub starts 2024-03-11T22:12:43.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:43 smithi144 ceph-mon[33984]: 5.12 scrub ok 2024-03-11T22:12:43.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:43 smithi144 ceph-mon[33984]: pgmap v153: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T22:12:43.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:43 smithi144 ceph-mon[33984]: 6.d scrub starts 2024-03-11T22:12:43.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:43 smithi144 ceph-mon[33984]: 6.d scrub ok 2024-03-11T22:12:43.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:43 smithi144 ceph-mon[33984]: 3.a scrub starts 2024-03-11T22:12:43.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:43 smithi144 ceph-mon[33984]: 3.a scrub ok 2024-03-11T22:12:43.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[30178]: 5.12 scrub starts 2024-03-11T22:12:43.592 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[30178]: 5.12 scrub ok 2024-03-11T22:12:43.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[30178]: pgmap v153: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T22:12:43.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[30178]: 6.d scrub starts 2024-03-11T22:12:43.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[30178]: 6.d scrub ok 2024-03-11T22:12:43.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[30178]: 3.a scrub starts 2024-03-11T22:12:43.593 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[30178]: 3.a scrub ok 2024-03-11T22:12:43.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[39715]: 5.12 scrub starts 2024-03-11T22:12:43.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[39715]: 5.12 scrub ok 2024-03-11T22:12:43.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[39715]: pgmap v153: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T22:12:43.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[39715]: 6.d scrub starts 2024-03-11T22:12:43.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[39715]: 6.d scrub ok 2024-03-11T22:12:43.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[39715]: 3.a scrub starts 2024-03-11T22:12:43.593 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:43 smithi045 ceph-mon[39715]: 3.a scrub ok 2024-03-11T22:12:43.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T22:12:43.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:43.709 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:43.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:43.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:43.811 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:43.839 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:43.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:43.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:43.941 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:43.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:43.996 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:43.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:44.097 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:44.130 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:44.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:44.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:44.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:44.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:44.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:44.369 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:44.397 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:44.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:44.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:44 smithi045 ceph-mon[30178]: 3.4 scrub starts 2024-03-11T22:12:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:44 smithi045 ceph-mon[30178]: 3.4 scrub ok 2024-03-11T22:12:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:44 smithi045 ceph-mon[30178]: 3.18 scrub starts 2024-03-11T22:12:44.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:44 smithi045 ceph-mon[30178]: 3.18 scrub ok 2024-03-11T22:12:44.417 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:44 smithi045 ceph-mon[39715]: 3.4 scrub starts 2024-03-11T22:12:44.417 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:44 smithi045 ceph-mon[39715]: 3.4 scrub ok 2024-03-11T22:12:44.417 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:44 smithi045 ceph-mon[39715]: 3.18 scrub starts 2024-03-11T22:12:44.417 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:44 smithi045 ceph-mon[39715]: 3.18 scrub ok 2024-03-11T22:12:44.499 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:44.526 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:44.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:44.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:44.628 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:44.655 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:44.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:44.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:44.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:44.795 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:44.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:44.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:44 smithi144 ceph-mon[33984]: 3.4 scrub starts 2024-03-11T22:12:44.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:44 smithi144 ceph-mon[33984]: 3.4 scrub ok 2024-03-11T22:12:44.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:44 smithi144 ceph-mon[33984]: 3.18 scrub starts 2024-03-11T22:12:44.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:44 smithi144 ceph-mon[33984]: 3.18 scrub ok 2024-03-11T22:12:44.897 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:44.927 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:44.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:45.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:45.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:45.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:45.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:45.210 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:45.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:45.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:45.312 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:45.330 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:45.338 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:45.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:45.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:45.412 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T22:12:45.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:45.497 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:45.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:45.600 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:45.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[39715]: 5.d scrub starts 2024-03-11T22:12:45.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[39715]: 5.d scrub ok 2024-03-11T22:12:45.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[39715]: pgmap v154: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 924 op/s 2024-03-11T22:12:45.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:12:45.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:12:45.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[39715]: 6.1f scrub starts 2024-03-11T22:12:45.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[39715]: 6.1f scrub ok 2024-03-11T22:12:45.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[39715]: 3.d scrub starts 2024-03-11T22:12:45.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[39715]: 3.d scrub ok 2024-03-11T22:12:45.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[39715]: 5.11 scrub starts 2024-03-11T22:12:45.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[39715]: 5.11 scrub ok 2024-03-11T22:12:45.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1791105041' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:12:45.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[30178]: 5.d scrub starts 2024-03-11T22:12:45.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[30178]: 5.d scrub ok 2024-03-11T22:12:45.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[30178]: pgmap v154: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 924 op/s 2024-03-11T22:12:45.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:12:45.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:12:45.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[30178]: 6.1f scrub starts 2024-03-11T22:12:45.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[30178]: 6.1f scrub ok 2024-03-11T22:12:45.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[30178]: 3.d scrub starts 2024-03-11T22:12:45.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[30178]: 3.d scrub ok 2024-03-11T22:12:45.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[30178]: 5.11 scrub starts 2024-03-11T22:12:45.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[30178]: 5.11 scrub ok 2024-03-11T22:12:45.618 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:45 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1791105041' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:12:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:45.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:45.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:45.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:45.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:45.771 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:45.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:45.822 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:45 smithi144 ceph-mon[33984]: 5.d scrub starts 2024-03-11T22:12:45.822 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:45 smithi144 ceph-mon[33984]: 5.d scrub ok 2024-03-11T22:12:45.822 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:45 smithi144 ceph-mon[33984]: pgmap v154: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 924 op/s 2024-03-11T22:12:45.822 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:12:45.822 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:12:45.823 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:45 smithi144 ceph-mon[33984]: 6.1f scrub starts 2024-03-11T22:12:45.823 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:45 smithi144 ceph-mon[33984]: 6.1f scrub ok 2024-03-11T22:12:45.823 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:45 smithi144 ceph-mon[33984]: 3.d scrub starts 2024-03-11T22:12:45.823 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:45 smithi144 ceph-mon[33984]: 3.d scrub ok 2024-03-11T22:12:45.823 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:45 smithi144 ceph-mon[33984]: 5.11 scrub starts 2024-03-11T22:12:45.823 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:45 smithi144 ceph-mon[33984]: 5.11 scrub ok 2024-03-11T22:12:45.823 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:45 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1791105041' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:12:45.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:45.911 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:45.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:46.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:46.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:46.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:46.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:46.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:12:45 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:12:46.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:46.191 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:46.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:46.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:46.336 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:46.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:46.438 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:46.466 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:46.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:46.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:46.568 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:46.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:46.597 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:46.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:46.699 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:46.708 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[30178]: 6.4 scrub starts 2024-03-11T22:12:46.708 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[30178]: 6.4 scrub ok 2024-03-11T22:12:46.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1791105041' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:12:46.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[30178]: osdmap e133: 8 total, 8 up, 8 in 2024-03-11T22:12:46.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[30178]: 6.6 deep-scrub starts 2024-03-11T22:12:46.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[30178]: 6.6 deep-scrub ok 2024-03-11T22:12:46.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[30178]: 3.5 scrub starts 2024-03-11T22:12:46.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[30178]: 3.5 scrub ok 2024-03-11T22:12:46.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:12:46.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:12:46.709 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[39715]: 6.4 scrub starts 2024-03-11T22:12:46.709 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[39715]: 6.4 scrub ok 2024-03-11T22:12:46.709 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1791105041' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:12:46.709 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[39715]: osdmap e133: 8 total, 8 up, 8 in 2024-03-11T22:12:46.710 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[39715]: 6.6 deep-scrub starts 2024-03-11T22:12:46.710 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[39715]: 6.6 deep-scrub ok 2024-03-11T22:12:46.710 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[39715]: 3.5 scrub starts 2024-03-11T22:12:46.710 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[39715]: 3.5 scrub ok 2024-03-11T22:12:46.710 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:12:46.710 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:46 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:12:46.725 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:46.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:46.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:46.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:46 smithi144 ceph-mon[33984]: 6.4 scrub starts 2024-03-11T22:12:46.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:46 smithi144 ceph-mon[33984]: 6.4 scrub ok 2024-03-11T22:12:46.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:46 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1791105041' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:12:46.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:46 smithi144 ceph-mon[33984]: osdmap e133: 8 total, 8 up, 8 in 2024-03-11T22:12:46.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:46 smithi144 ceph-mon[33984]: 6.6 deep-scrub starts 2024-03-11T22:12:46.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:46 smithi144 ceph-mon[33984]: 6.6 deep-scrub ok 2024-03-11T22:12:46.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:46 smithi144 ceph-mon[33984]: 3.5 scrub starts 2024-03-11T22:12:46.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:46 smithi144 ceph-mon[33984]: 3.5 scrub ok 2024-03-11T22:12:46.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:46 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:12:46.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:46 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:12:46.871 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:46.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:46.973 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:47.012 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:47.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:47.113 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:47.139 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:47.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:47.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:47.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:47.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:47.392 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:47.419 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:47.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:47.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:47.559 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:47.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:47.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:47.702 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:47.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:47.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[39715]: 2.1 scrub starts 2024-03-11T22:12:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[39715]: 2.1 scrub ok 2024-03-11T22:12:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[39715]: 5.19 scrub starts 2024-03-11T22:12:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[39715]: 5.19 scrub ok 2024-03-11T22:12:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[39715]: pgmap v156: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 993 op/s 2024-03-11T22:12:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[39715]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T22:12:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[39715]: 6.11 scrub starts 2024-03-11T22:12:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[39715]: 6.11 scrub ok 2024-03-11T22:12:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[39715]: 6.10 scrub starts 2024-03-11T22:12:47.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[39715]: 6.10 scrub ok 2024-03-11T22:12:47.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[39715]: 6.3 scrub starts 2024-03-11T22:12:47.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[39715]: 6.3 scrub ok 2024-03-11T22:12:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[30178]: 2.1 scrub starts 2024-03-11T22:12:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[30178]: 2.1 scrub ok 2024-03-11T22:12:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[30178]: 5.19 scrub starts 2024-03-11T22:12:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[30178]: 5.19 scrub ok 2024-03-11T22:12:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[30178]: pgmap v156: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 993 op/s 2024-03-11T22:12:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[30178]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T22:12:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[30178]: 6.11 scrub starts 2024-03-11T22:12:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[30178]: 6.11 scrub ok 2024-03-11T22:12:47.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[30178]: 6.10 scrub starts 2024-03-11T22:12:47.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[30178]: 6.10 scrub ok 2024-03-11T22:12:47.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[30178]: 6.3 scrub starts 2024-03-11T22:12:47.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:47 smithi045 ceph-mon[30178]: 6.3 scrub ok 2024-03-11T22:12:47.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:47.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:47.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:47 smithi144 ceph-mon[33984]: 2.1 scrub starts 2024-03-11T22:12:47.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:47 smithi144 ceph-mon[33984]: 2.1 scrub ok 2024-03-11T22:12:47.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:47 smithi144 ceph-mon[33984]: 5.19 scrub starts 2024-03-11T22:12:47.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:47 smithi144 ceph-mon[33984]: 5.19 scrub ok 2024-03-11T22:12:47.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:47 smithi144 ceph-mon[33984]: pgmap v156: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 993 op/s 2024-03-11T22:12:47.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:47 smithi144 ceph-mon[33984]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T22:12:47.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:47 smithi144 ceph-mon[33984]: 6.11 scrub starts 2024-03-11T22:12:47.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:47 smithi144 ceph-mon[33984]: 6.11 scrub ok 2024-03-11T22:12:47.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:47 smithi144 ceph-mon[33984]: 6.10 scrub starts 2024-03-11T22:12:47.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:47 smithi144 ceph-mon[33984]: 6.10 scrub ok 2024-03-11T22:12:47.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:47 smithi144 ceph-mon[33984]: 6.3 scrub starts 2024-03-11T22:12:47.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:47 smithi144 ceph-mon[33984]: 6.3 scrub ok 2024-03-11T22:12:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:47.855 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:47.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:47.956 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:47.985 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:47.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:47.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:48.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:48.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:48.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:48.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:48.234 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:48.263 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:48.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:48.364 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:48.392 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:48.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:48.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:48.494 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:48.519 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:48.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:48.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:48.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:48.657 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:48.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:48.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:48.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:48 smithi045 ceph-mon[39715]: 3.1d scrub starts 2024-03-11T22:12:48.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:48 smithi045 ceph-mon[39715]: 3.1d scrub ok 2024-03-11T22:12:48.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:48 smithi045 ceph-mon[30178]: 3.1d scrub starts 2024-03-11T22:12:48.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:48 smithi045 ceph-mon[30178]: 3.1d scrub ok 2024-03-11T22:12:48.796 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:48.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:48.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:48.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:48 smithi144 ceph-mon[33984]: 3.1d scrub starts 2024-03-11T22:12:48.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:48 smithi144 ceph-mon[33984]: 3.1d scrub ok 2024-03-11T22:12:48.898 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:48.926 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:48.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:48.968 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T22:12:48.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:12:49.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:49.081 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:49.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:49.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:49.183 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:49.208 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:49.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:49.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:49.354 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:49.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:49.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:49.484 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:49.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:49.586 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:49.617 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:49.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:49.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:49.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:49.746 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:49 smithi045 ceph-mon[39715]: pgmap v158: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:12:49.746 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:49 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:12:49.746 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:49 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:12:49.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:49 smithi045 ceph-mon[30178]: pgmap v158: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:12:49.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:49 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:12:49.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:49 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:12:49.759 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:49.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:49.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:49.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:49 smithi144 ceph-mon[33984]: pgmap v158: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:12:49.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:49 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:12:49.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:49 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:12:49.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:49.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:49.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:49.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:50.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:50.060 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:50.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:50.161 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:50.188 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:50.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:50.290 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:50.317 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:50.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:50.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:50.419 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:50.448 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:50.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:50.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:50.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:50.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:50.591 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:50.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:50.608 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[30178]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T22:12:50.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[30178]: 4.19 scrub starts 2024-03-11T22:12:50.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[30178]: 4.19 scrub ok 2024-03-11T22:12:50.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:12:50.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:12:50.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[30178]: 3.6 scrub starts 2024-03-11T22:12:50.609 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[30178]: 3.6 scrub ok 2024-03-11T22:12:50.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[39715]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T22:12:50.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[39715]: 4.19 scrub starts 2024-03-11T22:12:50.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[39715]: 4.19 scrub ok 2024-03-11T22:12:50.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:12:50.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:12:50.609 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[39715]: 3.6 scrub starts 2024-03-11T22:12:50.610 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:50 smithi045 ceph-mon[39715]: 3.6 scrub ok 2024-03-11T22:12:50.610 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:50.618 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:50.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:50.743 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:12:50.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:50.754 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:50.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:50.784 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:12:50.836 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:12:50.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:50 smithi144 ceph-mon[33984]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T22:12:50.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:50 smithi144 ceph-mon[33984]: 4.19 scrub starts 2024-03-11T22:12:50.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:50 smithi144 ceph-mon[33984]: 4.19 scrub ok 2024-03-11T22:12:50.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:50 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:12:50.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:50 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:12:50.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:50 smithi144 ceph-mon[33984]: 3.6 scrub starts 2024-03-11T22:12:50.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:50 smithi144 ceph-mon[33984]: 3.6 scrub ok 2024-03-11T22:12:50.856 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:50.879 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:12:50.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:50.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:50.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:50.919 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:12:50.962 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:12:51.001 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:51.007 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:12:51.049 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:51.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:51.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:51.055 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:12:51.056 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:12:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:12:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:12:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:12:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:12:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:12:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:12:51.057 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:12:51.152 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:51.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:51.185 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:51.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:51.286 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:51.314 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:51.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:51.416 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:51.448 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:51.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:51.486 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:12:51.550 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:51.580 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:51.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:51.683 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:51.718 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:51.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:51.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:51.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[30178]: pgmap v160: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 911 op/s 2024-03-11T22:12:51.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[30178]: osdmap e136: 8 total, 8 up, 8 in 2024-03-11T22:12:51.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[30178]: 6.1b scrub starts 2024-03-11T22:12:51.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[30178]: 6.1b scrub ok 2024-03-11T22:12:51.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1284356307' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:12:51.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:12:51.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3611636061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:12:51.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:12:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[39715]: pgmap v160: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 911 op/s 2024-03-11T22:12:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[39715]: osdmap e136: 8 total, 8 up, 8 in 2024-03-11T22:12:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[39715]: 6.1b scrub starts 2024-03-11T22:12:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[39715]: 6.1b scrub ok 2024-03-11T22:12:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1284356307' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:12:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:12:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3611636061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:12:51.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:51 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:12:51.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:51.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:51 smithi144 ceph-mon[33984]: pgmap v160: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 911 op/s 2024-03-11T22:12:51.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:51 smithi144 ceph-mon[33984]: osdmap e136: 8 total, 8 up, 8 in 2024-03-11T22:12:51.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:51 smithi144 ceph-mon[33984]: 6.1b scrub starts 2024-03-11T22:12:51.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:51 smithi144 ceph-mon[33984]: 6.1b scrub ok 2024-03-11T22:12:51.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:51 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1284356307' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:12:51.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:51 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:12:51.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:51 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3611636061' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:12:51.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:51 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:12:51.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:51.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:51.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:52.020 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:52.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:52.122 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:52.153 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:52.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:52.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:52.295 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:52.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:52.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:52.397 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:52.428 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:52.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:52.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:52.530 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:52.587 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:52.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:52.688 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:52.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:52.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:52.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:52.735 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:12:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:12:52.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[39715]: 6.f scrub starts 2024-03-11T22:12:52.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[39715]: 6.f scrub ok 2024-03-11T22:12:52.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:12:52.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:12:52.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:12:52.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:12:52.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[39715]: osdmap e137: 8 total, 8 up, 8 in 2024-03-11T22:12:52.736 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[30178]: 6.f scrub starts 2024-03-11T22:12:52.736 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[30178]: 6.f scrub ok 2024-03-11T22:12:52.736 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:12:52.736 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:12:52.736 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:12:52.736 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:12:52.736 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:52 smithi045 ceph-mon[30178]: osdmap e137: 8 total, 8 up, 8 in 2024-03-11T22:12:52.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:52.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:52 smithi144 ceph-mon[33984]: 6.f scrub starts 2024-03-11T22:12:52.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:52 smithi144 ceph-mon[33984]: 6.f scrub ok 2024-03-11T22:12:52.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:52 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:12:52.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:52 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:12:52.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:52 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:12:52.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:52 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:12:52.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:52 smithi144 ceph-mon[33984]: osdmap e137: 8 total, 8 up, 8 in 2024-03-11T22:12:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:52.865 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:52.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:52.967 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:53.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:53.002 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:53.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:53.103 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:53.135 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:53.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:53.236 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:53.275 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:53.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:53.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:53.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:53.422 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:53.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:53.524 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:53.558 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:53.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:53.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:53.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:53.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:53.692 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:53.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:53.739 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:53 smithi144 ceph-mon[33984]: pgmap v163: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 899 B/s rd, 0 B/s wr, 862 op/s 2024-03-11T22:12:53.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:53 smithi045 ceph-mon[39715]: pgmap v163: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 899 B/s rd, 0 B/s wr, 862 op/s 2024-03-11T22:12:53.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:53 smithi045 ceph-mon[30178]: pgmap v163: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 899 B/s rd, 0 B/s wr, 862 op/s 2024-03-11T22:12:53.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:53.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:53.842 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:53.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:53.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:53.971 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:53.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:53.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:54.073 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:54.114 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:54.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:54.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:54.253 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:54.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:54.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:54.401 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:54.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:54.502 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:54.535 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:54.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:54.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:12:54.636 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:54.663 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:54.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:54.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:54.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:54.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:54.911 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:54.953 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:54.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:55.054 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:55.081 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:55.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:55.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:55.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:55.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:55.224 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:55.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:55.325 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:55.356 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:55.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:55.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:55.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:55.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:55.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:55.584 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:12:55.584 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-03-11T22:12:55.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-03-11T22:12:55.584 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:12:55.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.584 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:26450 7.2 7:4d05f3c4:::benchmark_data_smithi045_83007_object26449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:12:55.584 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.975397+0000", 2024-03-11T22:12:55.584 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 120.537822807, 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.153187401, 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 26450 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975397+0000", 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.586 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.586 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975397+0000", 2024-03-11T22:12:55.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.586 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.586 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975398+0000", 2024-03-11T22:12:55.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2810000000000001e-06 2024-03-11T22:12:55.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.586 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975422+0000", 2024-03-11T22:12:55.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4409999999999998e-05 2024-03-11T22:12:55.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975433+0000", 2024-03-11T22:12:55.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.096e-05 2024-03-11T22:12:55.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975442+0000", 2024-03-11T22:12:55.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5709999999999998e-06 2024-03-11T22:12:55.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975500+0000", 2024-03-11T22:12:55.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8295999999999999e-05 2024-03-11T22:12:55.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975629+0000", 2024-03-11T22:12:55.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012867099999999999 2024-03-11T22:12:55.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:12:55.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975755+0000", 2024-03-11T22:12:55.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012651700000000001 2024-03-11T22:12:55.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126092+0000", 2024-03-11T22:12:55.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.150336689 2024-03-11T22:12:55.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:12:55.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126107+0000", 2024-03-11T22:12:55.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4783999999999999e-05 2024-03-11T22:12:55.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128442+0000", 2024-03-11T22:12:55.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023357540000000002 2024-03-11T22:12:55.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128541+0000", 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8388999999999996e-05 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128584+0000", 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3078999999999999e-05 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:26453 7.a 7:53c80b4b:::benchmark_data_smithi045_83007_object26452:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.975879+0000", 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 120.53734065099999, 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15598842800000001, 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 26453 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975879+0000", 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975879+0000", 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975880+0000", 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6800000000000009e-07 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975902+0000", 2024-03-11T22:12:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2144999999999999e-05 2024-03-11T22:12:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975913+0000", 2024-03-11T22:12:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1433e-05 2024-03-11T22:12:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.594 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975921+0000", 2024-03-11T22:12:55.595 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7600000000000002e-06 2024-03-11T22:12:55.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.976199+0000", 2024-03-11T22:12:55.595 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00027757599999999997 2024-03-11T22:12:55.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.976413+0000", 2024-03-11T22:12:55.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021448999999999999 2024-03-11T22:12:55.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.976552+0000", 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013905699999999999 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126114+0000", 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14956153999999999 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126151+0000", 2024-03-11T22:12:55.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6961000000000001e-05 2024-03-11T22:12:55.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.598 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.130760+0000", 2024-03-11T22:12:55.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046094489999999998 2024-03-11T22:12:55.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:12:55.598 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.131087+0000", 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00032702499999999999 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.131867+0000", 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00078002400000000004 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:26450 7.2 e106/99 ondisk, result = 0)", 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.977100+0000", 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 120.536118982, 2024-03-11T22:12:55.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14901504199999999, 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977100+0000", 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977100+0000", 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977101+0000", 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4099999999999998e-07 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977106+0000", 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4730000000000002e-06 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977116+0000", 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0492000000000001e-05 2024-03-11T22:12:55.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977123+0000", 2024-03-11T22:12:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9040000000000003e-06 2024-03-11T22:12:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126047+0000", 2024-03-11T22:12:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.148923889 2024-03-11T22:12:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126057+0000", 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7620000000000007e-06 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126115+0000", 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8780999999999998e-05 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:26450 7.2 e106/99 ondisk, result = 0)", 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.977360+0000", 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 120.535858991, 2024-03-11T22:12:55.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15123065199999999, 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977360+0000", 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977360+0000", 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977361+0000", 2024-03-11T22:12:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9800000000000003e-07 2024-03-11T22:12:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977365+0000", 2024-03-11T22:12:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3019999999999996e-06 2024-03-11T22:12:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977383+0000", 2024-03-11T22:12:55.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7302000000000001e-05 2024-03-11T22:12:55.606 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.606 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.606 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.606 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977392+0000", 2024-03-11T22:12:55.606 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0990000000000005e-06 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.127691+0000", 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.150299025 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128436+0000", 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00074558000000000003 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128591+0000", 2024-03-11T22:12:55.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015464600000000001 2024-03-11T22:12:55.608 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.608 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.608 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.608 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:26453 7.a e106/99 ondisk, result = 0)", 2024-03-11T22:12:55.608 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.977682+0000", 2024-03-11T22:12:55.608 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 120.535537303, 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.148475739, 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977682+0000", 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977682+0000", 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977683+0000", 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6899999999999997e-07 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977685+0000", 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5600000000000001e-06 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.124416+0000", 2024-03-11T22:12:55.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14673112999999999 2024-03-11T22:12:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.124434+0000", 2024-03-11T22:12:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7260000000000001e-05 2024-03-11T22:12:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126138+0000", 2024-03-11T22:12:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001704583 2024-03-11T22:12:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.611 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126145+0000", 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.922e-06 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126158+0000", 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2714999999999999e-05 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:27609 7.3 7:c42738fc:::benchmark_data_smithi045_83007_object27608:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:12:55.612 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.711341+0000", 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.801878078, 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14098761900000001, 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27609 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.711341+0000", 2024-03-11T22:12:55.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.711341+0000", 2024-03-11T22:12:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.711342+0000", 2024-03-11T22:12:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.09e-07 2024-03-11T22:12:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.711363+0000", 2024-03-11T22:12:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0480999999999999e-05 2024-03-11T22:12:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.711374+0000", 2024-03-11T22:12:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1286e-05 2024-03-11T22:12:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.711381+0000", 2024-03-11T22:12:55.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3050000000000004e-06 2024-03-11T22:12:55.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.712884+0000", 2024-03-11T22:12:55.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015027930000000001 2024-03-11T22:12:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.712995+0000", 2024-03-11T22:12:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110862 2024-03-11T22:12:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,4", 2024-03-11T22:12:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.713118+0000", 2024-03-11T22:12:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012350800000000001 2024-03-11T22:12:55.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:12:55.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.714684+0000", 2024-03-11T22:12:55.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015655809999999999 2024-03-11T22:12:55.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.714877+0000", 2024-03-11T22:12:55.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000193396 2024-03-11T22:12:55.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852227+0000", 2024-03-11T22:12:55.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13734940300000001 2024-03-11T22:12:55.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:12:55.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852283+0000", 2024-03-11T22:12:55.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6270000000000002e-05 2024-03-11T22:12:55.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852329+0000", 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5825000000000003e-05 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:27612 7.3 7:c545c31a:::benchmark_data_smithi045_83007_object27611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.712823+0000", 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.800396479, 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.140459746, 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.620 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27612 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.712823+0000", 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.712823+0000", 2024-03-11T22:12:55.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.712824+0000", 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1599999999999999e-06 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.712854+0000", 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0202e-05 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.712867+0000", 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3203e-05 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.712876+0000", 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7609999999999996e-06 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715555+0000", 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026784230000000001 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715669+0000", 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011468000000000001 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,4", 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715802+0000", 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013231500000000001 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852340+0000", 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.136538885 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852380+0000", 2024-03-11T22:12:55.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9932999999999999e-05 2024-03-11T22:12:55.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.853197+0000", 2024-03-11T22:12:55.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00081682800000000002 2024-03-11T22:12:55.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:12:55.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.853246+0000", 2024-03-11T22:12:55.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8767999999999998e-05 2024-03-11T22:12:55.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.853283+0000", 2024-03-11T22:12:55.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6588000000000003e-05 2024-03-11T22:12:55.626 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.626 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.626 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.627 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:27617 7.a 7:52ceabb1:::benchmark_data_smithi045_83007_object27616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:12:55.627 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.714354+0000", 2024-03-11T22:12:55.627 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.798865745, 2024-03-11T22:12:55.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.138425942, 2024-03-11T22:12:55.627 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.627 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:12:55.627 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:55.627 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:55.627 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27617 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.714354+0000", 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.714354+0000", 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.714355+0000", 2024-03-11T22:12:55.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1650000000000001e-06 2024-03-11T22:12:55.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.714378+0000", 2024-03-11T22:12:55.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3006999999999998e-05 2024-03-11T22:12:55.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.714389+0000", 2024-03-11T22:12:55.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0764000000000001e-05 2024-03-11T22:12:55.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.714397+0000", 2024-03-11T22:12:55.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9929999999999995e-06 2024-03-11T22:12:55.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715072+0000", 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00067514599999999995 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715149+0000", 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6953999999999995e-05 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715251+0000", 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000102191 2024-03-11T22:12:55.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852477+0000", 2024-03-11T22:12:55.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13722648500000001 2024-03-11T22:12:55.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:12:55.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852554+0000", 2024-03-11T22:12:55.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6809000000000003e-05 2024-03-11T22:12:55.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852679+0000", 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000125127 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852736+0000", 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7082e-05 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852780+0000", 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3219000000000001e-05 2024-03-11T22:12:55.633 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:27622 7.3 7:cf7d5727:::benchmark_data_smithi045_83007_object27621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.715347+0000", 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.79787220999999, 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14173917899999999, 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27622 2024-03-11T22:12:55.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715347+0000", 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715347+0000", 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715348+0000", 2024-03-11T22:12:55.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4399999999999999e-07 2024-03-11T22:12:55.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715369+0000", 2024-03-11T22:12:55.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1509e-05 2024-03-11T22:12:55.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715380+0000", 2024-03-11T22:12:55.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0387000000000001e-05 2024-03-11T22:12:55.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715388+0000", 2024-03-11T22:12:55.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8159999999999994e-06 2024-03-11T22:12:55.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718248+0000", 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028601830000000001 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718395+0000", 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014705 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,4", 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849978+0000", 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13158319199999999 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.853330+0000", 2024-03-11T22:12:55.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003351737 2024-03-11T22:12:55.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.856715+0000", 2024-03-11T22:12:55.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033848289999999998 2024-03-11T22:12:55.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.643 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:12:55.643 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.857003+0000", 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028860799999999999 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.857048+0000", 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4508000000000001e-05 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.857086+0000", 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8516000000000002e-05 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.644 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:27623 7.a 7:54014f37:::benchmark_data_smithi045_83007_object27622:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.715404+0000", 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.79781519399999, 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13782271500000001, 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27623 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.645 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715404+0000", 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715404+0000", 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715405+0000", 2024-03-11T22:12:55.646 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2300000000000001e-07 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715422+0000", 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7322000000000001e-05 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715429+0000", 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3039999999999999e-06 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.647 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715435+0000", 2024-03-11T22:12:55.648 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5010000000000004e-06 2024-03-11T22:12:55.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.648 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.648 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.717129+0000", 2024-03-11T22:12:55.648 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001693981 2024-03-11T22:12:55.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.648 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.648 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.717242+0000", 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011298699999999999 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.717508+0000", 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000266457 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852528+0000", 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13501985699999999 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852561+0000", 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3189999999999999e-05 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.853140+0000", 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00057828199999999995 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.853189+0000", 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9141999999999998e-05 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.651 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.651 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.853227+0000", 2024-03-11T22:12:55.651 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8169e-05 2024-03-11T22:12:55.651 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.651 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.651 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.651 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:27624 7.2 7:41266608:::benchmark_data_smithi045_83007_object27623:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:12:55.652 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.715446+0000", 2024-03-11T22:12:55.652 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.797773535, 2024-03-11T22:12:55.652 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14151458, 2024-03-11T22:12:55.652 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.652 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:12:55.652 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:55.652 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:55.652 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:55.653 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27624 2024-03-11T22:12:55.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.653 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.653 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.653 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715446+0000", 2024-03-11T22:12:55.653 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715446+0000", 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715446+0000", 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8500000000000002e-07 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715465+0000", 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8621999999999999e-05 2024-03-11T22:12:55.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715472+0000", 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7109999999999998e-06 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715477+0000", 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0220000000000004e-06 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718404+0000", 2024-03-11T22:12:55.655 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002927383 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.850226+0000", 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13182201900000001 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.850390+0000", 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000163914 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:12:55.656 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852567+0000", 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021767980000000002 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.856715+0000", 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0041479949999999998 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.856858+0000", 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014373599999999999 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:12:55.657 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.856916+0000", 2024-03-11T22:12:55.658 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8118999999999997e-05 2024-03-11T22:12:55.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.658 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.658 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.856960+0000", 2024-03-11T22:12:55.658 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3875999999999998e-05 2024-03-11T22:12:55.658 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.658 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.658 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.658 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:27609 7.3 e106/99 ondisk, result = 0)", 2024-03-11T22:12:55.658 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.715598+0000", 2024-03-11T22:12:55.659 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.797621601, 2024-03-11T22:12:55.659 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13674412499999999, 2024-03-11T22:12:55.659 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.659 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:12:55.659 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.659 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.659 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715598+0000", 2024-03-11T22:12:55.659 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.659 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.659 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715598+0000", 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715598+0000", 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4900000000000005e-07 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715603+0000", 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4340000000000002e-06 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715613+0000", 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0239e-05 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.715620+0000", 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9249999999999998e-06 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852212+0000", 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13659185900000001 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.661 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852221+0000", 2024-03-11T22:12:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3840000000000003e-06 2024-03-11T22:12:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852342+0000", 2024-03-11T22:12:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000120535 2024-03-11T22:12:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.662 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:27617 7.a e106/99 ondisk, result = 0)", 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.716610+0000", 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.79660892299999, 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13588023599999999, 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716610+0000", 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716610+0000", 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716611+0000", 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2900000000000003e-07 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716616+0000", 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5029999999999999e-06 2024-03-11T22:12:55.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716625+0000", 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1779999999999992e-06 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716631+0000", 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.981e-06 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852461+0000", 2024-03-11T22:12:55.665 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13583018299999999 2024-03-11T22:12:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852471+0000", 2024-03-11T22:12:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8509999999999994e-06 2024-03-11T22:12:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852491+0000", 2024-03-11T22:12:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9911e-05 2024-03-11T22:12:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.666 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:27612 7.3 e106/99 ondisk, result = 0)", 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.716972+0000", 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.796246981, 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.135415171, 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716972+0000", 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716972+0000", 2024-03-11T22:12:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716973+0000", 2024-03-11T22:12:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2200000000000003e-07 2024-03-11T22:12:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716977+0000", 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3259999999999997e-06 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716988+0000", 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0101999999999999e-05 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.716994+0000", 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.798e-06 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.669 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852369+0000", 2024-03-11T22:12:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.135374253 2024-03-11T22:12:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852375+0000", 2024-03-11T22:12:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8399999999999997e-06 2024-03-11T22:12:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852388+0000", 2024-03-11T22:12:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.203e-05 2024-03-11T22:12:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.670 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.14790.0:27626 7.0 e106/99 7:0e450ccf:::benchmark_data_smithi045_83007_object27625:head v 106'1689, mlcod=106'1687)", 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.718076+0000", 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.795142934, 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13179716499999999, 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718076+0000", 2024-03-11T22:12:55.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718076+0000", 2024-03-11T22:12:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718077+0000", 2024-03-11T22:12:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4499999999999996e-07 2024-03-11T22:12:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718136+0000", 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9095999999999998e-05 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718147+0000", 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0759e-05 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718154+0000", 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9990000000000002e-06 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718234+0000", 2024-03-11T22:12:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0029000000000002e-05 2024-03-11T22:12:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718277+0000", 2024-03-11T22:12:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2527999999999997e-05 2024-03-11T22:12:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:12:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849821+0000", 2024-03-11T22:12:55.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.131544411 2024-03-11T22:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849874+0000", 2024-03-11T22:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2497999999999998e-05 2024-03-11T22:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:27623 7.a e106/99 ondisk, result = 0)", 2024-03-11T22:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.718480+0000", 2024-03-11T22:12:55.675 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.79473953900001, 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13405535199999999, 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718480+0000", 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718480+0000", 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.677 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718480+0000", 2024-03-11T22:12:55.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.37e-07 2024-03-11T22:12:55.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.677 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718485+0000", 2024-03-11T22:12:55.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7319999999999998e-06 2024-03-11T22:12:55.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.677 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849848+0000", 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13136279000000001 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849856+0000", 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4130000000000006e-06 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852516+0000", 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026595149999999999 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.678 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852522+0000", 2024-03-11T22:12:55.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6800000000000004e-06 2024-03-11T22:12:55.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.852535+0000", 2024-03-11T22:12:55.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2685e-05 2024-03-11T22:12:55.679 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.679 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.679 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.679 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.14790.0:27627 7.c e106/99 7:3bf1c813:::benchmark_data_smithi045_83007_object27626:head v 106'1736, mlcod=106'1734)", 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:55.718647+0000", 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.794572002, 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13651666200000001, 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718647+0000", 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718647+0000", 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718648+0000", 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9999999999999997e-07 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.718690+0000", 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1831e-05 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849485+0000", 2024-03-11T22:12:55.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.130794934 2024-03-11T22:12:55.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849496+0000", 2024-03-11T22:12:55.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1731e-05 2024-03-11T22:12:55.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849605+0000", 2024-03-11T22:12:55.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010833 2024-03-11T22:12:55.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.683 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.683 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.849628+0000", 2024-03-11T22:12:55.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3642000000000002e-05 2024-03-11T22:12:55.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.683 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:12:55.683 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.855082+0000", 2024-03-11T22:12:55.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0054538520000000004 2024-03-11T22:12:55.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.683 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.683 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.855164+0000", 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1742000000000004e-05 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:28070 7.a 7:512190ac:::benchmark_data_smithi045_83007_object28069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:56.072959+0000", 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.44026081600001, 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13307163, 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:12:55.684 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 28070 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.072959+0000", 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.072959+0000", 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.072960+0000", 2024-03-11T22:12:55.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6029999999999999e-06 2024-03-11T22:12:55.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.072983+0000", 2024-03-11T22:12:55.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3187e-05 2024-03-11T22:12:55.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.073000+0000", 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7170999999999999e-05 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.073010+0000", 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3789999999999995e-06 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.073070+0000", 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0041000000000003e-05 2024-03-11T22:12:55.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.073268+0000", 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000197617 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.073425+0000", 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000157288 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.126268+0000", 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.052843590000000003 2024-03-11T22:12:55.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.126283+0000", 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4307e-05 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.205916+0000", 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.079633135999999993 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.205968+0000", 2024-03-11T22:12:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2642000000000002e-05 2024-03-11T22:12:55.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.690 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.690 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.206030+0000", 2024-03-11T22:12:55.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1668999999999995e-05 2024-03-11T22:12:55.690 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.690 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.690 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.690 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:28070 7.a e106/99 ondisk, result = 0)", 2024-03-11T22:12:55.690 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:56.077380+0000", 2024-03-11T22:12:55.690 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.435839458, 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.12865827099999999, 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.077380+0000", 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.077380+0000", 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.077381+0000", 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.187e-06 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.077392+0000", 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0572e-05 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.125211+0000", 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.047819510000000003 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.125218+0000", 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2620000000000001e-06 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.205902+0000", 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.080683484999999999 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.205910+0000", 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3350000000000007e-06 2024-03-11T22:12:55.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.694 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.694 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.206038+0000", 2024-03-11T22:12:55.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012792000000000001 2024-03-11T22:12:55.694 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.694 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.694 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.694 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.14790.0:28074 7.1 e107/99 7:868323d2:::benchmark_data_smithi045_83007_object28073:head v 107'1783, mlcod=106'1781)", 2024-03-11T22:12:55.694 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:56.077957+0000", 2024-03-11T22:12:55.694 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 119.435262169, 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13255535800000001, 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.077957+0000", 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.077957+0000", 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:12:55.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:12:55.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.077958+0000", 2024-03-11T22:12:55.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7399999999999999e-07 2024-03-11T22:12:55.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:12:55.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.077993+0000", 2024-03-11T22:12:55.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5580999999999997e-05 2024-03-11T22:12:55.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:12:55.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.125947+0000", 2024-03-11T22:12:55.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.047953867999999997 2024-03-11T22:12:55.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:12:55.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.125955+0000", 2024-03-11T22:12:55.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.79e-06 2024-03-11T22:12:55.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:12:55.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.205897+0000", 2024-03-11T22:12:55.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.079942526 2024-03-11T22:12:55.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.205916+0000", 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9083e-05 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.210476+0000", 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0045594010000000003 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:56.210513+0000", 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6735e-05 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.698 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.699 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.699 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:12:55.699 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:12:55.699 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:12:55.699 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:55.728 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:55.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:55.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:55.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:55 smithi045 ceph-mon[39715]: pgmap v164: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 818 op/s 2024-03-11T22:12:55.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:55 smithi045 ceph-mon[30178]: pgmap v164: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 818 op/s 2024-03-11T22:12:55.830 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:55.835 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:55 smithi144 ceph-mon[33984]: pgmap v164: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 818 op/s 2024-03-11T22:12:55.868 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:55.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:55.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:55.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:12:55.971 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:55.997 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:55.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:56.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:12:55 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:12:56.098 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:56.126 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:56.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:56.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:56.278 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:56.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:56.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:56.379 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:56.410 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:56.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:56.512 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:56.547 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:56.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:56.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:56.695 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T22:12:56.723 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:56.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:56.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:56.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:56.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:56.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:56.985 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:57.030 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:57.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:57.037 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:12:57.132 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:57.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:57.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:57 smithi045 ceph-mon[30178]: pgmap v165: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-03-11T22:12:57.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:57 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:12:57.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:57 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:12:57.175 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:57.176 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:57 smithi045 ceph-mon[39715]: pgmap v165: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-03-11T22:12:57.176 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:57 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:12:57.176 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:57 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:12:57.179 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:57.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:57.187 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:57.262 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:12:57.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:57.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:57.344 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:57.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:57.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:57.502 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:12:57.503 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:57 smithi144 ceph-mon[33984]: pgmap v165: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-03-11T22:12:57.503 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:57 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:12:57.503 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:57 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:12:57.506 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:57.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:57.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:57.609 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:57.656 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:57.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:57.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:57.702 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:12:57.712 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:12:57.758 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:57.786 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:57.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:57.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:57.888 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:57.937 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:57.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:57.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:58.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:58.081 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:58.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:58.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:58.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:58.226 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:58.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:58.328 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:12:58.357 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:58.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:58.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:12:58.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:58.497 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:58.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:58.598 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:12:58.628 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:58.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:12:58.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:58.754 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:58.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:58.856 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:12:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:58.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:12:58.988 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:12:59.022 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:59.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:59.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:12:59.064 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:12:59.064 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:12:59.064 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.064 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:162986 7.1 7:80caf485:::benchmark_data_smithi045_83007_object44515:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-11T22:12:59.064 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:59.059188+0000", 2024-03-11T22:12:59.064 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.007201051, 2024-03-11T22:12:59.064 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0072326109999999999, 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 162986 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.059188+0000", 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.065 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.066 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:12:59.066 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.059188+0000", 2024-03-11T22:12:59.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:12:59.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.066 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.066 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:12:59.066 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.059189+0000", 2024-03-11T22:12:59.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0449999999999999e-06 2024-03-11T22:12:59.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.066 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.066 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:12:59.066 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.059193+0000", 2024-03-11T22:12:59.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.98e-06 2024-03-11T22:12:59.067 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.067 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:12:59.067 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.059202+0000", 2024-03-11T22:12:59.067 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1740000000000006e-06 2024-03-11T22:12:59.067 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.067 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:12:59.067 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.059208+0000", 2024-03-11T22:12:59.067 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5280000000000003e-06 2024-03-11T22:12:59.067 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.067 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.059241+0000", 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3779999999999998e-05 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.059345+0000", 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010349 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.059451+0000", 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000105919 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.068 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.061811+0000", 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002360638 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.062058+0000", 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024650799999999999 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.069 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:162990 7.1 7:87485a7e:::benchmark_data_smithi045_83007_object44519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:59.064234+0000", 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.0021549809999999998, 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0022903649999999999, 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 162990 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:12:59.070 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.064234+0000", 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.064234+0000", 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.064235+0000", 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3480000000000001e-06 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:12:59.071 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.064240+0000", 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1880000000000001e-06 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.064256+0000", 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5091e-05 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.064265+0000", 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3810000000000005e-06 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.072 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.064322+0000", 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7494999999999998e-05 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.064416+0000", 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3718e-05 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.064498+0000", 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2002000000000005e-05 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:12:59.073 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:162984 7.9 e137/99 7:920bb262:::benchmark_data_smithi045_83007_object44513:head v 137'10126, mlcod=137'10123)", 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:59.066041+0000", 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.00034742099999999999, 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00057211399999999998, 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066041+0000", 2024-03-11T22:12:59.074 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066041+0000", 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066043+0000", 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4580000000000001e-06 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066050+0000", 2024-03-11T22:12:59.075 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9929999999999998e-06 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066065+0000", 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4949999999999999e-05 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066072+0000", 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4240000000000004e-06 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066140+0000", 2024-03-11T22:12:59.076 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8045000000000003e-05 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066160+0000", 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.997e-05 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop_reply(client.14790.0:162986 7.1 e137/99)", 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:59.066569+0000", 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4294967295.9998193, 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000114089, 2024-03-11T22:12:59.077 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066569+0000", 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066569+0000", 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:12:59.079 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066570+0000", 2024-03-11T22:12:59.079 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8700000000000005e-07 2024-03-11T22:12:59.079 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.079 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.079 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:12:59.079 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066574+0000", 2024-03-11T22:12:59.079 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3680000000000004e-06 2024-03-11T22:12:59.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.080 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.080 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:12:59.080 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066585+0000", 2024-03-11T22:12:59.080 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0808999999999999e-05 2024-03-11T22:12:59.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:12:59.080 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:12:59.080 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:12:59.080 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:59.066592+0000", 2024-03-11T22:12:59.080 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0419999999999997e-06 2024-03-11T22:12:59.081 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:12:59.081 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:12:59.081 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:12:59.081 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:12:59.081 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-03-11T22:12:59.081 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 4 2024-03-11T22:12:59.081 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:12:59.125 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:59.168 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:59.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:59.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:59.270 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:12:59.277 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:59 smithi045 ceph-mon[39715]: from='client.24776 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:12:59.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:12:59 smithi045 ceph-mon[39715]: pgmap v166: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-11T22:12:59.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:59 smithi045 ceph-mon[30178]: from='client.24776 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:12:59.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:12:59 smithi045 ceph-mon[30178]: pgmap v166: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-11T22:12:59.297 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:12:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:59.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:12:59.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:59 smithi144 ceph-mon[33984]: from='client.24776 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:12:59.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:12:59 smithi144 ceph-mon[33984]: pgmap v166: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-11T22:12:59.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:59.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:59.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:59.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:59.548 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:12:59.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:59.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:59.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:12:59.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:12:59.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:59.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:59.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:12:59.846 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:12:59.893 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:12:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:12:59.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:12:59.898 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T22:12:59.996 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:00.037 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:00.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:00.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:00.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:00.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:00.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:00.274 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:13:00.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:00.311 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:00.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:00.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:00.412 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:00.438 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:00 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:13:00.438 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:00 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:13:00.438 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:00 smithi144 ceph-mon[33984]: osdmap e138: 8 total, 8 up, 8 in 2024-03-11T22:13:00.438 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:00.439 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:13:00.442 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:00.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:00.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:00 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:13:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:00 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:13:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:00 smithi045 ceph-mon[30178]: osdmap e138: 8 total, 8 up, 8 in 2024-03-11T22:13:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:00.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:13:00.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:00 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:13:00.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:00 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:13:00.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:00 smithi045 ceph-mon[39715]: osdmap e138: 8 total, 8 up, 8 in 2024-03-11T22:13:00.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:00.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:13:00.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:00.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:00.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:00.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:00.669 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:00.707 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:00.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:00.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:00.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:00.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:00.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:00.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:00.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:00.989 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:00.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:01.090 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:01.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:01.116 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:01.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:01.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:01.257 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:01.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:01.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:01.392 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:01.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:01.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:01.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:01.536 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:01.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:01.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:01 smithi144 ceph-mon[33984]: pgmap v168: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 592 B/s rd, 0 B/s wr, 621 op/s 2024-03-11T22:13:01.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:01.688 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:01.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:01.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:01.733 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:01 smithi045 ceph-mon[30178]: pgmap v168: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 592 B/s rd, 0 B/s wr, 621 op/s 2024-03-11T22:13:01.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:01 smithi045 ceph-mon[39715]: pgmap v168: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 592 B/s rd, 0 B/s wr, 621 op/s 2024-03-11T22:13:01.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:01.839 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:01.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:01.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:01.941 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:02.001 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:02.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:02.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:02.145 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:02.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:02.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:02.247 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:02.277 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:02.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:02.379 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:02.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:02.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:02.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:02.523 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:02.555 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:02.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:02.599 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:13:02.600 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-03-11T22:13:02.600 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-03-11T22:13:02.600 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-03-11T22:13:02.600 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:13:02.657 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:02.684 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:02.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:02.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:02.777 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:13:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:13:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:13:02.786 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:02.815 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:02.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:02.868 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-03-11T22:12:56.119338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5086189940,"num_objects":77799,"num_object_clones":0,"num_object_copies":233397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77799,"num_whiteouts":0,"num_read":1154,"num_read_kb":1000,"num_write":159952,"num_write_kb":7584414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92337,"ondisk_log_size":92337,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15750632,"kb_used_data":15022248,"kb_used_omap":12,"kb_used_meta":728307,"kb_avail":734210584,"statfs":{"total":767960285184,"available":751831638016,"internally_reserved":0,"allocated":15382781952,"data_stored":15380261596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":745786931},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,12,6,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-653524992,"num_objects":-9972,"num_object_clones":0,"num_object_copies":-29916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9972,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9972,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.665881"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":146,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.510526+0000","last_change":"2024-03-11T22:11:30.350663+0000","last_active":"2024-03-11T22:12:51.510526+0000","last_peered":"2024-03-11T22:12:51.510526+0000","last_clean":"2024-03-11T22:12:51.510526+0000","last_became_active":"2024-03-11T22:08:39.074695+0000","last_became_peered":"2024-03-11T22:08:39.074695+0000","last_unstale":"2024-03-11T22:12:51.510526+0000","last_undegraded":"2024-03-11T22:12:51.510526+0000","last_fullsized":"2024-03-11T22:12:51.510526+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:30.350540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.350540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.504594+0000","last_change":"2024-03-11T22:12:45.929428+0000","last_active":"2024-03-11T22:12:51.504594+0000","last_peered":"2024-03-11T22:12:51.504594+0000","last_clean":"2024-03-11T22:12:51.504594+0000","last_became_active":"2024-03-11T22:08:37.073775+0000","last_became_peered":"2024-03-11T22:08:37.073775+0000","last_unstale":"2024-03-11T22:12:51.504594+0000","last_undegraded":"2024-03-11T22:12:51.504594+0000","last_fullsized":"2024-03-11T22:12:51.504594+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.928689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.928689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:15.589951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":167,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.502607+0000","last_change":"2024-03-11T22:11:45.340779+0000","last_active":"2024-03-11T22:12:51.502607+0000","last_peered":"2024-03-11T22:12:51.502607+0000","last_clean":"2024-03-11T22:12:51.502607+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:12:51.502607+0000","last_undegraded":"2024-03-11T22:12:51.502607+0000","last_fullsized":"2024-03-11T22:12:51.502607+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:45.340657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:45.340657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:06.074324+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.512694+0000","last_change":"2024-03-11T22:10:13.507406+0000","last_active":"2024-03-11T22:12:51.512694+0000","last_peered":"2024-03-11T22:12:51.512694+0000","last_clean":"2024-03-11T22:12:51.512694+0000","last_became_active":"2024-03-11T22:08:35.053179+0000","last_became_peered":"2024-03-11T22:08:35.053179+0000","last_unstale":"2024-03-11T22:12:51.512694+0000","last_undegraded":"2024-03-11T22:12:51.512694+0000","last_fullsized":"2024-03-11T22:12:51.512694+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:10:13.507312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:13.507312+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.501333+0000","last_change":"2024-03-11T22:12:50.533192+0000","last_active":"2024-03-11T22:12:51.501333+0000","last_peered":"2024-03-11T22:12:51.501333+0000","last_clean":"2024-03-11T22:12:51.501333+0000","last_became_active":"2024-03-11T22:08:39.079045+0000","last_became_peered":"2024-03-11T22:08:39.079045+0000","last_unstale":"2024-03-11T22:12:51.501333+0000","last_undegraded":"2024-03-11T22:12:51.501333+0000","last_fullsized":"2024-03-11T22:12:51.501333+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.533067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.533067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:07.226337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.513129+0000","last_change":"2024-03-11T22:11:43.150997+0000","last_active":"2024-03-11T22:12:51.513129+0000","last_peered":"2024-03-11T22:12:51.513129+0000","last_clean":"2024-03-11T22:12:51.513129+0000","last_became_active":"2024-03-11T22:08:37.081861+0000","last_became_peered":"2024-03-11T22:08:37.081861+0000","last_unstale":"2024-03-11T22:12:51.513129+0000","last_undegraded":"2024-03-11T22:12:51.513129+0000","last_fullsized":"2024-03-11T22:12:51.513129+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:43.150872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:43.150872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:59.009020+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":173,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.499240+0000","last_change":"2024-03-11T22:11:22.946804+0000","last_active":"2024-03-11T22:12:51.499240+0000","last_peered":"2024-03-11T22:12:51.499240+0000","last_clean":"2024-03-11T22:12:51.499240+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:12:51.499240+0000","last_undegraded":"2024-03-11T22:12:51.499240+0000","last_fullsized":"2024-03-11T22:12:51.499240+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:11:22.946664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:22.946664+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":187,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.500408+0000","last_change":"2024-03-11T22:12:49.510144+0000","last_active":"2024-03-11T22:12:51.500408+0000","last_peered":"2024-03-11T22:12:51.500408+0000","last_clean":"2024-03-11T22:12:51.500408+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:12:51.500408+0000","last_undegraded":"2024-03-11T22:12:51.500408+0000","last_fullsized":"2024-03-11T22:12:51.500408+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:12:49.509860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.509860+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:15.173673+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":146,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.499781+0000","last_change":"2024-03-11T22:10:15.238661+0000","last_active":"2024-03-11T22:12:51.499781+0000","last_peered":"2024-03-11T22:12:51.499781+0000","last_clean":"2024-03-11T22:12:51.499781+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:12:51.499781+0000","last_undegraded":"2024-03-11T22:12:51.499781+0000","last_fullsized":"2024-03-11T22:12:51.499781+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:15.238567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:15.238567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482403+0000","last_change":"2024-03-11T22:12:19.337740+0000","last_active":"2024-03-11T22:12:51.482403+0000","last_peered":"2024-03-11T22:12:51.482403+0000","last_clean":"2024-03-11T22:12:51.482403+0000","last_became_active":"2024-03-11T22:08:37.081996+0000","last_became_peered":"2024-03-11T22:08:37.081996+0000","last_unstale":"2024-03-11T22:12:51.482403+0000","last_undegraded":"2024-03-11T22:12:51.482403+0000","last_fullsized":"2024-03-11T22:12:51.482403+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:19.337662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:12:19.337662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:35.201935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":174,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482496+0000","last_change":"2024-03-11T22:12:47.432839+0000","last_active":"2024-03-11T22:12:51.482496+0000","last_peered":"2024-03-11T22:12:51.482496+0000","last_clean":"2024-03-11T22:12:51.482496+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:12:51.482496+0000","last_undegraded":"2024-03-11T22:12:51.482496+0000","last_fullsized":"2024-03-11T22:12:51.482496+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:47.432769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:47.432769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:59.023738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":178,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.510714+0000","last_change":"2024-03-11T22:11:45.169864+0000","last_active":"2024-03-11T22:12:51.510714+0000","last_peered":"2024-03-11T22:12:51.510714+0000","last_clean":"2024-03-11T22:12:51.510714+0000","last_became_active":"2024-03-11T22:08:35.060802+0000","last_became_peered":"2024-03-11T22:08:35.060802+0000","last_unstale":"2024-03-11T22:12:51.510714+0000","last_undegraded":"2024-03-11T22:12:51.510714+0000","last_fullsized":"2024-03-11T22:12:51.510714+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:11:45.169706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:45.169706+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:02.654471+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482087+0000","last_change":"2024-03-11T22:11:47.166498+0000","last_active":"2024-03-11T22:12:51.482087+0000","last_peered":"2024-03-11T22:12:51.482087+0000","last_clean":"2024-03-11T22:12:51.482087+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:12:51.482087+0000","last_undegraded":"2024-03-11T22:12:51.482087+0000","last_fullsized":"2024-03-11T22:12:51.482087+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:47.166360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:47.166360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:13.931342+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":88,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.510924+0000","last_change":"2024-03-11T22:10:35.411012+0000","last_active":"2024-03-11T22:12:51.510924+0000","last_peered":"2024-03-11T22:12:51.510924+0000","last_clean":"2024-03-11T22:12:51.510924+0000","last_became_active":"2024-03-11T22:10:35.410624+0000","last_became_peered":"2024-03-11T22:10:35.410624+0000","last_unstale":"2024-03-11T22:12:51.510924+0000","last_undegraded":"2024-03-11T22:12:51.510924+0000","last_fullsized":"2024-03-11T22:12:51.510924+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:00.270166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.270166+0000","last_clean_scrub_stamp":"2024-03-11T22:10:00.270166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482386+0000","last_change":"2024-03-11T22:12:24.388572+0000","last_active":"2024-03-11T22:12:51.482386+0000","last_peered":"2024-03-11T22:12:51.482386+0000","last_clean":"2024-03-11T22:12:51.482386+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:12:51.482386+0000","last_undegraded":"2024-03-11T22:12:51.482386+0000","last_fullsized":"2024-03-11T22:12:51.482386+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:24.388497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.388497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:43.170333+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"95'7","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.511214+0000","last_change":"2024-03-11T22:10:18.579990+0000","last_active":"2024-03-11T22:12:51.511214+0000","last_peered":"2024-03-11T22:12:51.511214+0000","last_clean":"2024-03-11T22:12:51.511214+0000","last_became_active":"2024-03-11T22:08:35.060777+0000","last_became_peered":"2024-03-11T22:08:35.060777+0000","last_unstale":"2024-03-11T22:12:51.511214+0000","last_undegraded":"2024-03-11T22:12:51.511214+0000","last_fullsized":"2024-03-11T22:12:51.511214+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'7","last_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:10:18.579896+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":155,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.513424+0000","last_change":"2024-03-11T22:12:32.044430+0000","last_active":"2024-03-11T22:12:51.513424+0000","last_peered":"2024-03-11T22:12:51.513424+0000","last_clean":"2024-03-11T22:12:51.513424+0000","last_became_active":"2024-03-11T22:08:39.075609+0000","last_became_peered":"2024-03-11T22:08:39.075609+0000","last_unstale":"2024-03-11T22:12:51.513424+0000","last_undegraded":"2024-03-11T22:12:51.513424+0000","last_fullsized":"2024-03-11T22:12:51.513424+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:32.044320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:32.044320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:34.643459+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.489034+0000","last_change":"2024-03-11T22:12:30.962293+0000","last_active":"2024-03-11T22:12:51.489034+0000","last_peered":"2024-03-11T22:12:51.489034+0000","last_clean":"2024-03-11T22:12:51.489034+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:12:51.489034+0000","last_undegraded":"2024-03-11T22:12:51.489034+0000","last_fullsized":"2024-03-11T22:12:51.489034+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:30.962171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:30.962171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.129955+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.500568+0000","last_change":"2024-03-11T22:12:22.649189+0000","last_active":"2024-03-11T22:12:51.500568+0000","last_peered":"2024-03-11T22:12:51.500568+0000","last_clean":"2024-03-11T22:12:51.500568+0000","last_became_active":"2024-03-11T22:08:33.035624+0000","last_became_peered":"2024-03-11T22:08:33.035624+0000","last_unstale":"2024-03-11T22:12:51.500568+0000","last_undegraded":"2024-03-11T22:12:51.500568+0000","last_fullsized":"2024-03-11T22:12:51.500568+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:22.649073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:22.649073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:32.856439+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":195,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.505766+0000","last_change":"2024-03-11T22:12:27.670423+0000","last_active":"2024-03-11T22:12:51.505766+0000","last_peered":"2024-03-11T22:12:51.505766+0000","last_clean":"2024-03-11T22:12:51.505766+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:12:51.505766+0000","last_undegraded":"2024-03-11T22:12:51.505766+0000","last_fullsized":"2024-03-11T22:12:51.505766+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:12:27.670284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:27.670284+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:39.408475+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":155,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.505735+0000","last_change":"2024-03-11T22:12:44.464428+0000","last_active":"2024-03-11T22:12:51.505735+0000","last_peered":"2024-03-11T22:12:51.505735+0000","last_clean":"2024-03-11T22:12:51.505735+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:12:51.505735+0000","last_undegraded":"2024-03-11T22:12:51.505735+0000","last_fullsized":"2024-03-11T22:12:51.505735+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.464308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.464308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:02.369972+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.511460+0000","last_change":"2024-03-11T22:11:27.391809+0000","last_active":"2024-03-11T22:12:51.511460+0000","last_peered":"2024-03-11T22:12:51.511460+0000","last_clean":"2024-03-11T22:12:51.511460+0000","last_became_active":"2024-03-11T22:08:37.080014+0000","last_became_peered":"2024-03-11T22:08:37.080014+0000","last_unstale":"2024-03-11T22:12:51.511460+0000","last_undegraded":"2024-03-11T22:12:51.511460+0000","last_fullsized":"2024-03-11T22:12:51.511460+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:27.391727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:27.391727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.510829+0000","last_change":"2024-03-11T22:10:03.469439+0000","last_active":"2024-03-11T22:12:51.510829+0000","last_peered":"2024-03-11T22:12:51.510829+0000","last_clean":"2024-03-11T22:12:51.510829+0000","last_became_active":"2024-03-11T22:08:33.299189+0000","last_became_peered":"2024-03-11T22:08:33.299189+0000","last_unstale":"2024-03-11T22:12:51.510829+0000","last_undegraded":"2024-03-11T22:12:51.510829+0000","last_fullsized":"2024-03-11T22:12:51.510829+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:03.469346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.469346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":185,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.498799+0000","last_change":"2024-03-11T22:12:28.292078+0000","last_active":"2024-03-11T22:12:51.498799+0000","last_peered":"2024-03-11T22:12:51.498799+0000","last_clean":"2024-03-11T22:12:51.498799+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:12:51.498799+0000","last_undegraded":"2024-03-11T22:12:51.498799+0000","last_fullsized":"2024-03-11T22:12:51.498799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:12:28.291919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:28.291919+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:33.113224+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":107,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482391+0000","last_change":"2024-03-11T22:11:22.021723+0000","last_active":"2024-03-11T22:12:51.482391+0000","last_peered":"2024-03-11T22:12:51.482391+0000","last_clean":"2024-03-11T22:12:51.482391+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:12:51.482391+0000","last_undegraded":"2024-03-11T22:12:51.482391+0000","last_fullsized":"2024-03-11T22:12:51.482391+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:11:22.021581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:22.021581+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":231,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.495195+0000","last_change":"2024-03-11T22:12:35.862998+0000","last_active":"2024-03-11T22:12:51.495195+0000","last_peered":"2024-03-11T22:12:51.495195+0000","last_clean":"2024-03-11T22:12:51.495195+0000","last_became_active":"2024-03-11T22:08:37.078588+0000","last_became_peered":"2024-03-11T22:08:37.078588+0000","last_unstale":"2024-03-11T22:12:51.495195+0000","last_undegraded":"2024-03-11T22:12:51.495195+0000","last_fullsized":"2024-03-11T22:12:51.495195+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:12:35.862855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.862855+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:00.615190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":167,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.504692+0000","last_change":"2024-03-11T22:12:24.919926+0000","last_active":"2024-03-11T22:12:51.504692+0000","last_peered":"2024-03-11T22:12:51.504692+0000","last_clean":"2024-03-11T22:12:51.504692+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:12:51.504692+0000","last_undegraded":"2024-03-11T22:12:51.504692+0000","last_fullsized":"2024-03-11T22:12:51.504692+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:24.919682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.919682+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:49.474264+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.500970+0000","last_change":"2024-03-11T22:10:32.196590+0000","last_active":"2024-03-11T22:12:51.500970+0000","last_peered":"2024-03-11T22:12:51.500970+0000","last_clean":"2024-03-11T22:12:51.500970+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:12:51.500970+0000","last_undegraded":"2024-03-11T22:12:51.500970+0000","last_fullsized":"2024-03-11T22:12:51.500970+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:10:32.196500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:32.196500+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.489253+0000","last_change":"2024-03-11T22:11:30.890018+0000","last_active":"2024-03-11T22:12:51.489253+0000","last_peered":"2024-03-11T22:12:51.489253+0000","last_clean":"2024-03-11T22:12:51.489253+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:12:51.489253+0000","last_undegraded":"2024-03-11T22:12:51.489253+0000","last_fullsized":"2024-03-11T22:12:51.489253+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:30.889504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.889504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":159,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.504751+0000","last_change":"2024-03-11T22:11:37.288072+0000","last_active":"2024-03-11T22:12:51.504751+0000","last_peered":"2024-03-11T22:12:51.504751+0000","last_clean":"2024-03-11T22:12:51.504751+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:12:51.504751+0000","last_undegraded":"2024-03-11T22:12:51.504751+0000","last_fullsized":"2024-03-11T22:12:51.504751+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:37.287951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:37.287951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.505590+0000","last_change":"2024-03-11T22:12:43.451182+0000","last_active":"2024-03-11T22:12:51.505590+0000","last_peered":"2024-03-11T22:12:51.505590+0000","last_clean":"2024-03-11T22:12:51.505590+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:12:51.505590+0000","last_undegraded":"2024-03-11T22:12:51.505590+0000","last_fullsized":"2024-03-11T22:12:51.505590+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:43.451074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:43.451074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:06.309610+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":182,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.495048+0000","last_change":"2024-03-11T22:11:11.339653+0000","last_active":"2024-03-11T22:12:51.495048+0000","last_peered":"2024-03-11T22:12:51.495048+0000","last_clean":"2024-03-11T22:12:51.495048+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:12:51.495048+0000","last_undegraded":"2024-03-11T22:12:51.495048+0000","last_fullsized":"2024-03-11T22:12:51.495048+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:11:11.339500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:11.339500+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":97,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.507028+0000","last_change":"2024-03-11T22:11:25.409252+0000","last_active":"2024-03-11T22:12:51.507028+0000","last_peered":"2024-03-11T22:12:51.507028+0000","last_clean":"2024-03-11T22:12:51.507028+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:12:51.507028+0000","last_undegraded":"2024-03-11T22:12:51.507028+0000","last_fullsized":"2024-03-11T22:12:51.507028+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:25.407823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.407823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:57.776716+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.493476+0000","last_change":"2024-03-11T22:12:44.941384+0000","last_active":"2024-03-11T22:12:51.493476+0000","last_peered":"2024-03-11T22:12:51.493476+0000","last_clean":"2024-03-11T22:12:51.493476+0000","last_became_active":"2024-03-11T22:08:37.075341+0000","last_became_peered":"2024-03-11T22:08:37.075341+0000","last_unstale":"2024-03-11T22:12:51.493476+0000","last_undegraded":"2024-03-11T22:12:51.493476+0000","last_fullsized":"2024-03-11T22:12:51.493476+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.941275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.941275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:04.779620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.502794+0000","last_change":"2024-03-11T22:10:18.176194+0000","last_active":"2024-03-11T22:12:51.502794+0000","last_peered":"2024-03-11T22:12:51.502794+0000","last_clean":"2024-03-11T22:12:51.502794+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:12:51.502794+0000","last_undegraded":"2024-03-11T22:12:51.502794+0000","last_fullsized":"2024-03-11T22:12:51.502794+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:18.176081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:18.176081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":180,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.505686+0000","last_change":"2024-03-11T22:11:25.927507+0000","last_active":"2024-03-11T22:12:51.505686+0000","last_peered":"2024-03-11T22:12:51.505686+0000","last_clean":"2024-03-11T22:12:51.505686+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:12:51.505686+0000","last_undegraded":"2024-03-11T22:12:51.505686+0000","last_fullsized":"2024-03-11T22:12:51.505686+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-11T22:11:25.926811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.926811+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.501613+0000","last_change":"2024-03-11T22:11:46.050869+0000","last_active":"2024-03-11T22:12:51.501613+0000","last_peered":"2024-03-11T22:12:51.501613+0000","last_clean":"2024-03-11T22:12:51.501613+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:12:51.501613+0000","last_undegraded":"2024-03-11T22:12:51.501613+0000","last_fullsized":"2024-03-11T22:12:51.501613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:46.050747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:46.050747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:05.807761+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":97,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.512641+0000","last_change":"2024-03-11T22:11:42.175942+0000","last_active":"2024-03-11T22:12:51.512641+0000","last_peered":"2024-03-11T22:12:51.512641+0000","last_clean":"2024-03-11T22:12:51.512641+0000","last_became_active":"2024-03-11T22:10:35.415813+0000","last_became_peered":"2024-03-11T22:10:35.415813+0000","last_unstale":"2024-03-11T22:12:51.512641+0000","last_undegraded":"2024-03-11T22:12:51.512641+0000","last_fullsized":"2024-03-11T22:12:51.512641+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:11:42.175852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:59.608248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482700+0000","last_change":"2024-03-11T22:11:32.118166+0000","last_active":"2024-03-11T22:12:51.482700+0000","last_peered":"2024-03-11T22:12:51.482700+0000","last_clean":"2024-03-11T22:12:51.482700+0000","last_became_active":"2024-03-11T22:08:33.039349+0000","last_became_peered":"2024-03-11T22:08:33.039349+0000","last_unstale":"2024-03-11T22:12:51.482700+0000","last_undegraded":"2024-03-11T22:12:51.482700+0000","last_fullsized":"2024-03-11T22:12:51.482700+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:32.118050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:32.118050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:52.098067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":173,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.500320+0000","last_change":"2024-03-11T22:11:21.929969+0000","last_active":"2024-03-11T22:12:51.500320+0000","last_peered":"2024-03-11T22:12:51.500320+0000","last_clean":"2024-03-11T22:12:51.500320+0000","last_became_active":"2024-03-11T22:08:35.085928+0000","last_became_peered":"2024-03-11T22:08:35.085928+0000","last_unstale":"2024-03-11T22:12:51.500320+0000","last_undegraded":"2024-03-11T22:12:51.500320+0000","last_fullsized":"2024-03-11T22:12:51.500320+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:11:21.929789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:21.929789+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.499436+0000","last_change":"2024-03-11T22:12:46.829082+0000","last_active":"2024-03-11T22:12:51.499436+0000","last_peered":"2024-03-11T22:12:51.499436+0000","last_clean":"2024-03-11T22:12:51.499436+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:12:51.499436+0000","last_undegraded":"2024-03-11T22:12:51.499436+0000","last_fullsized":"2024-03-11T22:12:51.499436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.828942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.828942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:59.694733+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.501249+0000","last_change":"2024-03-11T22:11:37.076127+0000","last_active":"2024-03-11T22:12:51.501249+0000","last_peered":"2024-03-11T22:12:51.501249+0000","last_clean":"2024-03-11T22:12:51.501249+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:12:51.501249+0000","last_undegraded":"2024-03-11T22:12:51.501249+0000","last_fullsized":"2024-03-11T22:12:51.501249+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:11:37.075868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:02.532287+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":97,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.499119+0000","last_change":"2024-03-11T22:11:38.050014+0000","last_active":"2024-03-11T22:12:51.499119+0000","last_peered":"2024-03-11T22:12:51.499119+0000","last_clean":"2024-03-11T22:12:51.499119+0000","last_became_active":"2024-03-11T22:10:35.414121+0000","last_became_peered":"2024-03-11T22:10:35.414121+0000","last_unstale":"2024-03-11T22:12:51.499119+0000","last_undegraded":"2024-03-11T22:12:51.499119+0000","last_fullsized":"2024-03-11T22:12:51.499119+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:38.049889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:38.049889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:04.959324+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":178,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.489196+0000","last_change":"2024-03-11T22:11:26.901776+0000","last_active":"2024-03-11T22:12:51.489196+0000","last_peered":"2024-03-11T22:12:51.489196+0000","last_clean":"2024-03-11T22:12:51.489196+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:12:51.489196+0000","last_undegraded":"2024-03-11T22:12:51.489196+0000","last_fullsized":"2024-03-11T22:12:51.489196+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:11:26.901528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:26.901528+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:56.234825+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.500040+0000","last_change":"2024-03-11T22:12:46.492608+0000","last_active":"2024-03-11T22:12:51.500040+0000","last_peered":"2024-03-11T22:12:51.500040+0000","last_clean":"2024-03-11T22:12:51.500040+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:12:51.500040+0000","last_undegraded":"2024-03-11T22:12:51.500040+0000","last_fullsized":"2024-03-11T22:12:51.500040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.492494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.492494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:11.071230+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.488951+0000","last_change":"2024-03-11T22:12:41.988328+0000","last_active":"2024-03-11T22:12:51.488951+0000","last_peered":"2024-03-11T22:12:51.488951+0000","last_clean":"2024-03-11T22:12:51.488951+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:12:51.488951+0000","last_undegraded":"2024-03-11T22:12:51.488951+0000","last_fullsized":"2024-03-11T22:12:51.488951+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:41.988094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:41.988094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:07.625482+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":172,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.511195+0000","last_change":"2024-03-11T22:12:18.154605+0000","last_active":"2024-03-11T22:12:51.511195+0000","last_peered":"2024-03-11T22:12:51.511195+0000","last_clean":"2024-03-11T22:12:51.511195+0000","last_became_active":"2024-03-11T22:08:33.036083+0000","last_became_peered":"2024-03-11T22:08:33.036083+0000","last_unstale":"2024-03-11T22:12:51.511195+0000","last_undegraded":"2024-03-11T22:12:51.511195+0000","last_fullsized":"2024-03-11T22:12:51.511195+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:18.154515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:18.154515+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:34.107328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":173,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.511378+0000","last_change":"2024-03-11T22:11:19.294807+0000","last_active":"2024-03-11T22:12:51.511378+0000","last_peered":"2024-03-11T22:12:51.511378+0000","last_clean":"2024-03-11T22:12:51.511378+0000","last_became_active":"2024-03-11T22:08:35.054726+0000","last_became_peered":"2024-03-11T22:08:35.054726+0000","last_unstale":"2024-03-11T22:12:51.511378+0000","last_undegraded":"2024-03-11T22:12:51.511378+0000","last_fullsized":"2024-03-11T22:12:51.511378+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:11:19.294671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:19.294671+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":146,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.500583+0000","last_change":"2024-03-11T22:11:13.259520+0000","last_active":"2024-03-11T22:12:51.500583+0000","last_peered":"2024-03-11T22:12:51.500583+0000","last_clean":"2024-03-11T22:12:51.500583+0000","last_became_active":"2024-03-11T22:08:39.082938+0000","last_became_peered":"2024-03-11T22:08:39.082938+0000","last_unstale":"2024-03-11T22:12:51.500583+0000","last_undegraded":"2024-03-11T22:12:51.500583+0000","last_fullsized":"2024-03-11T22:12:51.500583+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:13.259407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:13.259407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":219,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482654+0000","last_change":"2024-03-11T22:11:30.122434+0000","last_active":"2024-03-11T22:12:51.482654+0000","last_peered":"2024-03-11T22:12:51.482654+0000","last_clean":"2024-03-11T22:12:51.482654+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:12:51.482654+0000","last_undegraded":"2024-03-11T22:12:51.482654+0000","last_fullsized":"2024-03-11T22:12:51.482654+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:11:30.122330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.122330+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":98,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.511334+0000","last_change":"2024-03-11T22:11:39.174355+0000","last_active":"2024-03-11T22:12:51.511334+0000","last_peered":"2024-03-11T22:12:51.511334+0000","last_clean":"2024-03-11T22:12:51.511334+0000","last_became_active":"2024-03-11T22:10:35.407845+0000","last_became_peered":"2024-03-11T22:10:35.407845+0000","last_unstale":"2024-03-11T22:12:51.511334+0000","last_undegraded":"2024-03-11T22:12:51.511334+0000","last_fullsized":"2024-03-11T22:12:51.511334+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:39.174103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:39.174103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":177,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.505942+0000","last_change":"2024-03-11T22:12:20.295562+0000","last_active":"2024-03-11T22:12:51.505942+0000","last_peered":"2024-03-11T22:12:51.505942+0000","last_clean":"2024-03-11T22:12:51.505942+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:12:51.505942+0000","last_undegraded":"2024-03-11T22:12:51.505942+0000","last_fullsized":"2024-03-11T22:12:51.505942+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:12:20.295427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:20.295427+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:42.096971+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":146,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.513739+0000","last_change":"2024-03-11T22:11:31.308182+0000","last_active":"2024-03-11T22:12:51.513739+0000","last_peered":"2024-03-11T22:12:51.513739+0000","last_clean":"2024-03-11T22:12:51.513739+0000","last_became_active":"2024-03-11T22:08:39.072089+0000","last_became_peered":"2024-03-11T22:08:39.072089+0000","last_unstale":"2024-03-11T22:12:51.513739+0000","last_undegraded":"2024-03-11T22:12:51.513739+0000","last_fullsized":"2024-03-11T22:12:51.513739+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:31.308103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:31.308103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":222,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.506095+0000","last_change":"2024-03-11T22:11:44.093057+0000","last_active":"2024-03-11T22:12:51.506095+0000","last_peered":"2024-03-11T22:12:51.506095+0000","last_clean":"2024-03-11T22:12:51.506095+0000","last_became_active":"2024-03-11T22:08:37.071202+0000","last_became_peered":"2024-03-11T22:08:37.071202+0000","last_unstale":"2024-03-11T22:12:51.506095+0000","last_undegraded":"2024-03-11T22:12:51.506095+0000","last_fullsized":"2024-03-11T22:12:51.506095+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:11:44.092784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:44.092784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.131593+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.500062+0000","last_change":"2024-03-11T22:12:29.629914+0000","last_active":"2024-03-11T22:12:51.500062+0000","last_peered":"2024-03-11T22:12:51.500062+0000","last_clean":"2024-03-11T22:12:51.500062+0000","last_became_active":"2024-03-11T22:08:33.032850+0000","last_became_peered":"2024-03-11T22:08:33.032850+0000","last_unstale":"2024-03-11T22:12:51.500062+0000","last_undegraded":"2024-03-11T22:12:51.500062+0000","last_fullsized":"2024-03-11T22:12:51.500062+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:29.629778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:29.629778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:44.363248+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":178,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482822+0000","last_change":"2024-03-11T22:11:38.077749+0000","last_active":"2024-03-11T22:12:51.482822+0000","last_peered":"2024-03-11T22:12:51.482822+0000","last_clean":"2024-03-11T22:12:51.482822+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:12:51.482822+0000","last_undegraded":"2024-03-11T22:12:51.482822+0000","last_fullsized":"2024-03-11T22:12:51.482822+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:11:38.077654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:38.077654+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:59.933607+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.506517+0000","last_change":"2024-03-11T22:12:40.777949+0000","last_active":"2024-03-11T22:12:51.506517+0000","last_peered":"2024-03-11T22:12:51.506517+0000","last_clean":"2024-03-11T22:12:51.506517+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:12:51.506517+0000","last_undegraded":"2024-03-11T22:12:51.506517+0000","last_fullsized":"2024-03-11T22:12:51.506517+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:40.777821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:40.777821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:46.765451+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.506116+0000","last_change":"2024-03-11T22:11:31.040056+0000","last_active":"2024-03-11T22:12:51.506116+0000","last_peered":"2024-03-11T22:12:51.506116+0000","last_clean":"2024-03-11T22:12:51.506116+0000","last_became_active":"2024-03-11T22:08:37.069611+0000","last_became_peered":"2024-03-11T22:08:37.069611+0000","last_unstale":"2024-03-11T22:12:51.506116+0000","last_undegraded":"2024-03-11T22:12:51.506116+0000","last_fullsized":"2024-03-11T22:12:51.506116+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:31.039934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:31.039934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:58.937928+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":90,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.512999+0000","last_change":"2024-03-11T22:10:35.416311+0000","last_active":"2024-03-11T22:12:51.512999+0000","last_peered":"2024-03-11T22:12:51.512999+0000","last_clean":"2024-03-11T22:12:51.512999+0000","last_became_active":"2024-03-11T22:10:35.415644+0000","last_became_peered":"2024-03-11T22:10:35.415644+0000","last_unstale":"2024-03-11T22:12:51.512999+0000","last_undegraded":"2024-03-11T22:12:51.512999+0000","last_fullsized":"2024-03-11T22:12:51.512999+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:12.415483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:12.415483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"93'13","reported_seq":182,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.504594+0000","last_change":"2024-03-11T22:11:35.407557+0000","last_active":"2024-03-11T22:12:51.504594+0000","last_peered":"2024-03-11T22:12:51.504594+0000","last_clean":"2024-03-11T22:12:51.504594+0000","last_became_active":"2024-03-11T22:08:35.085787+0000","last_became_peered":"2024-03-11T22:08:35.085787+0000","last_unstale":"2024-03-11T22:12:51.504594+0000","last_undegraded":"2024-03-11T22:12:51.504594+0000","last_fullsized":"2024-03-11T22:12:51.504594+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:11:35.407301+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.407301+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":96,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.494614+0000","last_change":"2024-03-11T22:11:36.490489+0000","last_active":"2024-03-11T22:12:51.494614+0000","last_peered":"2024-03-11T22:12:51.494614+0000","last_clean":"2024-03-11T22:12:51.494614+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:12:51.494614+0000","last_undegraded":"2024-03-11T22:12:51.494614+0000","last_fullsized":"2024-03-11T22:12:51.494614+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:36.490349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:15.415292+0000","last_clean_scrub_stamp":"2024-03-11T22:11:36.490349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:52.894648+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482805+0000","last_change":"2024-03-11T22:11:41.090567+0000","last_active":"2024-03-11T22:12:51.482805+0000","last_peered":"2024-03-11T22:12:51.482805+0000","last_clean":"2024-03-11T22:12:51.482805+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:12:51.482805+0000","last_undegraded":"2024-03-11T22:12:51.482805+0000","last_fullsized":"2024-03-11T22:12:51.482805+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:41.090454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:07.195289+0000","last_clean_scrub_stamp":"2024-03-11T22:11:41.090454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.492649+0000","last_change":"2024-03-11T22:11:31.436443+0000","last_active":"2024-03-11T22:12:51.492649+0000","last_peered":"2024-03-11T22:12:51.492649+0000","last_clean":"2024-03-11T22:12:51.492649+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:12:51.492649+0000","last_undegraded":"2024-03-11T22:12:51.492649+0000","last_fullsized":"2024-03-11T22:12:51.492649+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:31.436306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:11:31.436306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":171,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.506200+0000","last_change":"2024-03-11T22:12:21.278151+0000","last_active":"2024-03-11T22:12:51.506200+0000","last_peered":"2024-03-11T22:12:51.506200+0000","last_clean":"2024-03-11T22:12:51.506200+0000","last_became_active":"2024-03-11T22:08:35.085914+0000","last_became_peered":"2024-03-11T22:08:35.085914+0000","last_unstale":"2024-03-11T22:12:51.506200+0000","last_undegraded":"2024-03-11T22:12:51.506200+0000","last_fullsized":"2024-03-11T22:12:51.506200+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:12:21.270671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:21.270671+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:41.395498+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"137'12311","reported_seq":12620,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.286332+0000","last_change":"2024-03-11T22:12:25.136636+0000","last_active":"2024-03-11T22:12:55.286332+0000","last_peered":"2024-03-11T22:12:55.286332+0000","last_clean":"2024-03-11T22:12:55.286332+0000","last_became_active":"2024-03-11T22:10:36.410723+0000","last_became_peered":"2024-03-11T22:10:36.410723+0000","last_unstale":"2024-03-11T22:12:55.286332+0000","last_undegraded":"2024-03-11T22:12:55.286332+0000","last_fullsized":"2024-03-11T22:12:55.286332+0000","mapping_epoch":99,"log_start":"122'7300","ondisk_log_start":"122'7300","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'10496","last_scrub_stamp":"2024-03-11T22:12:25.136305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:25.136305+0000","objects_scrubbed":6681,"log_size":5011,"log_dups_size":0,"ondisk_log_size":5011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:31.345968+0000","scrub_duration":2244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314376192,"num_objects":4797,"num_object_clones":0,"num_object_copies":14391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9831,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482782+0000","last_change":"2024-03-11T22:12:38.434933+0000","last_active":"2024-03-11T22:12:51.482782+0000","last_peered":"2024-03-11T22:12:51.482782+0000","last_clean":"2024-03-11T22:12:51.482782+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:12:51.482782+0000","last_undegraded":"2024-03-11T22:12:51.482782+0000","last_fullsized":"2024-03-11T22:12:51.482782+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:38.434796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.434796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:04.782006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":99,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.513651+0000","last_change":"2024-03-11T22:10:35.412939+0000","last_active":"2024-03-11T22:12:51.513651+0000","last_peered":"2024-03-11T22:12:51.513651+0000","last_clean":"2024-03-11T22:12:51.513651+0000","last_became_active":"2024-03-11T22:10:35.412565+0000","last_became_peered":"2024-03-11T22:10:35.412565+0000","last_unstale":"2024-03-11T22:12:51.513651+0000","last_undegraded":"2024-03-11T22:12:51.513651+0000","last_fullsized":"2024-03-11T22:12:51.513651+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:10:05.284508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:05.284508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"91'22","reported_seq":194,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482068+0000","last_change":"2024-03-11T22:11:12.222700+0000","last_active":"2024-03-11T22:12:51.482068+0000","last_peered":"2024-03-11T22:12:51.482068+0000","last_clean":"2024-03-11T22:12:51.482068+0000","last_became_active":"2024-03-11T22:08:35.085916+0000","last_became_peered":"2024-03-11T22:08:35.085916+0000","last_unstale":"2024-03-11T22:12:51.482068+0000","last_undegraded":"2024-03-11T22:12:51.482068+0000","last_fullsized":"2024-03-11T22:12:51.482068+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'22","last_scrub_stamp":"2024-03-11T22:11:12.222573+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:11:12.222573+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"99'9","reported_seq":151,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.493417+0000","last_change":"2024-03-11T22:11:25.476405+0000","last_active":"2024-03-11T22:12:51.493417+0000","last_peered":"2024-03-11T22:12:51.493417+0000","last_clean":"2024-03-11T22:12:51.493417+0000","last_became_active":"2024-03-11T22:10:35.413089+0000","last_became_peered":"2024-03-11T22:10:35.413089+0000","last_unstale":"2024-03-11T22:12:51.493417+0000","last_undegraded":"2024-03-11T22:12:51.493417+0000","last_fullsized":"2024-03-11T22:12:51.493417+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'9","last_scrub_stamp":"2024-03-11T22:11:25.476283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.476283+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:57.923099+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"137'9902","reported_seq":10059,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.300531+0000","last_change":"2024-03-11T22:12:38.052754+0000","last_active":"2024-03-11T22:12:55.300531+0000","last_peered":"2024-03-11T22:12:55.300531+0000","last_clean":"2024-03-11T22:12:55.300531+0000","last_became_active":"2024-03-11T22:10:36.411631+0000","last_became_peered":"2024-03-11T22:10:36.411631+0000","last_unstale":"2024-03-11T22:12:55.300531+0000","last_undegraded":"2024-03-11T22:12:55.300531+0000","last_fullsized":"2024-03-11T22:12:55.300531+0000","mapping_epoch":99,"log_start":"118'5200","ondisk_log_start":"118'5200","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'8787","last_scrub_stamp":"2024-03-11T22:12:38.052573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.052573+0000","objects_scrubbed":6049,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:03.556832+0000","scrub_duration":2625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318898176,"num_objects":4866,"num_object_clones":0,"num_object_copies":14598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9902,"num_write_kb":472576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":156,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.501424+0000","last_change":"2024-03-11T22:12:36.449430+0000","last_active":"2024-03-11T22:12:51.501424+0000","last_peered":"2024-03-11T22:12:51.501424+0000","last_clean":"2024-03-11T22:12:51.501424+0000","last_became_active":"2024-03-11T22:08:39.083141+0000","last_became_peered":"2024-03-11T22:08:39.083141+0000","last_unstale":"2024-03-11T22:12:51.501424+0000","last_undegraded":"2024-03-11T22:12:51.501424+0000","last_fullsized":"2024-03-11T22:12:51.501424+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:12:36.449307+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:12:36.449307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:00.397020+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":98,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.513513+0000","last_change":"2024-03-11T22:12:29.074360+0000","last_active":"2024-03-11T22:12:51.513513+0000","last_peered":"2024-03-11T22:12:51.513513+0000","last_clean":"2024-03-11T22:12:51.513513+0000","last_became_active":"2024-03-11T22:10:35.413337+0000","last_became_peered":"2024-03-11T22:10:35.413337+0000","last_unstale":"2024-03-11T22:12:51.513513+0000","last_undegraded":"2024-03-11T22:12:51.513513+0000","last_fullsized":"2024-03-11T22:12:51.513513+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:29.074256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:29.074256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:31.925354+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":195,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.513060+0000","last_change":"2024-03-11T22:11:21.379985+0000","last_active":"2024-03-11T22:12:51.513060+0000","last_peered":"2024-03-11T22:12:51.513060+0000","last_clean":"2024-03-11T22:12:51.513060+0000","last_became_active":"2024-03-11T22:08:54.607153+0000","last_became_peered":"2024-03-11T22:08:54.607153+0000","last_unstale":"2024-03-11T22:12:51.513060+0000","last_undegraded":"2024-03-11T22:12:51.513060+0000","last_fullsized":"2024-03-11T22:12:51.513060+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:11:21.379862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:21.379862+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.507469+0000","last_change":"2024-03-11T22:12:38.757349+0000","last_active":"2024-03-11T22:12:51.507469+0000","last_peered":"2024-03-11T22:12:51.507469+0000","last_clean":"2024-03-11T22:12:51.507469+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:12:51.507469+0000","last_undegraded":"2024-03-11T22:12:51.507469+0000","last_fullsized":"2024-03-11T22:12:51.507469+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.757164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:48.916257+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"137'9924","reported_seq":10011,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.349639+0000","last_change":"2024-03-11T22:10:36.413014+0000","last_active":"2024-03-11T22:12:55.349639+0000","last_peered":"2024-03-11T22:12:55.349639+0000","last_clean":"2024-03-11T22:12:55.349639+0000","last_became_active":"2024-03-11T22:10:36.412662+0000","last_became_peered":"2024-03-11T22:10:36.412662+0000","last_unstale":"2024-03-11T22:12:55.349639+0000","last_undegraded":"2024-03-11T22:12:55.349639+0000","last_fullsized":"2024-03-11T22:12:55.349639+0000","mapping_epoch":99,"log_start":"114'4300","ondisk_log_start":"114'4300","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":5624,"log_dups_size":0,"ondisk_log_size":5624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:57.874263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319160320,"num_objects":4870,"num_object_clones":0,"num_object_copies":14610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9924,"num_write_kb":473408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":105,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.506893+0000","last_change":"2024-03-11T22:12:34.731700+0000","last_active":"2024-03-11T22:12:51.506893+0000","last_peered":"2024-03-11T22:12:51.506893+0000","last_clean":"2024-03-11T22:12:51.506893+0000","last_became_active":"2024-03-11T22:10:35.410893+0000","last_became_peered":"2024-03-11T22:10:35.410893+0000","last_unstale":"2024-03-11T22:12:51.506893+0000","last_undegraded":"2024-03-11T22:12:51.506893+0000","last_fullsized":"2024-03-11T22:12:51.506893+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:34.731409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:34.731409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:42.767039+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.506413+0000","last_change":"2024-03-11T22:11:28.377967+0000","last_active":"2024-03-11T22:12:51.506413+0000","last_peered":"2024-03-11T22:12:51.506413+0000","last_clean":"2024-03-11T22:12:51.506413+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:12:51.506413+0000","last_undegraded":"2024-03-11T22:12:51.506413+0000","last_fullsized":"2024-03-11T22:12:51.506413+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:28.377844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.710319+0000","last_clean_scrub_stamp":"2024-03-11T22:11:28.377844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":106,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482785+0000","last_change":"2024-03-11T22:12:44.474800+0000","last_active":"2024-03-11T22:12:51.482785+0000","last_peered":"2024-03-11T22:12:51.482785+0000","last_clean":"2024-03-11T22:12:51.482785+0000","last_became_active":"2024-03-11T22:10:35.418715+0000","last_became_peered":"2024-03-11T22:10:35.418715+0000","last_unstale":"2024-03-11T22:12:51.482785+0000","last_undegraded":"2024-03-11T22:12:51.482785+0000","last_fullsized":"2024-03-11T22:12:51.482785+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.474728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.474728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:44.686186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":201,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.493589+0000","last_change":"2024-03-11T22:12:36.843050+0000","last_active":"2024-03-11T22:12:51.493589+0000","last_peered":"2024-03-11T22:12:51.493589+0000","last_clean":"2024-03-11T22:12:51.493589+0000","last_became_active":"2024-03-11T22:08:35.087046+0000","last_became_peered":"2024-03-11T22:08:35.087046+0000","last_unstale":"2024-03-11T22:12:51.493589+0000","last_undegraded":"2024-03-11T22:12:51.493589+0000","last_fullsized":"2024-03-11T22:12:51.493589+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:12:36.842912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:36.842912+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:04.592208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"137'10087","reported_seq":10251,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.281392+0000","last_change":"2024-03-11T22:12:25.097684+0000","last_active":"2024-03-11T22:12:55.281392+0000","last_peered":"2024-03-11T22:12:55.281392+0000","last_clean":"2024-03-11T22:12:55.281392+0000","last_became_active":"2024-03-11T22:10:36.410947+0000","last_became_peered":"2024-03-11T22:10:36.410947+0000","last_unstale":"2024-03-11T22:12:55.281392+0000","last_undegraded":"2024-03-11T22:12:55.281392+0000","last_fullsized":"2024-03-11T22:12:55.281392+0000","mapping_epoch":99,"log_start":"111'3200","ondisk_log_start":"111'3200","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'8252","last_scrub_stamp":"2024-03-11T22:12:25.097553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:25.097553+0000","objects_scrubbed":6851,"log_size":6887,"log_dups_size":0,"ondisk_log_size":6887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:43.600745+0000","scrub_duration":2502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323158016,"num_objects":4931,"num_object_clones":0,"num_object_copies":14793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10087,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.503841+0000","last_change":"2024-03-11T22:12:28.683756+0000","last_active":"2024-03-11T22:12:51.503841+0000","last_peered":"2024-03-11T22:12:51.503841+0000","last_clean":"2024-03-11T22:12:51.503841+0000","last_became_active":"2024-03-11T22:08:39.085496+0000","last_became_peered":"2024-03-11T22:08:39.085496+0000","last_unstale":"2024-03-11T22:12:51.503841+0000","last_undegraded":"2024-03-11T22:12:51.503841+0000","last_fullsized":"2024-03-11T22:12:51.503841+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:28.683646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:28.683646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:52.658563+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.506769+0000","last_change":"2024-03-11T22:11:36.289334+0000","last_active":"2024-03-11T22:12:51.506769+0000","last_peered":"2024-03-11T22:12:51.506769+0000","last_clean":"2024-03-11T22:12:51.506769+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:12:51.506769+0000","last_undegraded":"2024-03-11T22:12:51.506769+0000","last_fullsized":"2024-03-11T22:12:51.506769+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:11:36.289016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482474+0000","last_change":"2024-03-11T22:12:34.391183+0000","last_active":"2024-03-11T22:12:51.482474+0000","last_peered":"2024-03-11T22:12:51.482474+0000","last_clean":"2024-03-11T22:12:51.482474+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:12:51.482474+0000","last_undegraded":"2024-03-11T22:12:51.482474+0000","last_fullsized":"2024-03-11T22:12:51.482474+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:34.391054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:34.391054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:45.133139+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":186,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.500726+0000","last_change":"2024-03-11T22:10:17.179119+0000","last_active":"2024-03-11T22:12:51.500726+0000","last_peered":"2024-03-11T22:12:51.500726+0000","last_clean":"2024-03-11T22:12:51.500726+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:12:51.500726+0000","last_undegraded":"2024-03-11T22:12:51.500726+0000","last_fullsized":"2024-03-11T22:12:51.500726+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:10:17.178987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:17.178987+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"137'9955","reported_seq":10114,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.126320+0000","last_change":"2024-03-11T22:12:30.660272+0000","last_active":"2024-03-11T22:12:55.126320+0000","last_peered":"2024-03-11T22:12:55.126320+0000","last_clean":"2024-03-11T22:12:55.126320+0000","last_became_active":"2024-03-11T22:10:36.416476+0000","last_became_peered":"2024-03-11T22:10:36.416476+0000","last_unstale":"2024-03-11T22:12:55.126320+0000","last_undegraded":"2024-03-11T22:12:55.126320+0000","last_fullsized":"2024-03-11T22:12:55.126320+0000","mapping_epoch":99,"log_start":"112'3700","ondisk_log_start":"112'3700","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8438","last_scrub_stamp":"2024-03-11T22:12:30.660018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:30.660018+0000","objects_scrubbed":6392,"log_size":6255,"log_dups_size":0,"ondisk_log_size":6255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:40.736516+0000","scrub_duration":1794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316080128,"num_objects":4823,"num_object_clones":0,"num_object_copies":14469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9955,"num_write_kb":472896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":155,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.499688+0000","last_change":"2024-03-11T22:12:45.439711+0000","last_active":"2024-03-11T22:12:51.499688+0000","last_peered":"2024-03-11T22:12:51.499688+0000","last_clean":"2024-03-11T22:12:51.499688+0000","last_became_active":"2024-03-11T22:08:39.083738+0000","last_became_peered":"2024-03-11T22:08:39.083738+0000","last_unstale":"2024-03-11T22:12:51.499688+0000","last_undegraded":"2024-03-11T22:12:51.499688+0000","last_fullsized":"2024-03-11T22:12:51.499688+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.439584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:47.160579+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":179,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482299+0000","last_change":"2024-03-11T22:11:42.128436+0000","last_active":"2024-03-11T22:12:51.482299+0000","last_peered":"2024-03-11T22:12:51.482299+0000","last_clean":"2024-03-11T22:12:51.482299+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:12:51.482299+0000","last_undegraded":"2024-03-11T22:12:51.482299+0000","last_fullsized":"2024-03-11T22:12:51.482299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:11:42.128356+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:11:42.128356+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.282739+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":159,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.511014+0000","last_change":"2024-03-11T22:09:55.458484+0000","last_active":"2024-03-11T22:12:51.511014+0000","last_peered":"2024-03-11T22:12:51.511014+0000","last_clean":"2024-03-11T22:12:51.511014+0000","last_became_active":"2024-03-11T22:08:33.035469+0000","last_became_peered":"2024-03-11T22:08:33.035469+0000","last_unstale":"2024-03-11T22:12:51.511014+0000","last_undegraded":"2024-03-11T22:12:51.511014+0000","last_fullsized":"2024-03-11T22:12:51.511014+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:55.458385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:55.458385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":264,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.488761+0000","last_change":"2024-03-11T22:11:24.895982+0000","last_active":"2024-03-11T22:12:51.488761+0000","last_peered":"2024-03-11T22:12:51.488761+0000","last_clean":"2024-03-11T22:12:51.488761+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:12:51.488761+0000","last_undegraded":"2024-03-11T22:12:51.488761+0000","last_fullsized":"2024-03-11T22:12:51.488761+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-11T22:11:24.892593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:24.892593+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:56.891123+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":159,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.500963+0000","last_change":"2024-03-11T22:11:14.224976+0000","last_active":"2024-03-11T22:12:51.500963+0000","last_peered":"2024-03-11T22:12:51.500963+0000","last_clean":"2024-03-11T22:12:51.500963+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:12:51.500963+0000","last_undegraded":"2024-03-11T22:12:51.500963+0000","last_fullsized":"2024-03-11T22:12:51.500963+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:14.224859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:14.224859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:57.889956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"137'10057","reported_seq":10144,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.387593+0000","last_change":"2024-03-11T22:10:36.413021+0000","last_active":"2024-03-11T22:12:55.387593+0000","last_peered":"2024-03-11T22:12:55.387593+0000","last_clean":"2024-03-11T22:12:55.387593+0000","last_became_active":"2024-03-11T22:10:36.412694+0000","last_became_peered":"2024-03-11T22:10:36.412694+0000","last_unstale":"2024-03-11T22:12:55.387593+0000","last_undegraded":"2024-03-11T22:12:55.387593+0000","last_fullsized":"2024-03-11T22:12:55.387593+0000","mapping_epoch":99,"log_start":"114'4500","ondisk_log_start":"114'4500","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:57.877989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322764800,"num_objects":4925,"num_object_clones":0,"num_object_copies":14775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":479424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":97,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.494801+0000","last_change":"2024-03-11T22:11:35.482382+0000","last_active":"2024-03-11T22:12:51.494801+0000","last_peered":"2024-03-11T22:12:51.494801+0000","last_clean":"2024-03-11T22:12:51.494801+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:12:51.494801+0000","last_undegraded":"2024-03-11T22:12:51.494801+0000","last_fullsized":"2024-03-11T22:12:51.494801+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:35.482269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.482269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":167,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.489975+0000","last_change":"2024-03-11T22:11:19.960632+0000","last_active":"2024-03-11T22:12:51.489975+0000","last_peered":"2024-03-11T22:12:51.489975+0000","last_clean":"2024-03-11T22:12:51.489975+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:12:51.489975+0000","last_undegraded":"2024-03-11T22:12:51.489975+0000","last_fullsized":"2024-03-11T22:12:51.489975+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:19.960398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:19.960398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":184,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.505864+0000","last_change":"2024-03-11T22:12:45.863561+0000","last_active":"2024-03-11T22:12:51.505864+0000","last_peered":"2024-03-11T22:12:51.505864+0000","last_clean":"2024-03-11T22:12:51.505864+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:12:51.505864+0000","last_undegraded":"2024-03-11T22:12:51.505864+0000","last_fullsized":"2024-03-11T22:12:51.505864+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.863440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.863440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:09.786177+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":190,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.488493+0000","last_change":"2024-03-11T22:11:23.875763+0000","last_active":"2024-03-11T22:12:51.488493+0000","last_peered":"2024-03-11T22:12:51.488493+0000","last_clean":"2024-03-11T22:12:51.488493+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:12:51.488493+0000","last_undegraded":"2024-03-11T22:12:51.488493+0000","last_fullsized":"2024-03-11T22:12:51.488493+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-03-11T22:11:23.875628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:23.875628+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:56.891227+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":167,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.509273+0000","last_change":"2024-03-11T22:12:31.669715+0000","last_active":"2024-03-11T22:12:51.509273+0000","last_peered":"2024-03-11T22:12:51.509273+0000","last_clean":"2024-03-11T22:12:51.509273+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:12:51.509273+0000","last_undegraded":"2024-03-11T22:12:51.509273+0000","last_fullsized":"2024-03-11T22:12:51.509273+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:31.669555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.669555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:47.635385+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"137'10016","reported_seq":10103,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.382154+0000","last_change":"2024-03-11T22:10:36.409137+0000","last_active":"2024-03-11T22:12:55.382154+0000","last_peered":"2024-03-11T22:12:55.382154+0000","last_clean":"2024-03-11T22:12:55.382154+0000","last_became_active":"2024-03-11T22:10:36.408867+0000","last_became_peered":"2024-03-11T22:10:36.408867+0000","last_unstale":"2024-03-11T22:12:55.382154+0000","last_undegraded":"2024-03-11T22:12:55.382154+0000","last_fullsized":"2024-03-11T22:12:55.382154+0000","mapping_epoch":99,"log_start":"114'4400","ondisk_log_start":"114'4400","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":5616,"log_dups_size":0,"ondisk_log_size":5616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:57.889856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316932096,"num_objects":4836,"num_object_clones":0,"num_object_copies":14508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10016,"num_write_kb":475264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482683+0000","last_change":"2024-03-11T22:11:46.117166+0000","last_active":"2024-03-11T22:12:51.482683+0000","last_peered":"2024-03-11T22:12:51.482683+0000","last_clean":"2024-03-11T22:12:51.482683+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:12:51.482683+0000","last_undegraded":"2024-03-11T22:12:51.482683+0000","last_fullsized":"2024-03-11T22:12:51.482683+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:46.117039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:46.117039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:02.937947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":414,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.499638+0000","last_change":"2024-03-11T22:11:34.382394+0000","last_active":"2024-03-11T22:12:51.499638+0000","last_peered":"2024-03-11T22:12:51.499638+0000","last_clean":"2024-03-11T22:12:51.499638+0000","last_became_active":"2024-03-11T22:10:35.412943+0000","last_became_peered":"2024-03-11T22:10:35.412943+0000","last_unstale":"2024-03-11T22:12:51.499638+0000","last_undegraded":"2024-03-11T22:12:51.499638+0000","last_fullsized":"2024-03-11T22:12:51.499638+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:11:34.382250+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.382250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":167,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.501759+0000","last_change":"2024-03-11T22:11:34.039067+0000","last_active":"2024-03-11T22:12:51.501759+0000","last_peered":"2024-03-11T22:12:51.501759+0000","last_clean":"2024-03-11T22:12:51.501759+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:12:51.501759+0000","last_undegraded":"2024-03-11T22:12:51.501759+0000","last_fullsized":"2024-03-11T22:12:51.501759+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.038930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":238,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.494890+0000","last_change":"2024-03-11T22:11:30.413882+0000","last_active":"2024-03-11T22:12:51.494890+0000","last_peered":"2024-03-11T22:12:51.494890+0000","last_clean":"2024-03-11T22:12:51.494890+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:12:51.494890+0000","last_undegraded":"2024-03-11T22:12:51.494890+0000","last_fullsized":"2024-03-11T22:12:51.494890+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-11T22:11:30.413737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.413737+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:57.923202+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":105,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.506625+0000","last_change":"2024-03-11T22:12:33.715464+0000","last_active":"2024-03-11T22:12:51.506625+0000","last_peered":"2024-03-11T22:12:51.506625+0000","last_clean":"2024-03-11T22:12:51.506625+0000","last_became_active":"2024-03-11T22:10:35.412547+0000","last_became_peered":"2024-03-11T22:10:35.412547+0000","last_unstale":"2024-03-11T22:12:51.506625+0000","last_undegraded":"2024-03-11T22:12:51.506625+0000","last_fullsized":"2024-03-11T22:12:51.506625+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:33.715347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:12:33.715347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:47.739223+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"137'10014","reported_seq":10163,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.366048+0000","last_change":"2024-03-11T22:11:42.189504+0000","last_active":"2024-03-11T22:12:55.366048+0000","last_peered":"2024-03-11T22:12:55.366048+0000","last_clean":"2024-03-11T22:12:55.366048+0000","last_became_active":"2024-03-11T22:10:36.406423+0000","last_became_peered":"2024-03-11T22:10:36.406423+0000","last_unstale":"2024-03-11T22:12:55.366048+0000","last_undegraded":"2024-03-11T22:12:55.366048+0000","last_fullsized":"2024-03-11T22:12:55.366048+0000","mapping_epoch":99,"log_start":"111'3100","ondisk_log_start":"111'3100","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5593","last_scrub_stamp":"2024-03-11T22:11:42.188394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:11:42.188394+0000","objects_scrubbed":5541,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.768322+0000","scrub_duration":1878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318242816,"num_objects":4856,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10014,"num_write_kb":475840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":163,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.488379+0000","last_change":"2024-03-11T22:12:44.933303+0000","last_active":"2024-03-11T22:12:51.488379+0000","last_peered":"2024-03-11T22:12:51.488379+0000","last_clean":"2024-03-11T22:12:51.488379+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:12:51.488379+0000","last_undegraded":"2024-03-11T22:12:51.488379+0000","last_fullsized":"2024-03-11T22:12:51.488379+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.927967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.927967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:13.766656+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":167,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.504061+0000","last_change":"2024-03-11T22:12:19.685277+0000","last_active":"2024-03-11T22:12:51.504061+0000","last_peered":"2024-03-11T22:12:51.504061+0000","last_clean":"2024-03-11T22:12:51.504061+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:12:51.504061+0000","last_undegraded":"2024-03-11T22:12:51.504061+0000","last_fullsized":"2024-03-11T22:12:51.504061+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:19.684663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:19.684663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:48.270422+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":411,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:53.821164+0000","last_change":"2024-03-11T22:11:34.857427+0000","last_active":"2024-03-11T22:12:53.821164+0000","last_peered":"2024-03-11T22:12:53.821164+0000","last_clean":"2024-03-11T22:12:53.821164+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:12:53.821164+0000","last_undegraded":"2024-03-11T22:12:53.821164+0000","last_fullsized":"2024-03-11T22:12:53.821164+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:11:34.857300+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.857300+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:02.109560+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":229,"num_read_kb":224,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":181,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.500744+0000","last_change":"2024-03-11T22:11:06.292270+0000","last_active":"2024-03-11T22:12:51.500744+0000","last_peered":"2024-03-11T22:12:51.500744+0000","last_clean":"2024-03-11T22:12:51.500744+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:12:51.500744+0000","last_undegraded":"2024-03-11T22:12:51.500744+0000","last_fullsized":"2024-03-11T22:12:51.500744+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:11:06.292138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:06.292138+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:57.889986+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":159,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482997+0000","last_change":"2024-03-11T22:12:27.459283+0000","last_active":"2024-03-11T22:12:51.482997+0000","last_peered":"2024-03-11T22:12:51.482997+0000","last_clean":"2024-03-11T22:12:51.482997+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:12:51.482997+0000","last_undegraded":"2024-03-11T22:12:51.482997+0000","last_fullsized":"2024-03-11T22:12:51.482997+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:27.459154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:27.459154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:36.420728+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"137'9809","reported_seq":9971,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.105807+0000","last_change":"2024-03-11T22:12:25.123093+0000","last_active":"2024-03-11T22:12:55.105807+0000","last_peered":"2024-03-11T22:12:55.105807+0000","last_clean":"2024-03-11T22:12:55.105807+0000","last_became_active":"2024-03-11T22:10:36.411289+0000","last_became_peered":"2024-03-11T22:10:36.411289+0000","last_unstale":"2024-03-11T22:12:55.105807+0000","last_undegraded":"2024-03-11T22:12:55.105807+0000","last_fullsized":"2024-03-11T22:12:55.105807+0000","mapping_epoch":99,"log_start":"114'4000","ondisk_log_start":"114'4000","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7985","last_scrub_stamp":"2024-03-11T22:12:25.122944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:25.122944+0000","objects_scrubbed":6655,"log_size":5809,"log_dups_size":0,"ondisk_log_size":5809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:35.031889+0000","scrub_duration":1760,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314245120,"num_objects":4795,"num_object_clones":0,"num_object_copies":14385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9809,"num_write_kb":467328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":155,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.496749+0000","last_change":"2024-03-11T22:11:28.871970+0000","last_active":"2024-03-11T22:12:51.496749+0000","last_peered":"2024-03-11T22:12:51.496749+0000","last_clean":"2024-03-11T22:12:51.496749+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:12:51.496749+0000","last_undegraded":"2024-03-11T22:12:51.496749+0000","last_fullsized":"2024-03-11T22:12:51.496749+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:28.871246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:28.871246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":167,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.488613+0000","last_change":"2024-03-11T22:12:42.951642+0000","last_active":"2024-03-11T22:12:51.488613+0000","last_peered":"2024-03-11T22:12:51.488613+0000","last_clean":"2024-03-11T22:12:51.488613+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:12:51.488613+0000","last_undegraded":"2024-03-11T22:12:51.488613+0000","last_fullsized":"2024-03-11T22:12:51.488613+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.951379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.951379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:10.673379+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":191,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.500398+0000","last_change":"2024-03-11T22:10:31.211684+0000","last_active":"2024-03-11T22:12:51.500398+0000","last_peered":"2024-03-11T22:12:51.500398+0000","last_clean":"2024-03-11T22:12:51.500398+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:12:51.500398+0000","last_undegraded":"2024-03-11T22:12:51.500398+0000","last_fullsized":"2024-03-11T22:12:51.500398+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:10:31.211547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:31.211547+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":167,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.493612+0000","last_change":"2024-03-11T22:12:40.863550+0000","last_active":"2024-03-11T22:12:51.493612+0000","last_peered":"2024-03-11T22:12:51.493612+0000","last_clean":"2024-03-11T22:12:51.493612+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:12:51.493612+0000","last_undegraded":"2024-03-11T22:12:51.493612+0000","last_fullsized":"2024-03-11T22:12:51.493612+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:12:40.863445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.360193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"137'9943","reported_seq":10030,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.047709+0000","last_change":"2024-03-11T22:10:36.416445+0000","last_active":"2024-03-11T22:12:55.047709+0000","last_peered":"2024-03-11T22:12:55.047709+0000","last_clean":"2024-03-11T22:12:55.047709+0000","last_became_active":"2024-03-11T22:10:36.416083+0000","last_became_peered":"2024-03-11T22:10:36.416083+0000","last_unstale":"2024-03-11T22:12:55.047709+0000","last_undegraded":"2024-03-11T22:12:55.047709+0000","last_fullsized":"2024-03-11T22:12:55.047709+0000","mapping_epoch":99,"log_start":"118'5100","ondisk_log_start":"118'5100","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311623680,"num_objects":4755,"num_object_clones":0,"num_object_copies":14265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9943,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":147,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.499548+0000","last_change":"2024-03-11T22:11:11.177052+0000","last_active":"2024-03-11T22:12:51.499548+0000","last_peered":"2024-03-11T22:12:51.499548+0000","last_clean":"2024-03-11T22:12:51.499548+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:12:51.499548+0000","last_undegraded":"2024-03-11T22:12:51.499548+0000","last_fullsized":"2024-03-11T22:12:51.499548+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:11.176930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:11.176930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":175,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482730+0000","last_change":"2024-03-11T22:12:45.466625+0000","last_active":"2024-03-11T22:12:51.482730+0000","last_peered":"2024-03-11T22:12:51.482730+0000","last_clean":"2024-03-11T22:12:51.482730+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:12:51.482730+0000","last_undegraded":"2024-03-11T22:12:51.482730+0000","last_fullsized":"2024-03-11T22:12:51.482730+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.466508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.466508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:09.930146+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":178,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.489039+0000","last_change":"2024-03-11T22:12:38.896205+0000","last_active":"2024-03-11T22:12:51.489039+0000","last_peered":"2024-03-11T22:12:51.489039+0000","last_clean":"2024-03-11T22:12:51.489039+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:12:51.489039+0000","last_undegraded":"2024-03-11T22:12:51.489039+0000","last_fullsized":"2024-03-11T22:12:51.489039+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:12:38.895763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.895763+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:08.358608+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":220,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.500895+0000","last_change":"2024-03-11T22:11:25.385148+0000","last_active":"2024-03-11T22:12:51.500895+0000","last_peered":"2024-03-11T22:12:51.500895+0000","last_clean":"2024-03-11T22:12:51.500895+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:12:51.500895+0000","last_undegraded":"2024-03-11T22:12:51.500895+0000","last_fullsized":"2024-03-11T22:12:51.500895+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:11:25.384965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.384965+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:57.890103+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"137'9906","reported_seq":10063,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.311021+0000","last_change":"2024-03-11T22:12:35.256058+0000","last_active":"2024-03-11T22:12:55.311021+0000","last_peered":"2024-03-11T22:12:55.311021+0000","last_clean":"2024-03-11T22:12:55.311021+0000","last_became_active":"2024-03-11T22:10:36.410859+0000","last_became_peered":"2024-03-11T22:10:36.410859+0000","last_unstale":"2024-03-11T22:12:55.311021+0000","last_undegraded":"2024-03-11T22:12:55.311021+0000","last_fullsized":"2024-03-11T22:12:55.311021+0000","mapping_epoch":99,"log_start":"118'5200","ondisk_log_start":"118'5200","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8663","last_scrub_stamp":"2024-03-11T22:12:35.255739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.255739+0000","objects_scrubbed":6218,"log_size":4706,"log_dups_size":0,"ondisk_log_size":4706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:47.000467+0000","scrub_duration":2947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321126400,"num_objects":4900,"num_object_clones":0,"num_object_copies":14700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9906,"num_write_kb":473792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":105,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.494509+0000","last_change":"2024-03-11T22:12:46.968879+0000","last_active":"2024-03-11T22:12:51.494509+0000","last_peered":"2024-03-11T22:12:51.494509+0000","last_clean":"2024-03-11T22:12:51.494509+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:12:51.494509+0000","last_undegraded":"2024-03-11T22:12:51.494509+0000","last_fullsized":"2024-03-11T22:12:51.494509+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.968807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.968807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:12.616431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":168,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.500852+0000","last_change":"2024-03-11T22:12:49.856525+0000","last_active":"2024-03-11T22:12:51.500852+0000","last_peered":"2024-03-11T22:12:51.500852+0000","last_clean":"2024-03-11T22:12:51.500852+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:12:51.500852+0000","last_undegraded":"2024-03-11T22:12:51.500852+0000","last_fullsized":"2024-03-11T22:12:51.500852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:49.856381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.856381+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:57.055758+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.512797+0000","last_change":"2024-03-11T22:09:53.414248+0000","last_active":"2024-03-11T22:12:51.512797+0000","last_peered":"2024-03-11T22:12:51.512797+0000","last_clean":"2024-03-11T22:12:51.512797+0000","last_became_active":"2024-03-11T22:08:35.053452+0000","last_became_peered":"2024-03-11T22:08:35.053452+0000","last_unstale":"2024-03-11T22:12:51.512797+0000","last_undegraded":"2024-03-11T22:12:51.512797+0000","last_fullsized":"2024-03-11T22:12:51.512797+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:09:53.414127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:53.414127+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":220,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.506044+0000","last_change":"2024-03-11T22:12:31.264333+0000","last_active":"2024-03-11T22:12:51.506044+0000","last_peered":"2024-03-11T22:12:51.506044+0000","last_clean":"2024-03-11T22:12:51.506044+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:12:51.506044+0000","last_undegraded":"2024-03-11T22:12:51.506044+0000","last_fullsized":"2024-03-11T22:12:51.506044+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:12:31.264213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.264213+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:45.274053+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"137'9913","reported_seq":10069,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.323169+0000","last_change":"2024-03-11T22:12:35.900151+0000","last_active":"2024-03-11T22:12:55.323169+0000","last_peered":"2024-03-11T22:12:55.323169+0000","last_clean":"2024-03-11T22:12:55.323169+0000","last_became_active":"2024-03-11T22:10:36.414380+0000","last_became_peered":"2024-03-11T22:10:36.414380+0000","last_unstale":"2024-03-11T22:12:55.323169+0000","last_undegraded":"2024-03-11T22:12:55.323169+0000","last_fullsized":"2024-03-11T22:12:55.323169+0000","mapping_epoch":99,"log_start":"118'5200","ondisk_log_start":"118'5200","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8688","last_scrub_stamp":"2024-03-11T22:12:35.899750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.899750+0000","objects_scrubbed":6120,"log_size":4713,"log_dups_size":0,"ondisk_log_size":4713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:44.219720+0000","scrub_duration":2560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315424768,"num_objects":4813,"num_object_clones":0,"num_object_copies":14439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9913,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":147,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.510650+0000","last_change":"2024-03-11T22:11:35.248168+0000","last_active":"2024-03-11T22:12:51.510650+0000","last_peered":"2024-03-11T22:12:51.510650+0000","last_clean":"2024-03-11T22:12:51.510650+0000","last_became_active":"2024-03-11T22:08:39.078244+0000","last_became_peered":"2024-03-11T22:08:39.078244+0000","last_unstale":"2024-03-11T22:12:51.510650+0000","last_undegraded":"2024-03-11T22:12:51.510650+0000","last_fullsized":"2024-03-11T22:12:51.510650+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:35.248081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.248081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":167,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.505951+0000","last_change":"2024-03-11T22:11:24.938132+0000","last_active":"2024-03-11T22:12:51.505951+0000","last_peered":"2024-03-11T22:12:51.505951+0000","last_clean":"2024-03-11T22:12:51.505951+0000","last_became_active":"2024-03-11T22:08:33.036469+0000","last_became_peered":"2024-03-11T22:08:33.036469+0000","last_unstale":"2024-03-11T22:12:51.505951+0000","last_undegraded":"2024-03-11T22:12:51.505951+0000","last_fullsized":"2024-03-11T22:12:51.505951+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:24.937886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:24.937886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":198,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.505895+0000","last_change":"2024-03-11T22:12:39.447777+0000","last_active":"2024-03-11T22:12:51.505895+0000","last_peered":"2024-03-11T22:12:51.505895+0000","last_clean":"2024-03-11T22:12:51.505895+0000","last_became_active":"2024-03-11T22:08:35.085979+0000","last_became_peered":"2024-03-11T22:08:35.085979+0000","last_unstale":"2024-03-11T22:12:51.505895+0000","last_undegraded":"2024-03-11T22:12:51.505895+0000","last_fullsized":"2024-03-11T22:12:51.505895+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:12:39.447312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:39.447312+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:00.055885+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":159,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.512533+0000","last_change":"2024-03-11T22:11:40.165879+0000","last_active":"2024-03-11T22:12:51.512533+0000","last_peered":"2024-03-11T22:12:51.512533+0000","last_clean":"2024-03-11T22:12:51.512533+0000","last_became_active":"2024-03-11T22:08:37.076522+0000","last_became_peered":"2024-03-11T22:08:37.076522+0000","last_unstale":"2024-03-11T22:12:51.512533+0000","last_undegraded":"2024-03-11T22:12:51.512533+0000","last_fullsized":"2024-03-11T22:12:51.512533+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:40.165760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:40.165760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.990830+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"137'9885","reported_seq":10043,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.112127+0000","last_change":"2024-03-11T22:12:31.355260+0000","last_active":"2024-03-11T22:12:55.112127+0000","last_peered":"2024-03-11T22:12:55.112127+0000","last_clean":"2024-03-11T22:12:55.112127+0000","last_became_active":"2024-03-11T22:10:36.414372+0000","last_became_peered":"2024-03-11T22:10:36.414372+0000","last_unstale":"2024-03-11T22:12:55.112127+0000","last_undegraded":"2024-03-11T22:12:55.112127+0000","last_fullsized":"2024-03-11T22:12:55.112127+0000","mapping_epoch":99,"log_start":"112'3600","ondisk_log_start":"112'3600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8481","last_scrub_stamp":"2024-03-11T22:12:31.354979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.354979+0000","objects_scrubbed":6285,"log_size":6285,"log_dups_size":0,"ondisk_log_size":6285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:50.389207+0000","scrub_duration":1504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317259776,"num_objects":4841,"num_object_clones":0,"num_object_copies":14523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9885,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482400+0000","last_change":"2024-03-11T22:12:42.469626+0000","last_active":"2024-03-11T22:12:51.482400+0000","last_peered":"2024-03-11T22:12:51.482400+0000","last_clean":"2024-03-11T22:12:51.482400+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:12:51.482400+0000","last_undegraded":"2024-03-11T22:12:51.482400+0000","last_fullsized":"2024-03-11T22:12:51.482400+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.469501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.469501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:51.863765+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":211,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.513577+0000","last_change":"2024-03-11T22:10:14.545445+0000","last_active":"2024-03-11T22:12:51.513577+0000","last_peered":"2024-03-11T22:12:51.513577+0000","last_clean":"2024-03-11T22:12:51.513577+0000","last_became_active":"2024-03-11T22:08:37.077142+0000","last_became_peered":"2024-03-11T22:08:37.077142+0000","last_unstale":"2024-03-11T22:12:51.513577+0000","last_undegraded":"2024-03-11T22:12:51.513577+0000","last_fullsized":"2024-03-11T22:12:51.513577+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:10:14.545345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:10:14.545345+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":167,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.505827+0000","last_change":"2024-03-11T22:12:24.278539+0000","last_active":"2024-03-11T22:12:51.505827+0000","last_peered":"2024-03-11T22:12:51.505827+0000","last_clean":"2024-03-11T22:12:51.505827+0000","last_became_active":"2024-03-11T22:08:33.037537+0000","last_became_peered":"2024-03-11T22:08:33.037537+0000","last_unstale":"2024-03-11T22:12:51.505827+0000","last_undegraded":"2024-03-11T22:12:51.505827+0000","last_fullsized":"2024-03-11T22:12:51.505827+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:24.278298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.278298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:40.405410+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":187,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.488623+0000","last_change":"2024-03-11T22:11:41.817430+0000","last_active":"2024-03-11T22:12:51.488623+0000","last_peered":"2024-03-11T22:12:51.488623+0000","last_clean":"2024-03-11T22:12:51.488623+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:12:51.488623+0000","last_undegraded":"2024-03-11T22:12:51.488623+0000","last_fullsized":"2024-03-11T22:12:51.488623+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:11:41.817285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:41.817285+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:03.620607+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"137'10207","reported_seq":10366,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.140219+0000","last_change":"2024-03-11T22:12:35.582515+0000","last_active":"2024-03-11T22:12:55.140219+0000","last_peered":"2024-03-11T22:12:55.140219+0000","last_clean":"2024-03-11T22:12:55.140219+0000","last_became_active":"2024-03-11T22:10:36.412847+0000","last_became_peered":"2024-03-11T22:10:36.412847+0000","last_unstale":"2024-03-11T22:12:55.140219+0000","last_undegraded":"2024-03-11T22:12:55.140219+0000","last_fullsized":"2024-03-11T22:12:55.140219+0000","mapping_epoch":99,"log_start":"112'3900","ondisk_log_start":"112'3900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'9001","last_scrub_stamp":"2024-03-11T22:12:35.582375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.582375+0000","objects_scrubbed":6270,"log_size":6307,"log_dups_size":0,"ondisk_log_size":6307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:02.117832+0000","scrub_duration":1709,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329646104,"num_objects":5031,"num_object_clones":0,"num_object_copies":15093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10207,"num_write_kb":487553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":146,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.505712+0000","last_change":"2024-03-11T22:11:08.902041+0000","last_active":"2024-03-11T22:12:51.505712+0000","last_peered":"2024-03-11T22:12:51.505712+0000","last_clean":"2024-03-11T22:12:51.505712+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:12:51.505712+0000","last_undegraded":"2024-03-11T22:12:51.505712+0000","last_fullsized":"2024-03-11T22:12:51.505712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:08.901927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:08.901927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.482945+0000","last_change":"2024-03-11T22:12:20.302654+0000","last_active":"2024-03-11T22:12:51.482945+0000","last_peered":"2024-03-11T22:12:51.482945+0000","last_clean":"2024-03-11T22:12:51.482945+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:12:51.482945+0000","last_undegraded":"2024-03-11T22:12:51.482945+0000","last_fullsized":"2024-03-11T22:12:51.482945+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:20.302574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:20.302574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:30.645408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":159,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.512697+0000","last_change":"2024-03-11T22:11:23.353503+0000","last_active":"2024-03-11T22:12:51.512697+0000","last_peered":"2024-03-11T22:12:51.512697+0000","last_clean":"2024-03-11T22:12:51.512697+0000","last_became_active":"2024-03-11T22:08:33.299778+0000","last_became_peered":"2024-03-11T22:08:33.299778+0000","last_unstale":"2024-03-11T22:12:51.512697+0000","last_undegraded":"2024-03-11T22:12:51.512697+0000","last_fullsized":"2024-03-11T22:12:51.512697+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:23.353379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:23.353379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":181,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.513326+0000","last_change":"2024-03-11T22:12:24.093258+0000","last_active":"2024-03-11T22:12:51.513326+0000","last_peered":"2024-03-11T22:12:51.513326+0000","last_clean":"2024-03-11T22:12:51.513326+0000","last_became_active":"2024-03-11T22:08:35.053123+0000","last_became_peered":"2024-03-11T22:08:35.053123+0000","last_unstale":"2024-03-11T22:12:51.513326+0000","last_undegraded":"2024-03-11T22:12:51.513326+0000","last_fullsized":"2024-03-11T22:12:51.513326+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:12:24.093112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.093112+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:53.468276+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"137'9914","reported_seq":10068,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.350290+0000","last_change":"2024-03-11T22:11:46.750236+0000","last_active":"2024-03-11T22:12:55.350290+0000","last_peered":"2024-03-11T22:12:55.350290+0000","last_clean":"2024-03-11T22:12:55.350290+0000","last_became_active":"2024-03-11T22:10:36.412847+0000","last_became_peered":"2024-03-11T22:10:36.412847+0000","last_unstale":"2024-03-11T22:12:55.350290+0000","last_undegraded":"2024-03-11T22:12:55.350290+0000","last_fullsized":"2024-03-11T22:12:55.350290+0000","mapping_epoch":99,"log_start":"111'3000","ondisk_log_start":"111'3000","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'5801","last_scrub_stamp":"2024-03-11T22:11:46.749984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:11:46.749984+0000","objects_scrubbed":5740,"log_size":6914,"log_dups_size":0,"ondisk_log_size":6914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:57.776747+0000","scrub_duration":2346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315359232,"num_objects":4812,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9914,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.506208+0000","last_change":"2024-03-11T22:12:50.861675+0000","last_active":"2024-03-11T22:12:51.506208+0000","last_peered":"2024-03-11T22:12:51.506208+0000","last_clean":"2024-03-11T22:12:51.506208+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:12:51.506208+0000","last_undegraded":"2024-03-11T22:12:51.506208+0000","last_fullsized":"2024-03-11T22:12:51.506208+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.861557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.861557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:11.522320+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.488894+0000","last_change":"2024-03-11T22:12:18.789167+0000","last_active":"2024-03-11T22:12:51.488894+0000","last_peered":"2024-03-11T22:12:51.488894+0000","last_clean":"2024-03-11T22:12:51.488894+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:12:51.488894+0000","last_undegraded":"2024-03-11T22:12:51.488894+0000","last_fullsized":"2024-03-11T22:12:51.488894+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:18.788609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:12:18.788609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:22.290691+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":175,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.493722+0000","last_change":"2024-03-11T22:12:42.910183+0000","last_active":"2024-03-11T22:12:51.493722+0000","last_peered":"2024-03-11T22:12:51.493722+0000","last_clean":"2024-03-11T22:12:51.493722+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:12:51.493722+0000","last_undegraded":"2024-03-11T22:12:51.493722+0000","last_fullsized":"2024-03-11T22:12:51.493722+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.910067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.910067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:49.869286+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":182,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.513318+0000","last_change":"2024-03-11T22:11:17.270935+0000","last_active":"2024-03-11T22:12:51.513318+0000","last_peered":"2024-03-11T22:12:51.513318+0000","last_clean":"2024-03-11T22:12:51.513318+0000","last_became_active":"2024-03-11T22:08:35.055770+0000","last_became_peered":"2024-03-11T22:08:35.055770+0000","last_unstale":"2024-03-11T22:12:51.513318+0000","last_undegraded":"2024-03-11T22:12:51.513318+0000","last_fullsized":"2024-03-11T22:12:51.513318+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-11T22:11:17.270813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:17.270813+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"137'9974","reported_seq":10060,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:55.377901+0000","last_change":"2024-03-11T22:10:36.408956+0000","last_active":"2024-03-11T22:12:55.377901+0000","last_peered":"2024-03-11T22:12:55.377901+0000","last_clean":"2024-03-11T22:12:55.377901+0000","last_became_active":"2024-03-11T22:10:36.408528+0000","last_became_peered":"2024-03-11T22:10:36.408528+0000","last_unstale":"2024-03-11T22:12:55.377901+0000","last_undegraded":"2024-03-11T22:12:55.377901+0000","last_fullsized":"2024-03-11T22:12:55.377901+0000","mapping_epoch":99,"log_start":"114'4400","ondisk_log_start":"114'4400","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":5574,"log_dups_size":0,"ondisk_log_size":5574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:57.874216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311296000,"num_objects":4750,"num_object_clones":0,"num_object_copies":14250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9974,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":146,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.513615+0000","last_change":"2024-03-11T22:10:23.555240+0000","last_active":"2024-03-11T22:12:51.513615+0000","last_peered":"2024-03-11T22:12:51.513615+0000","last_clean":"2024-03-11T22:12:51.513615+0000","last_became_active":"2024-03-11T22:08:39.072422+0000","last_became_peered":"2024-03-11T22:08:39.072422+0000","last_unstale":"2024-03-11T22:12:51.513615+0000","last_undegraded":"2024-03-11T22:12:51.513615+0000","last_fullsized":"2024-03-11T22:12:51.513615+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:23.555159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:23.555159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":228,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.512999+0000","last_change":"2024-03-11T22:12:43.819554+0000","last_active":"2024-03-11T22:12:51.512999+0000","last_peered":"2024-03-11T22:12:51.512999+0000","last_clean":"2024-03-11T22:12:51.512999+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:12:51.512999+0000","last_undegraded":"2024-03-11T22:12:51.512999+0000","last_fullsized":"2024-03-11T22:12:51.512999+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:12:43.819269+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_clean_scrub_stamp":"2024-03-11T22:12:43.819269+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:53.786169+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.501162+0000","last_change":"2024-03-11T22:11:40.097948+0000","last_active":"2024-03-11T22:12:51.501162+0000","last_peered":"2024-03-11T22:12:51.501162+0000","last_clean":"2024-03-11T22:12:51.501162+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:12:51.501162+0000","last_undegraded":"2024-03-11T22:12:51.501162+0000","last_fullsized":"2024-03-11T22:12:51.501162+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:40.094297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:40.094297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:12:55.342593+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":169,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-11T22:12:51.511358+0000","last_change":"2024-03-11T22:11:12.298926+0000","last_active":"2024-03-11T22:12:51.511358+0000","last_peered":"2024-03-11T22:12:51.511358+0000","last_clean":"2024-03-11T22:12:51.511358+0000","last_became_active":"2024-03-11T22:08:35.056698+0000","last_became_peered":"2024-03-11T22:08:35.056698+0000","last_unstale":"2024-03-11T22:12:51.511358+0000","last_undegraded":"2024-03-11T22:12:51.511358+0000","last_fullsized":"2024-03-11T22:12:51.511358+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:11:12.298787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:12.298787+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5085593624,"num_objects":77601,"num_object_clones":0,"num_object_copies":232803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77601,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159337,"num_write_kb":7581953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15263543296,"data_stored":15263531080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91717,"ondisk_log_size":91717,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":229,"num_read_kb":224,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005113,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1640056,"kb_used_data":1576944,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92105096,"statfs":{"total":95995035648,"available":94315618304,"internally_reserved":0,"allocated":1614790656,"data_stored":1614473155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64616907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332158,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1648872,"kb_used_data":1559392,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92096280,"statfs":{"total":95995035648,"available":94306590720,"internally_reserved":0,"allocated":1596817408,"data_stored":1596516125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1995284,"kb_used_data":1887756,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91749868,"statfs":{"total":95995035648,"available":93951864832,"internally_reserved":0,"allocated":1933062144,"data_stored":1932715457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2604664,"kb_used_data":2505456,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91140488,"statfs":{"total":95995035648,"available":93327859712,"internally_reserved":0,"allocated":2565586944,"data_stored":2565268799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1981276,"kb_used_data":1882068,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91763876,"statfs":{"total":95995035648,"available":93966209024,"internally_reserved":0,"allocated":1927237632,"data_stored":1926947895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2938244,"kb_used_data":2802940,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":90806908,"statfs":{"total":95995035648,"available":92986273792,"internally_reserved":0,"allocated":2870210560,"data_stored":2869880543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138541511},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1627896,"kb_used_data":1560624,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92117256,"statfs":{"total":95995035648,"available":94328070144,"internally_reserved":0,"allocated":1598078976,"data_stored":1597760472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659202,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1314340,"kb_used_data":1247068,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92430812,"statfs":{"total":95995035648,"available":94649151488,"internally_reserved":0,"allocated":1276997632,"data_stored":1276699150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2547453952,"data_stored":2547449880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1921060864,"data_stored":1921056792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1911947264,"data_stored":1911947264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2838953984,"data_stored":2838953984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1596653568,"data_stored":1596653568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1265565696,"data_stored":1265565696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1589252096,"data_stored":1589248024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1592655872,"data_stored":1592655872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:13:02.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:13:02.917 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:02.944 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:02.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:02.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:03.046 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:03.074 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:03.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:03.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:03.176 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:03.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:03.207 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:03.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:03.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:03.372 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:03.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:03.474 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:03.502 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:03.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:03.566 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T22:13:03.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:03.645 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:03.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:03.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:03.671 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:03 smithi045 ceph-mon[39715]: pgmap v169: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s 2024-03-11T22:13:03.672 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:03 smithi045 ceph-mon[30178]: pgmap v169: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s 2024-03-11T22:13:03.695 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:13:03.741 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:13:03.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:03.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:03.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:03.794 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:13:03.839 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:13:03.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:03 smithi144 ceph-mon[33984]: pgmap v169: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s 2024-03-11T22:13:03.867 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:13:03.888 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:13:03.890 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:03.893 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:03.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:03.924 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:03.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:03.931 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:13:03.985 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:13:04.025 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:04.032 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:13:04.034 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:13:04.034 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:13:04.034 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:13:04.034 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:13:04.035 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:13:04.035 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:13:04.035 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:13:04.035 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:13:04.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:04.055 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:04.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:04.156 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:04.186 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:04.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:04.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:04.342 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:04.371 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:04.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:04.437 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:13:04.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:04.515 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:04.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:04.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:04 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2133206905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:04.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:04 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3210982917' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:04.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:04 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:04.616 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:04.649 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:04.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:04.686 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:04 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2133206905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:04.686 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:04 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3210982917' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:04.686 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:04 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:04.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:04 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2133206905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:04.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:04 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3210982917' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:04.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:04 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:04.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:04.790 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:04.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:04.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:04.892 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:04.921 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:04.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:04.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:05.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:05.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:05.063 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:05.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:05.164 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:05.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:05.197 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:05.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:05.298 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:05.327 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:05.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:05.428 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:05.459 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:05.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:05.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:05.561 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:05.595 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:05.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:05.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:05 smithi045 ceph-mon[30178]: pgmap v170: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s 2024-03-11T22:13:05.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:05 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:13:05.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:05 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:13:05.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:05 smithi045 ceph-mon[30178]: osdmap e139: 8 total, 8 up, 8 in 2024-03-11T22:13:05.633 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:05 smithi045 ceph-mon[39715]: pgmap v170: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s 2024-03-11T22:13:05.633 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:05 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:13:05.633 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:05 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:13:05.633 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:05 smithi045 ceph-mon[39715]: osdmap e139: 8 total, 8 up, 8 in 2024-03-11T22:13:05.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:05.704 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:13:05.704 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:13:05.704 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:05 smithi144 ceph-mon[33984]: pgmap v170: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s 2024-03-11T22:13:05.704 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:05 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:13:05.704 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:05 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:13:05.705 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:05 smithi144 ceph-mon[33984]: osdmap e139: 8 total, 8 up, 8 in 2024-03-11T22:13:05.738 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:05.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:05.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:05.840 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:05.865 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:05.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:05.967 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:05.993 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:05.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:05.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:06.034 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:13:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:13:06.095 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:06.126 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:06.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:06.228 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:06.258 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:06.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:06.359 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:06.392 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:06.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:06.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:06.486 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T22:13:06.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:06.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:06.535 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:06.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:06.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:06.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:06.673 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:06.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:06.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:06.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:06.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:06.861 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:13:06.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:06.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:06.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:06.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:07.072 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:07.110 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:07.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:07.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:07.211 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:07.249 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:07.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:07.350 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:07.381 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:07.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:07.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:07.483 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:07.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:07.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:07 smithi045 ceph-mon[30178]: pgmap v172: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-11T22:13:07.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:07.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:07 smithi045 ceph-mon[39715]: pgmap v172: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-11T22:13:07.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:07.533 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:07.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:07.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:07.680 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:07.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:07.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:13:07.782 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:07.786 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:07.786 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:07 smithi144 ceph-mon[33984]: pgmap v172: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-11T22:13:07.809 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:07.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:07.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:07.911 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:07.938 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:07.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:08.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:08.097 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:08.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:08.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:08.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:08.242 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:08.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:08.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:08.343 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:08.370 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:08.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:08.393 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:13:08.401 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:13:08.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:08.512 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:08.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:08.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:08 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:13:08.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:08 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:13:08.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:08 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:13:08.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:08 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:13:08.615 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:08.645 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:08.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:08.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:08.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:13:08.790 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:08.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:08.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:08 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:13:08.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:08 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:13:08.892 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:08.919 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:08.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:09.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:09.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:09.092 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:09.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:09.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:09.241 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:09.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:09.343 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:09.368 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:09.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:09.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:09.509 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:09.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:09.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:09.548 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-03-11T22:13:08.124406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4432140660,"num_objects":67819,"num_object_clones":0,"num_object_copies":203457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67819,"num_whiteouts":0,"num_read":1164,"num_read_kb":1010,"num_write":169932,"num_write_kb":7584414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92017,"ondisk_log_size":92017,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13856820,"kb_used_data":13098780,"kb_used_omap":12,"kb_used_meta":757427,"kb_avail":736104396,"statfs":{"total":767960285184,"available":753770901504,"internally_reserved":0,"allocated":13413150720,"data_stored":13410568019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":775605811},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,9,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-654049280,"num_objects":-9980,"num_object_clones":0,"num_object_copies":-29940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9980,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9980,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.933276"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":150,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.425651+0000","last_change":"2024-03-11T22:11:30.350663+0000","last_active":"2024-03-11T22:13:04.425651+0000","last_peered":"2024-03-11T22:13:04.425651+0000","last_clean":"2024-03-11T22:13:04.425651+0000","last_became_active":"2024-03-11T22:08:39.074695+0000","last_became_peered":"2024-03-11T22:08:39.074695+0000","last_unstale":"2024-03-11T22:13:04.425651+0000","last_undegraded":"2024-03-11T22:13:04.425651+0000","last_fullsized":"2024-03-11T22:13:04.425651+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:30.350540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.350540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.058421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.437477+0000","last_change":"2024-03-11T22:12:45.929428+0000","last_active":"2024-03-11T22:13:04.437477+0000","last_peered":"2024-03-11T22:13:04.437477+0000","last_clean":"2024-03-11T22:13:04.437477+0000","last_became_active":"2024-03-11T22:08:37.073775+0000","last_became_peered":"2024-03-11T22:08:37.073775+0000","last_unstale":"2024-03-11T22:13:04.437477+0000","last_undegraded":"2024-03-11T22:13:04.437477+0000","last_fullsized":"2024-03-11T22:13:04.437477+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.928689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.928689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:15.589951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":171,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.446521+0000","last_change":"2024-03-11T22:11:45.340779+0000","last_active":"2024-03-11T22:13:04.446521+0000","last_peered":"2024-03-11T22:13:04.446521+0000","last_clean":"2024-03-11T22:13:04.446521+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:13:04.446521+0000","last_undegraded":"2024-03-11T22:13:04.446521+0000","last_fullsized":"2024-03-11T22:13:04.446521+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:45.340657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:45.340657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:06.074324+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.425436+0000","last_change":"2024-03-11T22:10:13.507406+0000","last_active":"2024-03-11T22:13:04.425436+0000","last_peered":"2024-03-11T22:13:04.425436+0000","last_clean":"2024-03-11T22:13:04.425436+0000","last_became_active":"2024-03-11T22:08:35.053179+0000","last_became_peered":"2024-03-11T22:08:35.053179+0000","last_unstale":"2024-03-11T22:13:04.425436+0000","last_undegraded":"2024-03-11T22:13:04.425436+0000","last_fullsized":"2024-03-11T22:13:04.425436+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:10:13.507312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:13.507312+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.058053+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":158,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.444149+0000","last_change":"2024-03-11T22:12:50.533192+0000","last_active":"2024-03-11T22:13:04.444149+0000","last_peered":"2024-03-11T22:13:04.444149+0000","last_clean":"2024-03-11T22:13:04.444149+0000","last_became_active":"2024-03-11T22:08:39.079045+0000","last_became_peered":"2024-03-11T22:08:39.079045+0000","last_unstale":"2024-03-11T22:13:04.444149+0000","last_undegraded":"2024-03-11T22:13:04.444149+0000","last_fullsized":"2024-03-11T22:13:04.444149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.533067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.533067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:07.226337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.425564+0000","last_change":"2024-03-11T22:11:43.150997+0000","last_active":"2024-03-11T22:13:04.425564+0000","last_peered":"2024-03-11T22:13:04.425564+0000","last_clean":"2024-03-11T22:13:04.425564+0000","last_became_active":"2024-03-11T22:08:37.081861+0000","last_became_peered":"2024-03-11T22:08:37.081861+0000","last_unstale":"2024-03-11T22:13:04.425564+0000","last_undegraded":"2024-03-11T22:13:04.425564+0000","last_fullsized":"2024-03-11T22:13:04.425564+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:43.150872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:43.150872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:06.050251+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.445544+0000","last_change":"2024-03-11T22:11:22.946804+0000","last_active":"2024-03-11T22:13:04.445544+0000","last_peered":"2024-03-11T22:13:04.445544+0000","last_clean":"2024-03-11T22:13:04.445544+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:13:04.445544+0000","last_undegraded":"2024-03-11T22:13:04.445544+0000","last_fullsized":"2024-03-11T22:13:04.445544+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:11:22.946664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:22.946664+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:04.523124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":191,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.444141+0000","last_change":"2024-03-11T22:12:49.510144+0000","last_active":"2024-03-11T22:13:04.444141+0000","last_peered":"2024-03-11T22:13:04.444141+0000","last_clean":"2024-03-11T22:13:04.444141+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:13:04.444141+0000","last_undegraded":"2024-03-11T22:13:04.444141+0000","last_fullsized":"2024-03-11T22:13:04.444141+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:12:49.509860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.509860+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:15.173673+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":150,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.431228+0000","last_change":"2024-03-11T22:10:15.238661+0000","last_active":"2024-03-11T22:13:04.431228+0000","last_peered":"2024-03-11T22:13:04.431228+0000","last_clean":"2024-03-11T22:13:04.431228+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:13:04.431228+0000","last_undegraded":"2024-03-11T22:13:04.431228+0000","last_fullsized":"2024-03-11T22:13:04.431228+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:15.238567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:15.238567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.431352+0000","last_change":"2024-03-11T22:12:19.337740+0000","last_active":"2024-03-11T22:13:04.431352+0000","last_peered":"2024-03-11T22:13:04.431352+0000","last_clean":"2024-03-11T22:13:04.431352+0000","last_became_active":"2024-03-11T22:08:37.081996+0000","last_became_peered":"2024-03-11T22:08:37.081996+0000","last_unstale":"2024-03-11T22:13:04.431352+0000","last_undegraded":"2024-03-11T22:13:04.431352+0000","last_fullsized":"2024-03-11T22:13:04.431352+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:19.337662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:12:19.337662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:35.201935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":178,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.430181+0000","last_change":"2024-03-11T22:12:47.432839+0000","last_active":"2024-03-11T22:13:04.430181+0000","last_peered":"2024-03-11T22:13:04.430181+0000","last_clean":"2024-03-11T22:13:04.430181+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:13:04.430181+0000","last_undegraded":"2024-03-11T22:13:04.430181+0000","last_fullsized":"2024-03-11T22:13:04.430181+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:47.432769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:47.432769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:59.023738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":182,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.425808+0000","last_change":"2024-03-11T22:11:45.169864+0000","last_active":"2024-03-11T22:13:04.425808+0000","last_peered":"2024-03-11T22:13:04.425808+0000","last_clean":"2024-03-11T22:13:04.425808+0000","last_became_active":"2024-03-11T22:08:35.060802+0000","last_became_peered":"2024-03-11T22:08:35.060802+0000","last_unstale":"2024-03-11T22:13:04.425808+0000","last_undegraded":"2024-03-11T22:13:04.425808+0000","last_fullsized":"2024-03-11T22:13:04.425808+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:11:45.169706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:45.169706+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.038712+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":158,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.428888+0000","last_change":"2024-03-11T22:11:47.166498+0000","last_active":"2024-03-11T22:13:04.428888+0000","last_peered":"2024-03-11T22:13:04.428888+0000","last_clean":"2024-03-11T22:13:04.428888+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:13:04.428888+0000","last_undegraded":"2024-03-11T22:13:04.428888+0000","last_fullsized":"2024-03-11T22:13:04.428888+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:47.166360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:47.166360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:13.931342+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":93,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.425976+0000","last_change":"2024-03-11T22:10:35.411012+0000","last_active":"2024-03-11T22:13:04.425976+0000","last_peered":"2024-03-11T22:13:04.425976+0000","last_clean":"2024-03-11T22:13:04.425976+0000","last_became_active":"2024-03-11T22:10:35.410624+0000","last_became_peered":"2024-03-11T22:10:35.410624+0000","last_unstale":"2024-03-11T22:13:04.425976+0000","last_undegraded":"2024-03-11T22:13:04.425976+0000","last_fullsized":"2024-03-11T22:13:04.425976+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:00.270166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.270166+0000","last_clean_scrub_stamp":"2024-03-11T22:10:00.270166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.057945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.429528+0000","last_change":"2024-03-11T22:12:24.388572+0000","last_active":"2024-03-11T22:13:04.429528+0000","last_peered":"2024-03-11T22:13:04.429528+0000","last_clean":"2024-03-11T22:13:04.429528+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:13:04.429528+0000","last_undegraded":"2024-03-11T22:13:04.429528+0000","last_fullsized":"2024-03-11T22:13:04.429528+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:24.388497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.388497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:43.170333+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"95'7","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.427108+0000","last_change":"2024-03-11T22:10:18.579990+0000","last_active":"2024-03-11T22:13:04.427108+0000","last_peered":"2024-03-11T22:13:04.427108+0000","last_clean":"2024-03-11T22:13:04.427108+0000","last_became_active":"2024-03-11T22:08:35.060777+0000","last_became_peered":"2024-03-11T22:08:35.060777+0000","last_unstale":"2024-03-11T22:13:04.427108+0000","last_undegraded":"2024-03-11T22:13:04.427108+0000","last_fullsized":"2024-03-11T22:13:04.427108+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'7","last_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:10:18.579896+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.057884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":159,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.426506+0000","last_change":"2024-03-11T22:12:32.044430+0000","last_active":"2024-03-11T22:13:04.426506+0000","last_peered":"2024-03-11T22:13:04.426506+0000","last_clean":"2024-03-11T22:13:04.426506+0000","last_became_active":"2024-03-11T22:08:39.075609+0000","last_became_peered":"2024-03-11T22:08:39.075609+0000","last_unstale":"2024-03-11T22:13:04.426506+0000","last_undegraded":"2024-03-11T22:13:04.426506+0000","last_fullsized":"2024-03-11T22:13:04.426506+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:32.044320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:32.044320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:34.643459+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.439117+0000","last_change":"2024-03-11T22:12:30.962293+0000","last_active":"2024-03-11T22:13:04.439117+0000","last_peered":"2024-03-11T22:13:04.439117+0000","last_clean":"2024-03-11T22:13:04.439117+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:13:04.439117+0000","last_undegraded":"2024-03-11T22:13:04.439117+0000","last_fullsized":"2024-03-11T22:13:04.439117+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:30.962171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:30.962171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.129955+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.432220+0000","last_change":"2024-03-11T22:12:22.649189+0000","last_active":"2024-03-11T22:13:04.432220+0000","last_peered":"2024-03-11T22:13:04.432220+0000","last_clean":"2024-03-11T22:13:04.432220+0000","last_became_active":"2024-03-11T22:08:33.035624+0000","last_became_peered":"2024-03-11T22:08:33.035624+0000","last_unstale":"2024-03-11T22:13:04.432220+0000","last_undegraded":"2024-03-11T22:13:04.432220+0000","last_fullsized":"2024-03-11T22:13:04.432220+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:22.649073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:22.649073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:32.856439+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":199,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.440549+0000","last_change":"2024-03-11T22:12:27.670423+0000","last_active":"2024-03-11T22:13:04.440549+0000","last_peered":"2024-03-11T22:13:04.440549+0000","last_clean":"2024-03-11T22:13:04.440549+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:13:04.440549+0000","last_undegraded":"2024-03-11T22:13:04.440549+0000","last_fullsized":"2024-03-11T22:13:04.440549+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:12:27.670284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:27.670284+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:39.408475+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":159,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.457030+0000","last_change":"2024-03-11T22:12:44.464428+0000","last_active":"2024-03-11T22:13:04.457030+0000","last_peered":"2024-03-11T22:13:04.457030+0000","last_clean":"2024-03-11T22:13:04.457030+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:13:04.457030+0000","last_undegraded":"2024-03-11T22:13:04.457030+0000","last_fullsized":"2024-03-11T22:13:04.457030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.464308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.464308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:02.369972+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":154,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.426126+0000","last_change":"2024-03-11T22:11:27.391809+0000","last_active":"2024-03-11T22:13:04.426126+0000","last_peered":"2024-03-11T22:13:04.426126+0000","last_clean":"2024-03-11T22:13:04.426126+0000","last_became_active":"2024-03-11T22:08:37.080014+0000","last_became_peered":"2024-03-11T22:08:37.080014+0000","last_unstale":"2024-03-11T22:13:04.426126+0000","last_undegraded":"2024-03-11T22:13:04.426126+0000","last_fullsized":"2024-03-11T22:13:04.426126+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:27.391727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:27.391727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.058225+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.425812+0000","last_change":"2024-03-11T22:10:03.469439+0000","last_active":"2024-03-11T22:13:04.425812+0000","last_peered":"2024-03-11T22:13:04.425812+0000","last_clean":"2024-03-11T22:13:04.425812+0000","last_became_active":"2024-03-11T22:08:33.299189+0000","last_became_peered":"2024-03-11T22:08:33.299189+0000","last_unstale":"2024-03-11T22:13:04.425812+0000","last_undegraded":"2024-03-11T22:13:04.425812+0000","last_fullsized":"2024-03-11T22:13:04.425812+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:03.469346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.469346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.057765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":189,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.443293+0000","last_change":"2024-03-11T22:12:28.292078+0000","last_active":"2024-03-11T22:13:04.443293+0000","last_peered":"2024-03-11T22:13:04.443293+0000","last_clean":"2024-03-11T22:13:04.443293+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:13:04.443293+0000","last_undegraded":"2024-03-11T22:13:04.443293+0000","last_fullsized":"2024-03-11T22:13:04.443293+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:12:28.291919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:28.291919+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:33.113224+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":112,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.429363+0000","last_change":"2024-03-11T22:11:22.021723+0000","last_active":"2024-03-11T22:13:04.429363+0000","last_peered":"2024-03-11T22:13:04.429363+0000","last_clean":"2024-03-11T22:13:04.429363+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:13:04.429363+0000","last_undegraded":"2024-03-11T22:13:04.429363+0000","last_fullsized":"2024-03-11T22:13:04.429363+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:11:22.021581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:22.021581+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.364616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":237,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.435141+0000","last_change":"2024-03-11T22:12:35.862998+0000","last_active":"2024-03-11T22:13:04.435141+0000","last_peered":"2024-03-11T22:13:04.435141+0000","last_clean":"2024-03-11T22:13:04.435141+0000","last_became_active":"2024-03-11T22:08:37.078588+0000","last_became_peered":"2024-03-11T22:08:37.078588+0000","last_unstale":"2024-03-11T22:13:04.435141+0000","last_undegraded":"2024-03-11T22:13:04.435141+0000","last_fullsized":"2024-03-11T22:13:04.435141+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:12:35.862855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.862855+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:00.615190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":171,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.437567+0000","last_change":"2024-03-11T22:12:24.919926+0000","last_active":"2024-03-11T22:13:04.437567+0000","last_peered":"2024-03-11T22:13:04.437567+0000","last_clean":"2024-03-11T22:13:04.437567+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:13:04.437567+0000","last_undegraded":"2024-03-11T22:13:04.437567+0000","last_fullsized":"2024-03-11T22:13:04.437567+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:24.919682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.919682+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:49.474264+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.445376+0000","last_change":"2024-03-11T22:10:32.196590+0000","last_active":"2024-03-11T22:13:04.445376+0000","last_peered":"2024-03-11T22:13:04.445376+0000","last_clean":"2024-03-11T22:13:04.445376+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:13:04.445376+0000","last_undegraded":"2024-03-11T22:13:04.445376+0000","last_fullsized":"2024-03-11T22:13:04.445376+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:10:32.196500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:32.196500+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":158,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.439304+0000","last_change":"2024-03-11T22:11:30.890018+0000","last_active":"2024-03-11T22:13:04.439304+0000","last_peered":"2024-03-11T22:13:04.439304+0000","last_clean":"2024-03-11T22:13:04.439304+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:13:04.439304+0000","last_undegraded":"2024-03-11T22:13:04.439304+0000","last_fullsized":"2024-03-11T22:13:04.439304+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:30.889504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.889504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:06.850803+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.445842+0000","last_change":"2024-03-11T22:11:37.288072+0000","last_active":"2024-03-11T22:13:04.445842+0000","last_peered":"2024-03-11T22:13:04.445842+0000","last_clean":"2024-03-11T22:13:04.445842+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:13:04.445842+0000","last_undegraded":"2024-03-11T22:13:04.445842+0000","last_fullsized":"2024-03-11T22:13:04.445842+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:37.287951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:37.287951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.443054+0000","last_change":"2024-03-11T22:12:43.451182+0000","last_active":"2024-03-11T22:13:04.443054+0000","last_peered":"2024-03-11T22:13:04.443054+0000","last_clean":"2024-03-11T22:13:04.443054+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:13:04.443054+0000","last_undegraded":"2024-03-11T22:13:04.443054+0000","last_fullsized":"2024-03-11T22:13:04.443054+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:43.451074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:43.451074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:06.309610+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":186,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.435119+0000","last_change":"2024-03-11T22:11:11.339653+0000","last_active":"2024-03-11T22:13:04.435119+0000","last_peered":"2024-03-11T22:13:04.435119+0000","last_clean":"2024-03-11T22:13:04.435119+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:13:04.435119+0000","last_undegraded":"2024-03-11T22:13:04.435119+0000","last_fullsized":"2024-03-11T22:13:04.435119+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:11:11.339500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:11.339500+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:07.746361+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":101,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.446359+0000","last_change":"2024-03-11T22:11:25.409252+0000","last_active":"2024-03-11T22:13:04.446359+0000","last_peered":"2024-03-11T22:13:04.446359+0000","last_clean":"2024-03-11T22:13:04.446359+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:13:04.446359+0000","last_undegraded":"2024-03-11T22:13:04.446359+0000","last_fullsized":"2024-03-11T22:13:04.446359+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:25.407823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.407823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.685919+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.432256+0000","last_change":"2024-03-11T22:12:44.941384+0000","last_active":"2024-03-11T22:13:04.432256+0000","last_peered":"2024-03-11T22:13:04.432256+0000","last_clean":"2024-03-11T22:13:04.432256+0000","last_became_active":"2024-03-11T22:08:37.075341+0000","last_became_peered":"2024-03-11T22:08:37.075341+0000","last_unstale":"2024-03-11T22:13:04.432256+0000","last_undegraded":"2024-03-11T22:13:04.432256+0000","last_fullsized":"2024-03-11T22:13:04.432256+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.941275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.941275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:04.779620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.442880+0000","last_change":"2024-03-11T22:10:18.176194+0000","last_active":"2024-03-11T22:13:04.442880+0000","last_peered":"2024-03-11T22:13:04.442880+0000","last_clean":"2024-03-11T22:13:04.442880+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:13:04.442880+0000","last_undegraded":"2024-03-11T22:13:04.442880+0000","last_fullsized":"2024-03-11T22:13:04.442880+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:18.176081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:18.176081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":184,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.443154+0000","last_change":"2024-03-11T22:11:25.927507+0000","last_active":"2024-03-11T22:13:04.443154+0000","last_peered":"2024-03-11T22:13:04.443154+0000","last_clean":"2024-03-11T22:13:04.443154+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:13:04.443154+0000","last_undegraded":"2024-03-11T22:13:04.443154+0000","last_fullsized":"2024-03-11T22:13:04.443154+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-11T22:11:25.926811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.926811+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:04.523181+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":158,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.444418+0000","last_change":"2024-03-11T22:11:46.050869+0000","last_active":"2024-03-11T22:13:04.444418+0000","last_peered":"2024-03-11T22:13:04.444418+0000","last_clean":"2024-03-11T22:13:04.444418+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:13:04.444418+0000","last_undegraded":"2024-03-11T22:13:04.444418+0000","last_fullsized":"2024-03-11T22:13:04.444418+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:46.050747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:46.050747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:05.807761+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":101,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.425374+0000","last_change":"2024-03-11T22:11:42.175942+0000","last_active":"2024-03-11T22:13:04.425374+0000","last_peered":"2024-03-11T22:13:04.425374+0000","last_clean":"2024-03-11T22:13:04.425374+0000","last_became_active":"2024-03-11T22:10:35.415813+0000","last_became_peered":"2024-03-11T22:10:35.415813+0000","last_unstale":"2024-03-11T22:13:04.425374+0000","last_undegraded":"2024-03-11T22:13:04.425374+0000","last_fullsized":"2024-03-11T22:13:04.425374+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:11:42.175852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:06.050281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.429725+0000","last_change":"2024-03-11T22:11:32.118166+0000","last_active":"2024-03-11T22:13:04.429725+0000","last_peered":"2024-03-11T22:13:04.429725+0000","last_clean":"2024-03-11T22:13:04.429725+0000","last_became_active":"2024-03-11T22:08:33.039349+0000","last_became_peered":"2024-03-11T22:08:33.039349+0000","last_unstale":"2024-03-11T22:13:04.429725+0000","last_undegraded":"2024-03-11T22:13:04.429725+0000","last_fullsized":"2024-03-11T22:13:04.429725+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:32.118050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:32.118050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.364707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.444050+0000","last_change":"2024-03-11T22:11:21.929969+0000","last_active":"2024-03-11T22:13:04.444050+0000","last_peered":"2024-03-11T22:13:04.444050+0000","last_clean":"2024-03-11T22:13:04.444050+0000","last_became_active":"2024-03-11T22:08:35.085928+0000","last_became_peered":"2024-03-11T22:08:35.085928+0000","last_unstale":"2024-03-11T22:13:04.444050+0000","last_undegraded":"2024-03-11T22:13:04.444050+0000","last_fullsized":"2024-03-11T22:13:04.444050+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:11:21.929789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:21.929789+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:04.523090+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":158,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.430891+0000","last_change":"2024-03-11T22:12:46.829082+0000","last_active":"2024-03-11T22:13:04.430891+0000","last_peered":"2024-03-11T22:13:04.430891+0000","last_clean":"2024-03-11T22:13:04.430891+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:13:04.430891+0000","last_undegraded":"2024-03-11T22:13:04.430891+0000","last_fullsized":"2024-03-11T22:13:04.430891+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.828942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.828942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:59.694733+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.444068+0000","last_change":"2024-03-11T22:11:37.076127+0000","last_active":"2024-03-11T22:13:04.444068+0000","last_peered":"2024-03-11T22:13:04.444068+0000","last_clean":"2024-03-11T22:13:04.444068+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:13:04.444068+0000","last_undegraded":"2024-03-11T22:13:04.444068+0000","last_fullsized":"2024-03-11T22:13:04.444068+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:11:37.075868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":102,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.443574+0000","last_change":"2024-03-11T22:11:38.050014+0000","last_active":"2024-03-11T22:13:04.443574+0000","last_peered":"2024-03-11T22:13:04.443574+0000","last_clean":"2024-03-11T22:13:04.443574+0000","last_became_active":"2024-03-11T22:10:35.414121+0000","last_became_peered":"2024-03-11T22:10:35.414121+0000","last_unstale":"2024-03-11T22:13:04.443574+0000","last_undegraded":"2024-03-11T22:13:04.443574+0000","last_fullsized":"2024-03-11T22:13:04.443574+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:38.049889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:38.049889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:04.959324+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":182,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.439155+0000","last_change":"2024-03-11T22:11:26.901776+0000","last_active":"2024-03-11T22:13:04.439155+0000","last_peered":"2024-03-11T22:13:04.439155+0000","last_clean":"2024-03-11T22:13:04.439155+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:13:04.439155+0000","last_undegraded":"2024-03-11T22:13:04.439155+0000","last_fullsized":"2024-03-11T22:13:04.439155+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:11:26.901528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:26.901528+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":158,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.443678+0000","last_change":"2024-03-11T22:12:46.492608+0000","last_active":"2024-03-11T22:13:04.443678+0000","last_peered":"2024-03-11T22:13:04.443678+0000","last_clean":"2024-03-11T22:13:04.443678+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:13:04.443678+0000","last_undegraded":"2024-03-11T22:13:04.443678+0000","last_fullsized":"2024-03-11T22:13:04.443678+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.492494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.492494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:11.071230+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.438878+0000","last_change":"2024-03-11T22:12:41.988328+0000","last_active":"2024-03-11T22:13:04.438878+0000","last_peered":"2024-03-11T22:13:04.438878+0000","last_clean":"2024-03-11T22:13:04.438878+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:13:04.438878+0000","last_undegraded":"2024-03-11T22:13:04.438878+0000","last_fullsized":"2024-03-11T22:13:04.438878+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:41.988094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:41.988094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:07.625482+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":176,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.426044+0000","last_change":"2024-03-11T22:12:18.154605+0000","last_active":"2024-03-11T22:13:04.426044+0000","last_peered":"2024-03-11T22:13:04.426044+0000","last_clean":"2024-03-11T22:13:04.426044+0000","last_became_active":"2024-03-11T22:08:33.036083+0000","last_became_peered":"2024-03-11T22:08:33.036083+0000","last_unstale":"2024-03-11T22:13:04.426044+0000","last_undegraded":"2024-03-11T22:13:04.426044+0000","last_fullsized":"2024-03-11T22:13:04.426044+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:18.154515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:18.154515+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:34.107328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.427234+0000","last_change":"2024-03-11T22:11:19.294807+0000","last_active":"2024-03-11T22:13:04.427234+0000","last_peered":"2024-03-11T22:13:04.427234+0000","last_clean":"2024-03-11T22:13:04.427234+0000","last_became_active":"2024-03-11T22:08:35.054726+0000","last_became_peered":"2024-03-11T22:08:35.054726+0000","last_unstale":"2024-03-11T22:13:04.427234+0000","last_undegraded":"2024-03-11T22:13:04.427234+0000","last_fullsized":"2024-03-11T22:13:04.427234+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:11:19.294671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:19.294671+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.058196+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":150,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.444646+0000","last_change":"2024-03-11T22:11:13.259520+0000","last_active":"2024-03-11T22:13:04.444646+0000","last_peered":"2024-03-11T22:13:04.444646+0000","last_clean":"2024-03-11T22:13:04.444646+0000","last_became_active":"2024-03-11T22:08:39.082938+0000","last_became_peered":"2024-03-11T22:08:39.082938+0000","last_unstale":"2024-03-11T22:13:04.444646+0000","last_undegraded":"2024-03-11T22:13:04.444646+0000","last_fullsized":"2024-03-11T22:13:04.444646+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:13.259407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:13.259407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":225,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.430258+0000","last_change":"2024-03-11T22:11:30.122434+0000","last_active":"2024-03-11T22:13:04.430258+0000","last_peered":"2024-03-11T22:13:04.430258+0000","last_clean":"2024-03-11T22:13:04.430258+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:13:04.430258+0000","last_undegraded":"2024-03-11T22:13:04.430258+0000","last_fullsized":"2024-03-11T22:13:04.430258+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:11:30.122330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.122330+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.364677+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":102,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.427140+0000","last_change":"2024-03-11T22:11:39.174355+0000","last_active":"2024-03-11T22:13:04.427140+0000","last_peered":"2024-03-11T22:13:04.427140+0000","last_clean":"2024-03-11T22:13:04.427140+0000","last_became_active":"2024-03-11T22:10:35.407845+0000","last_became_peered":"2024-03-11T22:10:35.407845+0000","last_unstale":"2024-03-11T22:13:04.427140+0000","last_undegraded":"2024-03-11T22:13:04.427140+0000","last_fullsized":"2024-03-11T22:13:04.427140+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:39.174103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:39.174103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.058295+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.443971+0000","last_change":"2024-03-11T22:12:20.295562+0000","last_active":"2024-03-11T22:13:04.443971+0000","last_peered":"2024-03-11T22:13:04.443971+0000","last_clean":"2024-03-11T22:13:04.443971+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:13:04.443971+0000","last_undegraded":"2024-03-11T22:13:04.443971+0000","last_fullsized":"2024-03-11T22:13:04.443971+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:12:20.295427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:20.295427+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:42.096971+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":150,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.426564+0000","last_change":"2024-03-11T22:11:31.308182+0000","last_active":"2024-03-11T22:13:04.426564+0000","last_peered":"2024-03-11T22:13:04.426564+0000","last_clean":"2024-03-11T22:13:04.426564+0000","last_became_active":"2024-03-11T22:08:39.072089+0000","last_became_peered":"2024-03-11T22:08:39.072089+0000","last_unstale":"2024-03-11T22:13:04.426564+0000","last_undegraded":"2024-03-11T22:13:04.426564+0000","last_fullsized":"2024-03-11T22:13:04.426564+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:31.308103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:31.308103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.058358+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":228,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.444077+0000","last_change":"2024-03-11T22:11:44.093057+0000","last_active":"2024-03-11T22:13:04.444077+0000","last_peered":"2024-03-11T22:13:04.444077+0000","last_clean":"2024-03-11T22:13:04.444077+0000","last_became_active":"2024-03-11T22:08:37.071202+0000","last_became_peered":"2024-03-11T22:08:37.071202+0000","last_unstale":"2024-03-11T22:13:04.444077+0000","last_undegraded":"2024-03-11T22:13:04.444077+0000","last_fullsized":"2024-03-11T22:13:04.444077+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:11:44.092784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:44.092784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.131593+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.431751+0000","last_change":"2024-03-11T22:12:29.629914+0000","last_active":"2024-03-11T22:13:04.431751+0000","last_peered":"2024-03-11T22:13:04.431751+0000","last_clean":"2024-03-11T22:13:04.431751+0000","last_became_active":"2024-03-11T22:08:33.032850+0000","last_became_peered":"2024-03-11T22:08:33.032850+0000","last_unstale":"2024-03-11T22:13:04.431751+0000","last_undegraded":"2024-03-11T22:13:04.431751+0000","last_fullsized":"2024-03-11T22:13:04.431751+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:29.629778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:29.629778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:44.363248+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":182,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.430497+0000","last_change":"2024-03-11T22:11:38.077749+0000","last_active":"2024-03-11T22:13:04.430497+0000","last_peered":"2024-03-11T22:13:04.430497+0000","last_clean":"2024-03-11T22:13:04.430497+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:13:04.430497+0000","last_undegraded":"2024-03-11T22:13:04.430497+0000","last_fullsized":"2024-03-11T22:13:04.430497+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:11:38.077654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:38.077654+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:05.424137+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":166,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.440799+0000","last_change":"2024-03-11T22:12:40.777949+0000","last_active":"2024-03-11T22:13:04.440799+0000","last_peered":"2024-03-11T22:13:04.440799+0000","last_clean":"2024-03-11T22:13:04.440799+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:13:04.440799+0000","last_undegraded":"2024-03-11T22:13:04.440799+0000","last_fullsized":"2024-03-11T22:13:04.440799+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:40.777821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:40.777821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:46.765451+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.457360+0000","last_change":"2024-03-11T22:11:31.040056+0000","last_active":"2024-03-11T22:13:04.457360+0000","last_peered":"2024-03-11T22:13:04.457360+0000","last_clean":"2024-03-11T22:13:04.457360+0000","last_became_active":"2024-03-11T22:08:37.069611+0000","last_became_peered":"2024-03-11T22:08:37.069611+0000","last_unstale":"2024-03-11T22:13:04.457360+0000","last_undegraded":"2024-03-11T22:13:04.457360+0000","last_fullsized":"2024-03-11T22:13:04.457360+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:31.039934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:31.039934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:04.523271+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":94,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.425779+0000","last_change":"2024-03-11T22:10:35.416311+0000","last_active":"2024-03-11T22:13:04.425779+0000","last_peered":"2024-03-11T22:13:04.425779+0000","last_clean":"2024-03-11T22:13:04.425779+0000","last_became_active":"2024-03-11T22:10:35.415644+0000","last_became_peered":"2024-03-11T22:10:35.415644+0000","last_unstale":"2024-03-11T22:13:04.425779+0000","last_undegraded":"2024-03-11T22:13:04.425779+0000","last_fullsized":"2024-03-11T22:13:04.425779+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:12.415483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:12.415483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.058015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"93'13","reported_seq":186,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.445624+0000","last_change":"2024-03-11T22:11:35.407557+0000","last_active":"2024-03-11T22:13:04.445624+0000","last_peered":"2024-03-11T22:13:04.445624+0000","last_clean":"2024-03-11T22:13:04.445624+0000","last_became_active":"2024-03-11T22:08:35.085787+0000","last_became_peered":"2024-03-11T22:08:35.085787+0000","last_unstale":"2024-03-11T22:13:04.445624+0000","last_undegraded":"2024-03-11T22:13:04.445624+0000","last_fullsized":"2024-03-11T22:13:04.445624+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:11:35.407301+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.407301+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":101,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.432125+0000","last_change":"2024-03-11T22:11:36.490489+0000","last_active":"2024-03-11T22:13:04.432125+0000","last_peered":"2024-03-11T22:13:04.432125+0000","last_clean":"2024-03-11T22:13:04.432125+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:13:04.432125+0000","last_undegraded":"2024-03-11T22:13:04.432125+0000","last_fullsized":"2024-03-11T22:13:04.432125+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:36.490349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:15.415292+0000","last_clean_scrub_stamp":"2024-03-11T22:11:36.490349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:07.746525+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.429825+0000","last_change":"2024-03-11T22:11:41.090567+0000","last_active":"2024-03-11T22:13:04.429825+0000","last_peered":"2024-03-11T22:13:04.429825+0000","last_clean":"2024-03-11T22:13:04.429825+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:13:04.429825+0000","last_undegraded":"2024-03-11T22:13:04.429825+0000","last_fullsized":"2024-03-11T22:13:04.429825+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:41.090454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:07.195289+0000","last_clean_scrub_stamp":"2024-03-11T22:11:41.090454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.364648+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.431102+0000","last_change":"2024-03-11T22:11:31.436443+0000","last_active":"2024-03-11T22:13:04.431102+0000","last_peered":"2024-03-11T22:13:04.431102+0000","last_clean":"2024-03-11T22:13:04.431102+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:13:04.431102+0000","last_undegraded":"2024-03-11T22:13:04.431102+0000","last_fullsized":"2024-03-11T22:13:04.431102+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:31.436306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:11:31.436306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:07.746462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.457439+0000","last_change":"2024-03-11T22:12:21.278151+0000","last_active":"2024-03-11T22:13:04.457439+0000","last_peered":"2024-03-11T22:13:04.457439+0000","last_clean":"2024-03-11T22:13:04.457439+0000","last_became_active":"2024-03-11T22:08:35.085914+0000","last_became_peered":"2024-03-11T22:08:35.085914+0000","last_unstale":"2024-03-11T22:13:04.457439+0000","last_undegraded":"2024-03-11T22:13:04.457439+0000","last_fullsized":"2024-03-11T22:13:04.457439+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:12:21.270671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:21.270671+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:41.395498+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"139'12906","reported_seq":13219,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.241853+0000","last_change":"2024-03-11T22:12:25.136636+0000","last_active":"2024-03-11T22:13:05.241853+0000","last_peered":"2024-03-11T22:13:05.241853+0000","last_clean":"2024-03-11T22:13:05.241853+0000","last_became_active":"2024-03-11T22:10:36.410723+0000","last_became_peered":"2024-03-11T22:10:36.410723+0000","last_unstale":"2024-03-11T22:13:05.241853+0000","last_undegraded":"2024-03-11T22:13:05.241853+0000","last_fullsized":"2024-03-11T22:13:05.241853+0000","mapping_epoch":99,"log_start":"123'7900","ondisk_log_start":"123'7900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'10496","last_scrub_stamp":"2024-03-11T22:12:25.136305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:25.136305+0000","objects_scrubbed":6681,"log_size":5006,"log_dups_size":0,"ondisk_log_size":5006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:31.345968+0000","scrub_duration":2244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275382272,"num_objects":4202,"num_object_clones":0,"num_object_copies":12606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10426,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":159,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.429361+0000","last_change":"2024-03-11T22:12:38.434933+0000","last_active":"2024-03-11T22:13:04.429361+0000","last_peered":"2024-03-11T22:13:04.429361+0000","last_clean":"2024-03-11T22:13:04.429361+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:13:04.429361+0000","last_undegraded":"2024-03-11T22:13:04.429361+0000","last_fullsized":"2024-03-11T22:13:04.429361+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:38.434796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.434796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:04.782006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.426489+0000","last_change":"2024-03-11T22:10:35.412939+0000","last_active":"2024-03-11T22:13:04.426489+0000","last_peered":"2024-03-11T22:13:04.426489+0000","last_clean":"2024-03-11T22:13:04.426489+0000","last_became_active":"2024-03-11T22:10:35.412565+0000","last_became_peered":"2024-03-11T22:10:35.412565+0000","last_unstale":"2024-03-11T22:13:04.426489+0000","last_undegraded":"2024-03-11T22:13:04.426489+0000","last_fullsized":"2024-03-11T22:13:04.426489+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:10:05.284508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:05.284508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.057798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"91'22","reported_seq":199,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.428896+0000","last_change":"2024-03-11T22:11:12.222700+0000","last_active":"2024-03-11T22:13:04.428896+0000","last_peered":"2024-03-11T22:13:04.428896+0000","last_clean":"2024-03-11T22:13:04.428896+0000","last_became_active":"2024-03-11T22:08:35.085916+0000","last_became_peered":"2024-03-11T22:08:35.085916+0000","last_unstale":"2024-03-11T22:13:04.428896+0000","last_undegraded":"2024-03-11T22:13:04.428896+0000","last_fullsized":"2024-03-11T22:13:04.428896+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'22","last_scrub_stamp":"2024-03-11T22:11:12.222573+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:11:12.222573+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.364577+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"99'9","reported_seq":157,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.431621+0000","last_change":"2024-03-11T22:11:25.476405+0000","last_active":"2024-03-11T22:13:04.431621+0000","last_peered":"2024-03-11T22:13:04.431621+0000","last_clean":"2024-03-11T22:13:04.431621+0000","last_became_active":"2024-03-11T22:10:35.413089+0000","last_became_peered":"2024-03-11T22:10:35.413089+0000","last_unstale":"2024-03-11T22:13:04.431621+0000","last_undegraded":"2024-03-11T22:13:04.431621+0000","last_fullsized":"2024-03-11T22:13:04.431621+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'9","last_scrub_stamp":"2024-03-11T22:11:25.476283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.476283+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:07.746403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"139'10543","reported_seq":10704,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.378365+0000","last_change":"2024-03-11T22:12:38.052754+0000","last_active":"2024-03-11T22:13:05.378365+0000","last_peered":"2024-03-11T22:13:05.378365+0000","last_clean":"2024-03-11T22:13:05.378365+0000","last_became_active":"2024-03-11T22:10:36.411631+0000","last_became_peered":"2024-03-11T22:10:36.411631+0000","last_unstale":"2024-03-11T22:13:05.378365+0000","last_undegraded":"2024-03-11T22:13:05.378365+0000","last_fullsized":"2024-03-11T22:13:05.378365+0000","mapping_epoch":99,"log_start":"121'5900","ondisk_log_start":"121'5900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'8787","last_scrub_stamp":"2024-03-11T22:12:38.052573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.052573+0000","objects_scrubbed":6049,"log_size":4643,"log_dups_size":0,"ondisk_log_size":4643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:03.556832+0000","scrub_duration":2625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276889600,"num_objects":4225,"num_object_clones":0,"num_object_copies":12675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10543,"num_write_kb":472576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":161,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.444279+0000","last_change":"2024-03-11T22:12:36.449430+0000","last_active":"2024-03-11T22:13:04.444279+0000","last_peered":"2024-03-11T22:13:04.444279+0000","last_clean":"2024-03-11T22:13:04.444279+0000","last_became_active":"2024-03-11T22:08:39.083141+0000","last_became_peered":"2024-03-11T22:08:39.083141+0000","last_unstale":"2024-03-11T22:13:04.444279+0000","last_undegraded":"2024-03-11T22:13:04.444279+0000","last_fullsized":"2024-03-11T22:13:04.444279+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:12:36.449307+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:12:36.449307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:00.397020+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":102,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.426386+0000","last_change":"2024-03-11T22:12:29.074360+0000","last_active":"2024-03-11T22:13:04.426386+0000","last_peered":"2024-03-11T22:13:04.426386+0000","last_clean":"2024-03-11T22:13:04.426386+0000","last_became_active":"2024-03-11T22:10:35.413337+0000","last_became_peered":"2024-03-11T22:10:35.413337+0000","last_unstale":"2024-03-11T22:13:04.426386+0000","last_undegraded":"2024-03-11T22:13:04.426386+0000","last_fullsized":"2024-03-11T22:13:04.426386+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:29.074256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:29.074256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:31.925354+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":200,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.425782+0000","last_change":"2024-03-11T22:11:21.379985+0000","last_active":"2024-03-11T22:13:04.425782+0000","last_peered":"2024-03-11T22:13:04.425782+0000","last_clean":"2024-03-11T22:13:04.425782+0000","last_became_active":"2024-03-11T22:08:54.607153+0000","last_became_peered":"2024-03-11T22:08:54.607153+0000","last_unstale":"2024-03-11T22:13:04.425782+0000","last_undegraded":"2024-03-11T22:13:04.425782+0000","last_fullsized":"2024-03-11T22:13:04.425782+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:11:21.379862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:21.379862+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.058264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":171,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.441648+0000","last_change":"2024-03-11T22:12:38.757349+0000","last_active":"2024-03-11T22:13:04.441648+0000","last_peered":"2024-03-11T22:13:04.441648+0000","last_clean":"2024-03-11T22:13:04.441648+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:13:04.441648+0000","last_undegraded":"2024-03-11T22:13:04.441648+0000","last_fullsized":"2024-03-11T22:13:04.441648+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.757164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:48.916257+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"139'10528","reported_seq":10619,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.352406+0000","last_change":"2024-03-11T22:10:36.413014+0000","last_active":"2024-03-11T22:13:05.352406+0000","last_peered":"2024-03-11T22:13:05.352406+0000","last_clean":"2024-03-11T22:13:05.352406+0000","last_became_active":"2024-03-11T22:10:36.412662+0000","last_became_peered":"2024-03-11T22:10:36.412662+0000","last_unstale":"2024-03-11T22:13:05.352406+0000","last_undegraded":"2024-03-11T22:13:05.352406+0000","last_fullsized":"2024-03-11T22:13:05.352406+0000","mapping_epoch":99,"log_start":"116'4900","ondisk_log_start":"116'4900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":5628,"log_dups_size":0,"ondisk_log_size":5628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279576576,"num_objects":4266,"num_object_clones":0,"num_object_copies":12798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":473408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":109,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.441419+0000","last_change":"2024-03-11T22:12:34.731700+0000","last_active":"2024-03-11T22:13:04.441419+0000","last_peered":"2024-03-11T22:13:04.441419+0000","last_clean":"2024-03-11T22:13:04.441419+0000","last_became_active":"2024-03-11T22:10:35.410893+0000","last_became_peered":"2024-03-11T22:10:35.410893+0000","last_unstale":"2024-03-11T22:13:04.441419+0000","last_undegraded":"2024-03-11T22:13:04.441419+0000","last_fullsized":"2024-03-11T22:13:04.441419+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:34.731409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:34.731409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:42.767039+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.442671+0000","last_change":"2024-03-11T22:11:28.377967+0000","last_active":"2024-03-11T22:13:04.442671+0000","last_peered":"2024-03-11T22:13:04.442671+0000","last_clean":"2024-03-11T22:13:04.442671+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:13:04.442671+0000","last_undegraded":"2024-03-11T22:13:04.442671+0000","last_fullsized":"2024-03-11T22:13:04.442671+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:28.377844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.710319+0000","last_clean_scrub_stamp":"2024-03-11T22:11:28.377844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.685892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":110,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.430421+0000","last_change":"2024-03-11T22:12:44.474800+0000","last_active":"2024-03-11T22:13:04.430421+0000","last_peered":"2024-03-11T22:13:04.430421+0000","last_clean":"2024-03-11T22:13:04.430421+0000","last_became_active":"2024-03-11T22:10:35.418715+0000","last_became_peered":"2024-03-11T22:10:35.418715+0000","last_unstale":"2024-03-11T22:13:04.430421+0000","last_undegraded":"2024-03-11T22:13:04.430421+0000","last_fullsized":"2024-03-11T22:13:04.430421+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.474728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.474728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:44.686186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":206,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.431397+0000","last_change":"2024-03-11T22:12:36.843050+0000","last_active":"2024-03-11T22:13:04.431397+0000","last_peered":"2024-03-11T22:13:04.431397+0000","last_clean":"2024-03-11T22:13:04.431397+0000","last_became_active":"2024-03-11T22:08:35.087046+0000","last_became_peered":"2024-03-11T22:08:35.087046+0000","last_unstale":"2024-03-11T22:13:04.431397+0000","last_undegraded":"2024-03-11T22:13:04.431397+0000","last_fullsized":"2024-03-11T22:13:04.431397+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:12:36.842912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:36.842912+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:04.592208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"139'10744","reported_seq":10912,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.379524+0000","last_change":"2024-03-11T22:12:25.097684+0000","last_active":"2024-03-11T22:13:05.379524+0000","last_peered":"2024-03-11T22:13:05.379524+0000","last_clean":"2024-03-11T22:13:05.379524+0000","last_became_active":"2024-03-11T22:10:36.410947+0000","last_became_peered":"2024-03-11T22:10:36.410947+0000","last_unstale":"2024-03-11T22:13:05.379524+0000","last_undegraded":"2024-03-11T22:13:05.379524+0000","last_fullsized":"2024-03-11T22:13:05.379524+0000","mapping_epoch":99,"log_start":"112'3900","ondisk_log_start":"112'3900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'8252","last_scrub_stamp":"2024-03-11T22:12:25.097553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:25.097553+0000","objects_scrubbed":6851,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:43.600745+0000","scrub_duration":2502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280100864,"num_objects":4274,"num_object_clones":0,"num_object_copies":12822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10744,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":159,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.432820+0000","last_change":"2024-03-11T22:12:28.683756+0000","last_active":"2024-03-11T22:13:04.432820+0000","last_peered":"2024-03-11T22:13:04.432820+0000","last_clean":"2024-03-11T22:13:04.432820+0000","last_became_active":"2024-03-11T22:08:39.085496+0000","last_became_peered":"2024-03-11T22:08:39.085496+0000","last_unstale":"2024-03-11T22:13:04.432820+0000","last_undegraded":"2024-03-11T22:13:04.432820+0000","last_fullsized":"2024-03-11T22:13:04.432820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:28.683646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:28.683646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:52.658563+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.446178+0000","last_change":"2024-03-11T22:11:36.289334+0000","last_active":"2024-03-11T22:13:04.446178+0000","last_peered":"2024-03-11T22:13:04.446178+0000","last_clean":"2024-03-11T22:13:04.446178+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:13:04.446178+0000","last_undegraded":"2024-03-11T22:13:04.446178+0000","last_fullsized":"2024-03-11T22:13:04.446178+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:11:36.289016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.685862+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":171,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.429618+0000","last_change":"2024-03-11T22:12:34.391183+0000","last_active":"2024-03-11T22:13:04.429618+0000","last_peered":"2024-03-11T22:13:04.429618+0000","last_clean":"2024-03-11T22:13:04.429618+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:13:04.429618+0000","last_undegraded":"2024-03-11T22:13:04.429618+0000","last_fullsized":"2024-03-11T22:13:04.429618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:34.391054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:34.391054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:45.133139+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":191,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.432408+0000","last_change":"2024-03-11T22:10:17.179119+0000","last_active":"2024-03-11T22:13:04.432408+0000","last_peered":"2024-03-11T22:13:04.432408+0000","last_clean":"2024-03-11T22:13:04.432408+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:13:04.432408+0000","last_undegraded":"2024-03-11T22:13:04.432408+0000","last_fullsized":"2024-03-11T22:13:04.432408+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:10:17.178987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:17.178987+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"139'10563","reported_seq":10726,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.145556+0000","last_change":"2024-03-11T22:12:30.660272+0000","last_active":"2024-03-11T22:13:05.145556+0000","last_peered":"2024-03-11T22:13:05.145556+0000","last_clean":"2024-03-11T22:13:05.145556+0000","last_became_active":"2024-03-11T22:10:36.416476+0000","last_became_peered":"2024-03-11T22:10:36.416476+0000","last_unstale":"2024-03-11T22:13:05.145556+0000","last_undegraded":"2024-03-11T22:13:05.145556+0000","last_fullsized":"2024-03-11T22:13:05.145556+0000","mapping_epoch":99,"log_start":"114'4300","ondisk_log_start":"114'4300","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8438","last_scrub_stamp":"2024-03-11T22:12:30.660018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:30.660018+0000","objects_scrubbed":6392,"log_size":6263,"log_dups_size":0,"ondisk_log_size":6263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:40.736516+0000","scrub_duration":1794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276234240,"num_objects":4215,"num_object_clones":0,"num_object_copies":12645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10563,"num_write_kb":472896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":159,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.445876+0000","last_change":"2024-03-11T22:12:45.439711+0000","last_active":"2024-03-11T22:13:04.445876+0000","last_peered":"2024-03-11T22:13:04.445876+0000","last_clean":"2024-03-11T22:13:04.445876+0000","last_became_active":"2024-03-11T22:08:39.083738+0000","last_became_peered":"2024-03-11T22:08:39.083738+0000","last_unstale":"2024-03-11T22:13:04.445876+0000","last_undegraded":"2024-03-11T22:13:04.445876+0000","last_fullsized":"2024-03-11T22:13:04.445876+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.439584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:47.160579+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":183,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.429096+0000","last_change":"2024-03-11T22:11:42.128436+0000","last_active":"2024-03-11T22:13:04.429096+0000","last_peered":"2024-03-11T22:13:04.429096+0000","last_clean":"2024-03-11T22:13:04.429096+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:13:04.429096+0000","last_undegraded":"2024-03-11T22:13:04.429096+0000","last_fullsized":"2024-03-11T22:13:04.429096+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:11:42.128356+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:11:42.128356+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.282739+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.426985+0000","last_change":"2024-03-11T22:09:55.458484+0000","last_active":"2024-03-11T22:13:04.426985+0000","last_peered":"2024-03-11T22:13:04.426985+0000","last_clean":"2024-03-11T22:13:04.426985+0000","last_became_active":"2024-03-11T22:08:33.035469+0000","last_became_peered":"2024-03-11T22:08:33.035469+0000","last_unstale":"2024-03-11T22:13:04.426985+0000","last_undegraded":"2024-03-11T22:13:04.426985+0000","last_fullsized":"2024-03-11T22:13:04.426985+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:55.458385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:55.458385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.057854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":270,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.440456+0000","last_change":"2024-03-11T22:11:24.895982+0000","last_active":"2024-03-11T22:13:04.440456+0000","last_peered":"2024-03-11T22:13:04.440456+0000","last_clean":"2024-03-11T22:13:04.440456+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:13:04.440456+0000","last_undegraded":"2024-03-11T22:13:04.440456+0000","last_fullsized":"2024-03-11T22:13:04.440456+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-11T22:11:24.892593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:24.892593+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:06.850678+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.437866+0000","last_change":"2024-03-11T22:11:14.224976+0000","last_active":"2024-03-11T22:13:04.437866+0000","last_peered":"2024-03-11T22:13:04.437866+0000","last_clean":"2024-03-11T22:13:04.437866+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:13:04.437866+0000","last_undegraded":"2024-03-11T22:13:04.437866+0000","last_fullsized":"2024-03-11T22:13:04.437866+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:14.224859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:14.224859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"139'10674","reported_seq":10765,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.402667+0000","last_change":"2024-03-11T22:10:36.413021+0000","last_active":"2024-03-11T22:13:05.402667+0000","last_peered":"2024-03-11T22:13:05.402667+0000","last_clean":"2024-03-11T22:13:05.402667+0000","last_became_active":"2024-03-11T22:10:36.412694+0000","last_became_peered":"2024-03-11T22:10:36.412694+0000","last_unstale":"2024-03-11T22:13:05.402667+0000","last_undegraded":"2024-03-11T22:13:05.402667+0000","last_fullsized":"2024-03-11T22:13:05.402667+0000","mapping_epoch":99,"log_start":"117'5100","ondisk_log_start":"117'5100","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":5574,"log_dups_size":0,"ondisk_log_size":5574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282329088,"num_objects":4308,"num_object_clones":0,"num_object_copies":12924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10674,"num_write_kb":479424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":101,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.432310+0000","last_change":"2024-03-11T22:11:35.482382+0000","last_active":"2024-03-11T22:13:04.432310+0000","last_peered":"2024-03-11T22:13:04.432310+0000","last_clean":"2024-03-11T22:13:04.432310+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:13:04.432310+0000","last_undegraded":"2024-03-11T22:13:04.432310+0000","last_fullsized":"2024-03-11T22:13:04.432310+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:35.482269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.482269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:07.746433+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":171,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.440944+0000","last_change":"2024-03-11T22:11:19.960632+0000","last_active":"2024-03-11T22:13:04.440944+0000","last_peered":"2024-03-11T22:13:04.440944+0000","last_clean":"2024-03-11T22:13:04.440944+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:13:04.440944+0000","last_undegraded":"2024-03-11T22:13:04.440944+0000","last_fullsized":"2024-03-11T22:13:04.440944+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:19.960398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:19.960398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:06.850710+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":188,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.441498+0000","last_change":"2024-03-11T22:12:45.863561+0000","last_active":"2024-03-11T22:13:04.441498+0000","last_peered":"2024-03-11T22:13:04.441498+0000","last_clean":"2024-03-11T22:13:04.441498+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:13:04.441498+0000","last_undegraded":"2024-03-11T22:13:04.441498+0000","last_fullsized":"2024-03-11T22:13:04.441498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.863440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.863440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:09.786177+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":194,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.437722+0000","last_change":"2024-03-11T22:11:23.875763+0000","last_active":"2024-03-11T22:13:04.437722+0000","last_peered":"2024-03-11T22:13:04.437722+0000","last_clean":"2024-03-11T22:13:04.437722+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:13:04.437722+0000","last_undegraded":"2024-03-11T22:13:04.437722+0000","last_fullsized":"2024-03-11T22:13:04.437722+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-03-11T22:11:23.875628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:23.875628+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:06.850775+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":171,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.443486+0000","last_change":"2024-03-11T22:12:31.669715+0000","last_active":"2024-03-11T22:13:04.443486+0000","last_peered":"2024-03-11T22:13:04.443486+0000","last_clean":"2024-03-11T22:13:04.443486+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:13:04.443486+0000","last_undegraded":"2024-03-11T22:13:04.443486+0000","last_fullsized":"2024-03-11T22:13:04.443486+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:31.669555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.669555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:47.635385+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"139'10627","reported_seq":10718,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.367830+0000","last_change":"2024-03-11T22:10:36.409137+0000","last_active":"2024-03-11T22:13:05.367830+0000","last_peered":"2024-03-11T22:13:05.367830+0000","last_clean":"2024-03-11T22:13:05.367830+0000","last_became_active":"2024-03-11T22:10:36.408867+0000","last_became_peered":"2024-03-11T22:10:36.408867+0000","last_unstale":"2024-03-11T22:13:05.367830+0000","last_undegraded":"2024-03-11T22:13:05.367830+0000","last_fullsized":"2024-03-11T22:13:05.367830+0000","mapping_epoch":99,"log_start":"116'5000","ondisk_log_start":"116'5000","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":5627,"log_dups_size":0,"ondisk_log_size":5627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276889600,"num_objects":4225,"num_object_clones":0,"num_object_copies":12675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10627,"num_write_kb":475264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":159,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.429598+0000","last_change":"2024-03-11T22:11:46.117166+0000","last_active":"2024-03-11T22:13:04.429598+0000","last_peered":"2024-03-11T22:13:04.429598+0000","last_clean":"2024-03-11T22:13:04.429598+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:13:04.429598+0000","last_undegraded":"2024-03-11T22:13:04.429598+0000","last_fullsized":"2024-03-11T22:13:04.429598+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:46.117039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:46.117039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.364751+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":418,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.431088+0000","last_change":"2024-03-11T22:11:34.382394+0000","last_active":"2024-03-11T22:13:04.431088+0000","last_peered":"2024-03-11T22:13:04.431088+0000","last_clean":"2024-03-11T22:13:04.431088+0000","last_became_active":"2024-03-11T22:10:35.412943+0000","last_became_peered":"2024-03-11T22:10:35.412943+0000","last_unstale":"2024-03-11T22:13:04.431088+0000","last_undegraded":"2024-03-11T22:13:04.431088+0000","last_fullsized":"2024-03-11T22:13:04.431088+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:11:34.382250+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.382250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":171,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.445601+0000","last_change":"2024-03-11T22:11:34.039067+0000","last_active":"2024-03-11T22:13:04.445601+0000","last_peered":"2024-03-11T22:13:04.445601+0000","last_clean":"2024-03-11T22:13:04.445601+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:13:04.445601+0000","last_undegraded":"2024-03-11T22:13:04.445601+0000","last_fullsized":"2024-03-11T22:13:04.445601+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.038930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:04.523208+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":244,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.432378+0000","last_change":"2024-03-11T22:11:30.413882+0000","last_active":"2024-03-11T22:13:04.432378+0000","last_peered":"2024-03-11T22:13:04.432378+0000","last_clean":"2024-03-11T22:13:04.432378+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:13:04.432378+0000","last_undegraded":"2024-03-11T22:13:04.432378+0000","last_fullsized":"2024-03-11T22:13:04.432378+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-11T22:11:30.413737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.413737+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:07.746496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":109,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.440893+0000","last_change":"2024-03-11T22:12:33.715464+0000","last_active":"2024-03-11T22:13:04.440893+0000","last_peered":"2024-03-11T22:13:04.440893+0000","last_clean":"2024-03-11T22:13:04.440893+0000","last_became_active":"2024-03-11T22:10:35.412547+0000","last_became_peered":"2024-03-11T22:10:35.412547+0000","last_unstale":"2024-03-11T22:13:04.440893+0000","last_undegraded":"2024-03-11T22:13:04.440893+0000","last_fullsized":"2024-03-11T22:13:04.440893+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:33.715347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:12:33.715347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:47.739223+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"139'10669","reported_seq":10823,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.309053+0000","last_change":"2024-03-11T22:11:42.189504+0000","last_active":"2024-03-11T22:13:05.309053+0000","last_peered":"2024-03-11T22:13:05.309053+0000","last_clean":"2024-03-11T22:13:05.309053+0000","last_became_active":"2024-03-11T22:10:36.406423+0000","last_became_peered":"2024-03-11T22:10:36.406423+0000","last_unstale":"2024-03-11T22:13:05.309053+0000","last_undegraded":"2024-03-11T22:13:05.309053+0000","last_fullsized":"2024-03-11T22:13:05.309053+0000","mapping_epoch":99,"log_start":"112'3800","ondisk_log_start":"112'3800","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5593","last_scrub_stamp":"2024-03-11T22:11:42.188394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:11:42.188394+0000","objects_scrubbed":5541,"log_size":6869,"log_dups_size":0,"ondisk_log_size":6869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.768322+0000","scrub_duration":1878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275316736,"num_objects":4201,"num_object_clones":0,"num_object_copies":12603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10669,"num_write_kb":475840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.440105+0000","last_change":"2024-03-11T22:12:44.933303+0000","last_active":"2024-03-11T22:13:04.440105+0000","last_peered":"2024-03-11T22:13:04.440105+0000","last_clean":"2024-03-11T22:13:04.440105+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:13:04.440105+0000","last_undegraded":"2024-03-11T22:13:04.440105+0000","last_fullsized":"2024-03-11T22:13:04.440105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.927967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.927967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:13.766656+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":171,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.436734+0000","last_change":"2024-03-11T22:12:19.685277+0000","last_active":"2024-03-11T22:13:04.436734+0000","last_peered":"2024-03-11T22:13:04.436734+0000","last_clean":"2024-03-11T22:13:04.436734+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:13:04.436734+0000","last_undegraded":"2024-03-11T22:13:04.436734+0000","last_fullsized":"2024-03-11T22:13:04.436734+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:19.684663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:19.684663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:48.270422+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":425,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.441264+0000","last_change":"2024-03-11T22:11:34.857427+0000","last_active":"2024-03-11T22:13:04.441264+0000","last_peered":"2024-03-11T22:13:04.441264+0000","last_clean":"2024-03-11T22:13:04.441264+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:13:04.441264+0000","last_undegraded":"2024-03-11T22:13:04.441264+0000","last_fullsized":"2024-03-11T22:13:04.441264+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:11:34.857300+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.857300+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":239,"num_read_kb":234,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":185,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.444748+0000","last_change":"2024-03-11T22:11:06.292270+0000","last_active":"2024-03-11T22:13:04.444748+0000","last_peered":"2024-03-11T22:13:04.444748+0000","last_clean":"2024-03-11T22:13:04.444748+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:13:04.444748+0000","last_undegraded":"2024-03-11T22:13:04.444748+0000","last_fullsized":"2024-03-11T22:13:04.444748+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:11:06.292138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:06.292138+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.430003+0000","last_change":"2024-03-11T22:12:27.459283+0000","last_active":"2024-03-11T22:13:04.430003+0000","last_peered":"2024-03-11T22:13:04.430003+0000","last_clean":"2024-03-11T22:13:04.430003+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:13:04.430003+0000","last_undegraded":"2024-03-11T22:13:04.430003+0000","last_fullsized":"2024-03-11T22:13:04.430003+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:27.459154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:27.459154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:36.420728+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"139'10430","reported_seq":10595,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.112271+0000","last_change":"2024-03-11T22:12:25.123093+0000","last_active":"2024-03-11T22:13:05.112271+0000","last_peered":"2024-03-11T22:13:05.112271+0000","last_clean":"2024-03-11T22:13:05.112271+0000","last_became_active":"2024-03-11T22:10:36.411289+0000","last_became_peered":"2024-03-11T22:10:36.411289+0000","last_unstale":"2024-03-11T22:13:05.112271+0000","last_undegraded":"2024-03-11T22:13:05.112271+0000","last_fullsized":"2024-03-11T22:13:05.112271+0000","mapping_epoch":99,"log_start":"114'4600","ondisk_log_start":"114'4600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7985","last_scrub_stamp":"2024-03-11T22:12:25.122944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:25.122944+0000","objects_scrubbed":6655,"log_size":5830,"log_dups_size":0,"ondisk_log_size":5830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:35.031889+0000","scrub_duration":1760,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273547264,"num_objects":4174,"num_object_clones":0,"num_object_copies":12522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10430,"num_write_kb":467328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":159,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.437370+0000","last_change":"2024-03-11T22:11:28.871970+0000","last_active":"2024-03-11T22:13:04.437370+0000","last_peered":"2024-03-11T22:13:04.437370+0000","last_clean":"2024-03-11T22:13:04.437370+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:13:04.437370+0000","last_undegraded":"2024-03-11T22:13:04.437370+0000","last_fullsized":"2024-03-11T22:13:04.437370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:28.871246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:28.871246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:06.850740+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":171,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.440366+0000","last_change":"2024-03-11T22:12:42.951642+0000","last_active":"2024-03-11T22:13:04.440366+0000","last_peered":"2024-03-11T22:13:04.440366+0000","last_clean":"2024-03-11T22:13:04.440366+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:13:04.440366+0000","last_undegraded":"2024-03-11T22:13:04.440366+0000","last_fullsized":"2024-03-11T22:13:04.440366+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.951379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.951379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:10.673379+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":195,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.432124+0000","last_change":"2024-03-11T22:10:31.211684+0000","last_active":"2024-03-11T22:13:04.432124+0000","last_peered":"2024-03-11T22:13:04.432124+0000","last_clean":"2024-03-11T22:13:04.432124+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:13:04.432124+0000","last_undegraded":"2024-03-11T22:13:04.432124+0000","last_fullsized":"2024-03-11T22:13:04.432124+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:10:31.211547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:31.211547+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":171,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.431489+0000","last_change":"2024-03-11T22:12:40.863550+0000","last_active":"2024-03-11T22:13:04.431489+0000","last_peered":"2024-03-11T22:13:04.431489+0000","last_clean":"2024-03-11T22:13:04.431489+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:13:04.431489+0000","last_undegraded":"2024-03-11T22:13:04.431489+0000","last_fullsized":"2024-03-11T22:13:04.431489+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:12:40.863445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.360193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"139'10534","reported_seq":10625,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.078030+0000","last_change":"2024-03-11T22:10:36.416445+0000","last_active":"2024-03-11T22:13:05.078030+0000","last_peered":"2024-03-11T22:13:05.078030+0000","last_clean":"2024-03-11T22:13:05.078030+0000","last_became_active":"2024-03-11T22:10:36.416083+0000","last_became_peered":"2024-03-11T22:10:36.416083+0000","last_unstale":"2024-03-11T22:13:05.078030+0000","last_undegraded":"2024-03-11T22:13:05.078030+0000","last_fullsized":"2024-03-11T22:13:05.078030+0000","mapping_epoch":99,"log_start":"119'5700","ondisk_log_start":"119'5700","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":4834,"log_dups_size":0,"ondisk_log_size":4834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.058095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272891904,"num_objects":4164,"num_object_clones":0,"num_object_copies":12492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10534,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.431000+0000","last_change":"2024-03-11T22:11:11.177052+0000","last_active":"2024-03-11T22:13:04.431000+0000","last_peered":"2024-03-11T22:13:04.431000+0000","last_clean":"2024-03-11T22:13:04.431000+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:13:04.431000+0000","last_undegraded":"2024-03-11T22:13:04.431000+0000","last_fullsized":"2024-03-11T22:13:04.431000+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:11.176930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:11.176930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":179,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.430339+0000","last_change":"2024-03-11T22:12:45.466625+0000","last_active":"2024-03-11T22:13:04.430339+0000","last_peered":"2024-03-11T22:13:04.430339+0000","last_clean":"2024-03-11T22:13:04.430339+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:13:04.430339+0000","last_undegraded":"2024-03-11T22:13:04.430339+0000","last_fullsized":"2024-03-11T22:13:04.430339+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.466508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.466508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:09.930146+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":182,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.438984+0000","last_change":"2024-03-11T22:12:38.896205+0000","last_active":"2024-03-11T22:13:04.438984+0000","last_peered":"2024-03-11T22:13:04.438984+0000","last_clean":"2024-03-11T22:13:04.438984+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:13:04.438984+0000","last_undegraded":"2024-03-11T22:13:04.438984+0000","last_fullsized":"2024-03-11T22:13:04.438984+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:12:38.895763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.895763+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:08.358608+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":226,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.432464+0000","last_change":"2024-03-11T22:11:25.385148+0000","last_active":"2024-03-11T22:13:04.432464+0000","last_peered":"2024-03-11T22:13:04.432464+0000","last_clean":"2024-03-11T22:13:04.432464+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:13:04.432464+0000","last_undegraded":"2024-03-11T22:13:04.432464+0000","last_fullsized":"2024-03-11T22:13:04.432464+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:11:25.384965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.384965+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"139'10535","reported_seq":10694,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.378771+0000","last_change":"2024-03-11T22:12:35.256058+0000","last_active":"2024-03-11T22:13:05.378771+0000","last_peered":"2024-03-11T22:13:05.378771+0000","last_clean":"2024-03-11T22:13:05.378771+0000","last_became_active":"2024-03-11T22:10:36.410859+0000","last_became_peered":"2024-03-11T22:10:36.410859+0000","last_unstale":"2024-03-11T22:13:05.378771+0000","last_undegraded":"2024-03-11T22:13:05.378771+0000","last_fullsized":"2024-03-11T22:13:05.378771+0000","mapping_epoch":99,"log_start":"121'5900","ondisk_log_start":"121'5900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8663","last_scrub_stamp":"2024-03-11T22:12:35.255739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.255739+0000","objects_scrubbed":6218,"log_size":4635,"log_dups_size":0,"ondisk_log_size":4635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:47.000467+0000","scrub_duration":2947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279904256,"num_objects":4271,"num_object_clones":0,"num_object_copies":12813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":473792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":109,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.431922+0000","last_change":"2024-03-11T22:12:46.968879+0000","last_active":"2024-03-11T22:13:04.431922+0000","last_peered":"2024-03-11T22:13:04.431922+0000","last_clean":"2024-03-11T22:13:04.431922+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:13:04.431922+0000","last_undegraded":"2024-03-11T22:13:04.431922+0000","last_fullsized":"2024-03-11T22:13:04.431922+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.968807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.968807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:12.616431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.444857+0000","last_change":"2024-03-11T22:12:49.856525+0000","last_active":"2024-03-11T22:13:04.444857+0000","last_peered":"2024-03-11T22:13:04.444857+0000","last_clean":"2024-03-11T22:13:04.444857+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:13:04.444857+0000","last_undegraded":"2024-03-11T22:13:04.444857+0000","last_fullsized":"2024-03-11T22:13:04.444857+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:49.856381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.856381+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:57.055758+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.425684+0000","last_change":"2024-03-11T22:09:53.414248+0000","last_active":"2024-03-11T22:13:04.425684+0000","last_peered":"2024-03-11T22:13:04.425684+0000","last_clean":"2024-03-11T22:13:04.425684+0000","last_became_active":"2024-03-11T22:08:35.053452+0000","last_became_peered":"2024-03-11T22:08:35.053452+0000","last_unstale":"2024-03-11T22:13:04.425684+0000","last_undegraded":"2024-03-11T22:13:04.425684+0000","last_fullsized":"2024-03-11T22:13:04.425684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:09:53.414127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:53.414127+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.057829+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":226,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.443413+0000","last_change":"2024-03-11T22:12:31.264333+0000","last_active":"2024-03-11T22:13:04.443413+0000","last_peered":"2024-03-11T22:13:04.443413+0000","last_clean":"2024-03-11T22:13:04.443413+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:13:04.443413+0000","last_undegraded":"2024-03-11T22:13:04.443413+0000","last_fullsized":"2024-03-11T22:13:04.443413+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:12:31.264213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.264213+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:45.274053+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"139'10534","reported_seq":10694,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.394769+0000","last_change":"2024-03-11T22:12:35.900151+0000","last_active":"2024-03-11T22:13:05.394769+0000","last_peered":"2024-03-11T22:13:05.394769+0000","last_clean":"2024-03-11T22:13:05.394769+0000","last_became_active":"2024-03-11T22:10:36.414380+0000","last_became_peered":"2024-03-11T22:10:36.414380+0000","last_unstale":"2024-03-11T22:13:05.394769+0000","last_undegraded":"2024-03-11T22:13:05.394769+0000","last_fullsized":"2024-03-11T22:13:05.394769+0000","mapping_epoch":99,"log_start":"121'5900","ondisk_log_start":"121'5900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8688","last_scrub_stamp":"2024-03-11T22:12:35.899750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.899750+0000","objects_scrubbed":6120,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:44.219720+0000","scrub_duration":2560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274726912,"num_objects":4192,"num_object_clones":0,"num_object_copies":12576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10534,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.425708+0000","last_change":"2024-03-11T22:11:35.248168+0000","last_active":"2024-03-11T22:13:04.425708+0000","last_peered":"2024-03-11T22:13:04.425708+0000","last_clean":"2024-03-11T22:13:04.425708+0000","last_became_active":"2024-03-11T22:08:39.078244+0000","last_became_peered":"2024-03-11T22:08:39.078244+0000","last_unstale":"2024-03-11T22:13:04.425708+0000","last_undegraded":"2024-03-11T22:13:04.425708+0000","last_fullsized":"2024-03-11T22:13:04.425708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:35.248081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.248081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.058325+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":171,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.457178+0000","last_change":"2024-03-11T22:11:24.938132+0000","last_active":"2024-03-11T22:13:04.457178+0000","last_peered":"2024-03-11T22:13:04.457178+0000","last_clean":"2024-03-11T22:13:04.457178+0000","last_became_active":"2024-03-11T22:08:33.036469+0000","last_became_peered":"2024-03-11T22:08:33.036469+0000","last_unstale":"2024-03-11T22:13:04.457178+0000","last_undegraded":"2024-03-11T22:13:04.457178+0000","last_fullsized":"2024-03-11T22:13:04.457178+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:24.937886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:24.937886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:04.523150+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":202,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.443324+0000","last_change":"2024-03-11T22:12:39.447777+0000","last_active":"2024-03-11T22:13:04.443324+0000","last_peered":"2024-03-11T22:13:04.443324+0000","last_clean":"2024-03-11T22:13:04.443324+0000","last_became_active":"2024-03-11T22:08:35.085979+0000","last_became_peered":"2024-03-11T22:08:35.085979+0000","last_unstale":"2024-03-11T22:13:04.443324+0000","last_undegraded":"2024-03-11T22:13:04.443324+0000","last_fullsized":"2024-03-11T22:13:04.443324+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:12:39.447312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:39.447312+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:00.055885+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.425583+0000","last_change":"2024-03-11T22:11:40.165879+0000","last_active":"2024-03-11T22:13:04.425583+0000","last_peered":"2024-03-11T22:13:04.425583+0000","last_clean":"2024-03-11T22:13:04.425583+0000","last_became_active":"2024-03-11T22:08:37.076522+0000","last_became_peered":"2024-03-11T22:08:37.076522+0000","last_unstale":"2024-03-11T22:13:04.425583+0000","last_undegraded":"2024-03-11T22:13:04.425583+0000","last_fullsized":"2024-03-11T22:13:04.425583+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:40.165760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:40.165760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.990830+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"139'10475","reported_seq":10637,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.098429+0000","last_change":"2024-03-11T22:12:31.355260+0000","last_active":"2024-03-11T22:13:05.098429+0000","last_peered":"2024-03-11T22:13:05.098429+0000","last_clean":"2024-03-11T22:13:05.098429+0000","last_became_active":"2024-03-11T22:10:36.414372+0000","last_became_peered":"2024-03-11T22:10:36.414372+0000","last_unstale":"2024-03-11T22:13:05.098429+0000","last_undegraded":"2024-03-11T22:13:05.098429+0000","last_fullsized":"2024-03-11T22:13:05.098429+0000","mapping_epoch":99,"log_start":"114'4200","ondisk_log_start":"114'4200","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8481","last_scrub_stamp":"2024-03-11T22:12:31.354979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.354979+0000","objects_scrubbed":6285,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:50.389207+0000","scrub_duration":1504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278593536,"num_objects":4251,"num_object_clones":0,"num_object_copies":12753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10475,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":158,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.430102+0000","last_change":"2024-03-11T22:12:42.469626+0000","last_active":"2024-03-11T22:13:04.430102+0000","last_peered":"2024-03-11T22:13:04.430102+0000","last_clean":"2024-03-11T22:13:04.430102+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:13:04.430102+0000","last_undegraded":"2024-03-11T22:13:04.430102+0000","last_fullsized":"2024-03-11T22:13:04.430102+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.469501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.469501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:51.863765+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":217,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.426568+0000","last_change":"2024-03-11T22:10:14.545445+0000","last_active":"2024-03-11T22:13:04.426568+0000","last_peered":"2024-03-11T22:13:04.426568+0000","last_clean":"2024-03-11T22:13:04.426568+0000","last_became_active":"2024-03-11T22:08:37.077142+0000","last_became_peered":"2024-03-11T22:08:37.077142+0000","last_unstale":"2024-03-11T22:13:04.426568+0000","last_undegraded":"2024-03-11T22:13:04.426568+0000","last_fullsized":"2024-03-11T22:13:04.426568+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:10:14.545345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:10:14.545345+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.057917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":171,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.443246+0000","last_change":"2024-03-11T22:12:24.278539+0000","last_active":"2024-03-11T22:13:04.443246+0000","last_peered":"2024-03-11T22:13:04.443246+0000","last_clean":"2024-03-11T22:13:04.443246+0000","last_became_active":"2024-03-11T22:08:33.037537+0000","last_became_peered":"2024-03-11T22:08:33.037537+0000","last_unstale":"2024-03-11T22:13:04.443246+0000","last_undegraded":"2024-03-11T22:13:04.443246+0000","last_fullsized":"2024-03-11T22:13:04.443246+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:24.278298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.278298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:40.405410+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":191,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.438072+0000","last_change":"2024-03-11T22:11:41.817430+0000","last_active":"2024-03-11T22:13:04.438072+0000","last_peered":"2024-03-11T22:13:04.438072+0000","last_clean":"2024-03-11T22:13:04.438072+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:13:04.438072+0000","last_undegraded":"2024-03-11T22:13:04.438072+0000","last_fullsized":"2024-03-11T22:13:04.438072+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:11:41.817285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:41.817285+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"139'10872","reported_seq":11034,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.153424+0000","last_change":"2024-03-11T22:12:35.582515+0000","last_active":"2024-03-11T22:13:05.153424+0000","last_peered":"2024-03-11T22:13:05.153424+0000","last_clean":"2024-03-11T22:13:05.153424+0000","last_became_active":"2024-03-11T22:10:36.412847+0000","last_became_peered":"2024-03-11T22:10:36.412847+0000","last_unstale":"2024-03-11T22:13:05.153424+0000","last_undegraded":"2024-03-11T22:13:05.153424+0000","last_fullsized":"2024-03-11T22:13:05.153424+0000","mapping_epoch":99,"log_start":"114'4600","ondisk_log_start":"114'4600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'9001","last_scrub_stamp":"2024-03-11T22:12:35.582375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.582375+0000","objects_scrubbed":6270,"log_size":6272,"log_dups_size":0,"ondisk_log_size":6272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:02.117832+0000","scrub_duration":1709,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286064664,"num_objects":4366,"num_object_clones":0,"num_object_copies":13098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4366,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10872,"num_write_kb":487553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":150,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.443798+0000","last_change":"2024-03-11T22:11:08.902041+0000","last_active":"2024-03-11T22:13:04.443798+0000","last_peered":"2024-03-11T22:13:04.443798+0000","last_clean":"2024-03-11T22:13:04.443798+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:13:04.443798+0000","last_undegraded":"2024-03-11T22:13:04.443798+0000","last_fullsized":"2024-03-11T22:13:04.443798+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:08.901927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:08.901927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:04.523051+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.429927+0000","last_change":"2024-03-11T22:12:20.302654+0000","last_active":"2024-03-11T22:13:04.429927+0000","last_peered":"2024-03-11T22:13:04.429927+0000","last_clean":"2024-03-11T22:13:04.429927+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:13:04.429927+0000","last_undegraded":"2024-03-11T22:13:04.429927+0000","last_fullsized":"2024-03-11T22:13:04.429927+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:20.302574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:20.302574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:30.645408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.425600+0000","last_change":"2024-03-11T22:11:23.353503+0000","last_active":"2024-03-11T22:13:04.425600+0000","last_peered":"2024-03-11T22:13:04.425600+0000","last_clean":"2024-03-11T22:13:04.425600+0000","last_became_active":"2024-03-11T22:08:33.299778+0000","last_became_peered":"2024-03-11T22:08:33.299778+0000","last_unstale":"2024-03-11T22:13:04.425600+0000","last_undegraded":"2024-03-11T22:13:04.425600+0000","last_fullsized":"2024-03-11T22:13:04.425600+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:23.353379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:23.353379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.058160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":185,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.426477+0000","last_change":"2024-03-11T22:12:24.093258+0000","last_active":"2024-03-11T22:13:04.426477+0000","last_peered":"2024-03-11T22:13:04.426477+0000","last_clean":"2024-03-11T22:13:04.426477+0000","last_became_active":"2024-03-11T22:08:35.053123+0000","last_became_peered":"2024-03-11T22:08:35.053123+0000","last_unstale":"2024-03-11T22:13:04.426477+0000","last_undegraded":"2024-03-11T22:13:04.426477+0000","last_fullsized":"2024-03-11T22:13:04.426477+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:12:24.093112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.093112+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:53.468276+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"139'10554","reported_seq":10712,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.368904+0000","last_change":"2024-03-11T22:11:46.750236+0000","last_active":"2024-03-11T22:13:05.368904+0000","last_peered":"2024-03-11T22:13:05.368904+0000","last_clean":"2024-03-11T22:13:05.368904+0000","last_became_active":"2024-03-11T22:10:36.412847+0000","last_became_peered":"2024-03-11T22:10:36.412847+0000","last_unstale":"2024-03-11T22:13:05.368904+0000","last_undegraded":"2024-03-11T22:13:05.368904+0000","last_fullsized":"2024-03-11T22:13:05.368904+0000","mapping_epoch":99,"log_start":"112'3700","ondisk_log_start":"112'3700","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'5801","last_scrub_stamp":"2024-03-11T22:11:46.749984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:11:46.749984+0000","objects_scrubbed":5740,"log_size":6854,"log_dups_size":0,"ondisk_log_size":6854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:08.701961+0000","scrub_duration":2346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273416192,"num_objects":4172,"num_object_clones":0,"num_object_copies":12516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10554,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":166,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.442036+0000","last_change":"2024-03-11T22:12:50.861675+0000","last_active":"2024-03-11T22:13:04.442036+0000","last_peered":"2024-03-11T22:13:04.442036+0000","last_clean":"2024-03-11T22:13:04.442036+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:13:04.442036+0000","last_undegraded":"2024-03-11T22:13:04.442036+0000","last_fullsized":"2024-03-11T22:13:04.442036+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.861557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.861557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:11.522320+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.440608+0000","last_change":"2024-03-11T22:12:18.789167+0000","last_active":"2024-03-11T22:13:04.440608+0000","last_peered":"2024-03-11T22:13:04.440608+0000","last_clean":"2024-03-11T22:13:04.440608+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:13:04.440608+0000","last_undegraded":"2024-03-11T22:13:04.440608+0000","last_fullsized":"2024-03-11T22:13:04.440608+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:18.788609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:12:18.788609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:22.290691+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":179,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.431588+0000","last_change":"2024-03-11T22:12:42.910183+0000","last_active":"2024-03-11T22:13:04.431588+0000","last_peered":"2024-03-11T22:13:04.431588+0000","last_clean":"2024-03-11T22:13:04.431588+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:13:04.431588+0000","last_undegraded":"2024-03-11T22:13:04.431588+0000","last_fullsized":"2024-03-11T22:13:04.431588+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.910067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.910067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:49.869286+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":186,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.426253+0000","last_change":"2024-03-11T22:11:17.270935+0000","last_active":"2024-03-11T22:13:04.426253+0000","last_peered":"2024-03-11T22:13:04.426253+0000","last_clean":"2024-03-11T22:13:04.426253+0000","last_became_active":"2024-03-11T22:08:35.055770+0000","last_became_peered":"2024-03-11T22:08:35.055770+0000","last_unstale":"2024-03-11T22:13:04.426253+0000","last_undegraded":"2024-03-11T22:13:04.426253+0000","last_fullsized":"2024-03-11T22:13:04.426253+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-11T22:11:17.270813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:17.270813+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.058391+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"139'10609","reported_seq":10697,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:05.390730+0000","last_change":"2024-03-11T22:10:36.408956+0000","last_active":"2024-03-11T22:13:05.390730+0000","last_peered":"2024-03-11T22:13:05.390730+0000","last_clean":"2024-03-11T22:13:05.390730+0000","last_became_active":"2024-03-11T22:10:36.408528+0000","last_became_peered":"2024-03-11T22:10:36.408528+0000","last_unstale":"2024-03-11T22:13:05.390730+0000","last_undegraded":"2024-03-11T22:13:05.390730+0000","last_fullsized":"2024-03-11T22:13:05.390730+0000","mapping_epoch":99,"log_start":"116'5000","ondisk_log_start":"116'5000","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":5609,"log_dups_size":0,"ondisk_log_size":5609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269680640,"num_objects":4115,"num_object_clones":0,"num_object_copies":12345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10609,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":150,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.426607+0000","last_change":"2024-03-11T22:10:23.555240+0000","last_active":"2024-03-11T22:13:04.426607+0000","last_peered":"2024-03-11T22:13:04.426607+0000","last_clean":"2024-03-11T22:13:04.426607+0000","last_became_active":"2024-03-11T22:08:39.072422+0000","last_became_peered":"2024-03-11T22:08:39.072422+0000","last_unstale":"2024-03-11T22:13:04.426607+0000","last_undegraded":"2024-03-11T22:13:04.426607+0000","last_fullsized":"2024-03-11T22:13:04.426607+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:23.555159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:23.555159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.057972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":234,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.441818+0000","last_change":"2024-03-11T22:12:43.819554+0000","last_active":"2024-03-11T22:13:04.441818+0000","last_peered":"2024-03-11T22:13:04.441818+0000","last_clean":"2024-03-11T22:13:04.441818+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:13:04.441818+0000","last_undegraded":"2024-03-11T22:13:04.441818+0000","last_fullsized":"2024-03-11T22:13:04.441818+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:12:43.819269+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_clean_scrub_stamp":"2024-03-11T22:12:43.819269+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:53.786169+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":171,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.443982+0000","last_change":"2024-03-11T22:11:40.097948+0000","last_active":"2024-03-11T22:13:04.443982+0000","last_peered":"2024-03-11T22:13:04.443982+0000","last_clean":"2024-03-11T22:13:04.443982+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:13:04.443982+0000","last_undegraded":"2024-03-11T22:13:04.443982+0000","last_fullsized":"2024-03-11T22:13:04.443982+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:40.094297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:40.094297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:04.523239+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-11T22:13:04.426158+0000","last_change":"2024-03-11T22:11:12.298926+0000","last_active":"2024-03-11T22:13:04.426158+0000","last_peered":"2024-03-11T22:13:04.426158+0000","last_clean":"2024-03-11T22:13:04.426158+0000","last_became_active":"2024-03-11T22:08:35.056698+0000","last_became_peered":"2024-03-11T22:08:35.056698+0000","last_unstale":"2024-03-11T22:13:04.426158+0000","last_undegraded":"2024-03-11T22:13:04.426158+0000","last_fullsized":"2024-03-11T22:13:04.426158+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:11:12.298787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:12.298787+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:09.058141+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4431544344,"num_objects":67621,"num_object_clones":0,"num_object_copies":202863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169317,"num_write_kb":7581953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13300936704,"data_stored":13300924488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91397,"ondisk_log_size":91397,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":239,"num_read_kb":234,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005115,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1434960,"kb_used_data":1355208,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":92310192,"statfs":{"total":95995035648,"available":94525636608,"internally_reserved":0,"allocated":1387732992,"data_stored":1387407175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":81656267},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332160,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1451332,"kb_used_data":1366008,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":92293820,"statfs":{"total":95995035648,"available":94508871680,"internally_reserved":0,"allocated":1398792192,"data_stored":1398482593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87357894},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1758256,"kb_used_data":1650724,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91986896,"statfs":{"total":95995035648,"available":94194581504,"internally_reserved":0,"allocated":1690341376,"data_stored":1689986373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2283220,"kb_used_data":2184008,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91461932,"statfs":{"total":95995035648,"available":93657018368,"internally_reserved":0,"allocated":2236424192,"data_stored":2236097731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1752696,"kb_used_data":1640876,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91992456,"statfs":{"total":95995035648,"available":94200274944,"internally_reserved":0,"allocated":1680257024,"data_stored":1679958971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2583968,"kb_used_data":2448276,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91161184,"statfs":{"total":95995035648,"available":93349052416,"internally_reserved":0,"allocated":2507034624,"data_stored":2506696291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138541511},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920843,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1427028,"kb_used_data":1363912,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92318124,"statfs":{"total":95995035648,"available":94533758976,"internally_reserved":0,"allocated":1396645888,"data_stored":1396319068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659204,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1165360,"kb_used_data":1089768,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92579792,"statfs":{"total":95995035648,"available":94801707008,"internally_reserved":0,"allocated":1115922432,"data_stored":1115619817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2217086976,"data_stored":2217082904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1677004800,"data_stored":1677000728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1660485632,"data_stored":1660485632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2474180608,"data_stored":2474180608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1394802688,"data_stored":1394802688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1101987840,"data_stored":1101987840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1389170688,"data_stored":1389166616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1386217472,"data_stored":1386217472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:13:09.555 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T22:13:09.555 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:13:09.555 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-11T22:13:09.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:13:09.586 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:09.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:09.712 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:09.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:09.754 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:09 smithi045 ceph-mon[30178]: pgmap v173: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-11T22:13:09.754 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:09 smithi045 ceph-mon[30178]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:13:09.754 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:09 smithi045 ceph-mon[30178]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T22:13:09.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:09 smithi045 ceph-mon[39715]: pgmap v173: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-11T22:13:09.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:09 smithi045 ceph-mon[39715]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:13:09.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:09 smithi045 ceph-mon[39715]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T22:13:09.813 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:09.841 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:09.841 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:09 smithi144 ceph-mon[33984]: pgmap v173: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-11T22:13:09.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:09 smithi144 ceph-mon[33984]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:13:09.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:09 smithi144 ceph-mon[33984]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T22:13:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:09.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:09.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:10.002 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:10.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:10.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:10.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:10.150 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:10.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:10.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:10.252 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:10.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:10.281 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:10.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:10.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:10.417 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:10.430 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:10.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:10.532 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:10.561 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:10.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:10.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:10.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:10.696 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:10.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:10.742 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:13:10.742 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:13:10.797 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:10.838 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:10.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:10.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:10.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:10.975 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:10.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:11.076 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:11.112 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:11.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:11.216 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:11.247 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:11.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:11.349 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:11.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:11.379 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:11.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:11.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:11.517 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T22:13:11.521 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:11.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:11.567 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:11 smithi045 ceph-mon[30178]: pgmap v175: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 667 op/s 2024-03-11T22:13:11.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:11 smithi045 ceph-mon[39715]: pgmap v175: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 667 op/s 2024-03-11T22:13:11.623 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:11.657 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:11.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:11.759 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:11.797 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:11.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:11.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:11.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:11 smithi144 ceph-mon[33984]: pgmap v175: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 667 op/s 2024-03-11T22:13:11.857 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:13:11.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:11.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:11.949 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:11.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:12.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:12.094 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:12.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:12.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:12.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:12.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:12.370 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:12.401 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:12.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:12.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:12.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:12.546 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:12.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:12.647 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:12.676 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:12.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:12.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:12.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:12.820 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:13:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:13:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:13:12.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:12.842 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:12.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:12.943 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:12.970 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:12.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:12.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:13.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:13.092 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:12 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:13.092 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:12 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:13.092 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:12 smithi144 ceph-mon[33984]: pgmap v176: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-11T22:13:13.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:13.118 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:13.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:13.219 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:13.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:13.249 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:13.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:13.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:12 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:13.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:12 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:13.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:12 smithi045 ceph-mon[39715]: pgmap v176: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-11T22:13:13.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:12 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:13.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:12 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:13.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:12 smithi045 ceph-mon[30178]: pgmap v176: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-11T22:13:13.350 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:13.381 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:13.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:13.485 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:13.514 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:13.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:13.616 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:13.643 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:13.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:13.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:13.745 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:13.772 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:13.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:13.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:13.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:13.917 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:13.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:14.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:14.061 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:14.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:14.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:14.092 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:13 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:13:14.092 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:13 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:13:14.092 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:13 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:14.113 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:13:14.113 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:13:14.163 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:14.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:14.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:14.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:14.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:13 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:13:14.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:13 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:13:14.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:13 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:14.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:13 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:13:14.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:13 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:13:14.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:13 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:14.306 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:14.336 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:14.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:14.380 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:14.438 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:14.469 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:14.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:14.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:14.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:14.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:14.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:14.717 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:14.749 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:14.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:14.769 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T22:13:14.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:14.893 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:14.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:14.995 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:15.008 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:13:15.008 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:15 smithi144 ceph-mon[33984]: pgmap v177: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T22:13:15.009 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:15 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4210317447' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:13:15.009 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:15 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:13:15.009 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:13:15.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:15.029 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:15.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:15.130 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:15.140 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:13:15.170 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:15.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:15.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:15.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:15 smithi045 ceph-mon[30178]: pgmap v177: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T22:13:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:15 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4210317447' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:13:15.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:15 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:13:15.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:13:15.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:15 smithi045 ceph-mon[39715]: pgmap v177: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T22:13:15.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:15 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4210317447' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:13:15.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:15 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:13:15.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:13:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:15.313 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:15.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:15.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:15.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:15.556 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:15.586 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:15.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:15.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:15.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:15.725 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:15.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:15.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:15.856 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:13:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:15.866 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:15.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:15.909 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:13:15.945 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:13:15 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:13:15.953 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:13:15.967 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:15.997 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:15.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:15.998 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:13:16.039 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:13:16.084 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:13:16.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:16.138 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:16 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:13:16.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:16 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:13:16.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:16 smithi045 ceph-mon[39715]: osdmap e141: 8 total, 8 up, 8 in 2024-03-11T22:13:16.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:16 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2643555980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:16.139 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:13:16.139 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:16 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:13:16.139 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:16 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:13:16.140 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:16 smithi045 ceph-mon[30178]: osdmap e141: 8 total, 8 up, 8 in 2024-03-11T22:13:16.140 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:16 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2643555980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:16.141 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:16.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:16.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:16.181 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:13:16.183 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:13:16.183 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:13:16.183 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:13:16.183 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:13:16.183 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:13:16.183 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:13:16.183 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:13:16.183 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:13:16.243 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:16.278 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:16.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:16.325 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:16 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:13:16.325 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:16 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:13:16.325 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:16 smithi144 ceph-mon[33984]: osdmap e141: 8 total, 8 up, 8 in 2024-03-11T22:13:16.325 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:16 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2643555980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:16.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:16.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:16.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:16.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:16.522 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:16.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:16.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:16.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:16.700 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:16.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:16.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:16.845 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:16.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:16.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:16.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:16.988 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:16.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:16.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:17.090 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:17.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:17.139 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:17.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:17.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:17.287 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:17.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:17.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:17.342 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:17 smithi144 ceph-mon[33984]: 4.b scrub starts 2024-03-11T22:13:17.342 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:17 smithi144 ceph-mon[33984]: 4.b scrub ok 2024-03-11T22:13:17.342 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:17 smithi144 ceph-mon[33984]: pgmap v179: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-11T22:13:17.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:17.432 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:17.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:17.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:17.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:17 smithi045 ceph-mon[39715]: 4.b scrub starts 2024-03-11T22:13:17.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:17 smithi045 ceph-mon[39715]: 4.b scrub ok 2024-03-11T22:13:17.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:17 smithi045 ceph-mon[39715]: pgmap v179: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-11T22:13:17.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:17.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:17 smithi045 ceph-mon[30178]: 4.b scrub starts 2024-03-11T22:13:17.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:17 smithi045 ceph-mon[30178]: 4.b scrub ok 2024-03-11T22:13:17.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:17 smithi045 ceph-mon[30178]: pgmap v179: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-11T22:13:17.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:17.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:17.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:17.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:17.721 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:17.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:17.754 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:13:17.754 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-03-11T22:13:17.754 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-03-11T22:13:17.754 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:13:17.754 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.754 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:27626 7.0 7:0e450ccf:::benchmark_data_smithi045_83007_object27625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:13:17.754 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:55.717490+0000", 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 142.037824019, 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13484212200000001, 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 27626 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717490+0000", 2024-03-11T22:13:17.755 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717490+0000", 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717492+0000", 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3859999999999999e-06 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.756 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717531+0000", 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9270000000000002e-05 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717544+0000", 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3094e-05 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717550+0000", 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6899999999999997e-06 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.757 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717606+0000", 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5828000000000001e-05 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717750+0000", 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014452900000000001 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717910+0000", 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015962800000000001 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.758 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718911+0000", 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010012249999999999 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.719301+0000", 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00039031800000000001 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.852224+0000", 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13292285600000001 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.759 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.852278+0000", 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3783000000000003e-05 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.852333+0000", 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4515000000000002e-05 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:28347 7.0 7:043ba8fd:::benchmark_data_smithi045_83007_object28346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:13:17.760 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:56.330173+0000", 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 141.42514112399999, 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098482391000000002, 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 28347 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330173+0000", 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.761 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330173+0000", 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330175+0000", 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.455e-06 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330221+0000", 2024-03-11T22:13:17.762 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6217000000000002e-05 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330235+0000", 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3811e-05 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330241+0000", 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9610000000000002e-06 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330285+0000", 2024-03-11T22:13:17.763 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4215999999999998e-05 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330392+0000", 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000107118 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330505+0000", 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000112728 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.332109+0000", 2024-03-11T22:13:17.764 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016036710000000001 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.332155+0000", 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6718999999999998e-05 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.428535+0000", 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.096379861999999997 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.428599+0000", 2024-03-11T22:13:17.765 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4277999999999997e-05 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.428656+0000", 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6354999999999998e-05 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:28348 7.0 7:02caa2fd:::benchmark_data_smithi045_83007_object28347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:56.330255+0000", 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 141.425059162, 2024-03-11T22:13:17.766 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098596250999999996, 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 28348 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330255+0000", 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.767 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330255+0000", 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330256+0000", 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2900000000000004e-07 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330287+0000", 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1269999999999997e-05 2024-03-11T22:13:17.768 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330294+0000", 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3679999999999996e-06 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330299+0000", 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1110000000000001e-06 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.331401+0000", 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001102331 2024-03-11T22:13:17.769 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.331519+0000", 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000117959 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.331601+0000", 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2250999999999996e-05 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.332252+0000", 2024-03-11T22:13:17.770 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00065089 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.332734+0000", 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00048246299999999998 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.428756+0000", 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.096021624 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.771 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.428812+0000", 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5971000000000003e-05 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.428852+0000", 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9483999999999997e-05 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:33008 7.0 7:0360b4fc:::benchmark_data_smithi045_83007_object33007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:59.002880+0000", 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 138.752434384, 2024-03-11T22:13:17.772 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.074918210999999998, 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 33008 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.002880+0000", 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.773 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.774 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.002880+0000", 2024-03-11T22:13:17.774 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.774 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.774 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.002881+0000", 2024-03-11T22:13:17.774 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7899999999999997e-07 2024-03-11T22:13:17.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.774 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.774 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.774 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.002992+0000", 2024-03-11T22:13:17.774 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000110776 2024-03-11T22:13:17.774 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.003003+0000", 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1719000000000001e-05 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.003011+0000", 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8769999999999993e-06 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.003074+0000", 2024-03-11T22:13:17.775 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3101000000000005e-05 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.003242+0000", 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016787599999999999 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.003384+0000", 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014189699999999999 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.004554+0000", 2024-03-11T22:13:17.776 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011698850000000001 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.007550+0000", 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0029955400000000001 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.077683+0000", 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.070133038999999994 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.777 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.077751+0000", 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8428000000000003e-05 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.077798+0000", 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7194000000000002e-05 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:33009 7.0 7:0df63874:::benchmark_data_smithi045_83007_object33008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:59.004047+0000", 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 138.75126784899999, 2024-03-11T22:13:17.778 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.073855392000000006, 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 33009 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.004047+0000", 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.779 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.780 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.780 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.004047+0000", 2024-03-11T22:13:17.780 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.780 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.780 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.780 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.780 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.004047+0000", 2024-03-11T22:13:17.781 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7599999999999996e-07 2024-03-11T22:13:17.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.781 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.781 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.781 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.004078+0000", 2024-03-11T22:13:17.781 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0397999999999999e-05 2024-03-11T22:13:17.781 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.781 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.004086+0000", 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3999999999999992e-06 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.004093+0000", 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2910000000000004e-06 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.004592+0000", 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00049967299999999996 2024-03-11T22:13:17.782 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.004706+0000", 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000113922 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.004880+0000", 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017345300000000001 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.006064+0000", 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001184091 2024-03-11T22:13:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.007614+0000", 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015500519999999999 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.077841+0000", 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.070227099000000001 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.077876+0000", 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5014000000000001e-05 2024-03-11T22:13:17.784 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.077902+0000", 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6222999999999999e-05 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:33019 7.0 7:03abe11d:::benchmark_data_smithi045_83007_object33018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:59.009585+0000", 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 138.745729657, 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.073259580000000005, 2024-03-11T22:13:17.785 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 33019 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.009585+0000", 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.786 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.009585+0000", 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.009592+0000", 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6660000000000008e-06 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.009648+0000", 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5263000000000003e-05 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.787 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.009660+0000", 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2199e-05 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.009667+0000", 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3540000000000002e-06 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.009710+0000", 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2882e-05 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.788 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.009791+0000", 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0844e-05 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.009885+0000", 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3913999999999996e-05 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.011035+0000", 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011503559999999999 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.789 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.011684+0000", 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00064846900000000004 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.082787+0000", 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.071102950999999998 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.082820+0000", 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3249e-05 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.790 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:59.082844+0000", 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4433000000000001e-05 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:41669 7.b e109/99 7:d954276e:::benchmark_data_smithi045_83007_object41668:head v 109'2565, mlcod=109'2560)", 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.905298+0000", 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 133.85001677599999, 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.067677389000000004, 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.791 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.905298+0000", 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.905298+0000", 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.905298+0000", 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7800000000000001e-07 2024-03-11T22:13:17.792 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.905379+0000", 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1029e-05 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.905389+0000", 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.5140000000000007e-06 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.905394+0000", 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8760000000000004e-06 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.793 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.794 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.794 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.905439+0000", 2024-03-11T22:13:17.794 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4898000000000003e-05 2024-03-11T22:13:17.794 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.794 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.795 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.795 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.905451+0000", 2024-03-11T22:13:17.795 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2004000000000001e-05 2024-03-11T22:13:17.795 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.795 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.795 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.795 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906417+0000", 2024-03-11T22:13:17.796 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00096612399999999995 2024-03-11T22:13:17.796 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.796 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.797 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.797 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.972975+0000", 2024-03-11T22:13:17.797 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.066558365999999994 2024-03-11T22:13:17.798 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.798 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.798 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.798 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.798 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.798 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:41670 7.0 7:0437c8c5:::benchmark_data_smithi045_83007_object41669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:13:17.798 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.906116+0000", 2024-03-11T22:13:17.798 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 133.84919889099999, 2024-03-11T22:13:17.798 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.069338330000000004, 2024-03-11T22:13:17.798 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.799 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.799 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.799 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.799 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.799 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 41670 2024-03-11T22:13:17.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.799 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.799 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.799 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.799 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906116+0000", 2024-03-11T22:13:17.799 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.799 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.800 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.800 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.800 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906116+0000", 2024-03-11T22:13:17.800 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.800 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.800 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.800 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.800 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906117+0000", 2024-03-11T22:13:17.800 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4050000000000001e-06 2024-03-11T22:13:17.800 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.800 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.800 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.801 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906165+0000", 2024-03-11T22:13:17.801 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8044999999999998e-05 2024-03-11T22:13:17.801 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.801 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.801 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.801 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906173+0000", 2024-03-11T22:13:17.801 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4139999999999994e-06 2024-03-11T22:13:17.801 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.801 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.801 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.802 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906178+0000", 2024-03-11T22:13:17.802 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9280000000000001e-06 2024-03-11T22:13:17.802 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.802 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.802 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.802 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906213+0000", 2024-03-11T22:13:17.802 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4224000000000003e-05 2024-03-11T22:13:17.802 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.802 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.802 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906306+0000", 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.357e-05 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906394+0000", 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8214000000000006e-05 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.975312+0000", 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.068917199999999998 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.803 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.975341+0000", 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9754e-05 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.975349+0000", 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7710000000000006e-06 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.975403+0000", 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4188999999999999e-05 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.804 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.975454+0000", 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0615999999999998e-05 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:41671 7.0 7:00a072c5:::benchmark_data_smithi045_83007_object41670:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.906304+0000", 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 133.849010534, 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.07132521, 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.805 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.806 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.806 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.806 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 41671 2024-03-11T22:13:17.806 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.806 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.806 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.806 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.806 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906304+0000", 2024-03-11T22:13:17.806 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.806 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.806 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906304+0000", 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906305+0000", 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3399999999999999e-07 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906372+0000", 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7747e-05 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.807 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906378+0000", 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.913e-06 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.906383+0000", 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3069999999999996e-06 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.974168+0000", 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.067785471999999999 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.808 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.974307+0000", 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013874199999999999 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.974424+0000", 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011673899999999999 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.975462+0000", 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010383969999999999 2024-03-11T22:13:17.809 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.810 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.810 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.810 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.977537+0000", 2024-03-11T22:13:17.810 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002074649 2024-03-11T22:13:17.810 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.810 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.810 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.810 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.977567+0000", 2024-03-11T22:13:17.810 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0546999999999997e-05 2024-03-11T22:13:17.810 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.977599+0000", 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1526999999999998e-05 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.977629+0000", 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0536e-05 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.811 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.812 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop_reply(client.14790.0:41670 7.0 e109/99 ondisk, result = 0)", 2024-03-11T22:13:17.812 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.907882+0000", 2024-03-11T22:13:17.812 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 133.84743270800001, 2024-03-11T22:13:17.812 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.067435448999999995, 2024-03-11T22:13:17.812 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.812 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:13:17.812 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.812 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.907882+0000", 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.907882+0000", 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.907882+0000", 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8299999999999997e-07 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.813 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.814 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.814 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.907885+0000", 2024-03-11T22:13:17.814 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7760000000000002e-06 2024-03-11T22:13:17.814 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.814 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.814 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.815 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.972581+0000", 2024-03-11T22:13:17.815 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.064695582000000001 2024-03-11T22:13:17.815 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.815 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.815 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.815 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.972603+0000", 2024-03-11T22:13:17.815 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2646e-05 2024-03-11T22:13:17.815 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.815 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.815 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.815 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.975303+0000", 2024-03-11T22:13:17.815 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026998030000000002 2024-03-11T22:13:17.815 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.975308+0000", 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1229999999999999e-06 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.975317+0000", 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.0359999999999995e-06 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.816 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop_reply(client.14790.0:41670 7.0 e109/99 ondisk, result = 0)", 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.907884+0000", 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 133.84743047500001, 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.067572702999999998, 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.907884+0000", 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.817 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.907884+0000", 2024-03-11T22:13:17.818 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.818 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.818 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.818 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.818 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.907884+0000", 2024-03-11T22:13:17.818 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3000000000000001e-07 2024-03-11T22:13:17.818 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.818 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.818 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.818 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.907887+0000", 2024-03-11T22:13:17.818 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2560000000000001e-06 2024-03-11T22:13:17.818 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.818 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.972662+0000", 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.064774841 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.972686+0000", 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4189000000000002e-05 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.975334+0000", 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002647951 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.819 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.820 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.820 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.975338+0000", 2024-03-11T22:13:17.820 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4599999999999996e-06 2024-03-11T22:13:17.820 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.820 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.820 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.820 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.975457+0000", 2024-03-11T22:13:17.820 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000118576 2024-03-11T22:13:17.820 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.820 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.820 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.820 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.821 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.821 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:104411 7.0 7:08c8d844:::benchmark_data_smithi045_83007_object104410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:13:17.821 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:59.171403+0000", 2024-03-11T22:13:17.821 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 78.583911950000001, 2024-03-11T22:13:17.821 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.089551477000000004, 2024-03-11T22:13:17.821 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.821 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.821 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.821 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.821 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.821 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 104411 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.171403+0000", 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.171403+0000", 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.822 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.171410+0000", 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5349999999999999e-06 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.171519+0000", 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000108968 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.171533+0000", 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3499999999999999e-05 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.823 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.171540+0000", 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7700000000000001e-06 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.171609+0000", 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8430999999999996e-05 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.171735+0000", 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012615099999999999 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.824 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.825 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.825 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.171884+0000", 2024-03-11T22:13:17.825 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014879199999999999 2024-03-11T22:13:17.825 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.825 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.825 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.825 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.174601+0000", 2024-03-11T22:13:17.825 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0027170969999999999 2024-03-11T22:13:17.825 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.825 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.825 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.825 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.243005+0000", 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.068403933 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.260830+0000", 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.017825331 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.260894+0000", 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3547999999999995e-05 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.826 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:59.260954+0000", 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0421000000000003e-05 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:142098 7.0 7:024e453e:::benchmark_data_smithi045_83007_object23627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:38.072174+0000", 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 39.683140530999999, 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.066214166000000005, 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.827 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 142098 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.072174+0000", 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.072174+0000", 2024-03-11T22:13:17.828 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.072175+0000", 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.128e-06 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.072179+0000", 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4800000000000001e-06 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.072189+0000", 2024-03-11T22:13:17.829 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0693000000000001e-05 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.072197+0000", 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7549999999999994e-06 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.072259+0000", 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2260000000000004e-05 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.072381+0000", 2024-03-11T22:13:17.830 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000121981 2024-03-11T22:13:17.831 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.831 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.831 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.831 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.072497+0000", 2024-03-11T22:13:17.831 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011606 2024-03-11T22:13:17.831 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.831 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.831 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.831 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.075421+0000", 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0029231909999999999 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.075927+0000", 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00050662299999999999 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.138324+0000", 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.062396487 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.832 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.138357+0000", 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3552e-05 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.138388+0000", 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0956e-05 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:142105 7.0 7:0b2b6ca9:::benchmark_data_smithi045_83007_object23634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:38.079288+0000", 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 39.676026471999997, 2024-03-11T22:13:17.833 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.073113069000000003, 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 142105 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.079288+0000", 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.834 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.079288+0000", 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.079290+0000", 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5859999999999999e-06 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.079294+0000", 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.313e-06 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.079306+0000", 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2075e-05 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.079314+0000", 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1689999999999992e-06 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.079379+0000", 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4661999999999999e-05 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.836 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.079509+0000", 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013002599999999999 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.079650+0000", 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000141089 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.087397+0000", 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0077473059999999998 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.837 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.089735+0000", 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002338124 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.152298+0000", 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.062562240000000005 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.152352+0000", 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4237000000000003e-05 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.838 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.152401+0000", 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9242000000000001e-05 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:142108 7.0 7:0f5b27cf:::benchmark_data_smithi045_83007_object23637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:38.080668+0000", 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 39.674646838999998, 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.071874564000000002, 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.839 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.840 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.840 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 142108 2024-03-11T22:13:17.840 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.840 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.840 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:17.842 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.842 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.080668+0000", 2024-03-11T22:13:17.842 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.842 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.842 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.842 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.080668+0000", 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.080669+0000", 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0499999999999999e-06 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.080673+0000", 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4100000000000001e-06 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.843 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.080684+0000", 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1005000000000001e-05 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.080692+0000", 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6480000000000003e-06 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.082503+0000", 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018115220000000001 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.844 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.082625+0000", 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000121489 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.082746+0000", 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000121088 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.087402+0000", 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0046557600000000001 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.845 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.089769+0000", 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0023672160000000001 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.152446+0000", 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.062676800000000005 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.152499+0000", 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3633e-05 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.846 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.152542+0000", 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2942999999999999e-05 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:142109 7.0 7:034bb10e:::benchmark_data_smithi045_83007_object23638:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:38.080707+0000", 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 39.674607258000002, 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.076344875000000006, 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.847 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 142109 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.080707+0000", 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.080707+0000", 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.848 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.080708+0000", 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4999999999999998e-07 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.080710+0000", 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5110000000000002e-06 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.080717+0000", 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0520000000000004e-06 2024-03-11T22:13:17.849 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.080722+0000", 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0010000000000001e-06 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.085264+0000", 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0045414720000000004 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.085382+0000", 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000118671 2024-03-11T22:13:17.850 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.085452+0000", 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9497000000000002e-05 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.088983+0000", 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0035307440000000002 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.090930+0000", 2024-03-11T22:13:17.851 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0019471969999999999 2024-03-11T22:13:17.852 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.852 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.852 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.852 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.156982+0000", 2024-03-11T22:13:17.852 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.066052451999999998 2024-03-11T22:13:17.852 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.852 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.852 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.852 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.157015+0000", 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3161000000000003e-05 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.157052+0000", 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6667e-05 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:142114 7.0 7:08297da2:::benchmark_data_smithi045_83007_object23643:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:38.085328+0000", 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 39.669986788999999, 2024-03-11T22:13:17.853 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.071860623999999998, 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 142114 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.085328+0000", 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.854 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.085328+0000", 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.085329+0000", 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3370000000000001e-06 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.085336+0000", 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.289e-06 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.855 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.085346+0000", 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7289999999999995e-06 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.085357+0000", 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1406000000000001e-05 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.087433+0000", 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00207508 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.856 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.087534+0000", 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010189899999999999 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.087622+0000", 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7584999999999996e-05 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.089068+0000", 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0014457199999999999 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.857 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.090951+0000", 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018833560000000001 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.157095+0000", 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.066143449000000007 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.157147+0000", 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2924999999999999e-05 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.858 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.157188+0000", 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0849000000000001e-05 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:142725 7.0 7:068aa2b7:::benchmark_data_smithi045_83007_object24254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:38.771159+0000", 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 38.984155569999999, 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.076900267999999994, 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.859 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 142725 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771159+0000", 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771159+0000", 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.860 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771160+0000", 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4929999999999999e-06 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771167+0000", 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2559999999999999e-06 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771183+0000", 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6024000000000001e-05 2024-03-11T22:13:17.861 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.862 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.862 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.862 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771191+0000", 2024-03-11T22:13:17.862 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3790000000000006e-06 2024-03-11T22:13:17.862 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.862 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.862 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.862 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771237+0000", 2024-03-11T22:13:17.862 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5565999999999997e-05 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771320+0000", 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3800000000000004e-05 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771393+0000", 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2527000000000005e-05 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.863 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.776672+0000", 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0052791890000000001 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.777883+0000", 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001211211 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.847990+0000", 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.070106185000000001 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.864 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.848029+0000", 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9201e-05 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.848059+0000", 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0437e-05 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:142729 7.0 7:042477ec:::benchmark_data_smithi045_83007_object24258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:38.771960+0000", 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 38.983354794999997, 2024-03-11T22:13:17.865 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.079086639, 2024-03-11T22:13:17.866 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.866 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.866 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.866 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.866 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.866 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 142729 2024-03-11T22:13:17.866 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.866 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.866 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.866 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771960+0000", 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771960+0000", 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771961+0000", 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6300000000000004e-07 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.867 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.868 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771964+0000", 2024-03-11T22:13:17.868 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7919999999999999e-06 2024-03-11T22:13:17.868 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.868 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.868 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.868 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771977+0000", 2024-03-11T22:13:17.868 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2398000000000001e-05 2024-03-11T22:13:17.868 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.868 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.869 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.869 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771984+0000", 2024-03-11T22:13:17.869 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5020000000000003e-06 2024-03-11T22:13:17.869 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.869 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.869 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.869 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.774122+0000", 2024-03-11T22:13:17.869 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0021375510000000001 2024-03-11T22:13:17.869 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.869 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.869 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.870 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.774245+0000", 2024-03-11T22:13:17.870 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012332000000000001 2024-03-11T22:13:17.870 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.870 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.870 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.870 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.774362+0000", 2024-03-11T22:13:17.870 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011685500000000001 2024-03-11T22:13:17.870 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.870 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.870 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.776809+0000", 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024470410000000001 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.777934+0000", 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011248479999999999 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.850939+0000", 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.073004747999999994 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.871 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.872 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.850996+0000", 2024-03-11T22:13:17.872 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7518000000000001e-05 2024-03-11T22:13:17.872 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.872 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.872 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.872 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.851046+0000", 2024-03-11T22:13:17.872 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0203e-05 2024-03-11T22:13:17.872 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.872 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.872 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.872 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.873 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.873 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:180786 7.0 7:0a169876:::benchmark_data_smithi045_83007_object62315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T22:13:17.873 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:16.626278+0000", 2024-03-11T22:13:17.873 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 1.1290364589999999, 2024-03-11T22:13:17.873 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.061758579000000001, 2024-03-11T22:13:17.873 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:17.873 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:17.873 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:17.873 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:17.873 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:17.873 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 180786 2024-03-11T22:13:17.873 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.626278+0000", 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.626278+0000", 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.626279+0000", 2024-03-11T22:13:17.874 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.249e-06 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.626284+0000", 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2509999999999996e-06 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.626294+0000", 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0029e-05 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.626299+0000", 2024-03-11T22:13:17.875 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6629999999999998e-06 2024-03-11T22:13:17.876 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.876 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.876 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:17.876 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.626350+0000", 2024-03-11T22:13:17.876 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0479999999999998e-05 2024-03-11T22:13:17.876 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.876 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.876 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:17.876 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.626462+0000", 2024-03-11T22:13:17.876 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000112679 2024-03-11T22:13:17.876 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.876 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.626584+0000", 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012183700000000001 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.629682+0000", 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0030974900000000001 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.630523+0000", 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00084115700000000004 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.877 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.687945+0000", 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.057421725999999999 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.687987+0000", 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2558000000000001e-05 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:16.688037+0000", 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9459999999999997e-05 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.878 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.879 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.879 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:17.879 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:17.879 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:13:17.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:17.890 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:17.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:17.991 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:18.019 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:18.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:18.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:18.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:18.174 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:18.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:18.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:18.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:18.317 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:18.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:18.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:18 smithi045 ceph-mon[39715]: 4.1e scrub starts 2024-03-11T22:13:18.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:18 smithi045 ceph-mon[39715]: 4.1e scrub ok 2024-03-11T22:13:18.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:18 smithi045 ceph-mon[39715]: 4.1f scrub starts 2024-03-11T22:13:18.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:18 smithi045 ceph-mon[39715]: 4.1f scrub ok 2024-03-11T22:13:18.397 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:18 smithi045 ceph-mon[30178]: 4.1e scrub starts 2024-03-11T22:13:18.397 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:18 smithi045 ceph-mon[30178]: 4.1e scrub ok 2024-03-11T22:13:18.397 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:18 smithi045 ceph-mon[30178]: 4.1f scrub starts 2024-03-11T22:13:18.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:18 smithi045 ceph-mon[30178]: 4.1f scrub ok 2024-03-11T22:13:18.418 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:18.445 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:18.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:18.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:18.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:18.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:18 smithi144 ceph-mon[33984]: 4.1e scrub starts 2024-03-11T22:13:18.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:18 smithi144 ceph-mon[33984]: 4.1e scrub ok 2024-03-11T22:13:18.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:18 smithi144 ceph-mon[33984]: 4.1f scrub starts 2024-03-11T22:13:18.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:18 smithi144 ceph-mon[33984]: 4.1f scrub ok 2024-03-11T22:13:18.597 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:18.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:18.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:18.716 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T22:13:18.741 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:18.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:18.843 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:18.875 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:18.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:18.976 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:19.007 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:19.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:19.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:19.020 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:13:19.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:19.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:19.149 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:19.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:19.250 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:19.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:13:19.276 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:19.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:19.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:19.378 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:19.404 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:19.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:19.413 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:13:19.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[39715]: 6.c scrub starts 2024-03-11T22:13:19.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[39715]: 6.c scrub ok 2024-03-11T22:13:19.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[39715]: 4.4 scrub starts 2024-03-11T22:13:19.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[39715]: 4.4 scrub ok 2024-03-11T22:13:19.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[39715]: 7.f scrub starts 2024-03-11T22:13:19.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[39715]: pgmap v180: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T22:13:19.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[39715]: 4.8 scrub starts 2024-03-11T22:13:19.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[39715]: 4.8 scrub ok 2024-03-11T22:13:19.415 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[39715]: 4.11 scrub starts 2024-03-11T22:13:19.415 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[39715]: 4.11 scrub ok 2024-03-11T22:13:19.415 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[39715]: 5.9 scrub starts 2024-03-11T22:13:19.415 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[39715]: 5.9 scrub ok 2024-03-11T22:13:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[30178]: 6.c scrub starts 2024-03-11T22:13:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[30178]: 6.c scrub ok 2024-03-11T22:13:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[30178]: 4.4 scrub starts 2024-03-11T22:13:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[30178]: 4.4 scrub ok 2024-03-11T22:13:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[30178]: 7.f scrub starts 2024-03-11T22:13:19.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[30178]: pgmap v180: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T22:13:19.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[30178]: 4.8 scrub starts 2024-03-11T22:13:19.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[30178]: 4.8 scrub ok 2024-03-11T22:13:19.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[30178]: 4.11 scrub starts 2024-03-11T22:13:19.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[30178]: 4.11 scrub ok 2024-03-11T22:13:19.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[30178]: 5.9 scrub starts 2024-03-11T22:13:19.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:19 smithi045 ceph-mon[30178]: 5.9 scrub ok 2024-03-11T22:13:19.424 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:13:19.469 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:19 smithi144 ceph-mon[33984]: 6.c scrub starts 2024-03-11T22:13:19.469 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:19 smithi144 ceph-mon[33984]: 6.c scrub ok 2024-03-11T22:13:19.469 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:19 smithi144 ceph-mon[33984]: 4.4 scrub starts 2024-03-11T22:13:19.469 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:19 smithi144 ceph-mon[33984]: 4.4 scrub ok 2024-03-11T22:13:19.469 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:19 smithi144 ceph-mon[33984]: 7.f scrub starts 2024-03-11T22:13:19.469 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:19 smithi144 ceph-mon[33984]: pgmap v180: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T22:13:19.469 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:19 smithi144 ceph-mon[33984]: 4.8 scrub starts 2024-03-11T22:13:19.469 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:19 smithi144 ceph-mon[33984]: 4.8 scrub ok 2024-03-11T22:13:19.469 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:19 smithi144 ceph-mon[33984]: 4.11 scrub starts 2024-03-11T22:13:19.469 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:19 smithi144 ceph-mon[33984]: 4.11 scrub ok 2024-03-11T22:13:19.469 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:19 smithi144 ceph-mon[33984]: 5.9 scrub starts 2024-03-11T22:13:19.469 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:19 smithi144 ceph-mon[33984]: 5.9 scrub ok 2024-03-11T22:13:19.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:19.558 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:19.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:19.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:19.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:19.730 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:19.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:19.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:19.832 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:19.866 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:19.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:19.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:19.968 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:19.998 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:19.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:20.100 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:20.128 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:20.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:20.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:20.230 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:20.265 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:20.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:20.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:13:20.341 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:20 smithi144 ceph-mon[33984]: from='client.24803 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:13:20.342 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:20 smithi144 ceph-mon[33984]: 3.1e scrub starts 2024-03-11T22:13:20.342 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:20 smithi144 ceph-mon[33984]: 3.1e scrub ok 2024-03-11T22:13:20.366 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:20.393 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:20.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:20.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:20.496 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:20.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:20 smithi045 ceph-mon[39715]: from='client.24803 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:13:20.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:20 smithi045 ceph-mon[39715]: 3.1e scrub starts 2024-03-11T22:13:20.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:20 smithi045 ceph-mon[39715]: 3.1e scrub ok 2024-03-11T22:13:20.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:20 smithi045 ceph-mon[30178]: from='client.24803 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:13:20.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:20 smithi045 ceph-mon[30178]: 3.1e scrub starts 2024-03-11T22:13:20.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:20 smithi045 ceph-mon[30178]: 3.1e scrub ok 2024-03-11T22:13:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:20.530 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:20.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:20.631 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:20.655 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:20.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:20.757 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:20.789 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:20.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:20.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:20.890 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:20.916 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:20.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:20.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:21.001 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:13:21.001 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:13:21.001 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.001 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:184765 7.d 7:bb3bcb71:::benchmark_data_smithi045_83007_object66294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:20.991016+0000", 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.011976564, 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012014663, 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 184765 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:21.002 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.991016+0000", 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.991016+0000", 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.991017+0000", 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.167e-06 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:21.003 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.991029+0000", 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1276e-05 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.991039+0000", 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0244000000000001e-05 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.991046+0000", 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7219999999999999e-06 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:21.004 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.991121+0000", 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4727999999999993e-05 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.991275+0000", 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015409500000000001 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.991408+0000", 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013318600000000001 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.005 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:21.006 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996677+0000", 2024-03-11T22:13:21.006 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.005268951 2024-03-11T22:13:21.006 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.006 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.006 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:21.006 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.001863+0000", 2024-03-11T22:13:21.006 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0051855709999999999 2024-03-11T22:13:21.006 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:21.006 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:21.006 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:21.006 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.006 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.006 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:184770 7.b e141/99 7:d508d29b:::benchmark_data_smithi045_83007_object66299:head v 141'13882, mlcod=141'13879)", 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:20.996646+0000", 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.0063464690000000004, 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0064790289999999999, 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996646+0000", 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:21.007 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996646+0000", 2024-03-11T22:13:21.008 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:21.008 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.008 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.008 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:21.008 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996647+0000", 2024-03-11T22:13:21.008 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2200000000000003e-07 2024-03-11T22:13:21.008 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.008 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.008 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:21.008 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996654+0000", 2024-03-11T22:13:21.008 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2280000000000001e-06 2024-03-11T22:13:21.008 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.008 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996670+0000", 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6232999999999999e-05 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996679+0000", 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0960000000000001e-06 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996724+0000", 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5578000000000003e-05 2024-03-11T22:13:21.009 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.010 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.010 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:21.010 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996743+0000", 2024-03-11T22:13:21.010 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9083999999999999e-05 2024-03-11T22:13:21.010 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:21.010 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:21.010 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:21.010 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.010 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.010 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:184773 7.c 7:38b8f9bf:::benchmark_data_smithi045_83007_object66302:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T22:13:21.010 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:20.996798+0000", 2024-03-11T22:13:21.010 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.0061942070000000002, 2024-03-11T22:13:21.010 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0063906459999999998, 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 184773 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996798+0000", 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.011 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.012 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:21.012 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996798+0000", 2024-03-11T22:13:21.012 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:21.012 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.012 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.012 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:21.012 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996800+0000", 2024-03-11T22:13:21.012 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3549999999999999e-06 2024-03-11T22:13:21.012 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.012 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.012 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:21.012 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996804+0000", 2024-03-11T22:13:21.013 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6999999999999999e-06 2024-03-11T22:13:21.013 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.013 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.013 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:21.013 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996816+0000", 2024-03-11T22:13:21.013 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1564e-05 2024-03-11T22:13:21.013 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.013 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.013 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:21.013 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996824+0000", 2024-03-11T22:13:21.013 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5109999999999997e-06 2024-03-11T22:13:21.013 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.013 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.996899+0000", 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5766000000000006e-05 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.997048+0000", 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000148679 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.997169+0000", 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000120815 2024-03-11T22:13:21.014 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.015 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.015 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:21.015 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.001898+0000", 2024-03-11T22:13:21.015 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0047293659999999996 2024-03-11T22:13:21.015 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:21.015 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:21.015 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:21.015 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.015 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.016 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:184772 7.b e141/99 7:d9fa348f:::benchmark_data_smithi045_83007_object66301:head v 141'13883, mlcod=141'13879)", 2024-03-11T22:13:21.016 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:21.000579+0000", 2024-03-11T22:13:21.016 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.0024137270000000001, 2024-03-11T22:13:21.016 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002693773, 2024-03-11T22:13:21.016 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:21.016 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:13:21.016 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:21.016 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.016 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:21.016 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.000579+0000", 2024-03-11T22:13:21.016 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:21.016 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.016 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.000579+0000", 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.000580+0000", 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.0999999999999997e-07 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.000586+0000", 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0090000000000004e-06 2024-03-11T22:13:21.017 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.000600+0000", 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.419e-05 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.000607+0000", 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0480000000000001e-06 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.000673+0000", 2024-03-11T22:13:21.018 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6356e-05 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.000692+0000", 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8260000000000001e-05 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:184776 7.c 7:34e6605b:::benchmark_data_smithi045_83007_object66305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:21.001743+0000", 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.001249312, 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015897699999999999, 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:21.019 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 184776 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.001743+0000", 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.001743+0000", 2024-03-11T22:13:21.020 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.001745+0000", 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4130000000000001e-06 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.001750+0000", 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5539999999999997e-06 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:21.021 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.001762+0000", 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2109e-05 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.001771+0000", 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1839999999999999e-06 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.001926+0000", 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015555300000000001 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:21.022 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.002057+0000", 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000130462 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.002174+0000", 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000117712 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:184780 7.d 7:b7691f67:::benchmark_data_smithi045_83007_object66309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:21.003175+0000", 2024-03-11T22:13:21.023 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4294967295.9998174, 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000233308, 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "reached pg", 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 184780 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.003175+0000", 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.024 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.003175+0000", 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.003176+0000", 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3900000000000004e-07 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.003181+0000", 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4839999999999998e-06 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.025 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.003195+0000", 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4027e-05 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.003202+0000", 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6860000000000006e-06 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:21.003325+0000", 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000122994 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:21.026 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:21.027 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:21.027 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:21.027 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-03-11T22:13:21.027 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 6 2024-03-11T22:13:21.027 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:13:21.027 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:21.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:21.061 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:21.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:21.079 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-03-11T22:13:18.130750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3767277940,"num_objects":57674,"num_object_clones":0,"num_object_copies":173022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57674,"num_whiteouts":0,"num_read":1174,"num_read_kb":1020,"num_write":180077,"num_write_kb":7584414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92362,"ondisk_log_size":92362,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11990296,"kb_used_data":11164564,"kb_used_omap":12,"kb_used_meta":825459,"kb_avail":737970920,"statfs":{"total":767960285184,"available":755682222080,"internally_reserved":0,"allocated":11432513536,"data_stored":11429889405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":845270579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-664862720,"num_objects":-10145,"num_object_clones":0,"num_object_copies":-30435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10145,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10145,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.696293"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":154,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004130+0000","last_change":"2024-03-11T22:11:30.350663+0000","last_active":"2024-03-11T22:13:15.004130+0000","last_peered":"2024-03-11T22:13:15.004130+0000","last_clean":"2024-03-11T22:13:15.004130+0000","last_became_active":"2024-03-11T22:08:39.074695+0000","last_became_peered":"2024-03-11T22:08:39.074695+0000","last_unstale":"2024-03-11T22:13:15.004130+0000","last_undegraded":"2024-03-11T22:13:15.004130+0000","last_fullsized":"2024-03-11T22:13:15.004130+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:30.350540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.350540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.070260+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.000616+0000","last_change":"2024-03-11T22:12:45.929428+0000","last_active":"2024-03-11T22:13:15.000616+0000","last_peered":"2024-03-11T22:13:15.000616+0000","last_clean":"2024-03-11T22:13:15.000616+0000","last_became_active":"2024-03-11T22:08:37.073775+0000","last_became_peered":"2024-03-11T22:08:37.073775+0000","last_unstale":"2024-03-11T22:13:15.000616+0000","last_undegraded":"2024-03-11T22:13:15.000616+0000","last_fullsized":"2024-03-11T22:13:15.000616+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.928689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.928689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:15.589951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.006437+0000","last_change":"2024-03-11T22:11:45.340779+0000","last_active":"2024-03-11T22:13:15.006437+0000","last_peered":"2024-03-11T22:13:15.006437+0000","last_clean":"2024-03-11T22:13:15.006437+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:13:15.006437+0000","last_undegraded":"2024-03-11T22:13:15.006437+0000","last_fullsized":"2024-03-11T22:13:15.006437+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:45.340657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:45.340657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":178,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.003955+0000","last_change":"2024-03-11T22:10:13.507406+0000","last_active":"2024-03-11T22:13:15.003955+0000","last_peered":"2024-03-11T22:13:15.003955+0000","last_clean":"2024-03-11T22:13:15.003955+0000","last_became_active":"2024-03-11T22:08:35.053179+0000","last_became_peered":"2024-03-11T22:08:35.053179+0000","last_unstale":"2024-03-11T22:13:15.003955+0000","last_undegraded":"2024-03-11T22:13:15.003955+0000","last_fullsized":"2024-03-11T22:13:15.003955+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:10:13.507312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:13.507312+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.069896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":162,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005454+0000","last_change":"2024-03-11T22:12:50.533192+0000","last_active":"2024-03-11T22:13:15.005454+0000","last_peered":"2024-03-11T22:13:15.005454+0000","last_clean":"2024-03-11T22:13:15.005454+0000","last_became_active":"2024-03-11T22:08:39.079045+0000","last_became_peered":"2024-03-11T22:08:39.079045+0000","last_unstale":"2024-03-11T22:13:15.005454+0000","last_undegraded":"2024-03-11T22:13:15.005454+0000","last_fullsized":"2024-03-11T22:13:15.005454+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.533067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.533067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:07.226337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004124+0000","last_change":"2024-03-11T22:11:43.150997+0000","last_active":"2024-03-11T22:13:15.004124+0000","last_peered":"2024-03-11T22:13:15.004124+0000","last_clean":"2024-03-11T22:13:15.004124+0000","last_became_active":"2024-03-11T22:08:37.081861+0000","last_became_peered":"2024-03-11T22:08:37.081861+0000","last_unstale":"2024-03-11T22:13:15.004124+0000","last_undegraded":"2024-03-11T22:13:15.004124+0000","last_fullsized":"2024-03-11T22:13:15.004124+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:43.150872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:43.150872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:18.090857+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":181,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005638+0000","last_change":"2024-03-11T22:11:22.946804+0000","last_active":"2024-03-11T22:13:15.005638+0000","last_peered":"2024-03-11T22:13:15.005638+0000","last_clean":"2024-03-11T22:13:15.005638+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:13:15.005638+0000","last_undegraded":"2024-03-11T22:13:15.005638+0000","last_fullsized":"2024-03-11T22:13:15.005638+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:11:22.946664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:22.946664+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":195,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.017015+0000","last_change":"2024-03-11T22:12:49.510144+0000","last_active":"2024-03-11T22:13:15.017015+0000","last_peered":"2024-03-11T22:13:15.017015+0000","last_clean":"2024-03-11T22:13:15.017015+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:13:15.017015+0000","last_undegraded":"2024-03-11T22:13:15.017015+0000","last_fullsized":"2024-03-11T22:13:15.017015+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:12:49.509860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.509860+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:15.173673+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":154,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.999726+0000","last_change":"2024-03-11T22:10:15.238661+0000","last_active":"2024-03-11T22:13:14.999726+0000","last_peered":"2024-03-11T22:13:14.999726+0000","last_clean":"2024-03-11T22:13:14.999726+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:13:14.999726+0000","last_undegraded":"2024-03-11T22:13:14.999726+0000","last_fullsized":"2024-03-11T22:13:14.999726+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:15.238567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:15.238567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.994174+0000","last_change":"2024-03-11T22:12:19.337740+0000","last_active":"2024-03-11T22:13:14.994174+0000","last_peered":"2024-03-11T22:13:14.994174+0000","last_clean":"2024-03-11T22:13:14.994174+0000","last_became_active":"2024-03-11T22:08:37.081996+0000","last_became_peered":"2024-03-11T22:08:37.081996+0000","last_unstale":"2024-03-11T22:13:14.994174+0000","last_undegraded":"2024-03-11T22:13:14.994174+0000","last_fullsized":"2024-03-11T22:13:14.994174+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:19.337662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:12:19.337662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:35.201935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":182,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.994230+0000","last_change":"2024-03-11T22:12:47.432839+0000","last_active":"2024-03-11T22:13:14.994230+0000","last_peered":"2024-03-11T22:13:14.994230+0000","last_clean":"2024-03-11T22:13:14.994230+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:13:14.994230+0000","last_undegraded":"2024-03-11T22:13:14.994230+0000","last_fullsized":"2024-03-11T22:13:14.994230+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:47.432769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:47.432769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:59.023738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":186,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004805+0000","last_change":"2024-03-11T22:11:45.169864+0000","last_active":"2024-03-11T22:13:15.004805+0000","last_peered":"2024-03-11T22:13:15.004805+0000","last_clean":"2024-03-11T22:13:15.004805+0000","last_became_active":"2024-03-11T22:08:35.060802+0000","last_became_peered":"2024-03-11T22:08:35.060802+0000","last_unstale":"2024-03-11T22:13:15.004805+0000","last_undegraded":"2024-03-11T22:13:15.004805+0000","last_fullsized":"2024-03-11T22:13:15.004805+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:11:45.169706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:45.169706+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.070290+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":162,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.993330+0000","last_change":"2024-03-11T22:11:47.166498+0000","last_active":"2024-03-11T22:13:14.993330+0000","last_peered":"2024-03-11T22:13:14.993330+0000","last_clean":"2024-03-11T22:13:14.993330+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:13:14.993330+0000","last_undegraded":"2024-03-11T22:13:14.993330+0000","last_fullsized":"2024-03-11T22:13:14.993330+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:47.166360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:47.166360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:19.499732+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":97,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005045+0000","last_change":"2024-03-11T22:10:35.411012+0000","last_active":"2024-03-11T22:13:15.005045+0000","last_peered":"2024-03-11T22:13:15.005045+0000","last_clean":"2024-03-11T22:13:15.005045+0000","last_became_active":"2024-03-11T22:10:35.410624+0000","last_became_peered":"2024-03-11T22:10:35.410624+0000","last_unstale":"2024-03-11T22:13:15.005045+0000","last_undegraded":"2024-03-11T22:13:15.005045+0000","last_fullsized":"2024-03-11T22:13:15.005045+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:00.270166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.270166+0000","last_clean_scrub_stamp":"2024-03-11T22:10:00.270166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.069815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.994834+0000","last_change":"2024-03-11T22:12:24.388572+0000","last_active":"2024-03-11T22:13:14.994834+0000","last_peered":"2024-03-11T22:13:14.994834+0000","last_clean":"2024-03-11T22:13:14.994834+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:13:14.994834+0000","last_undegraded":"2024-03-11T22:13:14.994834+0000","last_fullsized":"2024-03-11T22:13:14.994834+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:24.388497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.388497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:43.170333+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"95'7","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004404+0000","last_change":"2024-03-11T22:10:18.579990+0000","last_active":"2024-03-11T22:13:15.004404+0000","last_peered":"2024-03-11T22:13:15.004404+0000","last_clean":"2024-03-11T22:13:15.004404+0000","last_became_active":"2024-03-11T22:08:35.060777+0000","last_became_peered":"2024-03-11T22:08:35.060777+0000","last_unstale":"2024-03-11T22:13:15.004404+0000","last_undegraded":"2024-03-11T22:13:15.004404+0000","last_fullsized":"2024-03-11T22:13:15.004404+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'7","last_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:10:18.579896+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.069757+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":163,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004638+0000","last_change":"2024-03-11T22:12:32.044430+0000","last_active":"2024-03-11T22:13:15.004638+0000","last_peered":"2024-03-11T22:13:15.004638+0000","last_clean":"2024-03-11T22:13:15.004638+0000","last_became_active":"2024-03-11T22:08:39.075609+0000","last_became_peered":"2024-03-11T22:08:39.075609+0000","last_unstale":"2024-03-11T22:13:15.004638+0000","last_undegraded":"2024-03-11T22:13:15.004638+0000","last_fullsized":"2024-03-11T22:13:15.004638+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:32.044320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:32.044320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:34.643459+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.001129+0000","last_change":"2024-03-11T22:12:30.962293+0000","last_active":"2024-03-11T22:13:15.001129+0000","last_peered":"2024-03-11T22:13:15.001129+0000","last_clean":"2024-03-11T22:13:15.001129+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:13:15.001129+0000","last_undegraded":"2024-03-11T22:13:15.001129+0000","last_fullsized":"2024-03-11T22:13:15.001129+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:30.962171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:30.962171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.129955+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.000722+0000","last_change":"2024-03-11T22:12:22.649189+0000","last_active":"2024-03-11T22:13:15.000722+0000","last_peered":"2024-03-11T22:13:15.000722+0000","last_clean":"2024-03-11T22:13:15.000722+0000","last_became_active":"2024-03-11T22:08:33.035624+0000","last_became_peered":"2024-03-11T22:08:33.035624+0000","last_unstale":"2024-03-11T22:13:15.000722+0000","last_undegraded":"2024-03-11T22:13:15.000722+0000","last_fullsized":"2024-03-11T22:13:15.000722+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:22.649073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:22.649073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:32.856439+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":203,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.001689+0000","last_change":"2024-03-11T22:12:27.670423+0000","last_active":"2024-03-11T22:13:15.001689+0000","last_peered":"2024-03-11T22:13:15.001689+0000","last_clean":"2024-03-11T22:13:15.001689+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:13:15.001689+0000","last_undegraded":"2024-03-11T22:13:15.001689+0000","last_fullsized":"2024-03-11T22:13:15.001689+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:12:27.670284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:27.670284+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:39.408475+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":163,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.011185+0000","last_change":"2024-03-11T22:12:44.464428+0000","last_active":"2024-03-11T22:13:15.011185+0000","last_peered":"2024-03-11T22:13:15.011185+0000","last_clean":"2024-03-11T22:13:15.011185+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:13:15.011185+0000","last_undegraded":"2024-03-11T22:13:15.011185+0000","last_fullsized":"2024-03-11T22:13:15.011185+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.464308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.464308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:02.369972+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":158,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004408+0000","last_change":"2024-03-11T22:11:27.391809+0000","last_active":"2024-03-11T22:13:15.004408+0000","last_peered":"2024-03-11T22:13:15.004408+0000","last_clean":"2024-03-11T22:13:15.004408+0000","last_became_active":"2024-03-11T22:08:37.080014+0000","last_became_peered":"2024-03-11T22:08:37.080014+0000","last_unstale":"2024-03-11T22:13:15.004408+0000","last_undegraded":"2024-03-11T22:13:15.004408+0000","last_fullsized":"2024-03-11T22:13:15.004408+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:27.391727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:27.391727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.070079+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004896+0000","last_change":"2024-03-11T22:10:03.469439+0000","last_active":"2024-03-11T22:13:15.004896+0000","last_peered":"2024-03-11T22:13:15.004896+0000","last_clean":"2024-03-11T22:13:15.004896+0000","last_became_active":"2024-03-11T22:08:33.299189+0000","last_became_peered":"2024-03-11T22:08:33.299189+0000","last_unstale":"2024-03-11T22:13:15.004896+0000","last_undegraded":"2024-03-11T22:13:15.004896+0000","last_fullsized":"2024-03-11T22:13:15.004896+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:03.469346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:03.469346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.069625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":193,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.009859+0000","last_change":"2024-03-11T22:12:28.292078+0000","last_active":"2024-03-11T22:13:15.009859+0000","last_peered":"2024-03-11T22:13:15.009859+0000","last_clean":"2024-03-11T22:13:15.009859+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:13:15.009859+0000","last_undegraded":"2024-03-11T22:13:15.009859+0000","last_fullsized":"2024-03-11T22:13:15.009859+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:12:28.291919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:28.291919+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:33.113224+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":116,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.993831+0000","last_change":"2024-03-11T22:11:22.021723+0000","last_active":"2024-03-11T22:13:14.993831+0000","last_peered":"2024-03-11T22:13:14.993831+0000","last_clean":"2024-03-11T22:13:14.993831+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:13:14.993831+0000","last_undegraded":"2024-03-11T22:13:14.993831+0000","last_fullsized":"2024-03-11T22:13:14.993831+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:11:22.021581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:22.021581+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.389049+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":243,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.014629+0000","last_change":"2024-03-11T22:12:35.862998+0000","last_active":"2024-03-11T22:13:15.014629+0000","last_peered":"2024-03-11T22:13:15.014629+0000","last_clean":"2024-03-11T22:13:15.014629+0000","last_became_active":"2024-03-11T22:08:37.078588+0000","last_became_peered":"2024-03-11T22:08:37.078588+0000","last_unstale":"2024-03-11T22:13:15.014629+0000","last_undegraded":"2024-03-11T22:13:15.014629+0000","last_fullsized":"2024-03-11T22:13:15.014629+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:12:35.862855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.862855+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:00.615190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.000712+0000","last_change":"2024-03-11T22:12:24.919926+0000","last_active":"2024-03-11T22:13:15.000712+0000","last_peered":"2024-03-11T22:13:15.000712+0000","last_clean":"2024-03-11T22:13:15.000712+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:13:15.000712+0000","last_undegraded":"2024-03-11T22:13:15.000712+0000","last_fullsized":"2024-03-11T22:13:15.000712+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:24.919682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.919682+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:49.474264+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":178,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004959+0000","last_change":"2024-03-11T22:10:32.196590+0000","last_active":"2024-03-11T22:13:15.004959+0000","last_peered":"2024-03-11T22:13:15.004959+0000","last_clean":"2024-03-11T22:13:15.004959+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:13:15.004959+0000","last_undegraded":"2024-03-11T22:13:15.004959+0000","last_fullsized":"2024-03-11T22:13:15.004959+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:10:32.196500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:32.196500+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":162,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.001325+0000","last_change":"2024-03-11T22:11:30.890018+0000","last_active":"2024-03-11T22:13:15.001325+0000","last_peered":"2024-03-11T22:13:15.001325+0000","last_clean":"2024-03-11T22:13:15.001325+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:13:15.001325+0000","last_undegraded":"2024-03-11T22:13:15.001325+0000","last_fullsized":"2024-03-11T22:13:15.001325+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:30.889504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.889504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":167,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005329+0000","last_change":"2024-03-11T22:11:37.288072+0000","last_active":"2024-03-11T22:13:15.005329+0000","last_peered":"2024-03-11T22:13:15.005329+0000","last_clean":"2024-03-11T22:13:15.005329+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:13:15.005329+0000","last_undegraded":"2024-03-11T22:13:15.005329+0000","last_fullsized":"2024-03-11T22:13:15.005329+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:37.287951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:37.287951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.011109+0000","last_change":"2024-03-11T22:12:43.451182+0000","last_active":"2024-03-11T22:13:15.011109+0000","last_peered":"2024-03-11T22:13:15.011109+0000","last_clean":"2024-03-11T22:13:15.011109+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:13:15.011109+0000","last_undegraded":"2024-03-11T22:13:15.011109+0000","last_fullsized":"2024-03-11T22:13:15.011109+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:43.451074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:43.451074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:06.309610+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":190,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.014564+0000","last_change":"2024-03-11T22:11:11.339653+0000","last_active":"2024-03-11T22:13:15.014564+0000","last_peered":"2024-03-11T22:13:15.014564+0000","last_clean":"2024-03-11T22:13:15.014564+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:13:15.014564+0000","last_undegraded":"2024-03-11T22:13:15.014564+0000","last_fullsized":"2024-03-11T22:13:15.014564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:11:11.339500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:11.339500+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:16.899267+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":105,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.010779+0000","last_change":"2024-03-11T22:11:25.409252+0000","last_active":"2024-03-11T22:13:15.010779+0000","last_peered":"2024-03-11T22:13:15.010779+0000","last_clean":"2024-03-11T22:13:15.010779+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:13:15.010779+0000","last_undegraded":"2024-03-11T22:13:15.010779+0000","last_fullsized":"2024-03-11T22:13:15.010779+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:25.407823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.407823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:16.753047+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.009671+0000","last_change":"2024-03-11T22:12:44.941384+0000","last_active":"2024-03-11T22:13:15.009671+0000","last_peered":"2024-03-11T22:13:15.009671+0000","last_clean":"2024-03-11T22:13:15.009671+0000","last_became_active":"2024-03-11T22:08:37.075341+0000","last_became_peered":"2024-03-11T22:08:37.075341+0000","last_unstale":"2024-03-11T22:13:15.009671+0000","last_undegraded":"2024-03-11T22:13:15.009671+0000","last_fullsized":"2024-03-11T22:13:15.009671+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.941275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.941275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:04.779620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.006592+0000","last_change":"2024-03-11T22:10:18.176194+0000","last_active":"2024-03-11T22:13:15.006592+0000","last_peered":"2024-03-11T22:13:15.006592+0000","last_clean":"2024-03-11T22:13:15.006592+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:13:15.006592+0000","last_undegraded":"2024-03-11T22:13:15.006592+0000","last_fullsized":"2024-03-11T22:13:15.006592+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:18.176081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:18.176081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":189,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.011246+0000","last_change":"2024-03-11T22:11:25.927507+0000","last_active":"2024-03-11T22:13:15.011246+0000","last_peered":"2024-03-11T22:13:15.011246+0000","last_clean":"2024-03-11T22:13:15.011246+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:13:15.011246+0000","last_undegraded":"2024-03-11T22:13:15.011246+0000","last_fullsized":"2024-03-11T22:13:15.011246+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-11T22:11:25.926811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.926811+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":162,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005766+0000","last_change":"2024-03-11T22:11:46.050869+0000","last_active":"2024-03-11T22:13:15.005766+0000","last_peered":"2024-03-11T22:13:15.005766+0000","last_clean":"2024-03-11T22:13:15.005766+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:13:15.005766+0000","last_undegraded":"2024-03-11T22:13:15.005766+0000","last_fullsized":"2024-03-11T22:13:15.005766+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:46.050747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:46.050747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":105,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.003848+0000","last_change":"2024-03-11T22:11:42.175942+0000","last_active":"2024-03-11T22:13:15.003848+0000","last_peered":"2024-03-11T22:13:15.003848+0000","last_clean":"2024-03-11T22:13:15.003848+0000","last_became_active":"2024-03-11T22:10:35.415813+0000","last_became_peered":"2024-03-11T22:10:35.415813+0000","last_unstale":"2024-03-11T22:13:15.003848+0000","last_undegraded":"2024-03-11T22:13:15.003848+0000","last_fullsized":"2024-03-11T22:13:15.003848+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:11:42.175852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:18.090890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":174,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.994707+0000","last_change":"2024-03-11T22:11:32.118166+0000","last_active":"2024-03-11T22:13:14.994707+0000","last_peered":"2024-03-11T22:13:14.994707+0000","last_clean":"2024-03-11T22:13:14.994707+0000","last_became_active":"2024-03-11T22:08:33.039349+0000","last_became_peered":"2024-03-11T22:08:33.039349+0000","last_unstale":"2024-03-11T22:13:14.994707+0000","last_undegraded":"2024-03-11T22:13:14.994707+0000","last_fullsized":"2024-03-11T22:13:14.994707+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:32.118050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:32.118050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.389105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":181,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.016927+0000","last_change":"2024-03-11T22:11:21.929969+0000","last_active":"2024-03-11T22:13:15.016927+0000","last_peered":"2024-03-11T22:13:15.016927+0000","last_clean":"2024-03-11T22:13:15.016927+0000","last_became_active":"2024-03-11T22:08:35.085928+0000","last_became_peered":"2024-03-11T22:08:35.085928+0000","last_unstale":"2024-03-11T22:13:15.016927+0000","last_undegraded":"2024-03-11T22:13:15.016927+0000","last_fullsized":"2024-03-11T22:13:15.016927+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:11:21.929789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:21.929789+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":162,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.999356+0000","last_change":"2024-03-11T22:12:46.829082+0000","last_active":"2024-03-11T22:13:14.999356+0000","last_peered":"2024-03-11T22:13:14.999356+0000","last_clean":"2024-03-11T22:13:14.999356+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:13:14.999356+0000","last_undegraded":"2024-03-11T22:13:14.999356+0000","last_fullsized":"2024-03-11T22:13:14.999356+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.828942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.828942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:59.694733+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005358+0000","last_change":"2024-03-11T22:11:37.076127+0000","last_active":"2024-03-11T22:13:15.005358+0000","last_peered":"2024-03-11T22:13:15.005358+0000","last_clean":"2024-03-11T22:13:15.005358+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:13:15.005358+0000","last_undegraded":"2024-03-11T22:13:15.005358+0000","last_fullsized":"2024-03-11T22:13:15.005358+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:11:37.075868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":106,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.010151+0000","last_change":"2024-03-11T22:11:38.050014+0000","last_active":"2024-03-11T22:13:15.010151+0000","last_peered":"2024-03-11T22:13:15.010151+0000","last_clean":"2024-03-11T22:13:15.010151+0000","last_became_active":"2024-03-11T22:10:35.414121+0000","last_became_peered":"2024-03-11T22:10:35.414121+0000","last_unstale":"2024-03-11T22:13:15.010151+0000","last_undegraded":"2024-03-11T22:13:15.010151+0000","last_fullsized":"2024-03-11T22:13:15.010151+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:38.049889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:38.049889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":187,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.999964+0000","last_change":"2024-03-11T22:11:26.901776+0000","last_active":"2024-03-11T22:13:14.999964+0000","last_peered":"2024-03-11T22:13:14.999964+0000","last_clean":"2024-03-11T22:13:14.999964+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:13:14.999964+0000","last_undegraded":"2024-03-11T22:13:14.999964+0000","last_fullsized":"2024-03-11T22:13:14.999964+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:11:26.901528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:26.901528+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":162,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.016609+0000","last_change":"2024-03-11T22:12:46.492608+0000","last_active":"2024-03-11T22:13:15.016609+0000","last_peered":"2024-03-11T22:13:15.016609+0000","last_clean":"2024-03-11T22:13:15.016609+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:13:15.016609+0000","last_undegraded":"2024-03-11T22:13:15.016609+0000","last_fullsized":"2024-03-11T22:13:15.016609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.492494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.492494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:11.071230+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.999671+0000","last_change":"2024-03-11T22:12:41.988328+0000","last_active":"2024-03-11T22:13:14.999671+0000","last_peered":"2024-03-11T22:13:14.999671+0000","last_clean":"2024-03-11T22:13:14.999671+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:13:14.999671+0000","last_undegraded":"2024-03-11T22:13:14.999671+0000","last_fullsized":"2024-03-11T22:13:14.999671+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:41.988094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:41.988094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:07.625482+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":181,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004346+0000","last_change":"2024-03-11T22:12:18.154605+0000","last_active":"2024-03-11T22:13:15.004346+0000","last_peered":"2024-03-11T22:13:15.004346+0000","last_clean":"2024-03-11T22:13:15.004346+0000","last_became_active":"2024-03-11T22:08:33.036083+0000","last_became_peered":"2024-03-11T22:08:33.036083+0000","last_unstale":"2024-03-11T22:13:15.004346+0000","last_undegraded":"2024-03-11T22:13:15.004346+0000","last_fullsized":"2024-03-11T22:13:15.004346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:18.154515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:18.154515+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:34.107328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":181,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004593+0000","last_change":"2024-03-11T22:11:19.294807+0000","last_active":"2024-03-11T22:13:15.004593+0000","last_peered":"2024-03-11T22:13:15.004593+0000","last_clean":"2024-03-11T22:13:15.004593+0000","last_became_active":"2024-03-11T22:08:35.054726+0000","last_became_peered":"2024-03-11T22:08:35.054726+0000","last_unstale":"2024-03-11T22:13:15.004593+0000","last_undegraded":"2024-03-11T22:13:15.004593+0000","last_fullsized":"2024-03-11T22:13:15.004593+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:11:19.294671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:19.294671+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.070045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":154,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004492+0000","last_change":"2024-03-11T22:11:13.259520+0000","last_active":"2024-03-11T22:13:15.004492+0000","last_peered":"2024-03-11T22:13:15.004492+0000","last_clean":"2024-03-11T22:13:15.004492+0000","last_became_active":"2024-03-11T22:08:39.082938+0000","last_became_peered":"2024-03-11T22:08:39.082938+0000","last_unstale":"2024-03-11T22:13:15.004492+0000","last_undegraded":"2024-03-11T22:13:15.004492+0000","last_fullsized":"2024-03-11T22:13:15.004492+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:13.259407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:13.259407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":231,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.994295+0000","last_change":"2024-03-11T22:11:30.122434+0000","last_active":"2024-03-11T22:13:14.994295+0000","last_peered":"2024-03-11T22:13:14.994295+0000","last_clean":"2024-03-11T22:13:14.994295+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:13:14.994295+0000","last_undegraded":"2024-03-11T22:13:14.994295+0000","last_fullsized":"2024-03-11T22:13:14.994295+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:11:30.122330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.122330+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.389087+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":106,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004502+0000","last_change":"2024-03-11T22:11:39.174355+0000","last_active":"2024-03-11T22:13:15.004502+0000","last_peered":"2024-03-11T22:13:15.004502+0000","last_clean":"2024-03-11T22:13:15.004502+0000","last_became_active":"2024-03-11T22:10:35.407845+0000","last_became_peered":"2024-03-11T22:10:35.407845+0000","last_unstale":"2024-03-11T22:13:15.004502+0000","last_undegraded":"2024-03-11T22:13:15.004502+0000","last_fullsized":"2024-03-11T22:13:15.004502+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:39.174103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:39.174103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.070144+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":185,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.010548+0000","last_change":"2024-03-11T22:12:20.295562+0000","last_active":"2024-03-11T22:13:15.010548+0000","last_peered":"2024-03-11T22:13:15.010548+0000","last_clean":"2024-03-11T22:13:15.010548+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:13:15.010548+0000","last_undegraded":"2024-03-11T22:13:15.010548+0000","last_fullsized":"2024-03-11T22:13:15.010548+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:12:20.295427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:20.295427+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:42.096971+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":154,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005415+0000","last_change":"2024-03-11T22:11:31.308182+0000","last_active":"2024-03-11T22:13:15.005415+0000","last_peered":"2024-03-11T22:13:15.005415+0000","last_clean":"2024-03-11T22:13:15.005415+0000","last_became_active":"2024-03-11T22:08:39.072089+0000","last_became_peered":"2024-03-11T22:08:39.072089+0000","last_unstale":"2024-03-11T22:13:15.005415+0000","last_undegraded":"2024-03-11T22:13:15.005415+0000","last_fullsized":"2024-03-11T22:13:15.005415+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:31.308103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:31.308103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.070198+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":234,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.010638+0000","last_change":"2024-03-11T22:11:44.093057+0000","last_active":"2024-03-11T22:13:15.010638+0000","last_peered":"2024-03-11T22:13:15.010638+0000","last_clean":"2024-03-11T22:13:15.010638+0000","last_became_active":"2024-03-11T22:08:37.071202+0000","last_became_peered":"2024-03-11T22:08:37.071202+0000","last_unstale":"2024-03-11T22:13:15.010638+0000","last_undegraded":"2024-03-11T22:13:15.010638+0000","last_fullsized":"2024-03-11T22:13:15.010638+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:11:44.092784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:44.092784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.000228+0000","last_change":"2024-03-11T22:12:29.629914+0000","last_active":"2024-03-11T22:13:15.000228+0000","last_peered":"2024-03-11T22:13:15.000228+0000","last_clean":"2024-03-11T22:13:15.000228+0000","last_became_active":"2024-03-11T22:08:33.032850+0000","last_became_peered":"2024-03-11T22:08:33.032850+0000","last_unstale":"2024-03-11T22:13:15.000228+0000","last_undegraded":"2024-03-11T22:13:15.000228+0000","last_fullsized":"2024-03-11T22:13:15.000228+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:29.629778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:29.629778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:44.363248+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":186,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.994406+0000","last_change":"2024-03-11T22:11:38.077749+0000","last_active":"2024-03-11T22:13:14.994406+0000","last_peered":"2024-03-11T22:13:14.994406+0000","last_clean":"2024-03-11T22:13:14.994406+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:13:14.994406+0000","last_undegraded":"2024-03-11T22:13:14.994406+0000","last_fullsized":"2024-03-11T22:13:14.994406+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:11:38.077654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:38.077654+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:16.411994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":170,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.001944+0000","last_change":"2024-03-11T22:12:40.777949+0000","last_active":"2024-03-11T22:13:15.001944+0000","last_peered":"2024-03-11T22:13:15.001944+0000","last_clean":"2024-03-11T22:13:15.001944+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:13:15.001944+0000","last_undegraded":"2024-03-11T22:13:15.001944+0000","last_fullsized":"2024-03-11T22:13:15.001944+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:40.777821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:40.777821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:46.765451+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.011602+0000","last_change":"2024-03-11T22:11:31.040056+0000","last_active":"2024-03-11T22:13:15.011602+0000","last_peered":"2024-03-11T22:13:15.011602+0000","last_clean":"2024-03-11T22:13:15.011602+0000","last_became_active":"2024-03-11T22:08:37.069611+0000","last_became_peered":"2024-03-11T22:08:37.069611+0000","last_unstale":"2024-03-11T22:13:15.011602+0000","last_undegraded":"2024-03-11T22:13:15.011602+0000","last_fullsized":"2024-03-11T22:13:15.011602+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:31.039934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:31.039934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":98,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.009058+0000","last_change":"2024-03-11T22:10:35.416311+0000","last_active":"2024-03-11T22:13:15.009058+0000","last_peered":"2024-03-11T22:13:15.009058+0000","last_clean":"2024-03-11T22:13:15.009058+0000","last_became_active":"2024-03-11T22:10:35.415644+0000","last_became_peered":"2024-03-11T22:10:35.415644+0000","last_unstale":"2024-03-11T22:13:15.009058+0000","last_undegraded":"2024-03-11T22:13:15.009058+0000","last_fullsized":"2024-03-11T22:13:15.009058+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:12.415483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:12.415483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.069867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"93'13","reported_seq":190,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005167+0000","last_change":"2024-03-11T22:11:35.407557+0000","last_active":"2024-03-11T22:13:15.005167+0000","last_peered":"2024-03-11T22:13:15.005167+0000","last_clean":"2024-03-11T22:13:15.005167+0000","last_became_active":"2024-03-11T22:08:35.085787+0000","last_became_peered":"2024-03-11T22:08:35.085787+0000","last_unstale":"2024-03-11T22:13:15.005167+0000","last_undegraded":"2024-03-11T22:13:15.005167+0000","last_fullsized":"2024-03-11T22:13:15.005167+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:11:35.407301+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.407301+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":105,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.009980+0000","last_change":"2024-03-11T22:11:36.490489+0000","last_active":"2024-03-11T22:13:15.009980+0000","last_peered":"2024-03-11T22:13:15.009980+0000","last_clean":"2024-03-11T22:13:15.009980+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:13:15.009980+0000","last_undegraded":"2024-03-11T22:13:15.009980+0000","last_fullsized":"2024-03-11T22:13:15.009980+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:36.490349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:15.415292+0000","last_clean_scrub_stamp":"2024-03-11T22:11:36.490349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:16.899362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":166,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.994743+0000","last_change":"2024-03-11T22:11:41.090567+0000","last_active":"2024-03-11T22:13:14.994743+0000","last_peered":"2024-03-11T22:13:14.994743+0000","last_clean":"2024-03-11T22:13:14.994743+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:13:14.994743+0000","last_undegraded":"2024-03-11T22:13:14.994743+0000","last_fullsized":"2024-03-11T22:13:14.994743+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:41.090454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:07.195289+0000","last_clean_scrub_stamp":"2024-03-11T22:11:41.090454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.389067+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.008787+0000","last_change":"2024-03-11T22:11:31.436443+0000","last_active":"2024-03-11T22:13:15.008787+0000","last_peered":"2024-03-11T22:13:15.008787+0000","last_clean":"2024-03-11T22:13:15.008787+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:13:15.008787+0000","last_undegraded":"2024-03-11T22:13:15.008787+0000","last_fullsized":"2024-03-11T22:13:15.008787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:31.436306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:11:31.436306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:16.899329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":179,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.011687+0000","last_change":"2024-03-11T22:12:21.278151+0000","last_active":"2024-03-11T22:13:15.011687+0000","last_peered":"2024-03-11T22:13:15.011687+0000","last_clean":"2024-03-11T22:13:15.011687+0000","last_became_active":"2024-03-11T22:08:35.085914+0000","last_became_peered":"2024-03-11T22:08:35.085914+0000","last_unstale":"2024-03-11T22:13:15.011687+0000","last_undegraded":"2024-03-11T22:13:15.011687+0000","last_fullsized":"2024-03-11T22:13:15.011687+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:12:21.270671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:21.270671+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:41.395498+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"141'13544","reported_seq":13861,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.295352+0000","last_change":"2024-03-11T22:12:25.136636+0000","last_active":"2024-03-11T22:13:15.295352+0000","last_peered":"2024-03-11T22:13:15.295352+0000","last_clean":"2024-03-11T22:13:15.295352+0000","last_became_active":"2024-03-11T22:10:36.410723+0000","last_became_peered":"2024-03-11T22:10:36.410723+0000","last_unstale":"2024-03-11T22:13:15.295352+0000","last_undegraded":"2024-03-11T22:13:15.295352+0000","last_fullsized":"2024-03-11T22:13:15.295352+0000","mapping_epoch":99,"log_start":"124'8500","ondisk_log_start":"124'8500","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'10496","last_scrub_stamp":"2024-03-11T22:12:25.136305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:25.136305+0000","objects_scrubbed":6681,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:31.345968+0000","scrub_duration":2244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233570304,"num_objects":3564,"num_object_clones":0,"num_object_copies":10692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11064,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":163,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.993772+0000","last_change":"2024-03-11T22:12:38.434933+0000","last_active":"2024-03-11T22:13:14.993772+0000","last_peered":"2024-03-11T22:13:14.993772+0000","last_clean":"2024-03-11T22:13:14.993772+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:13:14.993772+0000","last_undegraded":"2024-03-11T22:13:14.993772+0000","last_fullsized":"2024-03-11T22:13:14.993772+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:38.434796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.434796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:04.782006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":107,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005264+0000","last_change":"2024-03-11T22:10:35.412939+0000","last_active":"2024-03-11T22:13:15.005264+0000","last_peered":"2024-03-11T22:13:15.005264+0000","last_clean":"2024-03-11T22:13:15.005264+0000","last_became_active":"2024-03-11T22:10:35.412565+0000","last_became_peered":"2024-03-11T22:10:35.412565+0000","last_unstale":"2024-03-11T22:13:15.005264+0000","last_undegraded":"2024-03-11T22:13:15.005264+0000","last_fullsized":"2024-03-11T22:13:15.005264+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:10:05.284508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:05.284508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.069663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"91'22","reported_seq":203,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.993291+0000","last_change":"2024-03-11T22:11:12.222700+0000","last_active":"2024-03-11T22:13:14.993291+0000","last_peered":"2024-03-11T22:13:14.993291+0000","last_clean":"2024-03-11T22:13:14.993291+0000","last_became_active":"2024-03-11T22:08:35.085916+0000","last_became_peered":"2024-03-11T22:08:35.085916+0000","last_unstale":"2024-03-11T22:13:14.993291+0000","last_undegraded":"2024-03-11T22:13:14.993291+0000","last_fullsized":"2024-03-11T22:13:14.993291+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'22","last_scrub_stamp":"2024-03-11T22:11:12.222573+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:11:12.222573+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.389010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"99'9","reported_seq":163,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.009501+0000","last_change":"2024-03-11T22:11:25.476405+0000","last_active":"2024-03-11T22:13:15.009501+0000","last_peered":"2024-03-11T22:13:15.009501+0000","last_clean":"2024-03-11T22:13:15.009501+0000","last_became_active":"2024-03-11T22:10:35.413089+0000","last_became_peered":"2024-03-11T22:10:35.413089+0000","last_unstale":"2024-03-11T22:13:15.009501+0000","last_undegraded":"2024-03-11T22:13:15.009501+0000","last_fullsized":"2024-03-11T22:13:15.009501+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'9","last_scrub_stamp":"2024-03-11T22:11:25.476283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.476283+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:16.899287+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"141'11201","reported_seq":11366,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.418288+0000","last_change":"2024-03-11T22:12:38.052754+0000","last_active":"2024-03-11T22:13:15.418288+0000","last_peered":"2024-03-11T22:13:15.418288+0000","last_clean":"2024-03-11T22:13:15.418288+0000","last_became_active":"2024-03-11T22:10:36.411631+0000","last_became_peered":"2024-03-11T22:10:36.411631+0000","last_unstale":"2024-03-11T22:13:15.418288+0000","last_undegraded":"2024-03-11T22:13:15.418288+0000","last_fullsized":"2024-03-11T22:13:15.418288+0000","mapping_epoch":99,"log_start":"124'6500","ondisk_log_start":"124'6500","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'8787","last_scrub_stamp":"2024-03-11T22:12:38.052573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.052573+0000","objects_scrubbed":6049,"log_size":4701,"log_dups_size":0,"ondisk_log_size":4701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:03.556832+0000","scrub_duration":2625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233766912,"num_objects":3567,"num_object_clones":0,"num_object_copies":10701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11201,"num_write_kb":472576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":165,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005559+0000","last_change":"2024-03-11T22:12:36.449430+0000","last_active":"2024-03-11T22:13:15.005559+0000","last_peered":"2024-03-11T22:13:15.005559+0000","last_clean":"2024-03-11T22:13:15.005559+0000","last_became_active":"2024-03-11T22:08:39.083141+0000","last_became_peered":"2024-03-11T22:08:39.083141+0000","last_unstale":"2024-03-11T22:13:15.005559+0000","last_undegraded":"2024-03-11T22:13:15.005559+0000","last_fullsized":"2024-03-11T22:13:15.005559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:12:36.449307+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:12:36.449307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:00.397020+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":106,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004531+0000","last_change":"2024-03-11T22:12:29.074360+0000","last_active":"2024-03-11T22:13:15.004531+0000","last_peered":"2024-03-11T22:13:15.004531+0000","last_clean":"2024-03-11T22:13:15.004531+0000","last_became_active":"2024-03-11T22:10:35.413337+0000","last_became_peered":"2024-03-11T22:10:35.413337+0000","last_unstale":"2024-03-11T22:13:15.004531+0000","last_undegraded":"2024-03-11T22:13:15.004531+0000","last_fullsized":"2024-03-11T22:13:15.004531+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:29.074256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:29.074256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:31.925354+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":204,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.009153+0000","last_change":"2024-03-11T22:11:21.379985+0000","last_active":"2024-03-11T22:13:15.009153+0000","last_peered":"2024-03-11T22:13:15.009153+0000","last_clean":"2024-03-11T22:13:15.009153+0000","last_became_active":"2024-03-11T22:08:54.607153+0000","last_became_peered":"2024-03-11T22:08:54.607153+0000","last_unstale":"2024-03-11T22:13:15.009153+0000","last_undegraded":"2024-03-11T22:13:15.009153+0000","last_fullsized":"2024-03-11T22:13:15.009153+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:11:21.379862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:21.379862+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.070117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004367+0000","last_change":"2024-03-11T22:12:38.757349+0000","last_active":"2024-03-11T22:13:15.004367+0000","last_peered":"2024-03-11T22:13:15.004367+0000","last_clean":"2024-03-11T22:13:15.004367+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:13:15.004367+0000","last_undegraded":"2024-03-11T22:13:15.004367+0000","last_fullsized":"2024-03-11T22:13:15.004367+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.757164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:48.916257+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"141'11156","reported_seq":11251,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.391313+0000","last_change":"2024-03-11T22:10:36.413014+0000","last_active":"2024-03-11T22:13:15.391313+0000","last_peered":"2024-03-11T22:13:15.391313+0000","last_clean":"2024-03-11T22:13:15.391313+0000","last_became_active":"2024-03-11T22:10:36.412662+0000","last_became_peered":"2024-03-11T22:10:36.412662+0000","last_unstale":"2024-03-11T22:13:15.391313+0000","last_undegraded":"2024-03-11T22:13:15.391313+0000","last_fullsized":"2024-03-11T22:13:15.391313+0000","mapping_epoch":99,"log_start":"118'5600","ondisk_log_start":"118'5600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":5556,"log_dups_size":0,"ondisk_log_size":5556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238419968,"num_objects":3638,"num_object_clones":0,"num_object_copies":10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11156,"num_write_kb":473408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":113,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004134+0000","last_change":"2024-03-11T22:12:34.731700+0000","last_active":"2024-03-11T22:13:15.004134+0000","last_peered":"2024-03-11T22:13:15.004134+0000","last_clean":"2024-03-11T22:13:15.004134+0000","last_became_active":"2024-03-11T22:10:35.410893+0000","last_became_peered":"2024-03-11T22:10:35.410893+0000","last_unstale":"2024-03-11T22:13:15.004134+0000","last_undegraded":"2024-03-11T22:13:15.004134+0000","last_fullsized":"2024-03-11T22:13:15.004134+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:34.731409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:34.731409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:42.767039+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":167,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.002392+0000","last_change":"2024-03-11T22:11:28.377967+0000","last_active":"2024-03-11T22:13:15.002392+0000","last_peered":"2024-03-11T22:13:15.002392+0000","last_clean":"2024-03-11T22:13:15.002392+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:13:15.002392+0000","last_undegraded":"2024-03-11T22:13:15.002392+0000","last_fullsized":"2024-03-11T22:13:15.002392+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:28.377844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.710319+0000","last_clean_scrub_stamp":"2024-03-11T22:11:28.377844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:16.753021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":114,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.994381+0000","last_change":"2024-03-11T22:12:44.474800+0000","last_active":"2024-03-11T22:13:14.994381+0000","last_peered":"2024-03-11T22:13:14.994381+0000","last_clean":"2024-03-11T22:13:14.994381+0000","last_became_active":"2024-03-11T22:10:35.418715+0000","last_became_peered":"2024-03-11T22:10:35.418715+0000","last_unstale":"2024-03-11T22:13:14.994381+0000","last_undegraded":"2024-03-11T22:13:14.994381+0000","last_fullsized":"2024-03-11T22:13:14.994381+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.474728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.474728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:44.686186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":210,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.009859+0000","last_change":"2024-03-11T22:12:36.843050+0000","last_active":"2024-03-11T22:13:15.009859+0000","last_peered":"2024-03-11T22:13:15.009859+0000","last_clean":"2024-03-11T22:13:15.009859+0000","last_became_active":"2024-03-11T22:08:35.087046+0000","last_became_peered":"2024-03-11T22:08:35.087046+0000","last_unstale":"2024-03-11T22:13:15.009859+0000","last_undegraded":"2024-03-11T22:13:15.009859+0000","last_fullsized":"2024-03-11T22:13:15.009859+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:12:36.842912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:36.842912+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:04.592208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"141'11380","reported_seq":11552,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.401905+0000","last_change":"2024-03-11T22:12:25.097684+0000","last_active":"2024-03-11T22:13:15.401905+0000","last_peered":"2024-03-11T22:13:15.401905+0000","last_clean":"2024-03-11T22:13:15.401905+0000","last_became_active":"2024-03-11T22:10:36.410947+0000","last_became_peered":"2024-03-11T22:10:36.410947+0000","last_unstale":"2024-03-11T22:13:15.401905+0000","last_undegraded":"2024-03-11T22:13:15.401905+0000","last_fullsized":"2024-03-11T22:13:15.401905+0000","mapping_epoch":99,"log_start":"114'4500","ondisk_log_start":"114'4500","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'8252","last_scrub_stamp":"2024-03-11T22:12:25.097553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:25.097553+0000","objects_scrubbed":6851,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:43.600745+0000","scrub_duration":2502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238419968,"num_objects":3638,"num_object_clones":0,"num_object_copies":10914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11380,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":163,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.014327+0000","last_change":"2024-03-11T22:12:28.683756+0000","last_active":"2024-03-11T22:13:15.014327+0000","last_peered":"2024-03-11T22:13:15.014327+0000","last_clean":"2024-03-11T22:13:15.014327+0000","last_became_active":"2024-03-11T22:08:39.085496+0000","last_became_peered":"2024-03-11T22:08:39.085496+0000","last_unstale":"2024-03-11T22:13:15.014327+0000","last_undegraded":"2024-03-11T22:13:15.014327+0000","last_fullsized":"2024-03-11T22:13:15.014327+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:28.683646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:28.683646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:52.658563+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":167,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.010600+0000","last_change":"2024-03-11T22:11:36.289334+0000","last_active":"2024-03-11T22:13:15.010600+0000","last_peered":"2024-03-11T22:13:15.010600+0000","last_clean":"2024-03-11T22:13:15.010600+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:13:15.010600+0000","last_undegraded":"2024-03-11T22:13:15.010600+0000","last_fullsized":"2024-03-11T22:13:15.010600+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:11:36.289016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:16.752992+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.994872+0000","last_change":"2024-03-11T22:12:34.391183+0000","last_active":"2024-03-11T22:13:14.994872+0000","last_peered":"2024-03-11T22:13:14.994872+0000","last_clean":"2024-03-11T22:13:14.994872+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:13:14.994872+0000","last_undegraded":"2024-03-11T22:13:14.994872+0000","last_fullsized":"2024-03-11T22:13:14.994872+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:34.391054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:34.391054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:45.133139+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":195,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.000866+0000","last_change":"2024-03-11T22:10:17.179119+0000","last_active":"2024-03-11T22:13:15.000866+0000","last_peered":"2024-03-11T22:13:15.000866+0000","last_clean":"2024-03-11T22:13:15.000866+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:13:15.000866+0000","last_undegraded":"2024-03-11T22:13:15.000866+0000","last_fullsized":"2024-03-11T22:13:15.000866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:10:17.178987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:17.178987+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"141'11178","reported_seq":11345,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.121046+0000","last_change":"2024-03-11T22:12:30.660272+0000","last_active":"2024-03-11T22:13:15.121046+0000","last_peered":"2024-03-11T22:13:15.121046+0000","last_clean":"2024-03-11T22:13:15.121046+0000","last_became_active":"2024-03-11T22:10:36.416476+0000","last_became_peered":"2024-03-11T22:10:36.416476+0000","last_unstale":"2024-03-11T22:13:15.121046+0000","last_undegraded":"2024-03-11T22:13:15.121046+0000","last_fullsized":"2024-03-11T22:13:15.121046+0000","mapping_epoch":99,"log_start":"116'4900","ondisk_log_start":"116'4900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8438","last_scrub_stamp":"2024-03-11T22:12:30.660018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:30.660018+0000","objects_scrubbed":6392,"log_size":6278,"log_dups_size":0,"ondisk_log_size":6278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:40.736516+0000","scrub_duration":1794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235929600,"num_objects":3600,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11178,"num_write_kb":472896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":163,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.006063+0000","last_change":"2024-03-11T22:12:45.439711+0000","last_active":"2024-03-11T22:13:15.006063+0000","last_peered":"2024-03-11T22:13:15.006063+0000","last_clean":"2024-03-11T22:13:15.006063+0000","last_became_active":"2024-03-11T22:08:39.083738+0000","last_became_peered":"2024-03-11T22:08:39.083738+0000","last_unstale":"2024-03-11T22:13:15.006063+0000","last_undegraded":"2024-03-11T22:13:15.006063+0000","last_fullsized":"2024-03-11T22:13:15.006063+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.439584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:47.160579+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":187,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.993540+0000","last_change":"2024-03-11T22:11:42.128436+0000","last_active":"2024-03-11T22:13:14.993540+0000","last_peered":"2024-03-11T22:13:14.993540+0000","last_clean":"2024-03-11T22:13:14.993540+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:13:14.993540+0000","last_undegraded":"2024-03-11T22:13:14.993540+0000","last_fullsized":"2024-03-11T22:13:14.993540+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:11:42.128356+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:11:42.128356+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.389137+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":167,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004197+0000","last_change":"2024-03-11T22:09:55.458484+0000","last_active":"2024-03-11T22:13:15.004197+0000","last_peered":"2024-03-11T22:13:15.004197+0000","last_clean":"2024-03-11T22:13:15.004197+0000","last_became_active":"2024-03-11T22:08:33.035469+0000","last_became_peered":"2024-03-11T22:08:33.035469+0000","last_unstale":"2024-03-11T22:13:15.004197+0000","last_undegraded":"2024-03-11T22:13:15.004197+0000","last_fullsized":"2024-03-11T22:13:15.004197+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:09:55.458385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:09:55.458385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.069724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":276,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.001271+0000","last_change":"2024-03-11T22:11:24.895982+0000","last_active":"2024-03-11T22:13:15.001271+0000","last_peered":"2024-03-11T22:13:15.001271+0000","last_clean":"2024-03-11T22:13:15.001271+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:13:15.001271+0000","last_undegraded":"2024-03-11T22:13:15.001271+0000","last_fullsized":"2024-03-11T22:13:15.001271+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-11T22:11:24.892593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:24.892593+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":167,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.003839+0000","last_change":"2024-03-11T22:11:14.224976+0000","last_active":"2024-03-11T22:13:15.003839+0000","last_peered":"2024-03-11T22:13:15.003839+0000","last_clean":"2024-03-11T22:13:15.003839+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:13:15.003839+0000","last_undegraded":"2024-03-11T22:13:15.003839+0000","last_fullsized":"2024-03-11T22:13:15.003839+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:14.224859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:14.224859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"141'11331","reported_seq":11426,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.397842+0000","last_change":"2024-03-11T22:10:36.413021+0000","last_active":"2024-03-11T22:13:15.397842+0000","last_peered":"2024-03-11T22:13:15.397842+0000","last_clean":"2024-03-11T22:13:15.397842+0000","last_became_active":"2024-03-11T22:10:36.412694+0000","last_became_peered":"2024-03-11T22:10:36.412694+0000","last_unstale":"2024-03-11T22:13:15.397842+0000","last_undegraded":"2024-03-11T22:13:15.397842+0000","last_fullsized":"2024-03-11T22:13:15.397842+0000","mapping_epoch":99,"log_start":"118'5700","ondisk_log_start":"118'5700","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":5631,"log_dups_size":0,"ondisk_log_size":5631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239271936,"num_objects":3651,"num_object_clones":0,"num_object_copies":10953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11331,"num_write_kb":479424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":105,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.010340+0000","last_change":"2024-03-11T22:11:35.482382+0000","last_active":"2024-03-11T22:13:15.010340+0000","last_peered":"2024-03-11T22:13:15.010340+0000","last_clean":"2024-03-11T22:13:15.010340+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:13:15.010340+0000","last_undegraded":"2024-03-11T22:13:15.010340+0000","last_fullsized":"2024-03-11T22:13:15.010340+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:35.482269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.482269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:16.899303+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.001425+0000","last_change":"2024-03-11T22:11:19.960632+0000","last_active":"2024-03-11T22:13:15.001425+0000","last_peered":"2024-03-11T22:13:15.001425+0000","last_clean":"2024-03-11T22:13:15.001425+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:13:15.001425+0000","last_undegraded":"2024-03-11T22:13:15.001425+0000","last_fullsized":"2024-03-11T22:13:15.001425+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:19.960398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:19.960398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":192,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.001478+0000","last_change":"2024-03-11T22:12:45.863561+0000","last_active":"2024-03-11T22:13:15.001478+0000","last_peered":"2024-03-11T22:13:15.001478+0000","last_clean":"2024-03-11T22:13:15.001478+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:13:15.001478+0000","last_undegraded":"2024-03-11T22:13:15.001478+0000","last_fullsized":"2024-03-11T22:13:15.001478+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.863440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.863440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:09.786177+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":198,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.000104+0000","last_change":"2024-03-11T22:11:23.875763+0000","last_active":"2024-03-11T22:13:15.000104+0000","last_peered":"2024-03-11T22:13:15.000104+0000","last_clean":"2024-03-11T22:13:15.000104+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:13:15.000104+0000","last_undegraded":"2024-03-11T22:13:15.000104+0000","last_fullsized":"2024-03-11T22:13:15.000104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-03-11T22:11:23.875628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:23.875628+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.006881+0000","last_change":"2024-03-11T22:12:31.669715+0000","last_active":"2024-03-11T22:13:15.006881+0000","last_peered":"2024-03-11T22:13:15.006881+0000","last_clean":"2024-03-11T22:13:15.006881+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:13:15.006881+0000","last_undegraded":"2024-03-11T22:13:15.006881+0000","last_fullsized":"2024-03-11T22:13:15.006881+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:31.669555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.669555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:47.635385+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"141'11294","reported_seq":11389,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.381062+0000","last_change":"2024-03-11T22:10:36.409137+0000","last_active":"2024-03-11T22:13:15.381062+0000","last_peered":"2024-03-11T22:13:15.381062+0000","last_clean":"2024-03-11T22:13:15.381062+0000","last_became_active":"2024-03-11T22:10:36.408867+0000","last_became_peered":"2024-03-11T22:10:36.408867+0000","last_unstale":"2024-03-11T22:13:15.381062+0000","last_undegraded":"2024-03-11T22:13:15.381062+0000","last_fullsized":"2024-03-11T22:13:15.381062+0000","mapping_epoch":99,"log_start":"118'5700","ondisk_log_start":"118'5700","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233177088,"num_objects":3558,"num_object_clones":0,"num_object_copies":10674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":475264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":163,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.994634+0000","last_change":"2024-03-11T22:11:46.117166+0000","last_active":"2024-03-11T22:13:14.994634+0000","last_peered":"2024-03-11T22:13:14.994634+0000","last_clean":"2024-03-11T22:13:14.994634+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:13:14.994634+0000","last_undegraded":"2024-03-11T22:13:14.994634+0000","last_fullsized":"2024-03-11T22:13:14.994634+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:46.117039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:46.117039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.389120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":422,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.999569+0000","last_change":"2024-03-11T22:11:34.382394+0000","last_active":"2024-03-11T22:13:14.999569+0000","last_peered":"2024-03-11T22:13:14.999569+0000","last_clean":"2024-03-11T22:13:14.999569+0000","last_became_active":"2024-03-11T22:10:35.412943+0000","last_became_peered":"2024-03-11T22:10:35.412943+0000","last_unstale":"2024-03-11T22:13:14.999569+0000","last_undegraded":"2024-03-11T22:13:14.999569+0000","last_fullsized":"2024-03-11T22:13:14.999569+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:11:34.382250+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.382250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005521+0000","last_change":"2024-03-11T22:11:34.039067+0000","last_active":"2024-03-11T22:13:15.005521+0000","last_peered":"2024-03-11T22:13:15.005521+0000","last_clean":"2024-03-11T22:13:15.005521+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:13:15.005521+0000","last_undegraded":"2024-03-11T22:13:15.005521+0000","last_fullsized":"2024-03-11T22:13:15.005521+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.038930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":250,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.010439+0000","last_change":"2024-03-11T22:11:30.413882+0000","last_active":"2024-03-11T22:13:15.010439+0000","last_peered":"2024-03-11T22:13:15.010439+0000","last_clean":"2024-03-11T22:13:15.010439+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:13:15.010439+0000","last_undegraded":"2024-03-11T22:13:15.010439+0000","last_fullsized":"2024-03-11T22:13:15.010439+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-11T22:11:30.413737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.413737+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:16.899347+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":113,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.002036+0000","last_change":"2024-03-11T22:12:33.715464+0000","last_active":"2024-03-11T22:13:15.002036+0000","last_peered":"2024-03-11T22:13:15.002036+0000","last_clean":"2024-03-11T22:13:15.002036+0000","last_became_active":"2024-03-11T22:10:35.412547+0000","last_became_peered":"2024-03-11T22:10:35.412547+0000","last_unstale":"2024-03-11T22:13:15.002036+0000","last_undegraded":"2024-03-11T22:13:15.002036+0000","last_fullsized":"2024-03-11T22:13:15.002036+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:33.715347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:12:33.715347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:47.739223+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"141'11301","reported_seq":11459,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.372750+0000","last_change":"2024-03-11T22:11:42.189504+0000","last_active":"2024-03-11T22:13:15.372750+0000","last_peered":"2024-03-11T22:13:15.372750+0000","last_clean":"2024-03-11T22:13:15.372750+0000","last_became_active":"2024-03-11T22:10:36.406423+0000","last_became_peered":"2024-03-11T22:10:36.406423+0000","last_unstale":"2024-03-11T22:13:15.372750+0000","last_undegraded":"2024-03-11T22:13:15.372750+0000","last_fullsized":"2024-03-11T22:13:15.372750+0000","mapping_epoch":99,"log_start":"114'4400","ondisk_log_start":"114'4400","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'5593","last_scrub_stamp":"2024-03-11T22:11:42.188394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:11:42.188394+0000","objects_scrubbed":5541,"log_size":6901,"log_dups_size":0,"ondisk_log_size":6901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:16.763138+0000","scrub_duration":1878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233897984,"num_objects":3569,"num_object_clones":0,"num_object_copies":10707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11301,"num_write_kb":475840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":171,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.000964+0000","last_change":"2024-03-11T22:12:44.933303+0000","last_active":"2024-03-11T22:13:15.000964+0000","last_peered":"2024-03-11T22:13:15.000964+0000","last_clean":"2024-03-11T22:13:15.000964+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:13:15.000964+0000","last_undegraded":"2024-03-11T22:13:15.000964+0000","last_fullsized":"2024-03-11T22:13:15.000964+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.927967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.927967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:13.766656+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.014575+0000","last_change":"2024-03-11T22:12:19.685277+0000","last_active":"2024-03-11T22:13:15.014575+0000","last_peered":"2024-03-11T22:13:15.014575+0000","last_clean":"2024-03-11T22:13:15.014575+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:13:15.014575+0000","last_undegraded":"2024-03-11T22:13:15.014575+0000","last_fullsized":"2024-03-11T22:13:15.014575+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:19.684663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:19.684663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:48.270422+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":439,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.000702+0000","last_change":"2024-03-11T22:11:34.857427+0000","last_active":"2024-03-11T22:13:15.000702+0000","last_peered":"2024-03-11T22:13:15.000702+0000","last_clean":"2024-03-11T22:13:15.000702+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:13:15.000702+0000","last_undegraded":"2024-03-11T22:13:15.000702+0000","last_fullsized":"2024-03-11T22:13:15.000702+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:11:34.857300+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.857300+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":249,"num_read_kb":244,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":189,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005412+0000","last_change":"2024-03-11T22:11:06.292270+0000","last_active":"2024-03-11T22:13:15.005412+0000","last_peered":"2024-03-11T22:13:15.005412+0000","last_clean":"2024-03-11T22:13:15.005412+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:13:15.005412+0000","last_undegraded":"2024-03-11T22:13:15.005412+0000","last_fullsized":"2024-03-11T22:13:15.005412+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:11:06.292138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:06.292138+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":167,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.995128+0000","last_change":"2024-03-11T22:12:27.459283+0000","last_active":"2024-03-11T22:13:14.995128+0000","last_peered":"2024-03-11T22:13:14.995128+0000","last_clean":"2024-03-11T22:13:14.995128+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:13:14.995128+0000","last_undegraded":"2024-03-11T22:13:14.995128+0000","last_fullsized":"2024-03-11T22:13:14.995128+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:27.459154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:27.459154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:36.420728+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"141'11039","reported_seq":11209,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.094947+0000","last_change":"2024-03-11T22:12:25.123093+0000","last_active":"2024-03-11T22:13:15.094947+0000","last_peered":"2024-03-11T22:13:15.094947+0000","last_clean":"2024-03-11T22:13:15.094947+0000","last_became_active":"2024-03-11T22:10:36.411289+0000","last_became_peered":"2024-03-11T22:10:36.411289+0000","last_unstale":"2024-03-11T22:13:15.094947+0000","last_undegraded":"2024-03-11T22:13:15.094947+0000","last_fullsized":"2024-03-11T22:13:15.094947+0000","mapping_epoch":99,"log_start":"118'5200","ondisk_log_start":"118'5200","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7985","last_scrub_stamp":"2024-03-11T22:12:25.122944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:25.122944+0000","objects_scrubbed":6655,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:35.031889+0000","scrub_duration":1760,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233635840,"num_objects":3565,"num_object_clones":0,"num_object_copies":10695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11039,"num_write_kb":467328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":163,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.000512+0000","last_change":"2024-03-11T22:11:28.871970+0000","last_active":"2024-03-11T22:13:15.000512+0000","last_peered":"2024-03-11T22:13:15.000512+0000","last_clean":"2024-03-11T22:13:15.000512+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:13:15.000512+0000","last_undegraded":"2024-03-11T22:13:15.000512+0000","last_fullsized":"2024-03-11T22:13:15.000512+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:28.871246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:28.871246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.001180+0000","last_change":"2024-03-11T22:12:42.951642+0000","last_active":"2024-03-11T22:13:15.001180+0000","last_peered":"2024-03-11T22:13:15.001180+0000","last_clean":"2024-03-11T22:13:15.001180+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:13:15.001180+0000","last_undegraded":"2024-03-11T22:13:15.001180+0000","last_fullsized":"2024-03-11T22:13:15.001180+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.951379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.951379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:10.673379+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":199,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.000626+0000","last_change":"2024-03-11T22:10:31.211684+0000","last_active":"2024-03-11T22:13:15.000626+0000","last_peered":"2024-03-11T22:13:15.000626+0000","last_clean":"2024-03-11T22:13:15.000626+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:13:15.000626+0000","last_undegraded":"2024-03-11T22:13:15.000626+0000","last_fullsized":"2024-03-11T22:13:15.000626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:10:31.211547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:31.211547+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.009948+0000","last_change":"2024-03-11T22:12:40.863550+0000","last_active":"2024-03-11T22:13:15.009948+0000","last_peered":"2024-03-11T22:13:15.009948+0000","last_clean":"2024-03-11T22:13:15.009948+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:13:15.009948+0000","last_undegraded":"2024-03-11T22:13:15.009948+0000","last_fullsized":"2024-03-11T22:13:15.009948+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:12:40.863445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.360193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"141'11134","reported_seq":11229,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.081196+0000","last_change":"2024-03-11T22:10:36.416445+0000","last_active":"2024-03-11T22:13:15.081196+0000","last_peered":"2024-03-11T22:13:15.081196+0000","last_clean":"2024-03-11T22:13:15.081196+0000","last_became_active":"2024-03-11T22:10:36.416083+0000","last_became_peered":"2024-03-11T22:10:36.416083+0000","last_unstale":"2024-03-11T22:13:15.081196+0000","last_undegraded":"2024-03-11T22:13:15.081196+0000","last_fullsized":"2024-03-11T22:13:15.081196+0000","mapping_epoch":99,"log_start":"122'6300","ondisk_log_start":"122'6300","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":4834,"log_dups_size":0,"ondisk_log_size":4834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233570304,"num_objects":3564,"num_object_clones":0,"num_object_copies":10692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11134,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":155,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.999474+0000","last_change":"2024-03-11T22:11:11.177052+0000","last_active":"2024-03-11T22:13:14.999474+0000","last_peered":"2024-03-11T22:13:14.999474+0000","last_clean":"2024-03-11T22:13:14.999474+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:13:14.999474+0000","last_undegraded":"2024-03-11T22:13:14.999474+0000","last_fullsized":"2024-03-11T22:13:14.999474+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:11.176930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:11.176930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":183,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.994315+0000","last_change":"2024-03-11T22:12:45.466625+0000","last_active":"2024-03-11T22:13:14.994315+0000","last_peered":"2024-03-11T22:13:14.994315+0000","last_clean":"2024-03-11T22:13:14.994315+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:13:14.994315+0000","last_undegraded":"2024-03-11T22:13:14.994315+0000","last_fullsized":"2024-03-11T22:13:14.994315+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.466508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.466508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:09.930146+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":186,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.999762+0000","last_change":"2024-03-11T22:12:38.896205+0000","last_active":"2024-03-11T22:13:14.999762+0000","last_peered":"2024-03-11T22:13:14.999762+0000","last_clean":"2024-03-11T22:13:14.999762+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:13:14.999762+0000","last_undegraded":"2024-03-11T22:13:14.999762+0000","last_fullsized":"2024-03-11T22:13:14.999762+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:12:38.895763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.895763+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:08.358608+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":232,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.000965+0000","last_change":"2024-03-11T22:11:25.385148+0000","last_active":"2024-03-11T22:13:15.000965+0000","last_peered":"2024-03-11T22:13:15.000965+0000","last_clean":"2024-03-11T22:13:15.000965+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:13:15.000965+0000","last_undegraded":"2024-03-11T22:13:15.000965+0000","last_fullsized":"2024-03-11T22:13:15.000965+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:11:25.384965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.384965+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"141'11166","reported_seq":11331,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.380652+0000","last_change":"2024-03-11T22:12:35.256058+0000","last_active":"2024-03-11T22:13:15.380652+0000","last_peered":"2024-03-11T22:13:15.380652+0000","last_clean":"2024-03-11T22:13:15.380652+0000","last_became_active":"2024-03-11T22:10:36.410859+0000","last_became_peered":"2024-03-11T22:10:36.410859+0000","last_unstale":"2024-03-11T22:13:15.380652+0000","last_undegraded":"2024-03-11T22:13:15.380652+0000","last_fullsized":"2024-03-11T22:13:15.380652+0000","mapping_epoch":99,"log_start":"124'6500","ondisk_log_start":"124'6500","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8663","last_scrub_stamp":"2024-03-11T22:12:35.255739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.255739+0000","objects_scrubbed":6218,"log_size":4666,"log_dups_size":0,"ondisk_log_size":4666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:47.000467+0000","scrub_duration":2947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238551040,"num_objects":3640,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11166,"num_write_kb":473792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":113,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.010358+0000","last_change":"2024-03-11T22:12:46.968879+0000","last_active":"2024-03-11T22:13:15.010358+0000","last_peered":"2024-03-11T22:13:15.010358+0000","last_clean":"2024-03-11T22:13:15.010358+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:13:15.010358+0000","last_undegraded":"2024-03-11T22:13:15.010358+0000","last_fullsized":"2024-03-11T22:13:15.010358+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.968807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.968807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:12.616431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":176,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004729+0000","last_change":"2024-03-11T22:12:49.856525+0000","last_active":"2024-03-11T22:13:15.004729+0000","last_peered":"2024-03-11T22:13:15.004729+0000","last_clean":"2024-03-11T22:13:15.004729+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:13:15.004729+0000","last_undegraded":"2024-03-11T22:13:15.004729+0000","last_fullsized":"2024-03-11T22:13:15.004729+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:49.856381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.856381+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:57.055758+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":184,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.008899+0000","last_change":"2024-03-11T22:09:53.414248+0000","last_active":"2024-03-11T22:13:15.008899+0000","last_peered":"2024-03-11T22:13:15.008899+0000","last_clean":"2024-03-11T22:13:15.008899+0000","last_became_active":"2024-03-11T22:08:35.053452+0000","last_became_peered":"2024-03-11T22:08:35.053452+0000","last_unstale":"2024-03-11T22:13:15.008899+0000","last_undegraded":"2024-03-11T22:13:15.008899+0000","last_fullsized":"2024-03-11T22:13:15.008899+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:09:53.414127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:09:53.414127+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.069696+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":232,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.011578+0000","last_change":"2024-03-11T22:12:31.264333+0000","last_active":"2024-03-11T22:13:15.011578+0000","last_peered":"2024-03-11T22:13:15.011578+0000","last_clean":"2024-03-11T22:13:15.011578+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:13:15.011578+0000","last_undegraded":"2024-03-11T22:13:15.011578+0000","last_fullsized":"2024-03-11T22:13:15.011578+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:12:31.264213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.264213+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:45.274053+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"141'11182","reported_seq":11346,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.404829+0000","last_change":"2024-03-11T22:12:35.900151+0000","last_active":"2024-03-11T22:13:15.404829+0000","last_peered":"2024-03-11T22:13:15.404829+0000","last_clean":"2024-03-11T22:13:15.404829+0000","last_became_active":"2024-03-11T22:10:36.414380+0000","last_became_peered":"2024-03-11T22:10:36.414380+0000","last_unstale":"2024-03-11T22:13:15.404829+0000","last_undegraded":"2024-03-11T22:13:15.404829+0000","last_fullsized":"2024-03-11T22:13:15.404829+0000","mapping_epoch":99,"log_start":"124'6500","ondisk_log_start":"124'6500","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8688","last_scrub_stamp":"2024-03-11T22:12:35.899750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.899750+0000","objects_scrubbed":6120,"log_size":4682,"log_dups_size":0,"ondisk_log_size":4682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:44.219720+0000","scrub_duration":2560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232259584,"num_objects":3544,"num_object_clones":0,"num_object_copies":10632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11182,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":155,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004196+0000","last_change":"2024-03-11T22:11:35.248168+0000","last_active":"2024-03-11T22:13:15.004196+0000","last_peered":"2024-03-11T22:13:15.004196+0000","last_clean":"2024-03-11T22:13:15.004196+0000","last_became_active":"2024-03-11T22:08:39.078244+0000","last_became_peered":"2024-03-11T22:08:39.078244+0000","last_unstale":"2024-03-11T22:13:15.004196+0000","last_undegraded":"2024-03-11T22:13:15.004196+0000","last_fullsized":"2024-03-11T22:13:15.004196+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:35.248081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.248081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.070172+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.011404+0000","last_change":"2024-03-11T22:11:24.938132+0000","last_active":"2024-03-11T22:13:15.011404+0000","last_peered":"2024-03-11T22:13:15.011404+0000","last_clean":"2024-03-11T22:13:15.011404+0000","last_became_active":"2024-03-11T22:08:33.036469+0000","last_became_peered":"2024-03-11T22:08:33.036469+0000","last_unstale":"2024-03-11T22:13:15.011404+0000","last_undegraded":"2024-03-11T22:13:15.011404+0000","last_fullsized":"2024-03-11T22:13:15.011404+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:24.937886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:24.937886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":206,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.011481+0000","last_change":"2024-03-11T22:12:39.447777+0000","last_active":"2024-03-11T22:13:15.011481+0000","last_peered":"2024-03-11T22:13:15.011481+0000","last_clean":"2024-03-11T22:13:15.011481+0000","last_became_active":"2024-03-11T22:08:35.085979+0000","last_became_peered":"2024-03-11T22:08:35.085979+0000","last_unstale":"2024-03-11T22:13:15.011481+0000","last_undegraded":"2024-03-11T22:13:15.011481+0000","last_fullsized":"2024-03-11T22:13:15.011481+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:12:39.447312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:39.447312+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:00.055885+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":167,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.009237+0000","last_change":"2024-03-11T22:11:40.165879+0000","last_active":"2024-03-11T22:13:15.009237+0000","last_peered":"2024-03-11T22:13:15.009237+0000","last_clean":"2024-03-11T22:13:15.009237+0000","last_became_active":"2024-03-11T22:08:37.076522+0000","last_became_peered":"2024-03-11T22:08:37.076522+0000","last_unstale":"2024-03-11T22:13:15.009237+0000","last_undegraded":"2024-03-11T22:13:15.009237+0000","last_fullsized":"2024-03-11T22:13:15.009237+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:40.165760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:40.165760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.070316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"141'11119","reported_seq":11285,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.151159+0000","last_change":"2024-03-11T22:12:31.355260+0000","last_active":"2024-03-11T22:13:15.151159+0000","last_peered":"2024-03-11T22:13:15.151159+0000","last_clean":"2024-03-11T22:13:15.151159+0000","last_became_active":"2024-03-11T22:10:36.414372+0000","last_became_peered":"2024-03-11T22:10:36.414372+0000","last_unstale":"2024-03-11T22:13:15.151159+0000","last_undegraded":"2024-03-11T22:13:15.151159+0000","last_fullsized":"2024-03-11T22:13:15.151159+0000","mapping_epoch":99,"log_start":"116'4800","ondisk_log_start":"116'4800","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8481","last_scrub_stamp":"2024-03-11T22:12:31.354979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.354979+0000","objects_scrubbed":6285,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:50.389207+0000","scrub_duration":1504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236388352,"num_objects":3607,"num_object_clones":0,"num_object_copies":10821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11119,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":163,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.994174+0000","last_change":"2024-03-11T22:12:42.469626+0000","last_active":"2024-03-11T22:13:14.994174+0000","last_peered":"2024-03-11T22:13:14.994174+0000","last_clean":"2024-03-11T22:13:14.994174+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:13:14.994174+0000","last_undegraded":"2024-03-11T22:13:14.994174+0000","last_fullsized":"2024-03-11T22:13:14.994174+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.469501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.469501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:51.863765+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":224,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005350+0000","last_change":"2024-03-11T22:10:14.545445+0000","last_active":"2024-03-11T22:13:15.005350+0000","last_peered":"2024-03-11T22:13:15.005350+0000","last_clean":"2024-03-11T22:13:15.005350+0000","last_became_active":"2024-03-11T22:08:37.077142+0000","last_became_peered":"2024-03-11T22:08:37.077142+0000","last_unstale":"2024-03-11T22:13:15.005350+0000","last_undegraded":"2024-03-11T22:13:15.005350+0000","last_fullsized":"2024-03-11T22:13:15.005350+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:10:14.545345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:10:14.545345+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.069788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.011397+0000","last_change":"2024-03-11T22:12:24.278539+0000","last_active":"2024-03-11T22:13:15.011397+0000","last_peered":"2024-03-11T22:13:15.011397+0000","last_clean":"2024-03-11T22:13:15.011397+0000","last_became_active":"2024-03-11T22:08:33.037537+0000","last_became_peered":"2024-03-11T22:08:33.037537+0000","last_unstale":"2024-03-11T22:13:15.011397+0000","last_undegraded":"2024-03-11T22:13:15.011397+0000","last_fullsized":"2024-03-11T22:13:15.011397+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:24.278298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.278298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:40.405410+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":196,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.000261+0000","last_change":"2024-03-11T22:11:41.817430+0000","last_active":"2024-03-11T22:13:15.000261+0000","last_peered":"2024-03-11T22:13:15.000261+0000","last_clean":"2024-03-11T22:13:15.000261+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:13:15.000261+0000","last_undegraded":"2024-03-11T22:13:15.000261+0000","last_fullsized":"2024-03-11T22:13:15.000261+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:11:41.817285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:41.817285+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"141'11544","reported_seq":11711,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.146004+0000","last_change":"2024-03-11T22:12:35.582515+0000","last_active":"2024-03-11T22:13:15.146004+0000","last_peered":"2024-03-11T22:13:15.146004+0000","last_clean":"2024-03-11T22:13:15.146004+0000","last_became_active":"2024-03-11T22:10:36.412847+0000","last_became_peered":"2024-03-11T22:10:36.412847+0000","last_unstale":"2024-03-11T22:13:15.146004+0000","last_undegraded":"2024-03-11T22:13:15.146004+0000","last_fullsized":"2024-03-11T22:13:15.146004+0000","mapping_epoch":99,"log_start":"117'5200","ondisk_log_start":"117'5200","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'9001","last_scrub_stamp":"2024-03-11T22:12:35.582375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.582375+0000","objects_scrubbed":6270,"log_size":6344,"log_dups_size":0,"ondisk_log_size":6344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:02.117832+0000","scrub_duration":1709,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242024472,"num_objects":3694,"num_object_clones":0,"num_object_copies":11082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11544,"num_write_kb":487553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":155,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.010362+0000","last_change":"2024-03-11T22:11:08.902041+0000","last_active":"2024-03-11T22:13:15.010362+0000","last_peered":"2024-03-11T22:13:15.010362+0000","last_clean":"2024-03-11T22:13:15.010362+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:13:15.010362+0000","last_undegraded":"2024-03-11T22:13:15.010362+0000","last_fullsized":"2024-03-11T22:13:15.010362+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:08.901927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:08.901927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":167,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:14.994964+0000","last_change":"2024-03-11T22:12:20.302654+0000","last_active":"2024-03-11T22:13:14.994964+0000","last_peered":"2024-03-11T22:13:14.994964+0000","last_clean":"2024-03-11T22:13:14.994964+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:13:14.994964+0000","last_undegraded":"2024-03-11T22:13:14.994964+0000","last_fullsized":"2024-03-11T22:13:14.994964+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:20.302574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:20.302574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:30.645408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":167,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.008624+0000","last_change":"2024-03-11T22:11:23.353503+0000","last_active":"2024-03-11T22:13:15.008624+0000","last_peered":"2024-03-11T22:13:15.008624+0000","last_clean":"2024-03-11T22:13:15.008624+0000","last_became_active":"2024-03-11T22:08:33.299778+0000","last_became_peered":"2024-03-11T22:08:33.299778+0000","last_unstale":"2024-03-11T22:13:15.008624+0000","last_undegraded":"2024-03-11T22:13:15.008624+0000","last_fullsized":"2024-03-11T22:13:15.008624+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:23.353379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:23.353379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.069987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":190,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004717+0000","last_change":"2024-03-11T22:12:24.093258+0000","last_active":"2024-03-11T22:13:15.004717+0000","last_peered":"2024-03-11T22:13:15.004717+0000","last_clean":"2024-03-11T22:13:15.004717+0000","last_became_active":"2024-03-11T22:08:35.053123+0000","last_became_peered":"2024-03-11T22:08:35.053123+0000","last_unstale":"2024-03-11T22:13:15.004717+0000","last_undegraded":"2024-03-11T22:13:15.004717+0000","last_fullsized":"2024-03-11T22:13:15.004717+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:12:24.093112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.093112+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:53.468276+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"141'11142","reported_seq":11304,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.391535+0000","last_change":"2024-03-11T22:11:46.750236+0000","last_active":"2024-03-11T22:13:15.391535+0000","last_peered":"2024-03-11T22:13:15.391535+0000","last_clean":"2024-03-11T22:13:15.391535+0000","last_became_active":"2024-03-11T22:10:36.412847+0000","last_became_peered":"2024-03-11T22:10:36.412847+0000","last_unstale":"2024-03-11T22:13:15.391535+0000","last_undegraded":"2024-03-11T22:13:15.391535+0000","last_fullsized":"2024-03-11T22:13:15.391535+0000","mapping_epoch":99,"log_start":"114'4300","ondisk_log_start":"114'4300","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'5801","last_scrub_stamp":"2024-03-11T22:11:46.749984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:11:46.749984+0000","objects_scrubbed":5740,"log_size":6842,"log_dups_size":0,"ondisk_log_size":6842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:16.753079+0000","scrub_duration":2346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234881024,"num_objects":3584,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11142,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":171,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.001521+0000","last_change":"2024-03-11T22:12:50.861675+0000","last_active":"2024-03-11T22:13:15.001521+0000","last_peered":"2024-03-11T22:13:15.001521+0000","last_clean":"2024-03-11T22:13:15.001521+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:13:15.001521+0000","last_undegraded":"2024-03-11T22:13:15.001521+0000","last_fullsized":"2024-03-11T22:13:15.001521+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.861557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.861557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:11.522320+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":167,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.001439+0000","last_change":"2024-03-11T22:12:18.789167+0000","last_active":"2024-03-11T22:13:15.001439+0000","last_peered":"2024-03-11T22:13:15.001439+0000","last_clean":"2024-03-11T22:13:15.001439+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:13:15.001439+0000","last_undegraded":"2024-03-11T22:13:15.001439+0000","last_fullsized":"2024-03-11T22:13:15.001439+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:18.788609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:12:18.788609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:22.290691+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":183,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.010039+0000","last_change":"2024-03-11T22:12:42.910183+0000","last_active":"2024-03-11T22:13:15.010039+0000","last_peered":"2024-03-11T22:13:15.010039+0000","last_clean":"2024-03-11T22:13:15.010039+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:13:15.010039+0000","last_undegraded":"2024-03-11T22:13:15.010039+0000","last_fullsized":"2024-03-11T22:13:15.010039+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.910067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.910067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:49.869286+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":191,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004683+0000","last_change":"2024-03-11T22:11:17.270935+0000","last_active":"2024-03-11T22:13:15.004683+0000","last_peered":"2024-03-11T22:13:15.004683+0000","last_clean":"2024-03-11T22:13:15.004683+0000","last_became_active":"2024-03-11T22:08:35.055770+0000","last_became_peered":"2024-03-11T22:08:35.055770+0000","last_unstale":"2024-03-11T22:13:15.004683+0000","last_undegraded":"2024-03-11T22:13:15.004683+0000","last_fullsized":"2024-03-11T22:13:15.004683+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-11T22:11:17.270813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:17.270813+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.070233+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"141'11231","reported_seq":11326,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.397454+0000","last_change":"2024-03-11T22:10:36.408956+0000","last_active":"2024-03-11T22:13:15.397454+0000","last_peered":"2024-03-11T22:13:15.397454+0000","last_clean":"2024-03-11T22:13:15.397454+0000","last_became_active":"2024-03-11T22:10:36.408528+0000","last_became_peered":"2024-03-11T22:10:36.408528+0000","last_unstale":"2024-03-11T22:13:15.397454+0000","last_undegraded":"2024-03-11T22:13:15.397454+0000","last_fullsized":"2024-03-11T22:13:15.397454+0000","mapping_epoch":99,"log_start":"118'5600","ondisk_log_start":"118'5600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":5631,"log_dups_size":0,"ondisk_log_size":5631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228917248,"num_objects":3493,"num_object_clones":0,"num_object_copies":10479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11231,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":155,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005476+0000","last_change":"2024-03-11T22:10:23.555240+0000","last_active":"2024-03-11T22:13:15.005476+0000","last_peered":"2024-03-11T22:13:15.005476+0000","last_clean":"2024-03-11T22:13:15.005476+0000","last_became_active":"2024-03-11T22:08:39.072422+0000","last_became_peered":"2024-03-11T22:08:39.072422+0000","last_unstale":"2024-03-11T22:13:15.005476+0000","last_undegraded":"2024-03-11T22:13:15.005476+0000","last_fullsized":"2024-03-11T22:13:15.005476+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:23.555159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:23.555159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":241,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.006431+0000","last_change":"2024-03-11T22:12:43.819554+0000","last_active":"2024-03-11T22:13:15.006431+0000","last_peered":"2024-03-11T22:13:15.006431+0000","last_clean":"2024-03-11T22:13:15.006431+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:13:15.006431+0000","last_undegraded":"2024-03-11T22:13:15.006431+0000","last_fullsized":"2024-03-11T22:13:15.006431+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:12:43.819269+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_clean_scrub_stamp":"2024-03-11T22:12:43.819269+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:53.786169+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":175,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.005259+0000","last_change":"2024-03-11T22:11:40.097948+0000","last_active":"2024-03-11T22:13:15.005259+0000","last_peered":"2024-03-11T22:13:15.005259+0000","last_clean":"2024-03-11T22:13:15.005259+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:13:15.005259+0000","last_undegraded":"2024-03-11T22:13:15.005259+0000","last_fullsized":"2024-03-11T22:13:15.005259+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:40.094297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:40.094297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":178,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-03-11T22:13:15.004383+0000","last_change":"2024-03-11T22:11:12.298926+0000","last_active":"2024-03-11T22:13:15.004383+0000","last_peered":"2024-03-11T22:13:15.004383+0000","last_clean":"2024-03-11T22:13:15.004383+0000","last_became_active":"2024-03-11T22:08:35.056698+0000","last_became_peered":"2024-03-11T22:08:35.056698+0000","last_unstale":"2024-03-11T22:13:15.004383+0000","last_undegraded":"2024-03-11T22:13:15.004383+0000","last_fullsized":"2024-03-11T22:13:15.004383+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:11:12.298787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:12.298787+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:15.069958+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3766681624,"num_objects":57476,"num_object_clones":0,"num_object_copies":172428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179462,"num_write_kb":7581953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11308249088,"data_stored":11308236872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91742,"ondisk_log_size":91742,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":249,"num_read_kb":244,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005117,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1259420,"kb_used_data":1169940,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92485732,"statfs":{"total":95995035648,"available":94705389568,"internally_reserved":0,"allocated":1198018560,"data_stored":1197688610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":91617739},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332162,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1259532,"kb_used_data":1160324,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92485620,"statfs":{"total":95995035648,"available":94705274880,"internally_reserved":0,"allocated":1188171776,"data_stored":1187858044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1506424,"kb_used_data":1403056,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92238728,"statfs":{"total":95995035648,"available":94452457472,"internally_reserved":0,"allocated":1436729344,"data_stored":1436370208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1973980,"kb_used_data":1856724,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":91771172,"statfs":{"total":95995035648,"available":93973680128,"internally_reserved":0,"allocated":1901285376,"data_stored":1900954782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1507712,"kb_used_data":1395960,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92237440,"statfs":{"total":95995035648,"available":94451138560,"internally_reserved":0,"allocated":1429463040,"data_stored":1429160854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2228264,"kb_used_data":2088800,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91516888,"statfs":{"total":95995035648,"available":93713293312,"internally_reserved":0,"allocated":2138931200,"data_stored":2138588734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142801351},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1255596,"kb_used_data":1161824,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":92489556,"statfs":{"total":95995035648,"available":94709305344,"internally_reserved":0,"allocated":1189707776,"data_stored":1189372640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95877574},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659206,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":999368,"kb_used_data":927936,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92745784,"statfs":{"total":95995035648,"available":94971682816,"internally_reserved":0,"allocated":950206464,"data_stored":949895533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1884229632,"data_stored":1884225560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1424953344,"data_stored":1424949272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1411907584,"data_stored":1411907584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2101215232,"data_stored":2101215232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1188364288,"data_stored":1188364288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":938999808,"data_stored":938999808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1182011392,"data_stored":1182007320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1176567808,"data_stored":1176567808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:13:21.085 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-11T22:13:21.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-11T22:13:21.161 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:21.261 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:21.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:21.262 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:21.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:21.289 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:21.326 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:21 smithi045 ceph-mon[39715]: 7.f scrub ok 2024-03-11T22:13:21.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:21 smithi045 ceph-mon[39715]: 3.0 scrub starts 2024-03-11T22:13:21.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:21 smithi045 ceph-mon[39715]: 3.0 scrub ok 2024-03-11T22:13:21.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:21 smithi045 ceph-mon[39715]: 5.a scrub starts 2024-03-11T22:13:21.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:21 smithi045 ceph-mon[39715]: 5.a scrub ok 2024-03-11T22:13:21.327 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:21 smithi045 ceph-mon[39715]: pgmap v181: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T22:13:21.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:21 smithi045 ceph-mon[30178]: 7.f scrub ok 2024-03-11T22:13:21.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:21 smithi045 ceph-mon[30178]: 3.0 scrub starts 2024-03-11T22:13:21.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:21 smithi045 ceph-mon[30178]: 3.0 scrub ok 2024-03-11T22:13:21.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:21 smithi045 ceph-mon[30178]: 5.a scrub starts 2024-03-11T22:13:21.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:21 smithi045 ceph-mon[30178]: 5.a scrub ok 2024-03-11T22:13:21.329 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:21 smithi045 ceph-mon[30178]: pgmap v181: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T22:13:21.336 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:21 smithi144 ceph-mon[33984]: 7.f scrub ok 2024-03-11T22:13:21.336 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:21 smithi144 ceph-mon[33984]: 3.0 scrub starts 2024-03-11T22:13:21.336 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:21 smithi144 ceph-mon[33984]: 3.0 scrub ok 2024-03-11T22:13:21.336 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:21 smithi144 ceph-mon[33984]: 5.a scrub starts 2024-03-11T22:13:21.336 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:21 smithi144 ceph-mon[33984]: 5.a scrub ok 2024-03-11T22:13:21.336 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:21 smithi144 ceph-mon[33984]: pgmap v181: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T22:13:21.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:21.422 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:21.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:21.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:21.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:21.577 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:21.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:21.678 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:21.712 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:21.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:21.759 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T22:13:21.813 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:21.857 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:21.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:21.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:21.959 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:21.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:21.992 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:21.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:22.066 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:13:22.093 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:22.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:22.127 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:22.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:22.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:22.276 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:22.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:22.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:22.378 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:22.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:22.403 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:22.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:22.426 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:22 smithi144 ceph-mon[33984]: 6.1 scrub starts 2024-03-11T22:13:22.427 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:22 smithi144 ceph-mon[33984]: 6.1 scrub ok 2024-03-11T22:13:22.427 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:22 smithi144 ceph-mon[33984]: 6.1c scrub starts 2024-03-11T22:13:22.427 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:22 smithi144 ceph-mon[33984]: 6.1c scrub ok 2024-03-11T22:13:22.427 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:22 smithi144 ceph-mon[33984]: 6.5 scrub starts 2024-03-11T22:13:22.427 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:22 smithi144 ceph-mon[33984]: 6.5 scrub ok 2024-03-11T22:13:22.504 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:22.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:22 smithi045 ceph-mon[30178]: 6.1 scrub starts 2024-03-11T22:13:22.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:22 smithi045 ceph-mon[30178]: 6.1 scrub ok 2024-03-11T22:13:22.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:22 smithi045 ceph-mon[30178]: 6.1c scrub starts 2024-03-11T22:13:22.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:22 smithi045 ceph-mon[30178]: 6.1c scrub ok 2024-03-11T22:13:22.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:22 smithi045 ceph-mon[30178]: 6.5 scrub starts 2024-03-11T22:13:22.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:22 smithi045 ceph-mon[30178]: 6.5 scrub ok 2024-03-11T22:13:22.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:22 smithi045 ceph-mon[39715]: 6.1 scrub starts 2024-03-11T22:13:22.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:22 smithi045 ceph-mon[39715]: 6.1 scrub ok 2024-03-11T22:13:22.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:22 smithi045 ceph-mon[39715]: 6.1c scrub starts 2024-03-11T22:13:22.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:22 smithi045 ceph-mon[39715]: 6.1c scrub ok 2024-03-11T22:13:22.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:22 smithi045 ceph-mon[39715]: 6.5 scrub starts 2024-03-11T22:13:22.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:22 smithi045 ceph-mon[39715]: 6.5 scrub ok 2024-03-11T22:13:22.532 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:22.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:22.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:22.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:22.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:22.680 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:22.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:22.781 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:22.812 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:22.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:22.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:22.951 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:22.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:22.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:23.028 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:13:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:13:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:13:23.053 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:23.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:23.082 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:23.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:23.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:23.224 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:23.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:23.326 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:23.357 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:23.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:23.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:23 smithi045 ceph-mon[39715]: 3.1a scrub starts 2024-03-11T22:13:23.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:23 smithi045 ceph-mon[39715]: 3.1a scrub ok 2024-03-11T22:13:23.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:23 smithi045 ceph-mon[39715]: pgmap v182: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 982 op/s 2024-03-11T22:13:23.448 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:23 smithi045 ceph-mon[30178]: 3.1a scrub starts 2024-03-11T22:13:23.448 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:23 smithi045 ceph-mon[30178]: 3.1a scrub ok 2024-03-11T22:13:23.448 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:23 smithi045 ceph-mon[30178]: pgmap v182: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 982 op/s 2024-03-11T22:13:23.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:23.463 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:23 smithi144 ceph-mon[33984]: 3.1a scrub starts 2024-03-11T22:13:23.464 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:23 smithi144 ceph-mon[33984]: 3.1a scrub ok 2024-03-11T22:13:23.464 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:23 smithi144 ceph-mon[33984]: pgmap v182: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 982 op/s 2024-03-11T22:13:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:23.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:23.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:23.602 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:23.631 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:23.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:23.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:23.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:23.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:23.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:23.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:23.892 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:23.920 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:23.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:24.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:24.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:24.129 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:13:24.129 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-03-11T22:13:24.129 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-03-11T22:13:24.129 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-03-11T22:13:24.130 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:13:24.161 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:24.193 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:24.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:24.295 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:24.323 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:24.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:24.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:24.486 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:24.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:24.586 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:24.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:24.616 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:24.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:24.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:24.759 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:24.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:24.814 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T22:13:24.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:24.907 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:24.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:25.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:25.062 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:25.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:25.134 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:13:25.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:25.206 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:25.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:25.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:25.308 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:25.337 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:25.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:25.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:25.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:25.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:25.488 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:25.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:25.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:25 smithi045 ceph-mon[30178]: 7.9 scrub starts 2024-03-11T22:13:25.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:25 smithi045 ceph-mon[30178]: 3.f scrub starts 2024-03-11T22:13:25.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:25 smithi045 ceph-mon[30178]: 3.f scrub ok 2024-03-11T22:13:25.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:25 smithi045 ceph-mon[30178]: pgmap v183: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 982 op/s 2024-03-11T22:13:25.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:25 smithi045 ceph-mon[39715]: 7.9 scrub starts 2024-03-11T22:13:25.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:25 smithi045 ceph-mon[39715]: 3.f scrub starts 2024-03-11T22:13:25.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:25 smithi045 ceph-mon[39715]: 3.f scrub ok 2024-03-11T22:13:25.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:25 smithi045 ceph-mon[39715]: pgmap v183: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 982 op/s 2024-03-11T22:13:25.545 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:25 smithi144 ceph-mon[33984]: 7.9 scrub starts 2024-03-11T22:13:25.546 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:25 smithi144 ceph-mon[33984]: 3.f scrub starts 2024-03-11T22:13:25.546 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:25 smithi144 ceph-mon[33984]: 3.f scrub ok 2024-03-11T22:13:25.546 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:25 smithi144 ceph-mon[33984]: pgmap v183: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 982 op/s 2024-03-11T22:13:25.579 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:25.589 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:25.617 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:25.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:25.718 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:25.744 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:25.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:25.846 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:25.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:25.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:25.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:26.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:26.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:26.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:26.124 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:26.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:26.155 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:26.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:26.201 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:13:26.203 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:13:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:13:26.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:26.301 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:26.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:26.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:26.403 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:26.436 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:26.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[39715]: 4.7 scrub starts 2024-03-11T22:13:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[39715]: 4.7 scrub ok 2024-03-11T22:13:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[39715]: 5.b scrub starts 2024-03-11T22:13:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[39715]: 5.b scrub ok 2024-03-11T22:13:26.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[39715]: 6.18 scrub starts 2024-03-11T22:13:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[39715]: 6.18 scrub ok 2024-03-11T22:13:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[39715]: 7.9 scrub ok 2024-03-11T22:13:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[39715]: 5.16 scrub starts 2024-03-11T22:13:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[39715]: 5.16 scrub ok 2024-03-11T22:13:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3494642408' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:26.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:26.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[30178]: 4.7 scrub starts 2024-03-11T22:13:26.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[30178]: 4.7 scrub ok 2024-03-11T22:13:26.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[30178]: 5.b scrub starts 2024-03-11T22:13:26.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[30178]: 5.b scrub ok 2024-03-11T22:13:26.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[30178]: 6.18 scrub starts 2024-03-11T22:13:26.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[30178]: 6.18 scrub ok 2024-03-11T22:13:26.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[30178]: 7.9 scrub ok 2024-03-11T22:13:26.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[30178]: 5.16 scrub starts 2024-03-11T22:13:26.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[30178]: 5.16 scrub ok 2024-03-11T22:13:26.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3494642408' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:26.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:26 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:26.538 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:26.569 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:26.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:26.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:26 smithi144 ceph-mon[33984]: 4.7 scrub starts 2024-03-11T22:13:26.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:26 smithi144 ceph-mon[33984]: 4.7 scrub ok 2024-03-11T22:13:26.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:26 smithi144 ceph-mon[33984]: 5.b scrub starts 2024-03-11T22:13:26.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:26 smithi144 ceph-mon[33984]: 5.b scrub ok 2024-03-11T22:13:26.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:26 smithi144 ceph-mon[33984]: 6.18 scrub starts 2024-03-11T22:13:26.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:26 smithi144 ceph-mon[33984]: 6.18 scrub ok 2024-03-11T22:13:26.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:26 smithi144 ceph-mon[33984]: 7.9 scrub ok 2024-03-11T22:13:26.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:26 smithi144 ceph-mon[33984]: 5.16 scrub starts 2024-03-11T22:13:26.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:26 smithi144 ceph-mon[33984]: 5.16 scrub ok 2024-03-11T22:13:26.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:26 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3494642408' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:26.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:26 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:26.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:26.718 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:26.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:26.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:26.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:26.846 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:26.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:26.948 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:26.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:26.982 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:26.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:27.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:27.103 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:13:27.103 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-03-11T22:13:27.103 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-03-11T22:13:27.103 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:13:27.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.103 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24335.0:1130 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94036442452352 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-03-11T22:13:27.103 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:34.398788+0000", 2024-03-11T22:13:27.103 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 172.704495702, 2024-03-11T22:13:27.103 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.019798728, 2024-03-11T22:13:27.104 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.104 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:13:27.104 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.104 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24335", 2024-03-11T22:13:27.104 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/3033943457", 2024-03-11T22:13:27.104 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 1130 2024-03-11T22:13:27.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.104 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.104 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398788+0000", 2024-03-11T22:13:27.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398788+0000", 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398791+0000", 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2590000000000001e-06 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398805+0000", 2024-03-11T22:13:27.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3721e-05 2024-03-11T22:13:27.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.106 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.106 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398835+0000", 2024-03-11T22:13:27.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0238000000000001e-05 2024-03-11T22:13:27.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.106 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.106 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398851+0000", 2024-03-11T22:13:27.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5838999999999999e-05 2024-03-11T22:13:27.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.106 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398945+0000", 2024-03-11T22:13:27.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3986000000000004e-05 2024-03-11T22:13:27.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-03-11T22:13:27.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398953+0000", 2024-03-11T22:13:27.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9119999999999998e-06 2024-03-11T22:13:27.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.406225+0000", 2024-03-11T22:13:27.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0072720850000000002 2024-03-11T22:13:27.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.406238+0000", 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2743e-05 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:35.414245+0000", 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.008007466 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:35.414414+0000", 2024-03-11T22:13:27.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016910300000000001 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:35.418587+0000", 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0041723760000000002 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:26438 7.7 7:e97d8559:::benchmark_data_smithi045_83007_object26437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:54.971431+0000", 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 152.131853234, 2024-03-11T22:13:27.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.15983009400000001, 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 26438 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971431+0000", 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971431+0000", 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971432+0000", 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.265e-06 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971483+0000", 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1215999999999999e-05 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971499+0000", 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5633000000000001e-05 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971507+0000", 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8539999999999997e-06 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971568+0000", 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1066000000000001e-05 2024-03-11T22:13:27.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971680+0000", 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000112273 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971792+0000", 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011223200000000001 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.972903+0000", 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001111293 2024-03-11T22:13:27.113 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.975356+0000", 2024-03-11T22:13:27.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024528470000000002 2024-03-11T22:13:27.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.131147+0000", 2024-03-11T22:13:27.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.155790505 2024-03-11T22:13:27.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.131211+0000", 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3792000000000002e-05 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.131261+0000", 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0117999999999997e-05 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:27627 7.c 7:3bf1c813:::benchmark_data_smithi045_83007_object27626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:55.718204+0000", 2024-03-11T22:13:27.115 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 151.38508023599999, 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13720726899999999, 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 27627 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718204+0000", 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.117 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.117 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.118 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718204+0000", 2024-03-11T22:13:27.118 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.118 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.118 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.118 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718205+0000", 2024-03-11T22:13:27.118 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6050000000000001e-06 2024-03-11T22:13:27.118 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.118 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.118 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718251+0000", 2024-03-11T22:13:27.118 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6177999999999997e-05 2024-03-11T22:13:27.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.119 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.119 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718265+0000", 2024-03-11T22:13:27.119 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3200999999999999e-05 2024-03-11T22:13:27.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.119 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.119 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718272+0000", 2024-03-11T22:13:27.119 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0539999999999997e-06 2024-03-11T22:13:27.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.119 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718328+0000", 2024-03-11T22:13:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6705e-05 2024-03-11T22:13:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718465+0000", 2024-03-11T22:13:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013699599999999999 2024-03-11T22:13:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:13:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718557+0000", 2024-03-11T22:13:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1218999999999995e-05 2024-03-11T22:13:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.719504+0000", 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00094716799999999997 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.719651+0000", 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000147496 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.855305+0000", 2024-03-11T22:13:27.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13565408200000001 2024-03-11T22:13:27.122 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.122 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.122 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.122 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.855364+0000", 2024-03-11T22:13:27.122 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8455999999999997e-05 2024-03-11T22:13:27.122 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.122 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.122 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.122 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.855411+0000", 2024-03-11T22:13:27.122 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7108999999999999e-05 2024-03-11T22:13:27.122 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.122 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.122 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:28077 7.c 7:32c5189f:::benchmark_data_smithi045_83007_object28076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:56.079615+0000", 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 151.02366857000001, 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13117891700000001, 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 28077 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.123 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079615+0000", 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079615+0000", 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079617+0000", 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4440000000000001e-06 2024-03-11T22:13:27.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079648+0000", 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1707e-05 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079660+0000", 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1491999999999999e-05 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079668+0000", 2024-03-11T22:13:27.125 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1869999999999997e-06 2024-03-11T22:13:27.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.126 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.126 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079717+0000", 2024-03-11T22:13:27.126 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9237999999999999e-05 2024-03-11T22:13:27.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.132 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.132 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.133 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079871+0000", 2024-03-11T22:13:27.133 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015396299999999999 2024-03-11T22:13:27.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.133 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.133 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:13:27.133 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.080033+0000", 2024-03-11T22:13:27.133 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016142199999999999 2024-03-11T22:13:27.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.133 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.133 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.133 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.081317+0000", 2024-03-11T22:13:27.133 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001284232 2024-03-11T22:13:27.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.134 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.134 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.134 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.081507+0000", 2024-03-11T22:13:27.134 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018976400000000001 2024-03-11T22:13:27.134 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.134 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.135 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.135 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210699+0000", 2024-03-11T22:13:27.135 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.12919265399999999 2024-03-11T22:13:27.135 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.135 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.135 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.135 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210746+0000", 2024-03-11T22:13:27.135 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6906000000000001e-05 2024-03-11T22:13:27.135 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.135 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.135 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.135 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210794+0000", 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7908000000000003e-05 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:28080 7.7 7:eb74b751:::benchmark_data_smithi045_83007_object28079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:56.079682+0000", 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 151.023602033, 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.131107114, 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.136 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 28080 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079682+0000", 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079682+0000", 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079682+0000", 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0200000000000002e-07 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079704+0000", 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1733e-05 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079711+0000", 2024-03-11T22:13:27.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0609999999999998e-06 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079718+0000", 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3080000000000002e-06 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.080007+0000", 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00028894499999999998 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.139 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.080112+0000", 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000104268 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.080196+0000", 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4815999999999996e-05 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.081217+0000", 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010204439999999999 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.140 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.208091+0000", 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.12687417200000001 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210683+0000", 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0025918550000000001 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210746+0000", 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2639999999999997e-05 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.141 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.142 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210789+0000", 2024-03-11T22:13:27.142 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.337e-05 2024-03-11T22:13:27.142 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.142 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.142 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.142 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.142 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:28341 7.d 7:b4c93f93:::benchmark_data_smithi045_83007_object28340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:13:27.142 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:56.327641+0000", 2024-03-11T22:13:27.142 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 150.77564262499999, 2024-03-11T22:13:27.142 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098652376999999999, 2024-03-11T22:13:27.142 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.142 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 28341 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.327641+0000", 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.143 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.327641+0000", 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.327642+0000", 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2900000000000002e-07 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.327720+0000", 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8009000000000005e-05 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.144 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.145 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.327733+0000", 2024-03-11T22:13:27.145 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3004e-05 2024-03-11T22:13:27.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.145 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.327744+0000", 2024-03-11T22:13:27.145 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1116e-05 2024-03-11T22:13:27.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.145 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.327783+0000", 2024-03-11T22:13:27.145 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8401e-05 2024-03-11T22:13:27.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.146 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.146 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.327868+0000", 2024-03-11T22:13:27.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5698999999999994e-05 2024-03-11T22:13:27.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.146 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:13:27.146 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.327959+0000", 2024-03-11T22:13:27.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.0519000000000005e-05 2024-03-11T22:13:27.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.146 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.146 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.329120+0000", 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001161051 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.329914+0000", 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00079431800000000002 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.426218+0000", 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.096303888000000004 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.147 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.426262+0000", 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4264000000000002e-05 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.426294+0000", 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1279000000000003e-05 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:28350 7.d 7:bdc6b5c5:::benchmark_data_smithi045_83007_object28349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:56.330684+0000", 2024-03-11T22:13:27.148 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 150.77259984, 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098797530999999994, 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 28350 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330684+0000", 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330684+0000", 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330686+0000", 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7570000000000001e-06 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330836+0000", 2024-03-11T22:13:27.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000150371 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330851+0000", 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5087e-05 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330859+0000", 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4309999999999997e-06 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.151 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330915+0000", 2024-03-11T22:13:27.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6567000000000002e-05 2024-03-11T22:13:27.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.152 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.152 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.152 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.331043+0000", 2024-03-11T22:13:27.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012755000000000001 2024-03-11T22:13:27.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.152 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.152 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:13:27.152 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.331184+0000", 2024-03-11T22:13:27.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000140903 2024-03-11T22:13:27.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.152 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.153 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.153 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.332243+0000", 2024-03-11T22:13:27.153 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010596850000000001 2024-03-11T22:13:27.153 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.153 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.153 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.153 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.332506+0000", 2024-03-11T22:13:27.153 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026266100000000003 2024-03-11T22:13:27.153 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.153 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.153 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.153 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.429418+0000", 2024-03-11T22:13:27.153 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.096912015000000004 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.429455+0000", 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6711999999999997e-05 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.429482+0000", 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6792e-05 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.155 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.155 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:41624 7.d 7:bc5fdf06:::benchmark_data_smithi045_83007_object41623:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:13:27.155 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.868679+0000", 2024-03-11T22:13:27.155 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 143.234604405, 2024-03-11T22:13:27.155 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.112665528, 2024-03-11T22:13:27.155 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.155 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.155 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.156 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.156 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.156 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 41624 2024-03-11T22:13:27.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.156 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.156 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.156 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868679+0000", 2024-03-11T22:13:27.156 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.157 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.157 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.157 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868679+0000", 2024-03-11T22:13:27.157 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.157 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.157 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.157 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.157 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868680+0000", 2024-03-11T22:13:27.157 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6499999999999999e-07 2024-03-11T22:13:27.157 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.157 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.157 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.158 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868724+0000", 2024-03-11T22:13:27.158 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4027999999999999e-05 2024-03-11T22:13:27.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.158 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.158 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.158 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868732+0000", 2024-03-11T22:13:27.158 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6240000000000002e-06 2024-03-11T22:13:27.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.158 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.158 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.158 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868741+0000", 2024-03-11T22:13:27.158 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8060000000000001e-06 2024-03-11T22:13:27.159 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.159 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.159 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.159 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868771+0000", 2024-03-11T22:13:27.159 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0066e-05 2024-03-11T22:13:27.159 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.159 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.159 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.160 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868871+0000", 2024-03-11T22:13:27.160 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010005600000000001 2024-03-11T22:13:27.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.160 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:13:27.160 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868991+0000", 2024-03-11T22:13:27.160 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000120525 2024-03-11T22:13:27.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.160 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.161 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981178+0000", 2024-03-11T22:13:27.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11218715 2024-03-11T22:13:27.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.161 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.161 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.161 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981193+0000", 2024-03-11T22:13:27.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5003e-05 2024-03-11T22:13:27.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.161 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.161 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981229+0000", 2024-03-11T22:13:27.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5941e-05 2024-03-11T22:13:27.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981306+0000", 2024-03-11T22:13:27.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6853000000000004e-05 2024-03-11T22:13:27.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981345+0000", 2024-03-11T22:13:27.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8810999999999998e-05 2024-03-11T22:13:27.162 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.163 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.163 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.163 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.163 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:41628 7.d 7:b3acd120:::benchmark_data_smithi045_83007_object41627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:13:27.163 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.868789+0000", 2024-03-11T22:13:27.163 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 143.23449476600001, 2024-03-11T22:13:27.163 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11711809199999999, 2024-03-11T22:13:27.163 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.163 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.164 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.164 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.164 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.164 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 41628 2024-03-11T22:13:27.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.164 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.164 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868789+0000", 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868789+0000", 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868790+0000", 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9800000000000004e-07 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868858+0000", 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8860000000000001e-05 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868864+0000", 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2249999999999999e-06 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868867+0000", 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.613e-06 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.166 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.870148+0000", 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0012809970000000001 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.870273+0000", 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012430699999999999 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.979501+0000", 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.109228742 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.168 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.168 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.985659+0000", 2024-03-11T22:13:27.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0061581730000000003 2024-03-11T22:13:27.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.168 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.168 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.985719+0000", 2024-03-11T22:13:27.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9308999999999997e-05 2024-03-11T22:13:27.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.985807+0000", 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7916000000000003e-05 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.985864+0000", 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7082e-05 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.985907+0000", 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.337e-05 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.171 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:41629 7.c 7:386f5e1e:::benchmark_data_smithi045_83007_object41628:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.868874+0000", 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 143.23441001, 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.112468074, 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 41629 2024-03-11T22:13:27.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.173 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868874+0000", 2024-03-11T22:13:27.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868874+0000", 2024-03-11T22:13:27.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.176 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.176 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868874+0000", 2024-03-11T22:13:27.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4200000000000002e-07 2024-03-11T22:13:27.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.176 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.176 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868904+0000", 2024-03-11T22:13:27.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9839999999999999e-05 2024-03-11T22:13:27.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868908+0000", 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4379999999999997e-06 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868915+0000", 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5169999999999996e-06 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868960+0000", 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4909e-05 2024-03-11T22:13:27.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.869049+0000", 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8858999999999997e-05 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.869144+0000", 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.5629000000000002e-05 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.979159+0000", 2024-03-11T22:13:27.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.110014764 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.979730+0000", 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00057078200000000004 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981217+0000", 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001486996 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981290+0000", 2024-03-11T22:13:27.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3380999999999994e-05 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981342+0000", 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1616999999999997e-05 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:41626 7.b e109/99 7:d90bee90:::benchmark_data_smithi045_83007_object41625:head v 109'2562, mlcod=109'2560)", 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.869598+0000", 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 143.23368539099999, 2024-03-11T22:13:27.181 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.109763785, 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.869598+0000", 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.869598+0000", 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.869599+0000", 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3e-07 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.869673+0000", 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4277999999999996e-05 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.869683+0000", 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0139000000000001e-05 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.869690+0000", 2024-03-11T22:13:27.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4590000000000003e-06 2024-03-11T22:13:27.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.869744+0000", 2024-03-11T22:13:27.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4548e-05 2024-03-11T22:13:27.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.869765+0000", 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0129e-05 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.979198+0000", 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10943383 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.979362+0000", 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016377199999999999 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.185 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop_reply(client.14790.0:41629 7.c e109/99 ondisk, result = 0)", 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.870636+0000", 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 143.23264796699999, 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10910194099999999, 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.870636+0000", 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.186 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.870636+0000", 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.870637+0000", 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5799999999999999e-07 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.870640+0000", 2024-03-11T22:13:27.187 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5389999999999999e-06 2024-03-11T22:13:27.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.188 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.188 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.188 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.979574+0000", 2024-03-11T22:13:27.188 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.108933891 2024-03-11T22:13:27.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.188 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.188 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.188 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.979585+0000", 2024-03-11T22:13:27.188 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0693000000000001e-05 2024-03-11T22:13:27.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.188 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.188 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.979717+0000", 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000131987 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.979724+0000", 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6190000000000002e-06 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.979738+0000", 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3553999999999999e-05 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.189 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.190 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.190 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop_reply(client.14790.0:41624 7.d e109/99 ondisk, result = 0)", 2024-03-11T22:13:27.191 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.874698+0000", 2024-03-11T22:13:27.192 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 143.22858602400001, 2024-03-11T22:13:27.192 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106496203, 2024-03-11T22:13:27.192 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.192 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:13:27.192 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.192 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.192 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.192 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.874698+0000", 2024-03-11T22:13:27.192 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.192 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.192 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.192 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.192 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.874698+0000", 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.874698+0000", 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4899999999999995e-07 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.874702+0000", 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.63e-06 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.193 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.980823+0000", 2024-03-11T22:13:27.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10612107799999999 2024-03-11T22:13:27.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.980833+0000", 2024-03-11T22:13:27.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3470000000000005e-06 2024-03-11T22:13:27.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981166+0000", 2024-03-11T22:13:27.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00033305900000000002 2024-03-11T22:13:27.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981172+0000", 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9110000000000004e-06 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981194+0000", 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1529e-05 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.195 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:41645 7.1 e109/99 7:889a6bdd:::benchmark_data_smithi045_83007_object41644:head v 109'2655, mlcod=109'2653)", 2024-03-11T22:13:27.196 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.883384+0000", 2024-03-11T22:13:27.196 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 143.21989986400001, 2024-03-11T22:13:27.196 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100690614, 2024-03-11T22:13:27.196 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.196 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.196 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.196 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.196 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.196 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.883384+0000", 2024-03-11T22:13:27.196 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.196 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.196 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.197 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.197 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.883384+0000", 2024-03-11T22:13:27.197 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.197 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.198 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.198 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.883385+0000", 2024-03-11T22:13:27.198 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3400000000000001e-06 2024-03-11T22:13:27.198 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.198 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.198 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.198 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.883417+0000", 2024-03-11T22:13:27.198 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1510999999999997e-05 2024-03-11T22:13:27.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.199 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.980688+0000", 2024-03-11T22:13:27.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.097271195000000005 2024-03-11T22:13:27.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.199 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.980701+0000", 2024-03-11T22:13:27.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3261999999999999e-05 2024-03-11T22:13:27.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.200 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.200 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.200 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.980760+0000", 2024-03-11T22:13:27.200 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8603000000000001e-05 2024-03-11T22:13:27.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.200 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.200 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.200 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.980781+0000", 2024-03-11T22:13:27.200 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0716999999999998e-05 2024-03-11T22:13:27.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.200 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.200 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.200 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.983821+0000", 2024-03-11T22:13:27.201 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0030408750000000002 2024-03-11T22:13:27.201 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.201 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.201 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.201 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.984075+0000", 2024-03-11T22:13:27.201 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00025311099999999998 2024-03-11T22:13:27.201 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.201 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.201 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.201 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.201 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.201 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:141381 7.c 7:355201b1:::benchmark_data_smithi045_83007_object22910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:13:27.202 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:37.300554+0000", 2024-03-11T22:13:27.202 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 49.802729481999997, 2024-03-11T22:13:27.202 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.109008169, 2024-03-11T22:13:27.202 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.202 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.202 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.202 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.202 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.202 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 141381 2024-03-11T22:13:27.202 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.202 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.202 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.203 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.203 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.300554+0000", 2024-03-11T22:13:27.203 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.203 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.203 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.203 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.203 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.300554+0000", 2024-03-11T22:13:27.203 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.203 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.203 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.203 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.203 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.300555+0000", 2024-03-11T22:13:27.203 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0020000000000001e-06 2024-03-11T22:13:27.205 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.205 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.205 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.205 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.300560+0000", 2024-03-11T22:13:27.205 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7350000000000004e-06 2024-03-11T22:13:27.205 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.205 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.205 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.205 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.300570+0000", 2024-03-11T22:13:27.205 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8940000000000005e-06 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.300575+0000", 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3750000000000002e-06 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.300615+0000", 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0080999999999998e-05 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.206 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.300724+0000", 2024-03-11T22:13:27.207 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010840699999999999 2024-03-11T22:13:27.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.207 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.207 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:13:27.207 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.300833+0000", 2024-03-11T22:13:27.207 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000109172 2024-03-11T22:13:27.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.207 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.207 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.207 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.303600+0000", 2024-03-11T22:13:27.207 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0027673060000000002 2024-03-11T22:13:27.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.207 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.303662+0000", 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2142000000000002e-05 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.409379+0000", 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.105716147 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.409476+0000", 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7069000000000002e-05 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.409563+0000", 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6839e-05 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:141386 7.c 7:3f6866e0:::benchmark_data_smithi045_83007_object22915:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:37.304655+0000", 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 49.798628895, 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.112149399, 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.209 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 141386 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304655+0000", 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304655+0000", 2024-03-11T22:13:27.210 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304656+0000", 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3200000000000001e-06 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304660+0000", 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2080000000000002e-06 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.211 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304672+0000", 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1901e-05 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304679+0000", 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1069999999999999e-06 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304726+0000", 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6161000000000001e-05 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.212 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304813+0000", 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7842000000000004e-05 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304887+0000", 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3063000000000002e-05 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.308138+0000", 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003251463 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.213 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.308322+0000", 2024-03-11T22:13:27.214 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000184146 2024-03-11T22:13:27.214 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.214 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.214 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.214 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.416735+0000", 2024-03-11T22:13:27.214 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.108413038 2024-03-11T22:13:27.214 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.214 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.214 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.214 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.416771+0000", 2024-03-11T22:13:27.214 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6125999999999999e-05 2024-03-11T22:13:27.214 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.416804+0000", 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3024000000000001e-05 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:184406 7.d 7:b4edf805:::benchmark_data_smithi045_83007_object65935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:20.569377+0000", 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.5339069629999997, 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101832279, 2024-03-11T22:13:27.215 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 184406 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569377+0000", 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.216 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569377+0000", 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569379+0000", 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8619999999999999e-06 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569384+0000", 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5600000000000001e-06 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.217 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.218 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569398+0000", 2024-03-11T22:13:27.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3377000000000001e-05 2024-03-11T22:13:27.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.218 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.218 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569405+0000", 2024-03-11T22:13:27.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0550000000000002e-06 2024-03-11T22:13:27.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.218 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.218 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569474+0000", 2024-03-11T22:13:27.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9307999999999994e-05 2024-03-11T22:13:27.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569610+0000", 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013600799999999999 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569745+0000", 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013532400000000001 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.576008+0000", 2024-03-11T22:13:27.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0062627530000000002 2024-03-11T22:13:27.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.220 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.220 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.220 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.580795+0000", 2024-03-11T22:13:27.220 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.004786467 2024-03-11T22:13:27.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.220 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.220 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.220 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.671103+0000", 2024-03-11T22:13:27.220 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.090308699000000006 2024-03-11T22:13:27.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.221 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.221 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.671161+0000", 2024-03-11T22:13:27.221 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7791000000000003e-05 2024-03-11T22:13:27.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.221 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.221 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.671209+0000", 2024-03-11T22:13:27.221 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8075000000000003e-05 2024-03-11T22:13:27.221 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.221 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.221 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:184410 7.d 7:ba907d1b:::benchmark_data_smithi045_83007_object65939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:20.569721+0000", 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 6.5335628559999996, 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10622037500000001, 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 184410 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569721+0000", 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569721+0000", 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569722+0000", 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1099999999999995e-07 2024-03-11T22:13:27.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.224 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.224 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.224 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569725+0000", 2024-03-11T22:13:27.224 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2660000000000002e-06 2024-03-11T22:13:27.224 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.224 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.224 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569733+0000", 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7710000000000006e-06 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.569738+0000", 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9110000000000001e-06 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.572824+0000", 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0030867199999999998 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.225 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.226 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.572941+0000", 2024-03-11T22:13:27.226 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011680000000000001 2024-03-11T22:13:27.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.226 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:13:27.226 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.573061+0000", 2024-03-11T22:13:27.226 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000120307 2024-03-11T22:13:27.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.226 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.226 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.576093+0000", 2024-03-11T22:13:27.226 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0030314859999999999 2024-03-11T22:13:27.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.580911+0000", 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0048180269999999999 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.675820+0000", 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.094908893999999994 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.675887+0000", 2024-03-11T22:13:27.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7435999999999996e-05 2024-03-11T22:13:27.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.228 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.228 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.675941+0000", 2024-03-11T22:13:27.228 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4146e-05 2024-03-11T22:13:27.228 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.228 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.228 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.228 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:185992 7.d 7:bc7848b9:::benchmark_data_smithi045_83007_object67521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T22:13:27.228 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:22.315847+0000", 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.7874371450000002, 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10249235199999999, 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 185992 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.315847+0000", 2024-03-11T22:13:27.229 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.230 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.230 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.315847+0000", 2024-03-11T22:13:27.230 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.230 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.230 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.315847+0000", 2024-03-11T22:13:27.230 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8599999999999999e-07 2024-03-11T22:13:27.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.315849+0000", 2024-03-11T22:13:27.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.573e-06 2024-03-11T22:13:27.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.232 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.315852+0000", 2024-03-11T22:13:27.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9299999999999996e-06 2024-03-11T22:13:27.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.232 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.232 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.315856+0000", 2024-03-11T22:13:27.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.608e-06 2024-03-11T22:13:27.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.232 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.232 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.315897+0000", 2024-03-11T22:13:27.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1028000000000001e-05 2024-03-11T22:13:27.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.315977+0000", 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0187999999999999e-05 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.316058+0000", 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0618999999999995e-05 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.322436+0000", 2024-03-11T22:13:27.233 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0063783210000000002 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.322493+0000", 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.677e-05 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.418213+0000", 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.095720350999999995 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.418278+0000", 2024-03-11T22:13:27.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4871999999999998e-05 2024-03-11T22:13:27.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.235 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.235 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.418339+0000", 2024-03-11T22:13:27.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0806e-05 2024-03-11T22:13:27.235 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.235 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.235 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.236 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:185996 7.7 7:ef067263:::benchmark_data_smithi045_83007_object67525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T22:13:27.236 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:22.317287+0000", 2024-03-11T22:13:27.236 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 4.785996742, 2024-03-11T22:13:27.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11014377, 2024-03-11T22:13:27.236 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:27.236 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:27.236 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:13:27.236 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:13:27.236 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:27.236 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 185996 2024-03-11T22:13:27.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.237 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:27.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:27.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317287+0000", 2024-03-11T22:13:27.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:27.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317287+0000", 2024-03-11T22:13:27.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:27.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:27.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317288+0000", 2024-03-11T22:13:27.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6400000000000001e-07 2024-03-11T22:13:27.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:27.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317292+0000", 2024-03-11T22:13:27.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.566e-06 2024-03-11T22:13:27.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.238 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:27.238 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317307+0000", 2024-03-11T22:13:27.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4748e-05 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317315+0000", 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3830000000000003e-06 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317372+0000", 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7550000000000003e-05 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317605+0000", 2024-03-11T22:13:27.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00023298599999999999 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.320901+0000", 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003296284 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.323882+0000", 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0029803780000000001 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.374715+0000", 2024-03-11T22:13:27.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.050833138 2024-03-11T22:13:27.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.241 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:27.241 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.427331+0000", 2024-03-11T22:13:27.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.052615833000000001 2024-03-11T22:13:27.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.241 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:27.241 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.427383+0000", 2024-03-11T22:13:27.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2349999999999999e-05 2024-03-11T22:13:27.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:27.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:27.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:27.242 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.427431+0000", 2024-03-11T22:13:27.242 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7790000000000002e-05 2024-03-11T22:13:27.242 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.242 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.242 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.242 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:27.242 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:27.242 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:13:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:27.243 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:27.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:27.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:27.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:27.385 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:27.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:27.489 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:27.511 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:27.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[39715]: pgmap v184: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T22:13:27.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:13:27.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:13:27.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:13:27.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[39715]: osdmap e142: 8 total, 8 up, 8 in 2024-03-11T22:13:27.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[39715]: 3.7 scrub starts 2024-03-11T22:13:27.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[39715]: 3.7 scrub ok 2024-03-11T22:13:27.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:27.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[30178]: pgmap v184: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T22:13:27.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:13:27.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:13:27.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:13:27.513 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[30178]: osdmap e142: 8 total, 8 up, 8 in 2024-03-11T22:13:27.513 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[30178]: 3.7 scrub starts 2024-03-11T22:13:27.513 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:27 smithi045 ceph-mon[30178]: 3.7 scrub ok 2024-03-11T22:13:27.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:27.529 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:27.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:27.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:27.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:27 smithi144 ceph-mon[33984]: pgmap v184: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T22:13:27.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:27 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:13:27.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:27 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:13:27.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:27 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:13:27.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:27 smithi144 ceph-mon[33984]: osdmap e142: 8 total, 8 up, 8 in 2024-03-11T22:13:27.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:27 smithi144 ceph-mon[33984]: 3.7 scrub starts 2024-03-11T22:13:27.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:27 smithi144 ceph-mon[33984]: 3.7 scrub ok 2024-03-11T22:13:27.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:27.677 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:27.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:27.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:27.822 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:27.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:27.927 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:27.957 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:27.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:28.059 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:28.104 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:28.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:28.158 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T22:13:28.206 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:28.210 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:13:28.210 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:13:28.223 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:13:28.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:28.239 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:28.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:28.264 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:13:28.305 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:13:28.340 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:28.346 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:13:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:28.378 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:28.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:28.392 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:13:28.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:28 smithi045 ceph-mon[30178]: 4.3 scrub starts 2024-03-11T22:13:28.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:28 smithi045 ceph-mon[30178]: 4.3 scrub ok 2024-03-11T22:13:28.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:28 smithi045 ceph-mon[30178]: 4.1 scrub starts 2024-03-11T22:13:28.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:28 smithi045 ceph-mon[30178]: 4.1 scrub ok 2024-03-11T22:13:28.393 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:28 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2313196460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:28.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:28 smithi045 ceph-mon[39715]: 4.3 scrub starts 2024-03-11T22:13:28.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:28 smithi045 ceph-mon[39715]: 4.3 scrub ok 2024-03-11T22:13:28.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:28 smithi045 ceph-mon[39715]: 4.1 scrub starts 2024-03-11T22:13:28.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:28 smithi045 ceph-mon[39715]: 4.1 scrub ok 2024-03-11T22:13:28.393 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:28 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2313196460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:28.431 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:13:28.437 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:13:28.437 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:13:28.438 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:13:28.438 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:13:28.438 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:13:28.438 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:13:28.438 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:13:28.438 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:13:28.473 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:13:28.479 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:28.509 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:28.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:28.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:28 smithi144 ceph-mon[33984]: 4.3 scrub starts 2024-03-11T22:13:28.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:28 smithi144 ceph-mon[33984]: 4.3 scrub ok 2024-03-11T22:13:28.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:28 smithi144 ceph-mon[33984]: 4.1 scrub starts 2024-03-11T22:13:28.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:28 smithi144 ceph-mon[33984]: 4.1 scrub ok 2024-03-11T22:13:28.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:28 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2313196460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:28.611 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:28.640 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:28.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:28.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:28.780 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:28.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:28.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:28.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:28.930 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:28.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:29.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:29.081 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:29.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:29.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:29.225 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:29.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:29.328 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:29.355 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:29.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:29.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:29.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:29.502 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:29 smithi144 ceph-mon[33984]: 6.12 scrub starts 2024-03-11T22:13:29.502 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:29 smithi144 ceph-mon[33984]: 6.12 scrub ok 2024-03-11T22:13:29.502 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:29 smithi144 ceph-mon[33984]: 3.3 scrub starts 2024-03-11T22:13:29.502 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:29 smithi144 ceph-mon[33984]: 3.3 scrub ok 2024-03-11T22:13:29.502 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:29 smithi144 ceph-mon[33984]: 3.10 scrub starts 2024-03-11T22:13:29.502 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:29 smithi144 ceph-mon[33984]: 3.10 scrub ok 2024-03-11T22:13:29.502 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:29 smithi144 ceph-mon[33984]: pgmap v186: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 948 op/s 2024-03-11T22:13:29.502 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:29 smithi144 ceph-mon[33984]: 5.15 scrub starts 2024-03-11T22:13:29.502 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:29 smithi144 ceph-mon[33984]: 5.15 scrub ok 2024-03-11T22:13:29.505 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[30178]: 6.12 scrub starts 2024-03-11T22:13:29.505 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[30178]: 6.12 scrub ok 2024-03-11T22:13:29.505 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[30178]: 3.3 scrub starts 2024-03-11T22:13:29.506 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[30178]: 3.3 scrub ok 2024-03-11T22:13:29.506 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[30178]: 3.10 scrub starts 2024-03-11T22:13:29.506 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[30178]: 3.10 scrub ok 2024-03-11T22:13:29.506 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[30178]: pgmap v186: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 948 op/s 2024-03-11T22:13:29.506 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[30178]: 5.15 scrub starts 2024-03-11T22:13:29.506 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[30178]: 5.15 scrub ok 2024-03-11T22:13:29.506 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[39715]: 6.12 scrub starts 2024-03-11T22:13:29.506 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[39715]: 6.12 scrub ok 2024-03-11T22:13:29.506 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[39715]: 3.3 scrub starts 2024-03-11T22:13:29.506 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[39715]: 3.3 scrub ok 2024-03-11T22:13:29.506 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[39715]: 3.10 scrub starts 2024-03-11T22:13:29.507 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[39715]: 3.10 scrub ok 2024-03-11T22:13:29.507 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[39715]: pgmap v186: 148 pgs: 148 active+clean; 2.9 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 948 op/s 2024-03-11T22:13:29.507 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[39715]: 5.15 scrub starts 2024-03-11T22:13:29.507 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:29 smithi045 ceph-mon[39715]: 5.15 scrub ok 2024-03-11T22:13:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:29.511 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:29.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:29.612 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:29.640 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:29.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:29.742 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:29.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:29.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:29.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:29.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:29.933 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:29.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:30.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:30.068 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:30.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:30.170 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:30.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:30.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:30.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:30.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:30.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:30.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:30.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:30.500 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:30 smithi144 ceph-mon[33984]: 7.e deep-scrub starts 2024-03-11T22:13:30.500 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:30 smithi144 ceph-mon[33984]: 7.4 scrub starts 2024-03-11T22:13:30.500 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:30 smithi144 ceph-mon[33984]: 4.5 scrub starts 2024-03-11T22:13:30.500 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:30 smithi144 ceph-mon[33984]: 4.5 scrub ok 2024-03-11T22:13:30.501 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:30.501 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:13:30.501 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:30 smithi144 ceph-mon[33984]: 3.16 scrub starts 2024-03-11T22:13:30.501 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:30 smithi144 ceph-mon[33984]: 3.16 scrub ok 2024-03-11T22:13:30.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[30178]: 7.e deep-scrub starts 2024-03-11T22:13:30.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[30178]: 7.4 scrub starts 2024-03-11T22:13:30.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[30178]: 4.5 scrub starts 2024-03-11T22:13:30.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[30178]: 4.5 scrub ok 2024-03-11T22:13:30.528 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:30.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:13:30.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[30178]: 3.16 scrub starts 2024-03-11T22:13:30.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[30178]: 3.16 scrub ok 2024-03-11T22:13:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[39715]: 7.e deep-scrub starts 2024-03-11T22:13:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[39715]: 7.4 scrub starts 2024-03-11T22:13:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[39715]: 4.5 scrub starts 2024-03-11T22:13:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[39715]: 4.5 scrub ok 2024-03-11T22:13:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:13:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[39715]: 3.16 scrub starts 2024-03-11T22:13:30.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:30 smithi045 ceph-mon[39715]: 3.16 scrub ok 2024-03-11T22:13:30.549 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:13:30.549 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:13:30.596 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:30.622 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:30.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:30.724 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:30.753 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:30.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:30.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:30.855 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:30.884 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:30.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:30.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:30.986 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:31.013 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:31.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:31.115 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:31.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:31.146 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:31.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:31.195 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T22:13:31.247 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:31.280 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:31.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:31.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:31.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:31.420 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:31.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:set pool 1 pg_num to 11 2024-03-11T22:13:31.484 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:31 smithi045 ceph-mon[30178]: 7.4 scrub ok 2024-03-11T22:13:31.484 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:31 smithi045 ceph-mon[30178]: pgmap v187: 148 pgs: 148 active+clean; 2.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 952 op/s 2024-03-11T22:13:31.484 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:31 smithi045 ceph-mon[30178]: 4.15 scrub starts 2024-03-11T22:13:31.484 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:31 smithi045 ceph-mon[30178]: 4.15 scrub ok 2024-03-11T22:13:31.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:31 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2924771171' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T22:13:31.485 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:31 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T22:13:31.485 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:31 smithi045 ceph-mon[39715]: 7.4 scrub ok 2024-03-11T22:13:31.485 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:31 smithi045 ceph-mon[39715]: pgmap v187: 148 pgs: 148 active+clean; 2.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 952 op/s 2024-03-11T22:13:31.485 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:31 smithi045 ceph-mon[39715]: 4.15 scrub starts 2024-03-11T22:13:31.485 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:31 smithi045 ceph-mon[39715]: 4.15 scrub ok 2024-03-11T22:13:31.485 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:31 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2924771171' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T22:13:31.485 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:31 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T22:13:31.514 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:13:31.522 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:31.564 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:31.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:31.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:31.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:31 smithi144 ceph-mon[33984]: 7.4 scrub ok 2024-03-11T22:13:31.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:31 smithi144 ceph-mon[33984]: pgmap v187: 148 pgs: 148 active+clean; 2.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 952 op/s 2024-03-11T22:13:31.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:31 smithi144 ceph-mon[33984]: 4.15 scrub starts 2024-03-11T22:13:31.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:31 smithi144 ceph-mon[33984]: 4.15 scrub ok 2024-03-11T22:13:31.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:31 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2924771171' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T22:13:31.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:31 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T22:13:31.667 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:31.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:31.813 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:31.850 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:31.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:31.952 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:31.981 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:31.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:31.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:32.025 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:13:32.084 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:32.111 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:32.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:32.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:32.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:32.271 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:32.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:32.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:32.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:32.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:32.409 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:32.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:32.430 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:32.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:32.563 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:32.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:32.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:32.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:13:32.665 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:32.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:32.696 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:32.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:32.778 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:13:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:13:32.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[39715]: 6.1d scrub starts 2024-03-11T22:13:32.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[39715]: 6.1d scrub ok 2024-03-11T22:13:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[39715]: 7.5 scrub starts 2024-03-11T22:13:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-11T22:13:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[39715]: osdmap e143: 8 total, 8 up, 8 in 2024-03-11T22:13:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[39715]: 4.10 scrub starts 2024-03-11T22:13:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[39715]: 4.10 scrub ok 2024-03-11T22:13:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[39715]: 3.17 scrub starts 2024-03-11T22:13:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[39715]: 3.17 scrub ok 2024-03-11T22:13:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[39715]: 4.1b scrub starts 2024-03-11T22:13:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[39715]: 4.1b scrub ok 2024-03-11T22:13:32.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T22:13:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[30178]: 6.1d scrub starts 2024-03-11T22:13:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[30178]: 6.1d scrub ok 2024-03-11T22:13:32.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[30178]: 7.5 scrub starts 2024-03-11T22:13:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-11T22:13:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[30178]: osdmap e143: 8 total, 8 up, 8 in 2024-03-11T22:13:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[30178]: 4.10 scrub starts 2024-03-11T22:13:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[30178]: 4.10 scrub ok 2024-03-11T22:13:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[30178]: 3.17 scrub starts 2024-03-11T22:13:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[30178]: 3.17 scrub ok 2024-03-11T22:13:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[30178]: 4.1b scrub starts 2024-03-11T22:13:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[30178]: 4.1b scrub ok 2024-03-11T22:13:32.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:32 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T22:13:32.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:32.837 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:32.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:32.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:32.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:32 smithi144 ceph-mon[33984]: 6.1d scrub starts 2024-03-11T22:13:32.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:32 smithi144 ceph-mon[33984]: 6.1d scrub ok 2024-03-11T22:13:32.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:32 smithi144 ceph-mon[33984]: 7.5 scrub starts 2024-03-11T22:13:32.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:32 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-11T22:13:32.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:32 smithi144 ceph-mon[33984]: osdmap e143: 8 total, 8 up, 8 in 2024-03-11T22:13:32.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:32 smithi144 ceph-mon[33984]: 4.10 scrub starts 2024-03-11T22:13:32.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:32 smithi144 ceph-mon[33984]: 4.10 scrub ok 2024-03-11T22:13:32.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:32 smithi144 ceph-mon[33984]: 3.17 scrub starts 2024-03-11T22:13:32.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:32 smithi144 ceph-mon[33984]: 3.17 scrub ok 2024-03-11T22:13:32.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:32 smithi144 ceph-mon[33984]: 4.1b scrub starts 2024-03-11T22:13:32.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:32 smithi144 ceph-mon[33984]: 4.1b scrub ok 2024-03-11T22:13:32.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:32 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T22:13:32.939 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:32.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:32.964 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:32.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:33.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:33.107 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:33.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:33.208 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:33.238 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:33.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:33.339 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:33.365 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:33.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:33.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:33.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:33.511 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:33.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:33.612 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:33.640 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:33.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:33.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:33.774 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:33.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[39715]: 7.5 scrub ok 2024-03-11T22:13:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[39715]: 4.12 scrub starts 2024-03-11T22:13:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[39715]: 4.12 scrub ok 2024-03-11T22:13:33.775 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[39715]: pgmap v189: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:13:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:13:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:13:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T22:13:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[39715]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T22:13:33.776 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[39715]: 7.6 deep-scrub starts 2024-03-11T22:13:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[30178]: 7.5 scrub ok 2024-03-11T22:13:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[30178]: 4.12 scrub starts 2024-03-11T22:13:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[30178]: 4.12 scrub ok 2024-03-11T22:13:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[30178]: pgmap v189: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:13:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:13:33.776 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:13:33.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T22:13:33.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[30178]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T22:13:33.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:33 smithi045 ceph-mon[30178]: 7.6 deep-scrub starts 2024-03-11T22:13:33.821 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:13:33.821 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:13:33.821 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:33 smithi144 ceph-mon[33984]: 7.5 scrub ok 2024-03-11T22:13:33.821 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:33 smithi144 ceph-mon[33984]: 4.12 scrub starts 2024-03-11T22:13:33.821 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:33 smithi144 ceph-mon[33984]: 4.12 scrub ok 2024-03-11T22:13:33.822 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:33 smithi144 ceph-mon[33984]: pgmap v189: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:13:33.822 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:33 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:13:33.822 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:33 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:13:33.822 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:33 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T22:13:33.822 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:33 smithi144 ceph-mon[33984]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T22:13:33.822 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:33 smithi144 ceph-mon[33984]: 7.6 deep-scrub starts 2024-03-11T22:13:33.876 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:33.905 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:33.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:33.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:34.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:34.049 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:34.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:34.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:34.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:34.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:34.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:34.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:34.322 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:34.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:34.423 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:34.452 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:34.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:34.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:34.553 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:34.596 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:34.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:34.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:34.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:34.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:34.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:34.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:34 smithi045 ceph-mon[39715]: 3.2 scrub starts 2024-03-11T22:13:34.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:34 smithi045 ceph-mon[39715]: 3.2 scrub ok 2024-03-11T22:13:34.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:34 smithi045 ceph-mon[39715]: osdmap e145: 8 total, 8 up, 8 in 2024-03-11T22:13:34.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:34 smithi045 ceph-mon[39715]: 1.1 scrub starts 2024-03-11T22:13:34.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:34 smithi045 ceph-mon[39715]: 1.1 scrub ok 2024-03-11T22:13:34.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:34 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T22:13:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:34 smithi045 ceph-mon[30178]: 3.2 scrub starts 2024-03-11T22:13:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:34 smithi045 ceph-mon[30178]: 3.2 scrub ok 2024-03-11T22:13:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:34 smithi045 ceph-mon[30178]: osdmap e145: 8 total, 8 up, 8 in 2024-03-11T22:13:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:34 smithi045 ceph-mon[30178]: 1.1 scrub starts 2024-03-11T22:13:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:34 smithi045 ceph-mon[30178]: 1.1 scrub ok 2024-03-11T22:13:34.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:34 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T22:13:34.829 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:34.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:34.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:34 smithi144 ceph-mon[33984]: 3.2 scrub starts 2024-03-11T22:13:34.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:34 smithi144 ceph-mon[33984]: 3.2 scrub ok 2024-03-11T22:13:34.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:34 smithi144 ceph-mon[33984]: osdmap e145: 8 total, 8 up, 8 in 2024-03-11T22:13:34.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:34 smithi144 ceph-mon[33984]: 1.1 scrub starts 2024-03-11T22:13:34.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:34 smithi144 ceph-mon[33984]: 1.1 scrub ok 2024-03-11T22:13:34.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:34 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T22:13:34.878 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:34.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:34.980 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:35.011 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T22:13:35.015 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:35.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:35.117 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:35.147 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:35.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:35.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:35.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:35.287 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:35.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:35.337 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:13:35.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:35.437 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:35.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:35.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:35.575 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:35.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:35.606 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:35 smithi144 ceph-mon[33984]: pgmap v192: 158 pgs: 10 unknown, 1 active+clean+scrubbing+deep, 147 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 794 op/s 2024-03-11T22:13:35.606 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:35 smithi144 ceph-mon[33984]: 6.7 scrub starts 2024-03-11T22:13:35.606 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:35 smithi144 ceph-mon[33984]: 6.7 scrub ok 2024-03-11T22:13:35.606 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:35 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T22:13:35.606 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:35 smithi144 ceph-mon[33984]: osdmap e146: 8 total, 8 up, 8 in 2024-03-11T22:13:35.606 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:35 smithi144 ceph-mon[33984]: 1.0 deep-scrub starts 2024-03-11T22:13:35.606 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:35 smithi144 ceph-mon[33984]: 5.e scrub starts 2024-03-11T22:13:35.606 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:35 smithi144 ceph-mon[33984]: 1.0 deep-scrub ok 2024-03-11T22:13:35.606 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:35 smithi144 ceph-mon[33984]: 5.e scrub ok 2024-03-11T22:13:35.677 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:35.710 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:35.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:35.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[39715]: pgmap v192: 158 pgs: 10 unknown, 1 active+clean+scrubbing+deep, 147 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 794 op/s 2024-03-11T22:13:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[39715]: 6.7 scrub starts 2024-03-11T22:13:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[39715]: 6.7 scrub ok 2024-03-11T22:13:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T22:13:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[39715]: osdmap e146: 8 total, 8 up, 8 in 2024-03-11T22:13:35.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[39715]: 1.0 deep-scrub starts 2024-03-11T22:13:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[39715]: 5.e scrub starts 2024-03-11T22:13:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[39715]: 1.0 deep-scrub ok 2024-03-11T22:13:35.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[39715]: 5.e scrub ok 2024-03-11T22:13:35.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[30178]: pgmap v192: 158 pgs: 10 unknown, 1 active+clean+scrubbing+deep, 147 active+clean; 2.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 794 op/s 2024-03-11T22:13:35.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[30178]: 6.7 scrub starts 2024-03-11T22:13:35.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[30178]: 6.7 scrub ok 2024-03-11T22:13:35.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T22:13:35.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[30178]: osdmap e146: 8 total, 8 up, 8 in 2024-03-11T22:13:35.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[30178]: 1.0 deep-scrub starts 2024-03-11T22:13:35.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[30178]: 5.e scrub starts 2024-03-11T22:13:35.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[30178]: 1.0 deep-scrub ok 2024-03-11T22:13:35.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:35 smithi045 ceph-mon[30178]: 5.e scrub ok 2024-03-11T22:13:35.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:35.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:35.859 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:35.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:35.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:35.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:35.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:35.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:36.100 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:36.136 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:36.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:36.206 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:13:35 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:13:36.238 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:36.272 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:36.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:36.373 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:36.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:36.406 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:36.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:36.508 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:36.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:36.539 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:36.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:36.641 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:36.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:36.693 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:36.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:36.794 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:36.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:36 smithi045 ceph-mon[39715]: osdmap e147: 8 total, 8 up, 8 in 2024-03-11T22:13:36.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:36 smithi045 ceph-mon[39715]: 5.1a scrub starts 2024-03-11T22:13:36.807 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:36 smithi045 ceph-mon[39715]: 5.1a scrub ok 2024-03-11T22:13:36.807 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:36 smithi045 ceph-mon[30178]: osdmap e147: 8 total, 8 up, 8 in 2024-03-11T22:13:36.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:36 smithi045 ceph-mon[30178]: 5.1a scrub starts 2024-03-11T22:13:36.808 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:36 smithi045 ceph-mon[30178]: 5.1a scrub ok 2024-03-11T22:13:36.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:36.827 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:36.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:36.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:36 smithi144 ceph-mon[33984]: osdmap e147: 8 total, 8 up, 8 in 2024-03-11T22:13:36.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:36 smithi144 ceph-mon[33984]: 5.1a scrub starts 2024-03-11T22:13:36.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:36 smithi144 ceph-mon[33984]: 5.1a scrub ok 2024-03-11T22:13:36.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:36.979 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:36.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:37.080 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:37.114 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:37.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:37.164 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:13:37.205 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:13:37.215 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:37.246 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:13:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:37.252 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:37.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:37.252 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:13:37.252 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:13:37.252 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-11T22:13:37.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T22:13:37.274 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:13:37.311 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:13:37.351 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:13:37.353 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:37.393 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:13:37.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:37.398 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:37.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:37.432 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:13:37.434 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:13:37.434 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:13:37.434 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:13:37.435 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:13:37.435 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:13:37.435 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:13:37.435 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:13:37.435 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:13:37.499 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:37.536 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:37.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:37.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:37.563 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:13:37.617 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:13:37.617 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-03-11T22:13:37.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-03-11T22:13:37.617 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:13:37.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.617 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:82172 7.5 e118/99 7:ab5208dd:::benchmark_data_smithi045_83007_object82171:head v 118'5165, mlcod=118'5162)", 2024-03-11T22:13:37.617 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:35.758803+0000", 2024-03-11T22:13:37.618 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 121.85879149, 2024-03-11T22:13:37.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012433427, 2024-03-11T22:13:37.618 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.618 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.618 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:35.758803+0000", 2024-03-11T22:13:37.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:35.758803+0000", 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:35.758804+0000", 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6000000000000004e-07 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:35.758968+0000", 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016369900000000001 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:35.758976+0000", 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8800000000000008e-06 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:35.758981+0000", 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7180000000000001e-06 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:35.759061+0000", 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9476999999999998e-05 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:35.759078+0000", 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7564999999999999e-05 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:35.771147+0000", 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012068399 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:35.771237+0000", 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.0129000000000004e-05 2024-03-11T22:13:37.621 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:89037 7.6 e118/99 7:6948b031:::benchmark_data_smithi045_83007_object89036:head v 118'5617, mlcod=118'5615)", 2024-03-11T22:13:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:42.318725+0000", 2024-03-11T22:13:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 115.29887003, 2024-03-11T22:13:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011057948999999999, 2024-03-11T22:13:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:42.318725+0000", 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:42.318725+0000", 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:42.318726+0000", 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.037e-06 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:42.318758+0000", 2024-03-11T22:13:37.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1869000000000003e-05 2024-03-11T22:13:37.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:42.318786+0000", 2024-03-11T22:13:37.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8418000000000001e-05 2024-03-11T22:13:37.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:42.318795+0000", 2024-03-11T22:13:37.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9449999999999999e-06 2024-03-11T22:13:37.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.626 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.626 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:42.320025+0000", 2024-03-11T22:13:37.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0012301980000000001 2024-03-11T22:13:37.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.626 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.626 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:42.320060+0000", 2024-03-11T22:13:37.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4912e-05 2024-03-11T22:13:37.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:42.329750+0000", 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0096894170000000005 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:42.329783+0000", 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3152999999999999e-05 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.627 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:106787 7.c e124/99 7:33a5a9fb:::benchmark_data_smithi045_83007_object106786:head v 124'6631, mlcod=124'6629)", 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:01.771674+0000", 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 95.845920723999996, 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011945223, 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:01.771674+0000", 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:01.771674+0000", 2024-03-11T22:13:37.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:01.771675+0000", 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8899999999999998e-07 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:01.771719+0000", 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4372999999999997e-05 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:01.771729+0000", 2024-03-11T22:13:37.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6770000000000007e-06 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:01.771736+0000", 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8120000000000002e-06 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:01.771804+0000", 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8454999999999994e-05 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:01.771823+0000", 2024-03-11T22:13:37.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8380999999999999e-05 2024-03-11T22:13:37.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:01.783557+0000", 2024-03-11T22:13:37.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011734820999999999 2024-03-11T22:13:37.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:01.783619+0000", 2024-03-11T22:13:37.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1915000000000006e-05 2024-03-11T22:13:37.631 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.631 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.632 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.632 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:111078 7.f e125/99 7:fbe49a80:::benchmark_data_smithi045_83007_object111077:head v 125'6898, mlcod=125'6896)", 2024-03-11T22:13:37.632 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:06.219285+0000", 2024-03-11T22:13:37.632 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 91.398310374000005, 2024-03-11T22:13:37.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011416018999999999, 2024-03-11T22:13:37.632 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.632 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.632 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.219285+0000", 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.219285+0000", 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.219285+0000", 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1299999999999998e-07 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.634 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.219318+0000", 2024-03-11T22:13:37.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2078e-05 2024-03-11T22:13:37.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.634 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.219332+0000", 2024-03-11T22:13:37.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4032999999999999e-05 2024-03-11T22:13:37.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.634 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.219340+0000", 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1100000000000003e-06 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.219398+0000", 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8155000000000001e-05 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.219417+0000", 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9423999999999999e-05 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.230536+0000", 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011118503 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.230701+0000", 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016480299999999999 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:111638 7.6 e126/99 7:6aa82e2d:::benchmark_data_smithi045_83007_object111637:head v 126'7034, mlcod=126'7031)", 2024-03-11T22:13:37.636 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:06.793579+0000", 2024-03-11T22:13:37.637 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 90.824015458999995, 2024-03-11T22:13:37.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012698563, 2024-03-11T22:13:37.637 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.637 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.637 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.793579+0000", 2024-03-11T22:13:37.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.793579+0000", 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.793580+0000", 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1200000000000002e-07 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.803613+0000", 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010032355999999999 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.803625+0000", 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2223e-05 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.803635+0000", 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0149000000000001e-05 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.803690+0000", 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4979000000000003e-05 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.803711+0000", 2024-03-11T22:13:37.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1128000000000002e-05 2024-03-11T22:13:37.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.806220+0000", 2024-03-11T22:13:37.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002508536 2024-03-11T22:13:37.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:06.806278+0000", 2024-03-11T22:13:37.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8480000000000003e-05 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:133117 7.8 e129/99 7:1da79adf:::benchmark_data_smithi045_83007_object14646:head v 129'8466, mlcod=129'8464)", 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:28.463066+0000", 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 69.154529179999997, 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011265628, 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.641 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:28.463066+0000", 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:28.463066+0000", 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:28.463066+0000", 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0800000000000004e-07 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:28.463073+0000", 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1990000000000003e-06 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:28.463085+0000", 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2404e-05 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:28.463094+0000", 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5979999999999997e-06 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:28.463149+0000", 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5815e-05 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:28.463168+0000", 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8975999999999999e-05 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:28.474261+0000", 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011092972 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:28.474331+0000", 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9956000000000006e-05 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:140333 7.c e131/99 7:300d3d36:::benchmark_data_smithi045_83007_object21862:head v 131'8757, mlcod=131'8749)", 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:36.176116+0000", 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 61.441478558, 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011161938, 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.645 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.646 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:36.176116+0000", 2024-03-11T22:13:37.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.646 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:36.176116+0000", 2024-03-11T22:13:37.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.647 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:37.648 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.648 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.648 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:36.176117+0000", 2024-03-11T22:13:37.648 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6800000000000001e-07 2024-03-11T22:13:37.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.648 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.648 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.648 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:36.176121+0000", 2024-03-11T22:13:37.648 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1749999999999998e-06 2024-03-11T22:13:37.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.649 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.649 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:36.176131+0000", 2024-03-11T22:13:37.649 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0352e-05 2024-03-11T22:13:37.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.649 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.649 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:36.176138+0000", 2024-03-11T22:13:37.649 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4640000000000003e-06 2024-03-11T22:13:37.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.649 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:36.179048+0000", 2024-03-11T22:13:37.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002910433 2024-03-11T22:13:37.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:36.179067+0000", 2024-03-11T22:13:37.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8723999999999999e-05 2024-03-11T22:13:37.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:36.187234+0000", 2024-03-11T22:13:37.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0081674260000000002 2024-03-11T22:13:37.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:36.187278+0000", 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3995999999999997e-05 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:158951 7.c e137/99 7:3e10589c:::benchmark_data_smithi045_83007_object40480:head v 137'9875, mlcod=137'9872)", 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:54.929124+0000", 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 42.688470983000002, 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011073487, 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.651 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.929124+0000", 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.929124+0000", 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.929125+0000", 2024-03-11T22:13:37.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.92e-07 2024-03-11T22:13:37.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.653 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.653 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.929130+0000", 2024-03-11T22:13:37.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6729999999999997e-06 2024-03-11T22:13:37.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.653 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.653 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.929141+0000", 2024-03-11T22:13:37.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0764000000000001e-05 2024-03-11T22:13:37.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.929149+0000", 2024-03-11T22:13:37.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.357e-06 2024-03-11T22:13:37.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.929227+0000", 2024-03-11T22:13:37.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8335000000000001e-05 2024-03-11T22:13:37.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.929246+0000", 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9239e-05 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.939782+0000", 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010536057 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.940197+0000", 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00041526999999999997 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.655 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:158975 7.8 e137/99 7:10814b0e:::benchmark_data_smithi045_83007_object40504:head v 137'10057, mlcod=137'10051)", 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:54.944125+0000", 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 42.673470174999999, 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012853564, 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.944125+0000", 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.944125+0000", 2024-03-11T22:13:37.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.944126+0000", 2024-03-11T22:13:37.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0979999999999999e-06 2024-03-11T22:13:37.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.944131+0000", 2024-03-11T22:13:37.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9620000000000001e-06 2024-03-11T22:13:37.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.944146+0000", 2024-03-11T22:13:37.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.521e-05 2024-03-11T22:13:37.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.944154+0000", 2024-03-11T22:13:37.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3299999999999999e-06 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.944250+0000", 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.5668000000000006e-05 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.944269+0000", 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9108999999999999e-05 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.956876+0000", 2024-03-11T22:13:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012606717 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.956978+0000", 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010247 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:158976 7.8 e137/99 7:1dedfb11:::benchmark_data_smithi045_83007_object40505:head v 137'10058, mlcod=137'10052)", 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:54.947094+0000", 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 42.670501162999997, 2024-03-11T22:13:37.660 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.013789272, 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.947094+0000", 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.947094+0000", 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.947095+0000", 2024-03-11T22:13:37.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.077e-06 2024-03-11T22:13:37.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.947102+0000", 2024-03-11T22:13:37.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3420000000000002e-06 2024-03-11T22:13:37.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.663 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.663 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.947117+0000", 2024-03-11T22:13:37.663 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4756999999999999e-05 2024-03-11T22:13:37.663 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.663 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.663 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.663 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.947125+0000", 2024-03-11T22:13:37.663 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8739999999999995e-06 2024-03-11T22:13:37.663 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.663 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.663 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.663 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.948744+0000", 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016188680000000001 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.948762+0000", 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8309999999999999e-05 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.960760+0000", 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011997933000000001 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.960883+0000", 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012311099999999999 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:159265 7.f e137/99 7:fc2cf9c1:::benchmark_data_smithi045_83007_object40794:head v 137'9964, mlcod=137'9961)", 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:55.225161+0000", 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 42.392434321000003, 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011466245, 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.225161+0000", 2024-03-11T22:13:37.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.225161+0000", 2024-03-11T22:13:37.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.225161+0000", 2024-03-11T22:13:37.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5499999999999998e-07 2024-03-11T22:13:37.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.667 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.225166+0000", 2024-03-11T22:13:37.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4429999999999996e-06 2024-03-11T22:13:37.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.667 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.225174+0000", 2024-03-11T22:13:37.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7679999999999997e-06 2024-03-11T22:13:37.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.225179+0000", 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4680000000000003e-06 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.227713+0000", 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0025343789999999998 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.227734+0000", 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0914999999999999e-05 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.236573+0000", 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0088386709999999993 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.236627+0000", 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4045999999999997e-05 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.669 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:159269 7.5 e137/99 7:af4207e3:::benchmark_data_smithi045_83007_object40798:head v 137'10001, mlcod=137'9997)", 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:55.228563+0000", 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 42.38903182, 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011198387000000001, 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.228563+0000", 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.228563+0000", 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.228564+0000", 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1999999999999998e-07 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.228573+0000", 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8009999999999993e-06 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.228589+0000", 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6121e-05 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.228596+0000", 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2599999999999999e-06 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.228639+0000", 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3300000000000002e-05 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.228660+0000", 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1084999999999999e-05 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.239637+0000", 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010976661 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:55.239761+0000", 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000124339 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:160998 7.8 e137/99 7:16651b0d:::benchmark_data_smithi045_83007_object42527:head v 137'10192, mlcod=137'10190)", 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:56.965593+0000", 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 40.652002285000002, 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011585936, 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:56.965593+0000", 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:56.965593+0000", 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:56.965593+0000", 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8599999999999998e-07 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:56.965599+0000", 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6559999999999997e-06 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:56.965609+0000", 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.8069999999999994e-06 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:56.965615+0000", 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8540000000000002e-06 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:56.965694+0000", 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9054999999999994e-05 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:56.965718+0000", 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4250999999999999e-05 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:56.977101+0000", 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.01138284 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:56.977179+0000", 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7787000000000006e-05 2024-03-11T22:13:37.677 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.678 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.678 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.678 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:166297 7.c e138/99 7:38aebddc:::benchmark_data_smithi045_83007_object47826:head v 138'10293, mlcod=138'10290)", 2024-03-11T22:13:37.678 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:02.287652+0000", 2024-03-11T22:13:37.678 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 35.329942469000002, 2024-03-11T22:13:37.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011410445, 2024-03-11T22:13:37.678 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.678 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.287652+0000", 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.287652+0000", 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.287653+0000", 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9100000000000003e-07 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.287659+0000", 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2600000000000002e-06 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.287673+0000", 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3981999999999999e-05 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.287681+0000", 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.17e-06 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.287745+0000", 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4264999999999995e-05 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.287763+0000", 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8210999999999999e-05 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.299017+0000", 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011253667 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.299063+0000", 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6099e-05 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:166646 7.8 e138/99 7:13fe9421:::benchmark_data_smithi045_83007_object48175:head v 138'10554, mlcod=138'10551)", 2024-03-11T22:13:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:02.624992+0000", 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 34.992603004999999, 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011969989, 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.624992+0000", 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.624992+0000", 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.624993+0000", 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9800000000000003e-07 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.624998+0000", 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1080000000000001e-06 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.625006+0000", 2024-03-11T22:13:37.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3399999999999998e-06 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.625012+0000", 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0580000000000003e-06 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.626447+0000", 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0014345709999999999 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.626468+0000", 2024-03-11T22:13:37.685 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.092e-05 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.636819+0000", 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010351226999999999 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:02.636962+0000", 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000143067 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.686 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:184506 7.c e141/99 7:3587fc58:::benchmark_data_smithi045_83007_object66035:head v 141'11472, mlcod=141'11469)", 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:20.728895+0000", 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 16.888699586000001, 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010849414999999999, 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.728895+0000", 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.728895+0000", 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.728896+0000", 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.569999999999999e-07 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.728905+0000", 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2239999999999995e-06 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.728917+0000", 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2394e-05 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.728925+0000", 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6539999999999999e-06 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.730227+0000", 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013023209999999999 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.689 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.690 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.730246+0000", 2024-03-11T22:13:37.690 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8726999999999999e-05 2024-03-11T22:13:37.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.690 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.690 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.690 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.739703+0000", 2024-03-11T22:13:37.690 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.009457221 2024-03-11T22:13:37.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.690 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.690 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.690 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:20.739745+0000", 2024-03-11T22:13:37.690 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1916999999999999e-05 2024-03-11T22:13:37.690 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:190831 7.6 e142/99 7:61307de5:::benchmark_data_smithi045_83007_object72360:head v 142'12003, mlcod=142'12001)", 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:27.176260+0000", 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 10.441334788000001, 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012594737, 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176260+0000", 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176260+0000", 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176261+0000", 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.311e-06 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176274+0000", 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2242000000000001e-05 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176299+0000", 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5780000000000001e-05 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176309+0000", 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.2590000000000006e-06 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176406+0000", 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7355000000000006e-05 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176426+0000", 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0117000000000001e-05 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.188805+0000", 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012378538999999999 2024-03-11T22:13:37.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.695 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.695 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.188855+0000", 2024-03-11T22:13:37.695 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0133999999999998e-05 2024-03-11T22:13:37.695 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.695 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.695 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.695 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:190834 7.6 e142/99 7:6369eb7c:::benchmark_data_smithi045_83007_object72363:head v 142'12004, mlcod=142'12001)", 2024-03-11T22:13:37.695 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:27.176332+0000", 2024-03-11T22:13:37.695 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 10.441263247, 2024-03-11T22:13:37.695 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.01255645, 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176332+0000", 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176332+0000", 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176332+0000", 2024-03-11T22:13:37.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9400000000000001e-07 2024-03-11T22:13:37.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176336+0000", 2024-03-11T22:13:37.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5159999999999996e-06 2024-03-11T22:13:37.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176342+0000", 2024-03-11T22:13:37.698 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8699999999999997e-06 2024-03-11T22:13:37.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176347+0000", 2024-03-11T22:13:37.698 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7559999999999999e-06 2024-03-11T22:13:37.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.179417+0000", 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0030698449999999999 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.179436+0000", 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.891e-05 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.188860+0000", 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0094243090000000005 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.699 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.188888+0000", 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7949999999999998e-05 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:190833 7.8 e142/99 7:10f5ead6:::benchmark_data_smithi045_83007_object72362:head v 142'12079, mlcod=142'12077)", 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:27.176748+0000", 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 10.440847003, 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.014695094000000001, 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.700 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176748+0000", 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176748+0000", 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176749+0000", 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3900000000000004e-07 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176757+0000", 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5390000000000007e-06 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176766+0000", 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6750000000000008e-06 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176772+0000", 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9209999999999997e-06 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.702 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.178072+0000", 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013003120000000001 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.178095+0000", 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2648000000000001e-05 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.191322+0000", 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.013227536999999999 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.703 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.704 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.704 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.191443+0000", 2024-03-11T22:13:37.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000120723 2024-03-11T22:13:37.704 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.704 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.704 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.704 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.705 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:190839 7.8 e142/99 7:1c0908be:::benchmark_data_smithi045_83007_object72368:head v 142'12080, mlcod=142'12077)", 2024-03-11T22:13:37.705 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:27.180367+0000", 2024-03-11T22:13:37.705 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 10.437227443999999, 2024-03-11T22:13:37.705 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011139552, 2024-03-11T22:13:37.705 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:13:37.705 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:13:37.705 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:13:37.705 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.705 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:13:37.705 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.180367+0000", 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.180367+0000", 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.180368+0000", 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8199999999999999e-07 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.706 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.180380+0000", 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1585e-05 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.180395+0000", 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5491000000000001e-05 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.180404+0000", 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6209999999999993e-06 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.183040+0000", 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026358419999999998 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.183100+0000", 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0402000000000002e-05 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.191449+0000", 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0083484370000000002 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:13:37.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:13:37.709 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:13:37.709 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.191507+0000", 2024-03-11T22:13:37.709 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8391999999999999e-05 2024-03-11T22:13:37.709 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.709 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.709 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.709 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:13:37.709 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:13:37.709 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:13:37.730 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:37.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:37.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:37.832 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:37 smithi144 ceph-mon[33984]: pgmap v195: 158 pgs: 10 unknown, 2 active+clean+scrubbing+deep, 146 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 990 op/s 2024-03-11T22:13:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:37 smithi144 ceph-mon[33984]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T22:13:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:37 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4062211261' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:13:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:37 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2341342294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:37.861 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:37.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:37.962 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:37.991 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:37.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:37.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:38.011 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:38.012 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:37 smithi045 ceph-mon[39715]: pgmap v195: 158 pgs: 10 unknown, 2 active+clean+scrubbing+deep, 146 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 990 op/s 2024-03-11T22:13:38.012 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:37 smithi045 ceph-mon[39715]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T22:13:38.012 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:37 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4062211261' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:13:38.012 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:37 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2341342294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:38.012 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:38.013 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:37 smithi045 ceph-mon[30178]: pgmap v195: 158 pgs: 10 unknown, 2 active+clean+scrubbing+deep, 146 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 990 op/s 2024-03-11T22:13:38.013 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:37 smithi045 ceph-mon[30178]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T22:13:38.013 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:37 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4062211261' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:13:38.013 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:37 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2341342294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:38.093 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:38.123 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:38.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:38.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:38.249 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:38.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:38.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:38.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:38.491 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:38.518 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:38.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:38.620 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:38.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T22:13:38.644 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:38.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:38.665 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:38 smithi045 ceph-mon[30178]: 6.15 deep-scrub starts 2024-03-11T22:13:38.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:38 smithi045 ceph-mon[39715]: 6.15 deep-scrub starts 2024-03-11T22:13:38.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:38 smithi045 ceph-mon[39715]: 6.15 deep-scrub ok 2024-03-11T22:13:38.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:38 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:13:38.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:38 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4062211261' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:13:38.666 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:38 smithi045 ceph-mon[39715]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T22:13:38.666 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:38 smithi045 ceph-mon[39715]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T22:13:38.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:38.787 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:38.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:38.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:38.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:38 smithi144 ceph-mon[33984]: 6.15 deep-scrub starts 2024-03-11T22:13:38.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:38 smithi144 ceph-mon[33984]: 6.15 deep-scrub ok 2024-03-11T22:13:38.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:38 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:13:38.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:38 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4062211261' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:13:38.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:38 smithi144 ceph-mon[33984]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T22:13:38.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:38 smithi144 ceph-mon[33984]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T22:13:38.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:38.934 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:38.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:38.970 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:38 smithi045 ceph-mon[30178]: 6.15 deep-scrub ok 2024-03-11T22:13:38.970 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:38 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:13:38.970 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:38 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4062211261' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:13:38.970 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:38 smithi045 ceph-mon[30178]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T22:13:38.970 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:38 smithi045 ceph-mon[30178]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T22:13:39.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:39.060 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:39.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:39.162 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:39.194 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:39.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:39.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:39.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:39.323 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:39.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:39.425 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:39.455 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:39.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:39.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:39.557 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:39.587 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:39.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:39.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:39.596 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:39.662 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:39.688 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:39.730 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:39.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:39.757 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:39 smithi045 ceph-mon[39715]: pgmap v198: 158 pgs: 10 unknown, 2 active+clean+scrubbing+deep, 146 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 990 op/s 2024-03-11T22:13:39.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:39 smithi045 ceph-mon[30178]: pgmap v198: 158 pgs: 10 unknown, 2 active+clean+scrubbing+deep, 146 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 990 op/s 2024-03-11T22:13:39.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:39.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:39 smithi144 ceph-mon[33984]: pgmap v198: 158 pgs: 10 unknown, 2 active+clean+scrubbing+deep, 146 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 990 op/s 2024-03-11T22:13:39.861 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:39.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:39.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:39.966 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:39.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:39.993 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:39.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:40.094 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:40.120 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:40.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:40.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:40.222 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:40.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:40.249 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:40.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:40.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:40.382 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:40.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:40.484 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:40.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:40.533 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:40.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:40.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:40.669 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:40.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:40.770 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:40.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:40.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:40.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:40.910 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:40.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:40.940 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:40.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:41.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:13:41.042 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:41.070 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:41.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:41.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:41.172 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:41.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:41.216 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:41.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:41.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:41.349 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:41.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:41.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:41.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:41.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:41.531 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:13:41.600 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:41.644 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:41.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:41.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:41.786 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:41.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:41.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:41.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:41 smithi144 ceph-mon[33984]: pgmap v200: 158 pgs: 9 unknown, 2 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 6.1 KiB/s rd, 48 KiB/s wr, 234 op/s; 34 KiB/s, 0 objects/s recovering 2024-03-11T22:13:41.888 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:41.916 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:41.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:42.003 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:41 smithi045 ceph-mon[39715]: pgmap v200: 158 pgs: 9 unknown, 2 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 6.1 KiB/s rd, 48 KiB/s wr, 234 op/s; 34 KiB/s, 0 objects/s recovering 2024-03-11T22:13:42.004 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:41 smithi045 ceph-mon[30178]: pgmap v200: 158 pgs: 9 unknown, 2 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 6.1 KiB/s rd, 48 KiB/s wr, 234 op/s; 34 KiB/s, 0 objects/s recovering 2024-03-11T22:13:42.018 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:42.043 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:42.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:42.145 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:42.173 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:42.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:42.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:42.275 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:42.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:42.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:42.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:42.416 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:42.445 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:42.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:42.474 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:42.547 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:42.575 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:42.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:42.622 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:13:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:13:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:13:42.677 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:42.705 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:42.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:42.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:42.807 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:42.838 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:42.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:42.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:42.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:42.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:43.086 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:43.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:43.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:43.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:43.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:43.267 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:43.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:43.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:43.369 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:43.404 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:43.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:43.507 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:43.533 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:43.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:43.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:43.635 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:43.662 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:43.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:43.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:43.764 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:43.790 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:43.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:43.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:43.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:43 smithi045 ceph-mon[39715]: pgmap v201: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 29 KiB/s rd, 219 KiB/s wr, 672 op/s; 253 KiB/s, 0 objects/s recovering 2024-03-11T22:13:43.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:43 smithi045 ceph-mon[30178]: pgmap v201: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 29 KiB/s rd, 219 KiB/s wr, 672 op/s; 253 KiB/s, 0 objects/s recovering 2024-03-11T22:13:43.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:43 smithi144 ceph-mon[33984]: pgmap v201: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 29 KiB/s rd, 219 KiB/s wr, 672 op/s; 253 KiB/s, 0 objects/s recovering 2024-03-11T22:13:43.891 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:43.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:43.921 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:43.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:44.022 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:44.052 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:44.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:44.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:44.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:44.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:44.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:44.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:44.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:44.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:44.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:44.415 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:44.440 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:44.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:44.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:44.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:44.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:44.580 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:44.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:44.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:44.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:44.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:44.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:44.820 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:44.849 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:44.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:44.956 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:44.983 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:44.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:44.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:45.085 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:45.109 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:45.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:45.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:45.255 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:45.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:45.356 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:45.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:45.381 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:45.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:45.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:45.525 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:45.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:45.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:45.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:45 smithi045 ceph-mon[39715]: pgmap v202: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 23 KiB/s rd, 173 KiB/s wr, 531 op/s; 200 KiB/s, 0 objects/s recovering 2024-03-11T22:13:45.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:45.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:13:45.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:45 smithi045 ceph-mon[30178]: pgmap v202: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 23 KiB/s rd, 173 KiB/s wr, 531 op/s; 200 KiB/s, 0 objects/s recovering 2024-03-11T22:13:45.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:45.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:13:45.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:45.670 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:45.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:45.775 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:45.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:45.799 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:45.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:45.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:45 smithi144 ceph-mon[33984]: pgmap v202: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 23 KiB/s rd, 173 KiB/s wr, 531 op/s; 200 KiB/s, 0 objects/s recovering 2024-03-11T22:13:45.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:13:45.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:13:45.900 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:45.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:45.928 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:45.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:46.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:46.065 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:46.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:46.167 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:46.192 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:46.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:46.293 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:46.320 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:46.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:46.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:46.342 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:13:45 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:13:46.422 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:46.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:46.447 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:46.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:46.548 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:46.574 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:46.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:46.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:46.676 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:46.726 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:46.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:46.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:46.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:46.853 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:46.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:46.954 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:46.982 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:46.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:46.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:47.084 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:47.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:47.121 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:47.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:47.222 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:47.253 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:47.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:47.355 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:47.395 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:47.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:47.497 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:47.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:47.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:47.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:47 smithi144 ceph-mon[33984]: pgmap v203: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 153 KiB/s wr, 933 op/s; 177 KiB/s, 0 objects/s recovering 2024-03-11T22:13:47.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:47 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/67918099' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T22:13:47.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:47.672 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:47.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:47.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:47.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:47 smithi045 ceph-mon[30178]: pgmap v203: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 153 KiB/s wr, 933 op/s; 177 KiB/s, 0 objects/s recovering 2024-03-11T22:13:47.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/67918099' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T22:13:47.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:47.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:47 smithi045 ceph-mon[39715]: pgmap v203: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 21 KiB/s rd, 153 KiB/s wr, 933 op/s; 177 KiB/s, 0 objects/s recovering 2024-03-11T22:13:47.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:47 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/67918099' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T22:13:47.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:47.819 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:47.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:47.920 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:47.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:47.955 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:47.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:48.056 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:48.084 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:48.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:48.185 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:48.210 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:48.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:48.271 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:13:48.271 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:13:48.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.271 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205573 7.6 7.3a06476 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:13:48.271 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.339661+0000", 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.097760245999999995, 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.097790643999999996, 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205573 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339661+0000", 2024-03-11T22:13:48.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:13:48.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339661+0000", 2024-03-11T22:13:48.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:13:48.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339662+0000", 2024-03-11T22:13:48.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5700000000000002e-07 2024-03-11T22:13:48.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:13:48.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339666+0000", 2024-03-11T22:13:48.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.715e-06 2024-03-11T22:13:48.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:13:48.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339674+0000", 2024-03-11T22:13:48.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1249999999999993e-06 2024-03-11T22:13:48.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:13:48.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339679+0000", 2024-03-11T22:13:48.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1529999999999996e-06 2024-03-11T22:13:48.277 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.277 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.277 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205559 7.6 7:65c68fbc:::benchmark_data_smithi045_83007_object87088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:13:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.320961+0000", 2024-03-11T22:13:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.11646042400000001, 2024-03-11T22:13:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11653675199999999, 2024-03-11T22:13:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:13:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:13:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:13:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:13:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205559 2024-03-11T22:13:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.278 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:13:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:13:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.320961+0000", 2024-03-11T22:13:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:13:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.320961+0000", 2024-03-11T22:13:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:13:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.320963+0000", 2024-03-11T22:13:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8619999999999999e-06 2024-03-11T22:13:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:13:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.320969+0000", 2024-03-11T22:13:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5640000000000002e-06 2024-03-11T22:13:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:13:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.320985+0000", 2024-03-11T22:13:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5823000000000001e-05 2024-03-11T22:13:48.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:13:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.320994+0000", 2024-03-11T22:13:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0960000000000007e-06 2024-03-11T22:13:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:13:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.354300+0000", 2024-03-11T22:13:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.033305242999999998 2024-03-11T22:13:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:13:48.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.354416+0000", 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116085 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.354545+0000", 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012903099999999999 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.365883+0000", 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011338595 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.365990+0000", 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010686199999999999 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205560 7.6 7:66e89320:::benchmark_data_smithi045_83007_object87089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.321023+0000", 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.116398323, 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11652647200000001, 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:13:48.283 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205560 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.321023+0000", 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.321023+0000", 2024-03-11T22:13:48.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.321024+0000", 2024-03-11T22:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2399999999999998e-07 2024-03-11T22:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.321027+0000", 2024-03-11T22:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2059999999999999e-06 2024-03-11T22:13:48.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:13:48.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.321038+0000", 2024-03-11T22:13:48.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1428e-05 2024-03-11T22:13:48.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:13:48.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.321044+0000", 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1349999999999997e-06 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.359215+0000", 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038170994999999999 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.359349+0000", 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000133225 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:13:48.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.359484+0000", 2024-03-11T22:13:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013509700000000001 2024-03-11T22:13:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:205559 7.6 e150/99)", 2024-03-11T22:13:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.358199+0000", 2024-03-11T22:13:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.079222476999999999, 2024-03-11T22:13:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.079391736000000004, 2024-03-11T22:13:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:13:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:13:48.288 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:13:48.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:13:48.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.358199+0000", 2024-03-11T22:13:48.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:13:48.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.358199+0000", 2024-03-11T22:13:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:13:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.358200+0000", 2024-03-11T22:13:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1600000000000001e-07 2024-03-11T22:13:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:13:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.358205+0000", 2024-03-11T22:13:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9339999999999997e-06 2024-03-11T22:13:48.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:13:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.358215+0000", 2024-03-11T22:13:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0611000000000001e-05 2024-03-11T22:13:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:13:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.358223+0000", 2024-03-11T22:13:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2609999999999993e-06 2024-03-11T22:13:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.14790.0:205589 7.e e150/99 7:70d7852d:::benchmark_data_smithi045_83007_object87118:head v 150'14398, mlcod=150'14395)", 2024-03-11T22:13:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.360289+0000", 2024-03-11T22:13:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.077132151999999995, 2024-03-11T22:13:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.077332181, 2024-03-11T22:13:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:13:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:13:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:13:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:13:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.360289+0000", 2024-03-11T22:13:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:13:48.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.360289+0000", 2024-03-11T22:13:48.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:13:48.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.360290+0000", 2024-03-11T22:13:48.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4600000000000004e-07 2024-03-11T22:13:48.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.360295+0000", 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8029999999999996e-06 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.360305+0000", 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0207000000000001e-05 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.360312+0000", 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0129999999999996e-06 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.366045+0000", 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0057332759999999998 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.366063+0000", 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7975999999999999e-05 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:205560 7.6 e150/99)", 2024-03-11T22:13:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.362386+0000", 2024-03-11T22:13:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.075035266000000003, 2024-03-11T22:13:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.075267915000000005, 2024-03-11T22:13:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:13:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:13:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:13:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:13:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.362386+0000", 2024-03-11T22:13:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:13:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.362386+0000", 2024-03-11T22:13:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:13:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.362387+0000", 2024-03-11T22:13:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8299999999999996e-07 2024-03-11T22:13:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:13:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.362393+0000", 2024-03-11T22:13:48.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2260000000000002e-06 2024-03-11T22:13:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:13:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.362405+0000", 2024-03-11T22:13:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.163e-05 2024-03-11T22:13:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:13:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.362412+0000", 2024-03-11T22:13:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5100000000000001e-06 2024-03-11T22:13:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.301 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.14790.0:205592 7.e e150/99)", 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.363930+0000", 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.073491593999999993, 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.073750350000000006, 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.363930+0000", 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.363930+0000", 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.363930+0000", 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4600000000000005e-07 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.363946+0000", 2024-03-11T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5780000000000001e-05 2024-03-11T22:13:48.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:13:48.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.363959+0000", 2024-03-11T22:13:48.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.269e-05 2024-03-11T22:13:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:13:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.363967+0000", 2024-03-11T22:13:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8099999999999998e-06 2024-03-11T22:13:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205593 7.4 7:23a81867:::benchmark_data_smithi045_83007_object87122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:13:48.305 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.364077+0000", 2024-03-11T22:13:48.306 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.073344638000000004, 2024-03-11T22:13:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.073633115999999998, 2024-03-11T22:13:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:13:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:13:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:13:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:13:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205593 2024-03-11T22:13:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:13:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:13:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.364077+0000", 2024-03-11T22:13:48.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:13:48.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.364077+0000", 2024-03-11T22:13:48.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:13:48.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.364078+0000", 2024-03-11T22:13:48.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3799999999999999e-06 2024-03-11T22:13:48.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:13:48.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.364083+0000", 2024-03-11T22:13:48.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4399999999999998e-06 2024-03-11T22:13:48.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:13:48.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.364094+0000", 2024-03-11T22:13:48.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0992000000000001e-05 2024-03-11T22:13:48.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:13:48.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.364101+0000", 2024-03-11T22:13:48.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1470000000000004e-06 2024-03-11T22:13:48.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:13:48.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.365239+0000", 2024-03-11T22:13:48.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001137847 2024-03-11T22:13:48.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:13:48.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.365357+0000", 2024-03-11T22:13:48.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118725 2024-03-11T22:13:48.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-03-11T22:13:48.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.365467+0000", 2024-03-11T22:13:48.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000109957 2024-03-11T22:13:48.311 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.311 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.311 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.312 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205570 7.6 7.b27d6e36 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:13:48.312 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.334799+0000", 2024-03-11T22:13:48.312 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.102622084, 2024-03-11T22:13:48.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102953393, 2024-03-11T22:13:48.312 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:13:48.312 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:13:48.312 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:13:48.312 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:13:48.312 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:13:48.312 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205570 2024-03-11T22:13:48.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.312 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334799+0000", 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334799+0000", 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334800+0000", 2024-03-11T22:13:48.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9999999999999995e-07 2024-03-11T22:13:48.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:13:48.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334805+0000", 2024-03-11T22:13:48.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1220000000000003e-06 2024-03-11T22:13:48.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:13:48.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334818+0000", 2024-03-11T22:13:48.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2605e-05 2024-03-11T22:13:48.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.315 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:13:48.315 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334827+0000", 2024-03-11T22:13:48.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6980000000000004e-06 2024-03-11T22:13:48.315 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.315 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.315 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.315 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.315 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:13:48.315 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 9 2024-03-11T22:13:48.316 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:13:48.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:48.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:48.351 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:48.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:48.452 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:48.479 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:48.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:48.580 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:48.607 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:48.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:48.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:48.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:48.714 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:48.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:48.765 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:48.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:48.866 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:48.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:48.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:48.911 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 150, 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "f75aafa4-dff2-11ee-95c8-87774f69a715", 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-03-11T22:02:51.075940+0000", 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-03-11T22:13:38.551675+0000", 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout: "last_up_change": "2024-03-11T22:08:17.992246+0000", 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout: "last_in_change": "2024-03-11T22:08:00.853801+0000", 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_num": 5801984, 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_set": [ 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout: "noscrub", 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout: "pglog_hardlimit", 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout: "purged_snapdirs", 2024-03-11T22:13:48.912 INFO:teuthology.orchestra.run.smithi045.stdout: "recovery_deletes", 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: "sortbitwise" 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_version": 18, 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_snapshot": "", 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_max": 7, 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: "max_osd": 8, 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: "require_min_compat_client": "luminous", 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: "min_compat_client": "luminous", 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: "require_osd_release": "squid", 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: "allow_crimson": false, 2024-03-11T22:13:48.913 INFO:teuthology.orchestra.run.smithi045.stdout: "pools": [ 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 1, 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": ".mgr", 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:06:12.903386+0000", 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:13:48.914 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:13:48.915 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:13:48.915 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:13:48.915 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 11, 2024-03-11T22:13:48.915 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 11, 2024-03-11T22:13:48.915 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 11, 2024-03-11T22:13:48.915 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 11, 2024-03-11T22:13:48.915 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 11, 2024-03-11T22:13:48.915 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:13:48.915 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:13:48.915 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:13:48.916 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:13:48.916 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:13:48.916 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:13:48.916 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:13:48.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.916 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "146", 2024-03-11T22:13:48.916 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:13:48.916 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:13:48.916 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "144", 2024-03-11T22:13:48.916 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:13:48.916 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:13:48.916 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:13:48.916 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:13:48.917 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:13:48.918 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:13:48.918 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:13:48.918 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:13:48.918 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:13:48.918 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:13:48.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.918 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:13:48.918 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:13:48.919 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:13:48.919 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:13:48.919 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:13:48.919 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:13:48.919 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:13:48.919 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:13:48.919 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:13:48.919 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:13:48.919 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:13:48.919 INFO:teuthology.orchestra.run.smithi045.stdout: "options": { 2024-03-11T22:13:48.919 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_max": 32, 2024-03-11T22:13:48.920 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_min": 1 2024-03-11T22:13:48.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.920 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:13:48.920 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": {} 2024-03-11T22:13:48.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.920 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.9099999666213989, 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.9099999666213989, 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2.1800000667572021, 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2.1800000667572021, 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 2, 2024-03-11T22:13:48.921 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "datapool", 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:31.014060+0000", 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 8193, 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:13:48.922 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 3, 2024-03-11T22:13:48.923 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 3, 2024-03-11T22:13:48.923 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 3, 2024-03-11T22:13:48.923 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 3, 2024-03-11T22:13:48.923 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 3, 2024-03-11T22:13:48.923 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:13:48.923 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:13:48.923 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:13:48.923 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:13:48.923 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:13:48.923 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:13:48.923 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:13:48.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.923 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "71", 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 2, 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 71, 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:13:48.924 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:13:48.925 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:13:48.926 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:13:48.926 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:13:48.926 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd": {} 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 2.3299999237060547, 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 2.3299999237060547, 2024-03-11T22:13:48.927 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1, 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 3, 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": ".rgw.root", 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:31.591167+0000", 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:13:48.928 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:13:48.929 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:13:48.929 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:13:48.929 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:13:48.929 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:13:48.929 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:13:48.929 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:13:48.929 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:13:48.929 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:13:48.930 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 32, 2024-03-11T22:13:48.930 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 32, 2024-03-11T22:13:48.930 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 32, 2024-03-11T22:13:48.930 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 32, 2024-03-11T22:13:48.930 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 32, 2024-03-11T22:13:48.930 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:13:48.930 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:13:48.930 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:13:48.931 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:13:48.931 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:13:48.931 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:13:48.931 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:13:48.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.931 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "69", 2024-03-11T22:13:48.931 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:13:48.931 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:13:48.932 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:13:48.933 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw": {} 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:13:48.934 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.75, 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.75, 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2, 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2, 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 4, 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "default.rgw.log", 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:33.606516+0000", 2024-03-11T22:13:48.935 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:13:48.936 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:13:48.936 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:13:48.936 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:13:48.936 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:13:48.936 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:13:48.936 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:13:48.936 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:13:48.936 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:13:48.936 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:13:48.936 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:13:48.936 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:13:48.936 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 32, 2024-03-11T22:13:48.937 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 32, 2024-03-11T22:13:48.937 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 32, 2024-03-11T22:13:48.937 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 32, 2024-03-11T22:13:48.937 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 32, 2024-03-11T22:13:48.937 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:13:48.937 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:13:48.937 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:13:48.937 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:13:48.937 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:13:48.937 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:13:48.937 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:13:48.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.938 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "71", 2024-03-11T22:13:48.938 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:13:48.938 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:13:48.938 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:13:48.938 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:13:48.938 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:13:48.938 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:13:48.938 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:13:48.938 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:13:48.938 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:13:48.938 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:13:48.938 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:13:48.939 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:13:48.940 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw": {} 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.9700000286102295, 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.9700000286102295, 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2.25, 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2.25, 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T22:13:48.941 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 5, 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "default.rgw.control", 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:35.315522+0000", 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:13:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 32, 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 32, 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 32, 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 32, 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 32, 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:13:48.943 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "73", 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:13:48.944 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.945 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:13:48.946 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:13:48.946 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:13:48.946 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:13:48.946 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:13:48.946 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:13:48.946 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:13:48.946 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:13:48.946 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:13:48.946 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:13:48.946 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:13:48.946 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:13:48.946 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:13:48.946 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw": {} 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.3200000524520874, 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.3200000524520874, 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 1.5, 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 1.5, 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.948 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 6, 2024-03-11T22:13:48.948 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "default.rgw.meta", 2024-03-11T22:13:48.948 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:37.383856+0000", 2024-03-11T22:13:48.948 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:13:48.948 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:13:48.948 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:13:48.948 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:13:48.948 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:13:48.948 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:13:48.948 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:13:48.948 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:13:48.948 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 32, 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 32, 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 32, 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 32, 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 32, 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:13:48.949 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "75", 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:13:48.950 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:13:48.951 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: "options": { 2024-03-11T22:13:48.952 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_bias": 4 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw": {} 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.5399999618530273, 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.5399999618530273, 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 1.75, 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 1.75, 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T22:13:48.953 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 7, 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "unique_pool_0", 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:10:34.899512+0000", 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:13:48.954 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 16, 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 16, 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 16, 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 16, 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 16, 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:13:48.955 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "102", 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:13:48.956 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:13:48.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:13:48.958 INFO:teuthology.orchestra.run.smithi045.stdout: "rados": {} 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.75, 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.75, 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.1399999856948853, 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2, 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2, 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.875, 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:13:48.959 INFO:teuthology.orchestra.run.smithi045.stdout: "osds": [ 2024-03-11T22:13:48.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 0, 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "55a4e633-f2a6-4d27-b10c-de1c0f87cf32", 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 8, 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 149, 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:13:48.962 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6802", 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6803", 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:13:48.963 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6804", 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6805", 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6808", 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6809", 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.965 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.966 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6806", 2024-03-11T22:13:48.966 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:13:48.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.966 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.966 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6807", 2024-03-11T22:13:48.966 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:13:48.966 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.966 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.966 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6803/2216353820", 2024-03-11T22:13:48.966 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.45:6805/2216353820", 2024-03-11T22:13:48.966 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.45:6809/2216353820", 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.45:6807/2216353820", 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 1, 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "60274eeb-591a-43fe-a579-f13d13631e14", 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:13:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 13, 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 149, 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6810", 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6811", 2024-03-11T22:13:48.968 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:13:48.969 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.969 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.969 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:13:48.969 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.969 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.969 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6812", 2024-03-11T22:13:48.969 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:13:48.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.969 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.969 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6813", 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6816", 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.970 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6817", 2024-03-11T22:13:48.971 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:13:48.971 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.971 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.971 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:13:48.971 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.971 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.971 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6814", 2024-03-11T22:13:48.971 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:13:48.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.971 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.972 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6815", 2024-03-11T22:13:48.972 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:13:48.972 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.972 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.972 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6811/1241486091", 2024-03-11T22:13:48.972 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.45:6813/1241486091", 2024-03-11T22:13:48.972 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.45:6817/1241486091", 2024-03-11T22:13:48.972 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.45:6815/1241486091", 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 2, 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b", 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:13:48.973 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:13:48.974 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 18, 2024-03-11T22:13:48.974 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 99, 2024-03-11T22:13:48.974 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:13:48.974 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:13:48.974 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:13:48.974 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.974 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.974 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6818", 2024-03-11T22:13:48.974 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:13:48.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.974 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6819", 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6820", 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.975 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6821", 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6824", 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.976 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6825", 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6822", 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6823", 2024-03-11T22:13:48.977 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6819/4251983518", 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.45:6821/4251983518", 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.45:6825/4251983518", 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.45:6823/4251983518", 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.978 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 3, 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "e51a65f3-96f6-478c-8d26-df1ace4a5058", 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 26, 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 149, 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.979 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6826", 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6827", 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.980 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6828", 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6829", 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6832", 2024-03-11T22:13:48.981 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6833", 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6830", 2024-03-11T22:13:48.982 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6831", 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6827/3178755830", 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.45:6829/3178755830", 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.45:6833/3178755830", 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.45:6831/3178755830", 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:13:48.983 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 4, 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "32d0b71f-c696-4677-bdd3-b88a04c97604", 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 33, 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 149, 2024-03-11T22:13:48.984 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6800", 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6801", 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:13:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6802", 2024-03-11T22:13:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:13:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6803", 2024-03-11T22:13:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:13:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.986 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:13:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6806", 2024-03-11T22:13:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:13:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.987 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6807", 2024-03-11T22:13:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:13:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:13:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6804", 2024-03-11T22:13:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:13:48.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6805", 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.144:6801/912745159", 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.144:6803/912745159", 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.144:6807/912745159", 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.144:6805/912745159", 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:13:48.989 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 5, 2024-03-11T22:13:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "f46b91af-d67d-404c-aad1-75348e35d1c4", 2024-03-11T22:13:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:13:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:13:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:13:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:13:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:13:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:13:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 41, 2024-03-11T22:13:48.990 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 149, 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6808", 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6809", 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.991 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6810", 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6811", 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6814", 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6815", 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6812", 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6813", 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.994 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.144:6809/3162869174", 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.144:6811/3162869174", 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.144:6815/3162869174", 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.144:6813/3162869174", 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 6, 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "970b128d-cd56-4dff-8345-3196b46c1257", 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:13:48.995 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 51, 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 149, 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6816", 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6817", 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6818", 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.997 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6819", 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6822", 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.998 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6823", 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6820", 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6821", 2024-03-11T22:13:48.999 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.144:6817/3439446288", 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.144:6819/3439446288", 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.144:6823/3439446288", 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.144:6821/3439446288", 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.000 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 7, 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "40100aaf-830d-4137-b037-5cf167cba308", 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 0, 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 61, 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 73, 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.001 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6824", 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6825", 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6826", 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6827", 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.003 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6830", 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6831", 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:13:49.004 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6828", 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6829", 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.144:6825/2371064429", 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.144:6827/2371064429", 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.144:6831/2371064429", 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.144:6829/2371064429", 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:13:49.005 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_xinfo": [ 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 0, 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:12:45.714752+0000", 2024-03-11T22:13:49.006 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 1, 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:12:50.938407+0000", 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 2, 2024-03-11T22:13:49.007 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:12:29.652150+0000", 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 3, 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:13:49.008 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:12:48.679871+0000", 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 4, 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:13:32.444978+0000", 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 5, 2024-03-11T22:13:49.009 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:12:49.541541+0000", 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 6, 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:13:49.010 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:12:59.160130+0000", 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 7, 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:13:07.906897+0000", 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:13:49.011 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_upmap": [], 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_upmap_items": [ 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "4.9", 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: "mappings": [ 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: "from": 1, 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: "to": 5 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "4.b", 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: "mappings": [ 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.012 INFO:teuthology.orchestra.run.smithi045.stdout: "from": 1, 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: "to": 2 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "4.10", 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: "mappings": [ 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: "from": 1, 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: "to": 2 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:13:49.013 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_upmap_primaries": [], 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_temp": [], 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_temp": [], 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "blocklist": { 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/1901139892": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/1110781721": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6800/2299156110": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/2331445880": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6801/4256719031": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6801/2299156110": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6800/3882489981": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/272906373": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/11898002": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/2777740355": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:13:49.014 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/1903379086": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/926341889": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6800/4256719031": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/2513993410": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/2584267737": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/3509006862": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6801/3882489981": "2024-03-12T22:09:14.021723+0000" 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "range_blocklist": {}, 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profiles": { 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "default": { 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "crush-failure-domain": "osd", 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "k": "2", 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "m": "1", 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "plugin": "jerasure", 2024-03-11T22:13:49.015 INFO:teuthology.orchestra.run.smithi045.stdout: "technique": "reed_sol_van" 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps_queue": [], 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: "new_removed_snaps": [], 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: "new_purged_snaps": [], 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_node_flags": {}, 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: "device_class_flags": {}, 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": { 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode_enabled": false, 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_bucket_count": 0, 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: "degraded_stretch_mode": 0, 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: "recovering_stretch_mode": 0, 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode_bucket": 0 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:13:49.016 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:13:49.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:49.057 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:49.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:49.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:49.191 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:49.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:49.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:49.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:49.341 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:49.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:49.442 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:49.471 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:49.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:49.574 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:49.601 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:49.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:49.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:49 smithi045 ceph-mon[39715]: pgmap v204: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s rd, 132 KiB/s wr, 801 op/s; 152 KiB/s, 0 objects/s recovering 2024-03-11T22:13:49.693 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:49 smithi045 ceph-mon[30178]: pgmap v204: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s rd, 132 KiB/s wr, 801 op/s; 152 KiB/s, 0 objects/s recovering 2024-03-11T22:13:49.703 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:49.728 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:49.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:49.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:49.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:49 smithi144 ceph-mon[33984]: pgmap v204: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.9 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s rd, 132 KiB/s wr, 801 op/s; 152 KiB/s, 0 objects/s recovering 2024-03-11T22:13:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:49.877 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:49.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:49.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:50.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:50.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:50.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:50.113 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:50.141 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:50.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:50.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:50.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:50.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:50.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:50.371 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:50.401 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:50.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:50.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:50.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:50.535 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:50.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:50.636 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:50.665 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:50.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:50.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:50.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:50.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:50.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:50.951 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:50.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:50.970 INFO:tasks.thrashosds.thrasher:j is {'epoch': 150, 'fsid': 'f75aafa4-dff2-11ee-95c8-87774f69a715', 'created': '2024-03-11T22:02:51.075940+0000', 'modified': '2024-03-11T22:13:38.551675+0000', 'last_up_change': '2024-03-11T22:08:17.992246+0000', 'last_in_change': '2024-03-11T22:08:00.853801+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T22:06:12.903386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '144', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-11T22:08:31.014060+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T22:08:31.591167+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T22:08:33.606516+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T22:08:35.315522+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T22:08:37.383856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-11T22:10:34.899512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '55a4e633-f2a6-4d27-b10c-de1c0f87cf32', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6802', 'nonce': 2216353820}, {'type': 'v1', 'addr': '172.21.15.45:6803', 'nonce': 2216353820}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6804', 'nonce': 2216353820}, {'type': 'v1', 'addr': '172.21.15.45:6805', 'nonce': 2216353820}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6808', 'nonce': 2216353820}, {'type': 'v1', 'addr': '172.21.15.45:6809', 'nonce': 2216353820}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6806', 'nonce': 2216353820}, {'type': 'v1', 'addr': '172.21.15.45:6807', 'nonce': 2216353820}]}, 'public_addr': '172.21.15.45:6803/2216353820', 'cluster_addr': '172.21.15.45:6805/2216353820', 'heartbeat_back_addr': '172.21.15.45:6809/2216353820', 'heartbeat_front_addr': '172.21.15.45:6807/2216353820', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '60274eeb-591a-43fe-a579-f13d13631e14', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6810', 'nonce': 1241486091}, {'type': 'v1', 'addr': '172.21.15.45:6811', 'nonce': 1241486091}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6812', 'nonce': 1241486091}, {'type': 'v1', 'addr': '172.21.15.45:6813', 'nonce': 1241486091}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6816', 'nonce': 1241486091}, {'type': 'v1', 'addr': '172.21.15.45:6817', 'nonce': 1241486091}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6814', 'nonce': 1241486091}, {'type': 'v1', 'addr': '172.21.15.45:6815', 'nonce': 1241486091}]}, 'public_addr': '172.21.15.45:6811/1241486091', 'cluster_addr': '172.21.15.45:6813/1241486091', 'heartbeat_back_addr': '172.21.15.45:6817/1241486091', 'heartbeat_front_addr': '172.21.15.45:6815/1241486091', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6818', 'nonce': 4251983518}, {'type': 'v1', 'addr': '172.21.15.45:6819', 'nonce': 4251983518}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6820', 'nonce': 4251983518}, {'type': 'v1', 'addr': '172.21.15.45:6821', 'nonce': 4251983518}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6824', 'nonce': 4251983518}, {'type': 'v1', 'addr': '172.21.15.45:6825', 'nonce': 4251983518}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6822', 'nonce': 4251983518}, {'type': 'v1', 'addr': '172.21.15.45:6823', 'nonce': 4251983518}]}, 'public_addr': '172.21.15.45:6819/4251983518', 'cluster_addr': '172.21.15.45:6821/4251983518', 'heartbeat_back_addr': '172.21.15.45:6825/4251983518', 'heartbeat_front_addr': '172.21.15.45:6823/4251983518', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e51a65f3-96f6-478c-8d26-df1ace4a5058', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6826', 'nonce': 3178755830}, {'type': 'v1', 'addr': '172.21.15.45:6827', 'nonce': 3178755830}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6828', 'nonce': 3178755830}, {'type': 'v1', 'addr': '172.21.15.45:6829', 'nonce': 3178755830}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6832', 'nonce': 3178755830}, {'type': 'v1', 'addr': '172.21.15.45:6833', 'nonce': 3178755830}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6830', 'nonce': 3178755830}, {'type': 'v1', 'addr': '172.21.15.45:6831', 'nonce': 3178755830}]}, 'public_addr': '172.21.15.45:6827/3178755830', 'cluster_addr': '172.21.15.45:6829/3178755830', 'heartbeat_back_addr': '172.21.15.45:6833/3178755830', 'heartbeat_front_addr': '172.21.15.45:6831/3178755830', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '32d0b71f-c696-4677-bdd3-b88a04c97604', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6800', 'nonce': 912745159}, {'type': 'v1', 'addr': '172.21.15.144:6801', 'nonce': 912745159}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6802', 'nonce': 912745159}, {'type': 'v1', 'addr': '172.21.15.144:6803', 'nonce': 912745159}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6806', 'nonce': 912745159}, {'type': 'v1', 'addr': '172.21.15.144:6807', 'nonce': 912745159}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6804', 'nonce': 912745159}, {'type': 'v1', 'addr': '172.21.15.144:6805', 'nonce': 912745159}]}, 'public_addr': '172.21.15.144:6801/912745159', 'cluster_addr': '172.21.15.144:6803/912745159', 'heartbeat_back_addr': '172.21.15.144:6807/912745159', 'heartbeat_front_addr': '172.21.15.144:6805/912745159', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f46b91af-d67d-404c-aad1-75348e35d1c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6808', 'nonce': 3162869174}, {'type': 'v1', 'addr': '172.21.15.144:6809', 'nonce': 3162869174}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6810', 'nonce': 3162869174}, {'type': 'v1', 'addr': '172.21.15.144:6811', 'nonce': 3162869174}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6814', 'nonce': 3162869174}, {'type': 'v1', 'addr': '172.21.15.144:6815', 'nonce': 3162869174}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6812', 'nonce': 3162869174}, {'type': 'v1', 'addr': '172.21.15.144:6813', 'nonce': 3162869174}]}, 'public_addr': '172.21.15.144:6809/3162869174', 'cluster_addr': '172.21.15.144:6811/3162869174', 'heartbeat_back_addr': '172.21.15.144:6815/3162869174', 'heartbeat_front_addr': '172.21.15.144:6813/3162869174', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '970b128d-cd56-4dff-8345-3196b46c1257', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6816', 'nonce': 3439446288}, {'type': 'v1', 'addr': '172.21.15.144:6817', 'nonce': 3439446288}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6818', 'nonce': 3439446288}, {'type': 'v1', 'addr': '172.21.15.144:6819', 'nonce': 3439446288}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6822', 'nonce': 3439446288}, {'type': 'v1', 'addr': '172.21.15.144:6823', 'nonce': 3439446288}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6820', 'nonce': 3439446288}, {'type': 'v1', 'addr': '172.21.15.144:6821', 'nonce': 3439446288}]}, 'public_addr': '172.21.15.144:6817/3439446288', 'cluster_addr': '172.21.15.144:6819/3439446288', 'heartbeat_back_addr': '172.21.15.144:6823/3439446288', 'heartbeat_front_addr': '172.21.15.144:6821/3439446288', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '40100aaf-830d-4137-b037-5cf167cba308', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6824', 'nonce': 2371064429}, {'type': 'v1', 'addr': '172.21.15.144:6825', 'nonce': 2371064429}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6826', 'nonce': 2371064429}, {'type': 'v1', 'addr': '172.21.15.144:6827', 'nonce': 2371064429}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6830', 'nonce': 2371064429}, {'type': 'v1', 'addr': '172.21.15.144:6831', 'nonce': 2371064429}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6828', 'nonce': 2371064429}, {'type': 'v1', 'addr': '172.21.15.144:6829', 'nonce': 2371064429}]}, 'public_addr': '172.21.15.144:6825/2371064429', 'cluster_addr': '172.21.15.144:6827/2371064429', 'heartbeat_back_addr': '172.21.15.144:6831/2371064429', 'heartbeat_front_addr': '172.21.15.144:6829/2371064429', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:12:45.714752+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:12:50.938407+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:12:29.652150+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:12:48.679871+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:13:32.444978+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:12:49.541541+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:12:59.160130+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:13:07.906897+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.45:0/1901139892': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/1110781721': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:6800/2299156110': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:0/2331445880': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:6801/4256719031': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:6801/2299156110': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:6800/3882489981': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/272906373': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/11898002': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/2777740355': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:0/1903379086': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/926341889': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:6800/4256719031': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:0/2513993410': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:0/2584267737': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:0/3509006862': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:6801/3882489981': '2024-03-12T22:09:14.021723+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T22:13:50.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:13:51.053 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:51.097 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:51.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:51.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:51.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:51.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:51.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:51.339 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:51.367 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:51.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:51.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:51.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:51.502 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:51.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:51.604 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:51.631 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:51.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:51.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:51.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:51.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:51 smithi045 ceph-mon[39715]: pgmap v205: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.9 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 114 KiB/s wr, 707 op/s; 131 KiB/s, 0 objects/s recovering 2024-03-11T22:13:51.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:51.780 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:51.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:51.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:51 smithi045 ceph-mon[30178]: pgmap v205: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.9 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 114 KiB/s wr, 707 op/s; 131 KiB/s, 0 objects/s recovering 2024-03-11T22:13:51.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:51 smithi144 ceph-mon[33984]: pgmap v205: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.9 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 114 KiB/s wr, 707 op/s; 131 KiB/s, 0 objects/s recovering 2024-03-11T22:13:51.881 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:51.908 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:51.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:52.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:52.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T22:13:52.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:52.052 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:52.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:52.154 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:52.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:52.182 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:52.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:52.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:52.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:52.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:52.427 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:52.455 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:52.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:52.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:52.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:52.598 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:52.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:52.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:52.691 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:13:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:13:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:13:52.700 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:52.725 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:52.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:52.827 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:52.855 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:52.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:52.956 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:52.983 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:52.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:52.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:53.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:53.128 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:53.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:53.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:53.230 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:53.257 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:53.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:53.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:53.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:53.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:53.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:53.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:53.526 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:53.580 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:53.581 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:53.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:53.619 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:13:53.665 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:13:53.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:53.710 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:13:53.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:53.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:53.754 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:13:53.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:53 smithi045 ceph-mon[39715]: pgmap v206: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.7 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 91 KiB/s wr, 934 op/s; 114 KiB/s, 0 objects/s recovering 2024-03-11T22:13:53.754 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:53 smithi045 ceph-mon[39715]: 7.e deep-scrub ok 2024-03-11T22:13:53.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:53 smithi045 ceph-mon[30178]: pgmap v206: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.7 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 91 KiB/s wr, 934 op/s; 114 KiB/s, 0 objects/s recovering 2024-03-11T22:13:53.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:53 smithi045 ceph-mon[30178]: 7.e deep-scrub ok 2024-03-11T22:13:53.791 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:13:53.828 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:13:53.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:53 smithi144 ceph-mon[33984]: pgmap v206: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.7 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 13 KiB/s rd, 91 KiB/s wr, 934 op/s; 114 KiB/s, 0 objects/s recovering 2024-03-11T22:13:53.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:53 smithi144 ceph-mon[33984]: 7.e deep-scrub ok 2024-03-11T22:13:53.846 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:53.871 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:13:53.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:53.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:53.918 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:13:53.924 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:13:53.924 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:13:53.925 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:13:53.925 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:13:53.925 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:13:53.925 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:13:53.925 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:13:53.925 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:13:53.994 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:54.022 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:54.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:54.124 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:54.152 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:54.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:54.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:54.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:54.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:54.399 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:54.435 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:54.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:54.524 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:13:54.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:54.577 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:54.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:54.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:54.682 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:54.711 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:54.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:54.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:54.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:54 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2647867683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:54.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:54 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2434504732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:54.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:54 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:54.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:54 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2647867683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:54.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:54 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2434504732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:54.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:54 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:54.813 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:54.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:54 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2647867683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:54.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:54 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2434504732' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:13:54.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:54 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:13:54.853 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:54.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:54.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:54.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:54.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:54.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:54.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:55.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:55.138 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:55.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:55.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:55.240 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:55.272 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:55.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:55.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:55.457 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:55.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:55.559 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:55.587 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:55.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:55.689 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:55.716 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:55.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:55.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:55.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:55 smithi045 ceph-mon[39715]: pgmap v207: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.7 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 622 op/s 2024-03-11T22:13:55.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:55 smithi045 ceph-mon[39715]: 7.6 deep-scrub ok 2024-03-11T22:13:55.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:55 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:13:55.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:55 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:13:55.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:55 smithi045 ceph-mon[39715]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T22:13:55.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:55 smithi045 ceph-mon[30178]: pgmap v207: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.7 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 622 op/s 2024-03-11T22:13:55.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:55 smithi045 ceph-mon[30178]: 7.6 deep-scrub ok 2024-03-11T22:13:55.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:55 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:13:55.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:55 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:13:55.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:55 smithi045 ceph-mon[30178]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T22:13:55.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:55.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:55 smithi144 ceph-mon[33984]: pgmap v207: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 1.7 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 622 op/s 2024-03-11T22:13:55.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:55 smithi144 ceph-mon[33984]: 7.6 deep-scrub ok 2024-03-11T22:13:55.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:55 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:13:55.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:55 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:13:55.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:55 smithi144 ceph-mon[33984]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T22:13:55.855 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:55.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:55.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:55.957 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:55.987 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:55.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:56.088 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:56.116 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:56.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:56.218 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:56.249 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:56.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:56.342 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:13:55 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:13:56.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:13:56.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:56.387 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:56.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:13:56.488 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:56.517 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:56.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:56.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:56.619 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:56.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:56.655 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:56.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:56.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:56.794 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:56.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:56.895 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:56.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:56.924 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:56.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:57.025 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:57.065 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:57.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:57.167 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:57.198 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:57.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:57.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:57.322 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:57.323 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:57 smithi045 ceph-mon[39715]: pgmap v209: 158 pgs: 158 active+clean; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 789 op/s 2024-03-11T22:13:57.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:57.323 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:57 smithi045 ceph-mon[30178]: pgmap v209: 158 pgs: 158 active+clean; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 789 op/s 2024-03-11T22:13:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:57.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:57.451 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:57.478 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:57.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:57.579 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:13:57.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:13:57.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:57 smithi144 ceph-mon[33984]: pgmap v209: 158 pgs: 158 active+clean; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 789 op/s 2024-03-11T22:13:57.607 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:57.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:13:57.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:57.746 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:57.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:57.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:57.888 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:57.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:57.942 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:13:57.947 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:13:57.990 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:58.017 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:58.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:58.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:58.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:58.158 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:58.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:58.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:58.246 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:58 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:13:58.247 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:58 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:13:58.247 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:58 smithi045 ceph-mon[30178]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T22:13:58.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:58 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:13:58.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:58 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:13:58.247 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:58 smithi045 ceph-mon[39715]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T22:13:58.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:13:58.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:58.310 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:58.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:13:58.410 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:13:58.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:58.441 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:58.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:13:58.542 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:58.572 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:58.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:58.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:58.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:58 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:13:58.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:58 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:13:58.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:58 smithi144 ceph-mon[33984]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T22:13:58.674 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:58.721 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:58.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:58.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:58.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:13:58.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:58.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:58.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:58.989 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:59.016 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:59.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:59.118 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:13:59.145 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:59.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:13:59.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:13:59.290 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:59.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:59.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:13:59.302 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-03-11T22:13:57.146394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1538422313,"num_objects":23665,"num_object_clones":0,"num_object_copies":70995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23665,"num_whiteouts":0,"num_read":1175,"num_read_kb":1005,"num_write":214052,"num_write_kb":7583975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":2039783,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86744,"ondisk_log_size":86744,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5603816,"kb_used_data":4683792,"kb_used_omap":12,"kb_used_meta":919859,"kb_avail":744357400,"statfs":{"total":767960285184,"available":762221977600,"internally_reserved":0,"allocated":4796203008,"data_stored":4793614018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":941936179},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,4,15,6,17,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-516816896,"num_objects":-7886,"num_object_clones":0,"num_object_copies":-23658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7886,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7886,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.005769"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":175,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.530945+0000","last_change":"2024-03-11T22:11:30.350663+0000","last_active":"2024-03-11T22:13:54.530945+0000","last_peered":"2024-03-11T22:13:54.530945+0000","last_clean":"2024-03-11T22:13:54.530945+0000","last_became_active":"2024-03-11T22:08:39.074695+0000","last_became_peered":"2024-03-11T22:08:39.074695+0000","last_unstale":"2024-03-11T22:13:54.530945+0000","last_undegraded":"2024-03-11T22:13:54.530945+0000","last_fullsized":"2024-03-11T22:13:54.530945+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:30.350540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.350540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.529141+0000","last_change":"2024-03-11T22:12:45.929428+0000","last_active":"2024-03-11T22:13:54.529141+0000","last_peered":"2024-03-11T22:13:54.529141+0000","last_clean":"2024-03-11T22:13:54.529141+0000","last_became_active":"2024-03-11T22:08:37.073775+0000","last_became_peered":"2024-03-11T22:08:37.073775+0000","last_unstale":"2024-03-11T22:13:54.529141+0000","last_undegraded":"2024-03-11T22:13:54.529141+0000","last_fullsized":"2024-03-11T22:13:54.529141+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.928689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.928689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:15.589951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.547003+0000","last_change":"2024-03-11T22:11:45.340779+0000","last_active":"2024-03-11T22:13:54.547003+0000","last_peered":"2024-03-11T22:13:54.547003+0000","last_clean":"2024-03-11T22:13:54.547003+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:13:54.547003+0000","last_undegraded":"2024-03-11T22:13:54.547003+0000","last_fullsized":"2024-03-11T22:13:54.547003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:45.340657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:45.340657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.964056+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":199,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.530729+0000","last_change":"2024-03-11T22:10:13.507406+0000","last_active":"2024-03-11T22:13:54.530729+0000","last_peered":"2024-03-11T22:13:54.530729+0000","last_clean":"2024-03-11T22:13:54.530729+0000","last_became_active":"2024-03-11T22:08:35.053179+0000","last_became_peered":"2024-03-11T22:08:35.053179+0000","last_unstale":"2024-03-11T22:13:54.530729+0000","last_undegraded":"2024-03-11T22:13:54.530729+0000","last_fullsized":"2024-03-11T22:13:54.530729+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:10:13.507312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:10:13.507312+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.535115+0000","last_change":"2024-03-11T22:12:50.533192+0000","last_active":"2024-03-11T22:13:54.535115+0000","last_peered":"2024-03-11T22:13:54.535115+0000","last_clean":"2024-03-11T22:13:54.535115+0000","last_became_active":"2024-03-11T22:08:39.079045+0000","last_became_peered":"2024-03-11T22:08:39.079045+0000","last_unstale":"2024-03-11T22:13:54.535115+0000","last_undegraded":"2024-03-11T22:13:54.535115+0000","last_fullsized":"2024-03-11T22:13:54.535115+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.533067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.533067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:07.226337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.530842+0000","last_change":"2024-03-11T22:11:43.150997+0000","last_active":"2024-03-11T22:13:54.530842+0000","last_peered":"2024-03-11T22:13:54.530842+0000","last_clean":"2024-03-11T22:13:54.530842+0000","last_became_active":"2024-03-11T22:08:37.081861+0000","last_became_peered":"2024-03-11T22:08:37.081861+0000","last_unstale":"2024-03-11T22:13:54.530842+0000","last_undegraded":"2024-03-11T22:13:54.530842+0000","last_fullsized":"2024-03-11T22:13:54.530842+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:43.150872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:43.150872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":209,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.534626+0000","last_change":"2024-03-11T22:13:19.463087+0000","last_active":"2024-03-11T22:13:54.534626+0000","last_peered":"2024-03-11T22:13:54.534626+0000","last_clean":"2024-03-11T22:13:54.534626+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:13:54.534626+0000","last_undegraded":"2024-03-11T22:13:54.534626+0000","last_fullsized":"2024-03-11T22:13:54.534626+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:13:19.462955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.462955+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:30.805293+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":216,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531714+0000","last_change":"2024-03-11T22:12:49.510144+0000","last_active":"2024-03-11T22:13:54.531714+0000","last_peered":"2024-03-11T22:13:54.531714+0000","last_clean":"2024-03-11T22:13:54.531714+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:13:54.531714+0000","last_undegraded":"2024-03-11T22:13:54.531714+0000","last_fullsized":"2024-03-11T22:13:54.531714+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:12:49.509860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.509860+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:15.173673+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.550492+0000","last_change":"2024-03-11T22:13:25.042445+0000","last_active":"2024-03-11T22:13:54.550492+0000","last_peered":"2024-03-11T22:13:54.550492+0000","last_clean":"2024-03-11T22:13:54.550492+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:13:54.550492+0000","last_undegraded":"2024-03-11T22:13:54.550492+0000","last_fullsized":"2024-03-11T22:13:54.550492+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:25.042307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:25.042307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:43.208699+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526175+0000","last_change":"2024-03-11T22:12:19.337740+0000","last_active":"2024-03-11T22:13:54.526175+0000","last_peered":"2024-03-11T22:13:54.526175+0000","last_clean":"2024-03-11T22:13:54.526175+0000","last_became_active":"2024-03-11T22:08:37.081996+0000","last_became_peered":"2024-03-11T22:08:37.081996+0000","last_unstale":"2024-03-11T22:13:54.526175+0000","last_undegraded":"2024-03-11T22:13:54.526175+0000","last_fullsized":"2024-03-11T22:13:54.526175+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:19.337662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:12:19.337662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526443+0000","last_change":"2024-03-11T22:12:47.432839+0000","last_active":"2024-03-11T22:13:54.526443+0000","last_peered":"2024-03-11T22:13:54.526443+0000","last_clean":"2024-03-11T22:13:54.526443+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:13:54.526443+0000","last_undegraded":"2024-03-11T22:13:54.526443+0000","last_fullsized":"2024-03-11T22:13:54.526443+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:47.432769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:47.432769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:59.023738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":207,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531109+0000","last_change":"2024-03-11T22:11:45.169864+0000","last_active":"2024-03-11T22:13:54.531109+0000","last_peered":"2024-03-11T22:13:54.531109+0000","last_clean":"2024-03-11T22:13:54.531109+0000","last_became_active":"2024-03-11T22:08:35.060802+0000","last_became_peered":"2024-03-11T22:08:35.060802+0000","last_unstale":"2024-03-11T22:13:54.531109+0000","last_undegraded":"2024-03-11T22:13:54.531109+0000","last_fullsized":"2024-03-11T22:13:54.531109+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:11:45.169706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:45.169706+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525810+0000","last_change":"2024-03-11T22:11:47.166498+0000","last_active":"2024-03-11T22:13:54.525810+0000","last_peered":"2024-03-11T22:13:54.525810+0000","last_clean":"2024-03-11T22:13:54.525810+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:13:54.525810+0000","last_undegraded":"2024-03-11T22:13:54.525810+0000","last_fullsized":"2024-03-11T22:13:54.525810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:47.166360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:47.166360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":125,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531218+0000","last_change":"2024-03-11T22:13:36.039706+0000","last_active":"2024-03-11T22:13:54.531218+0000","last_peered":"2024-03-11T22:13:54.531218+0000","last_clean":"2024-03-11T22:13:54.531218+0000","last_became_active":"2024-03-11T22:10:35.410624+0000","last_became_peered":"2024-03-11T22:10:35.410624+0000","last_unstale":"2024-03-11T22:13:54.531218+0000","last_undegraded":"2024-03-11T22:13:54.531218+0000","last_fullsized":"2024-03-11T22:13:54.531218+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:36.039591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.270166+0000","last_clean_scrub_stamp":"2024-03-11T22:13:36.039591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:46.694507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526241+0000","last_change":"2024-03-11T22:12:24.388572+0000","last_active":"2024-03-11T22:13:54.526241+0000","last_peered":"2024-03-11T22:13:54.526241+0000","last_clean":"2024-03-11T22:13:54.526241+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:13:54.526241+0000","last_undegraded":"2024-03-11T22:13:54.526241+0000","last_fullsized":"2024-03-11T22:13:54.526241+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:24.388497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.388497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:57.548330+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"95'7","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531763+0000","last_change":"2024-03-11T22:13:32.041334+0000","last_active":"2024-03-11T22:13:54.531763+0000","last_peered":"2024-03-11T22:13:54.531763+0000","last_clean":"2024-03-11T22:13:54.531763+0000","last_became_active":"2024-03-11T22:08:35.060777+0000","last_became_peered":"2024-03-11T22:08:35.060777+0000","last_unstale":"2024-03-11T22:13:54.531763+0000","last_undegraded":"2024-03-11T22:13:54.531763+0000","last_fullsized":"2024-03-11T22:13:54.531763+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'7","last_scrub_stamp":"2024-03-11T22:13:32.041219+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:13:32.041219+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:57.879414+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531632+0000","last_change":"2024-03-11T22:12:32.044430+0000","last_active":"2024-03-11T22:13:54.531632+0000","last_peered":"2024-03-11T22:13:54.531632+0000","last_clean":"2024-03-11T22:13:54.531632+0000","last_became_active":"2024-03-11T22:08:39.075609+0000","last_became_peered":"2024-03-11T22:08:39.075609+0000","last_unstale":"2024-03-11T22:13:54.531632+0000","last_undegraded":"2024-03-11T22:13:54.531632+0000","last_fullsized":"2024-03-11T22:13:54.531632+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:32.044320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:32.044320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.522527+0000","last_change":"2024-03-11T22:12:30.962293+0000","last_active":"2024-03-11T22:13:54.522527+0000","last_peered":"2024-03-11T22:13:54.522527+0000","last_clean":"2024-03-11T22:13:54.522527+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:13:54.522527+0000","last_undegraded":"2024-03-11T22:13:54.522527+0000","last_fullsized":"2024-03-11T22:13:54.522527+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:30.962171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:30.962171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.129955+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.547557+0000","last_change":"2024-03-11T22:12:22.649189+0000","last_active":"2024-03-11T22:13:54.547557+0000","last_peered":"2024-03-11T22:13:54.547557+0000","last_clean":"2024-03-11T22:13:54.547557+0000","last_became_active":"2024-03-11T22:08:33.035624+0000","last_became_peered":"2024-03-11T22:08:33.035624+0000","last_unstale":"2024-03-11T22:13:54.547557+0000","last_undegraded":"2024-03-11T22:13:54.547557+0000","last_fullsized":"2024-03-11T22:13:54.547557+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:22.649073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:22.649073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.964082+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":224,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.523793+0000","last_change":"2024-03-11T22:12:27.670423+0000","last_active":"2024-03-11T22:13:54.523793+0000","last_peered":"2024-03-11T22:13:54.523793+0000","last_clean":"2024-03-11T22:13:54.523793+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:13:54.523793+0000","last_undegraded":"2024-03-11T22:13:54.523793+0000","last_fullsized":"2024-03-11T22:13:54.523793+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:12:27.670284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:27.670284+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.958112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.537530+0000","last_change":"2024-03-11T22:12:44.464428+0000","last_active":"2024-03-11T22:13:54.537530+0000","last_peered":"2024-03-11T22:13:54.537530+0000","last_clean":"2024-03-11T22:13:54.537530+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:13:54.537530+0000","last_undegraded":"2024-03-11T22:13:54.537530+0000","last_fullsized":"2024-03-11T22:13:54.537530+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.464308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.464308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:02.369972+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":179,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.532127+0000","last_change":"2024-03-11T22:11:27.391809+0000","last_active":"2024-03-11T22:13:54.532127+0000","last_peered":"2024-03-11T22:13:54.532127+0000","last_clean":"2024-03-11T22:13:54.532127+0000","last_became_active":"2024-03-11T22:08:37.080014+0000","last_became_peered":"2024-03-11T22:08:37.080014+0000","last_unstale":"2024-03-11T22:13:54.532127+0000","last_undegraded":"2024-03-11T22:13:54.532127+0000","last_fullsized":"2024-03-11T22:13:54.532127+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:27.391727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:27.391727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531114+0000","last_change":"2024-03-11T22:13:22.058907+0000","last_active":"2024-03-11T22:13:54.531114+0000","last_peered":"2024-03-11T22:13:54.531114+0000","last_clean":"2024-03-11T22:13:54.531114+0000","last_became_active":"2024-03-11T22:08:33.299189+0000","last_became_peered":"2024-03-11T22:08:33.299189+0000","last_unstale":"2024-03-11T22:13:54.531114+0000","last_undegraded":"2024-03-11T22:13:54.531114+0000","last_fullsized":"2024-03-11T22:13:54.531114+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:22.058795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:22.058795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:25.056228+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":214,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.532169+0000","last_change":"2024-03-11T22:12:28.292078+0000","last_active":"2024-03-11T22:13:54.532169+0000","last_peered":"2024-03-11T22:13:54.532169+0000","last_clean":"2024-03-11T22:13:54.532169+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:13:54.532169+0000","last_undegraded":"2024-03-11T22:13:54.532169+0000","last_fullsized":"2024-03-11T22:13:54.532169+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:12:28.291919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:28.291919+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":144,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526084+0000","last_change":"2024-03-11T22:13:21.532369+0000","last_active":"2024-03-11T22:13:54.526084+0000","last_peered":"2024-03-11T22:13:54.526084+0000","last_clean":"2024-03-11T22:13:54.526084+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:13:54.526084+0000","last_undegraded":"2024-03-11T22:13:54.526084+0000","last_fullsized":"2024-03-11T22:13:54.526084+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:13:21.532280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:21.532280+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:45.199055+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":271,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525089+0000","last_change":"2024-03-11T22:12:35.862998+0000","last_active":"2024-03-11T22:13:54.525089+0000","last_peered":"2024-03-11T22:13:54.525089+0000","last_clean":"2024-03-11T22:13:54.525089+0000","last_became_active":"2024-03-11T22:08:37.078588+0000","last_became_peered":"2024-03-11T22:08:37.078588+0000","last_unstale":"2024-03-11T22:13:54.525089+0000","last_undegraded":"2024-03-11T22:13:54.525089+0000","last_fullsized":"2024-03-11T22:13:54.525089+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:12:35.862855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.862855+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:00.615190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.529228+0000","last_change":"2024-03-11T22:12:24.919926+0000","last_active":"2024-03-11T22:13:54.529228+0000","last_peered":"2024-03-11T22:13:54.529228+0000","last_clean":"2024-03-11T22:13:54.529228+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:13:54.529228+0000","last_undegraded":"2024-03-11T22:13:54.529228+0000","last_fullsized":"2024-03-11T22:13:54.529228+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:24.919682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.919682+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:54.706624+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":206,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.559434+0000","last_change":"2024-03-11T22:13:16.935454+0000","last_active":"2024-03-11T22:13:54.559434+0000","last_peered":"2024-03-11T22:13:54.559434+0000","last_clean":"2024-03-11T22:13:54.559434+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:13:54.559434+0000","last_undegraded":"2024-03-11T22:13:54.559434+0000","last_fullsized":"2024-03-11T22:13:54.559434+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:13:16.935159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:16.935159+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:36.018341+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":191,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.522728+0000","last_change":"2024-03-11T22:13:30.794816+0000","last_active":"2024-03-11T22:13:54.522728+0000","last_peered":"2024-03-11T22:13:54.522728+0000","last_clean":"2024-03-11T22:13:54.522728+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:13:54.522728+0000","last_undegraded":"2024-03-11T22:13:54.522728+0000","last_fullsized":"2024-03-11T22:13:54.522728+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:30.794739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:30.794739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:42.625116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.560013+0000","last_change":"2024-03-11T22:11:37.288072+0000","last_active":"2024-03-11T22:13:54.560013+0000","last_peered":"2024-03-11T22:13:54.560013+0000","last_clean":"2024-03-11T22:13:54.560013+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:13:54.560013+0000","last_undegraded":"2024-03-11T22:13:54.560013+0000","last_fullsized":"2024-03-11T22:13:54.560013+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:37.287951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:37.287951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.963962+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.534976+0000","last_change":"2024-03-11T22:12:43.451182+0000","last_active":"2024-03-11T22:13:54.534976+0000","last_peered":"2024-03-11T22:13:54.534976+0000","last_clean":"2024-03-11T22:13:54.534976+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:13:54.534976+0000","last_undegraded":"2024-03-11T22:13:54.534976+0000","last_fullsized":"2024-03-11T22:13:54.534976+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:43.451074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:43.451074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:06.309610+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":218,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524984+0000","last_change":"2024-03-11T22:13:17.923943+0000","last_active":"2024-03-11T22:13:54.524984+0000","last_peered":"2024-03-11T22:13:54.524984+0000","last_clean":"2024-03-11T22:13:54.524984+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:13:54.524984+0000","last_undegraded":"2024-03-11T22:13:54.524984+0000","last_fullsized":"2024-03-11T22:13:54.524984+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:13:17.923797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:17.923797+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:38.893975+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":133,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526626+0000","last_change":"2024-03-11T22:13:27.763590+0000","last_active":"2024-03-11T22:13:54.526626+0000","last_peered":"2024-03-11T22:13:54.526626+0000","last_clean":"2024-03-11T22:13:54.526626+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:13:54.526626+0000","last_undegraded":"2024-03-11T22:13:54.526626+0000","last_fullsized":"2024-03-11T22:13:54.526626+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:27.763466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:27.763466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:48.983317+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524408+0000","last_change":"2024-03-11T22:12:44.941384+0000","last_active":"2024-03-11T22:13:54.524408+0000","last_peered":"2024-03-11T22:13:54.524408+0000","last_clean":"2024-03-11T22:13:54.524408+0000","last_became_active":"2024-03-11T22:08:37.075341+0000","last_became_peered":"2024-03-11T22:08:37.075341+0000","last_unstale":"2024-03-11T22:13:54.524408+0000","last_undegraded":"2024-03-11T22:13:54.524408+0000","last_fullsized":"2024-03-11T22:13:54.524408+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.941275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.941275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:04.779620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.547637+0000","last_change":"2024-03-11T22:13:32.033557+0000","last_active":"2024-03-11T22:13:54.547637+0000","last_peered":"2024-03-11T22:13:54.547637+0000","last_clean":"2024-03-11T22:13:54.547637+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:13:54.547637+0000","last_undegraded":"2024-03-11T22:13:54.547637+0000","last_fullsized":"2024-03-11T22:13:54.547637+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:32.033444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:32.033444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:32.453949+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":217,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.535070+0000","last_change":"2024-03-11T22:13:31.491151+0000","last_active":"2024-03-11T22:13:54.535070+0000","last_peered":"2024-03-11T22:13:54.535070+0000","last_clean":"2024-03-11T22:13:54.535070+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:13:54.535070+0000","last_undegraded":"2024-03-11T22:13:54.535070+0000","last_fullsized":"2024-03-11T22:13:54.535070+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-11T22:13:31.490878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:31.490878+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:50.850169+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.535412+0000","last_change":"2024-03-11T22:11:46.050869+0000","last_active":"2024-03-11T22:13:54.535412+0000","last_peered":"2024-03-11T22:13:54.535412+0000","last_clean":"2024-03-11T22:13:54.535412+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:13:54.535412+0000","last_undegraded":"2024-03-11T22:13:54.535412+0000","last_fullsized":"2024-03-11T22:13:54.535412+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:46.050747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:46.050747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":125,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.530669+0000","last_change":"2024-03-11T22:11:42.175942+0000","last_active":"2024-03-11T22:13:54.530669+0000","last_peered":"2024-03-11T22:13:54.530669+0000","last_clean":"2024-03-11T22:13:54.530669+0000","last_became_active":"2024-03-11T22:10:35.415813+0000","last_became_peered":"2024-03-11T22:10:35.415813+0000","last_unstale":"2024-03-11T22:13:54.530669+0000","last_undegraded":"2024-03-11T22:13:54.530669+0000","last_fullsized":"2024-03-11T22:13:54.530669+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:11:42.175852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526697+0000","last_change":"2024-03-11T22:13:29.550985+0000","last_active":"2024-03-11T22:13:54.526697+0000","last_peered":"2024-03-11T22:13:54.526697+0000","last_clean":"2024-03-11T22:13:54.526697+0000","last_became_active":"2024-03-11T22:08:33.039349+0000","last_became_peered":"2024-03-11T22:08:33.039349+0000","last_unstale":"2024-03-11T22:13:54.526697+0000","last_undegraded":"2024-03-11T22:13:54.526697+0000","last_fullsized":"2024-03-11T22:13:54.526697+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:29.550889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:29.550889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:57.436176+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":210,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531607+0000","last_change":"2024-03-11T22:13:18.486183+0000","last_active":"2024-03-11T22:13:54.531607+0000","last_peered":"2024-03-11T22:13:54.531607+0000","last_clean":"2024-03-11T22:13:54.531607+0000","last_became_active":"2024-03-11T22:08:35.085928+0000","last_became_peered":"2024-03-11T22:08:35.085928+0000","last_unstale":"2024-03-11T22:13:54.531607+0000","last_undegraded":"2024-03-11T22:13:54.531607+0000","last_fullsized":"2024-03-11T22:13:54.531607+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:13:18.486039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:18.486039+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:24.736759+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.550156+0000","last_change":"2024-03-11T22:12:46.829082+0000","last_active":"2024-03-11T22:13:54.550156+0000","last_peered":"2024-03-11T22:13:54.550156+0000","last_clean":"2024-03-11T22:13:54.550156+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:13:54.550156+0000","last_undegraded":"2024-03-11T22:13:54.550156+0000","last_fullsized":"2024-03-11T22:13:54.550156+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.828942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.828942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:59.694733+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.535024+0000","last_change":"2024-03-11T22:11:37.076127+0000","last_active":"2024-03-11T22:13:54.535024+0000","last_peered":"2024-03-11T22:13:54.535024+0000","last_clean":"2024-03-11T22:13:54.535024+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:13:54.535024+0000","last_undegraded":"2024-03-11T22:13:54.535024+0000","last_fullsized":"2024-03-11T22:13:54.535024+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:11:37.075868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":126,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.532416+0000","last_change":"2024-03-11T22:11:38.050014+0000","last_active":"2024-03-11T22:13:54.532416+0000","last_peered":"2024-03-11T22:13:54.532416+0000","last_clean":"2024-03-11T22:13:54.532416+0000","last_became_active":"2024-03-11T22:10:35.414121+0000","last_became_peered":"2024-03-11T22:10:35.414121+0000","last_unstale":"2024-03-11T22:13:54.532416+0000","last_undegraded":"2024-03-11T22:13:54.532416+0000","last_fullsized":"2024-03-11T22:13:54.532416+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:38.049889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:38.049889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":215,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525682+0000","last_change":"2024-03-11T22:13:31.796744+0000","last_active":"2024-03-11T22:13:54.525682+0000","last_peered":"2024-03-11T22:13:54.525682+0000","last_clean":"2024-03-11T22:13:54.525682+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:13:54.525682+0000","last_undegraded":"2024-03-11T22:13:54.525682+0000","last_fullsized":"2024-03-11T22:13:54.525682+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:13:31.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:31.796611+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:49.930017+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531227+0000","last_change":"2024-03-11T22:12:46.492608+0000","last_active":"2024-03-11T22:13:54.531227+0000","last_peered":"2024-03-11T22:13:54.531227+0000","last_clean":"2024-03-11T22:13:54.531227+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:13:54.531227+0000","last_undegraded":"2024-03-11T22:13:54.531227+0000","last_fullsized":"2024-03-11T22:13:54.531227+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.492494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.492494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:11.071230+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525451+0000","last_change":"2024-03-11T22:12:41.988328+0000","last_active":"2024-03-11T22:13:54.525451+0000","last_peered":"2024-03-11T22:13:54.525451+0000","last_clean":"2024-03-11T22:13:54.525451+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:13:54.525451+0000","last_undegraded":"2024-03-11T22:13:54.525451+0000","last_fullsized":"2024-03-11T22:13:54.525451+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:41.988094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:41.988094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:07.625482+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":201,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531802+0000","last_change":"2024-03-11T22:12:18.154605+0000","last_active":"2024-03-11T22:13:54.531802+0000","last_peered":"2024-03-11T22:13:54.531802+0000","last_clean":"2024-03-11T22:13:54.531802+0000","last_became_active":"2024-03-11T22:08:33.036083+0000","last_became_peered":"2024-03-11T22:08:33.036083+0000","last_unstale":"2024-03-11T22:13:54.531802+0000","last_undegraded":"2024-03-11T22:13:54.531802+0000","last_fullsized":"2024-03-11T22:13:54.531802+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:18.154515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:18.154515+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":202,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531878+0000","last_change":"2024-03-11T22:11:19.294807+0000","last_active":"2024-03-11T22:13:54.531878+0000","last_peered":"2024-03-11T22:13:54.531878+0000","last_clean":"2024-03-11T22:13:54.531878+0000","last_became_active":"2024-03-11T22:08:35.054726+0000","last_became_peered":"2024-03-11T22:08:35.054726+0000","last_unstale":"2024-03-11T22:13:54.531878+0000","last_undegraded":"2024-03-11T22:13:54.531878+0000","last_fullsized":"2024-03-11T22:13:54.531878+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:11:19.294671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:19.294671+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":175,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.547694+0000","last_change":"2024-03-11T22:11:13.259520+0000","last_active":"2024-03-11T22:13:54.547694+0000","last_peered":"2024-03-11T22:13:54.547694+0000","last_clean":"2024-03-11T22:13:54.547694+0000","last_became_active":"2024-03-11T22:08:39.082938+0000","last_became_peered":"2024-03-11T22:08:39.082938+0000","last_unstale":"2024-03-11T22:13:54.547694+0000","last_undegraded":"2024-03-11T22:13:54.547694+0000","last_fullsized":"2024-03-11T22:13:54.547694+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:13.259407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:13.259407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.963865+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":268,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526477+0000","last_change":"2024-03-11T22:13:28.520411+0000","last_active":"2024-03-11T22:13:54.526477+0000","last_peered":"2024-03-11T22:13:54.526477+0000","last_clean":"2024-03-11T22:13:54.526477+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:13:54.526477+0000","last_undegraded":"2024-03-11T22:13:54.526477+0000","last_fullsized":"2024-03-11T22:13:54.526477+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:13:28.520283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:13:28.520283+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:34.669903+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":126,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531679+0000","last_change":"2024-03-11T22:11:39.174355+0000","last_active":"2024-03-11T22:13:54.531679+0000","last_peered":"2024-03-11T22:13:54.531679+0000","last_clean":"2024-03-11T22:13:54.531679+0000","last_became_active":"2024-03-11T22:10:35.407845+0000","last_became_peered":"2024-03-11T22:10:35.407845+0000","last_unstale":"2024-03-11T22:13:54.531679+0000","last_undegraded":"2024-03-11T22:13:54.531679+0000","last_fullsized":"2024-03-11T22:13:54.531679+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:39.174103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:39.174103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":206,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.532915+0000","last_change":"2024-03-11T22:12:20.295562+0000","last_active":"2024-03-11T22:13:54.532915+0000","last_peered":"2024-03-11T22:13:54.532915+0000","last_clean":"2024-03-11T22:13:54.532915+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:13:54.532915+0000","last_undegraded":"2024-03-11T22:13:54.532915+0000","last_fullsized":"2024-03-11T22:13:54.532915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:12:20.295427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:20.295427+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":175,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531930+0000","last_change":"2024-03-11T22:11:31.308182+0000","last_active":"2024-03-11T22:13:54.531930+0000","last_peered":"2024-03-11T22:13:54.531930+0000","last_clean":"2024-03-11T22:13:54.531930+0000","last_became_active":"2024-03-11T22:08:39.072089+0000","last_became_peered":"2024-03-11T22:08:39.072089+0000","last_unstale":"2024-03-11T22:13:54.531930+0000","last_undegraded":"2024-03-11T22:13:54.531930+0000","last_fullsized":"2024-03-11T22:13:54.531930+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:31.308103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:31.308103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":263,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.533054+0000","last_change":"2024-03-11T22:11:44.093057+0000","last_active":"2024-03-11T22:13:54.533054+0000","last_peered":"2024-03-11T22:13:54.533054+0000","last_clean":"2024-03-11T22:13:54.533054+0000","last_became_active":"2024-03-11T22:08:37.071202+0000","last_became_peered":"2024-03-11T22:08:37.071202+0000","last_unstale":"2024-03-11T22:13:54.533054+0000","last_undegraded":"2024-03-11T22:13:54.533054+0000","last_fullsized":"2024-03-11T22:13:54.533054+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:11:44.092784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:44.092784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.546896+0000","last_change":"2024-03-11T22:12:29.629914+0000","last_active":"2024-03-11T22:13:54.546896+0000","last_peered":"2024-03-11T22:13:54.546896+0000","last_clean":"2024-03-11T22:13:54.546896+0000","last_became_active":"2024-03-11T22:08:33.032850+0000","last_became_peered":"2024-03-11T22:08:33.032850+0000","last_unstale":"2024-03-11T22:13:54.546896+0000","last_undegraded":"2024-03-11T22:13:54.546896+0000","last_fullsized":"2024-03-11T22:13:54.546896+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:29.629778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:29.629778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:54.999031+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":215,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526626+0000","last_change":"2024-03-11T22:13:30.515940+0000","last_active":"2024-03-11T22:13:54.526626+0000","last_peered":"2024-03-11T22:13:54.526626+0000","last_clean":"2024-03-11T22:13:54.526626+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:13:54.526626+0000","last_undegraded":"2024-03-11T22:13:54.526626+0000","last_fullsized":"2024-03-11T22:13:54.526626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:13:30.515836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:30.515836+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:33.944436+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":191,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524030+0000","last_change":"2024-03-11T22:12:40.777949+0000","last_active":"2024-03-11T22:13:54.524030+0000","last_peered":"2024-03-11T22:13:54.524030+0000","last_clean":"2024-03-11T22:13:54.524030+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:13:54.524030+0000","last_undegraded":"2024-03-11T22:13:54.524030+0000","last_fullsized":"2024-03-11T22:13:54.524030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:40.777821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:40.777821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.958200+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.537911+0000","last_change":"2024-03-11T22:11:31.040056+0000","last_active":"2024-03-11T22:13:54.537911+0000","last_peered":"2024-03-11T22:13:54.537911+0000","last_clean":"2024-03-11T22:13:54.537911+0000","last_became_active":"2024-03-11T22:08:37.069611+0000","last_became_peered":"2024-03-11T22:08:37.069611+0000","last_unstale":"2024-03-11T22:13:54.537911+0000","last_undegraded":"2024-03-11T22:13:54.537911+0000","last_fullsized":"2024-03-11T22:13:54.537911+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:31.039934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:31.039934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":118,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531305+0000","last_change":"2024-03-11T22:10:35.416311+0000","last_active":"2024-03-11T22:13:54.531305+0000","last_peered":"2024-03-11T22:13:54.531305+0000","last_clean":"2024-03-11T22:13:54.531305+0000","last_became_active":"2024-03-11T22:10:35.415644+0000","last_became_peered":"2024-03-11T22:10:35.415644+0000","last_unstale":"2024-03-11T22:13:54.531305+0000","last_undegraded":"2024-03-11T22:13:54.531305+0000","last_fullsized":"2024-03-11T22:13:54.531305+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:12.415483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:10:12.415483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"93'13","reported_seq":211,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.559846+0000","last_change":"2024-03-11T22:11:35.407557+0000","last_active":"2024-03-11T22:13:54.559846+0000","last_peered":"2024-03-11T22:13:54.559846+0000","last_clean":"2024-03-11T22:13:54.559846+0000","last_became_active":"2024-03-11T22:08:35.085787+0000","last_became_peered":"2024-03-11T22:08:35.085787+0000","last_unstale":"2024-03-11T22:13:54.559846+0000","last_undegraded":"2024-03-11T22:13:54.559846+0000","last_fullsized":"2024-03-11T22:13:54.559846+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:11:35.407301+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.407301+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.963993+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":133,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525187+0000","last_change":"2024-03-11T22:13:37.172811+0000","last_active":"2024-03-11T22:13:54.525187+0000","last_peered":"2024-03-11T22:13:54.525187+0000","last_clean":"2024-03-11T22:13:54.525187+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:13:54.525187+0000","last_undegraded":"2024-03-11T22:13:54.525187+0000","last_fullsized":"2024-03-11T22:13:54.525187+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:13:37.172700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:00.602583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526709+0000","last_change":"2024-03-11T22:13:25.550514+0000","last_active":"2024-03-11T22:13:54.526709+0000","last_peered":"2024-03-11T22:13:54.526709+0000","last_clean":"2024-03-11T22:13:54.526709+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:13:54.526709+0000","last_undegraded":"2024-03-11T22:13:54.526709+0000","last_fullsized":"2024-03-11T22:13:54.526709+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:25.550429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:07.195289+0000","last_clean_scrub_stamp":"2024-03-11T22:13:25.550429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:50.191276+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524121+0000","last_change":"2024-03-11T22:13:28.086496+0000","last_active":"2024-03-11T22:13:54.524121+0000","last_peered":"2024-03-11T22:13:54.524121+0000","last_clean":"2024-03-11T22:13:54.524121+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:13:54.524121+0000","last_undegraded":"2024-03-11T22:13:54.524121+0000","last_fullsized":"2024-03-11T22:13:54.524121+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:28.086387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:13:28.086387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:54.604819+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":200,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.538000+0000","last_change":"2024-03-11T22:12:21.278151+0000","last_active":"2024-03-11T22:13:54.538000+0000","last_peered":"2024-03-11T22:13:54.538000+0000","last_clean":"2024-03-11T22:13:54.538000+0000","last_became_active":"2024-03-11T22:08:35.085914+0000","last_became_peered":"2024-03-11T22:08:35.085914+0000","last_unstale":"2024-03-11T22:13:54.538000+0000","last_undegraded":"2024-03-11T22:13:54.538000+0000","last_fullsized":"2024-03-11T22:13:54.538000+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:12:21.270671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:21.270671+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"151'15621","reported_seq":15958,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.383140+0000","last_change":"2024-03-11T22:12:25.136636+0000","last_active":"2024-03-11T22:13:55.383140+0000","last_peered":"2024-03-11T22:13:55.383140+0000","last_clean":"2024-03-11T22:13:55.383140+0000","last_became_active":"2024-03-11T22:10:36.410723+0000","last_became_peered":"2024-03-11T22:10:36.410723+0000","last_unstale":"2024-03-11T22:13:55.383140+0000","last_undegraded":"2024-03-11T22:13:55.383140+0000","last_fullsized":"2024-03-11T22:13:55.383140+0000","mapping_epoch":99,"log_start":"129'10621","ondisk_log_start":"129'10621","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'10496","last_scrub_stamp":"2024-03-11T22:12:25.136305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:25.136305+0000","objects_scrubbed":6681,"log_size":5000,"log_dups_size":0,"ondisk_log_size":5000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97452032,"num_objects":1487,"num_object_clones":0,"num_object_copies":4461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13141,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526389+0000","last_change":"2024-03-11T22:12:38.434933+0000","last_active":"2024-03-11T22:13:54.526389+0000","last_peered":"2024-03-11T22:13:54.526389+0000","last_clean":"2024-03-11T22:13:54.526389+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:13:54.526389+0000","last_undegraded":"2024-03-11T22:13:54.526389+0000","last_fullsized":"2024-03-11T22:13:54.526389+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:38.434796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.434796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:04.782006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":135,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531848+0000","last_change":"2024-03-11T22:13:24.079957+0000","last_active":"2024-03-11T22:13:54.531848+0000","last_peered":"2024-03-11T22:13:54.531848+0000","last_clean":"2024-03-11T22:13:54.531848+0000","last_became_active":"2024-03-11T22:10:35.412565+0000","last_became_peered":"2024-03-11T22:10:35.412565+0000","last_unstale":"2024-03-11T22:13:54.531848+0000","last_undegraded":"2024-03-11T22:13:54.531848+0000","last_fullsized":"2024-03-11T22:13:54.531848+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:13:24.079822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:24.079822+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:44.565938+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"91'22","reported_seq":231,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525662+0000","last_change":"2024-03-11T22:13:18.458453+0000","last_active":"2024-03-11T22:13:54.525662+0000","last_peered":"2024-03-11T22:13:54.525662+0000","last_clean":"2024-03-11T22:13:54.525662+0000","last_became_active":"2024-03-11T22:08:35.085916+0000","last_became_peered":"2024-03-11T22:08:35.085916+0000","last_unstale":"2024-03-11T22:13:54.525662+0000","last_undegraded":"2024-03-11T22:13:54.525662+0000","last_fullsized":"2024-03-11T22:13:54.525662+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'22","last_scrub_stamp":"2024-03-11T22:13:18.458313+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:13:18.458313+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:29.608130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"99'9","reported_seq":199,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524304+0000","last_change":"2024-03-11T22:13:18.957027+0000","last_active":"2024-03-11T22:13:54.524304+0000","last_peered":"2024-03-11T22:13:54.524304+0000","last_clean":"2024-03-11T22:13:54.524304+0000","last_became_active":"2024-03-11T22:10:35.413089+0000","last_became_peered":"2024-03-11T22:10:35.413089+0000","last_unstale":"2024-03-11T22:13:54.524304+0000","last_undegraded":"2024-03-11T22:13:54.524304+0000","last_fullsized":"2024-03-11T22:13:54.524304+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'9","last_scrub_stamp":"2024-03-11T22:13:18.956933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:13:18.956933+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:33.528867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"151'13276","reported_seq":13461,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.480700+0000","last_change":"2024-03-11T22:12:38.052754+0000","last_active":"2024-03-11T22:13:55.480700+0000","last_peered":"2024-03-11T22:13:55.480700+0000","last_clean":"2024-03-11T22:13:55.480700+0000","last_became_active":"2024-03-11T22:10:36.411631+0000","last_became_peered":"2024-03-11T22:10:36.411631+0000","last_unstale":"2024-03-11T22:13:55.480700+0000","last_undegraded":"2024-03-11T22:13:55.480700+0000","last_fullsized":"2024-03-11T22:13:55.480700+0000","mapping_epoch":99,"log_start":"132'8958","ondisk_log_start":"132'8958","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'8787","last_scrub_stamp":"2024-03-11T22:12:38.052573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.052573+0000","objects_scrubbed":6049,"log_size":4318,"log_dups_size":0,"ondisk_log_size":4318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:03.556832+0000","scrub_duration":2625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97779712,"num_objects":1492,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13276,"num_write_kb":472576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.535211+0000","last_change":"2024-03-11T22:12:36.449430+0000","last_active":"2024-03-11T22:13:54.535211+0000","last_peered":"2024-03-11T22:13:54.535211+0000","last_clean":"2024-03-11T22:13:54.535211+0000","last_became_active":"2024-03-11T22:08:39.083141+0000","last_became_peered":"2024-03-11T22:08:39.083141+0000","last_unstale":"2024-03-11T22:13:54.535211+0000","last_undegraded":"2024-03-11T22:13:54.535211+0000","last_fullsized":"2024-03-11T22:13:54.535211+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:12:36.449307+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:12:36.449307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:00.397020+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":126,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531532+0000","last_change":"2024-03-11T22:12:29.074360+0000","last_active":"2024-03-11T22:13:54.531532+0000","last_peered":"2024-03-11T22:13:54.531532+0000","last_clean":"2024-03-11T22:13:54.531532+0000","last_became_active":"2024-03-11T22:10:35.413337+0000","last_became_peered":"2024-03-11T22:10:35.413337+0000","last_unstale":"2024-03-11T22:13:54.531532+0000","last_undegraded":"2024-03-11T22:13:54.531532+0000","last_fullsized":"2024-03-11T22:13:54.531532+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:29.074256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:29.074256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":224,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531409+0000","last_change":"2024-03-11T22:11:21.379985+0000","last_active":"2024-03-11T22:13:54.531409+0000","last_peered":"2024-03-11T22:13:54.531409+0000","last_clean":"2024-03-11T22:13:54.531409+0000","last_became_active":"2024-03-11T22:08:54.607153+0000","last_became_peered":"2024-03-11T22:08:54.607153+0000","last_unstale":"2024-03-11T22:13:54.531409+0000","last_undegraded":"2024-03-11T22:13:54.531409+0000","last_fullsized":"2024-03-11T22:13:54.531409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:11:21.379862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:21.379862+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.523657+0000","last_change":"2024-03-11T22:12:38.757349+0000","last_active":"2024-03-11T22:13:54.523657+0000","last_peered":"2024-03-11T22:13:54.523657+0000","last_clean":"2024-03-11T22:13:54.523657+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:13:54.523657+0000","last_undegraded":"2024-03-11T22:13:54.523657+0000","last_fullsized":"2024-03-11T22:13:54.523657+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.757164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.958277+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"151'13329","reported_seq":13484,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.630102+0000","last_change":"2024-03-11T22:13:25.207275+0000","last_active":"2024-03-11T22:13:55.630102+0000","last_peered":"2024-03-11T22:13:55.630102+0000","last_clean":"2024-03-11T22:13:55.630102+0000","last_became_active":"2024-03-11T22:10:36.412662+0000","last_became_peered":"2024-03-11T22:10:36.412662+0000","last_unstale":"2024-03-11T22:13:55.630102+0000","last_undegraded":"2024-03-11T22:13:55.630102+0000","last_fullsized":"2024-03-11T22:13:55.630102+0000","mapping_epoch":99,"log_start":"129'7970","ondisk_log_start":"129'7970","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'11754","last_scrub_stamp":"2024-03-11T22:13:25.206897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:13:25.206897+0000","objects_scrubbed":3073,"log_size":5359,"log_dups_size":0,"ondisk_log_size":5359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:52.018247+0000","scrub_duration":1174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96010240,"num_objects":1465,"num_object_clones":0,"num_object_copies":4395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13329,"num_write_kb":473408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":133,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.523927+0000","last_change":"2024-03-11T22:12:34.731700+0000","last_active":"2024-03-11T22:13:54.523927+0000","last_peered":"2024-03-11T22:13:54.523927+0000","last_clean":"2024-03-11T22:13:54.523927+0000","last_became_active":"2024-03-11T22:10:35.410893+0000","last_became_peered":"2024-03-11T22:10:35.410893+0000","last_unstale":"2024-03-11T22:13:54.523927+0000","last_undegraded":"2024-03-11T22:13:54.523927+0000","last_fullsized":"2024-03-11T22:13:54.523927+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:34.731409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:34.731409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.958142+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525643+0000","last_change":"2024-03-11T22:13:24.767308+0000","last_active":"2024-03-11T22:13:54.525643+0000","last_peered":"2024-03-11T22:13:54.525643+0000","last_clean":"2024-03-11T22:13:54.525643+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:13:54.525643+0000","last_undegraded":"2024-03-11T22:13:54.525643+0000","last_fullsized":"2024-03-11T22:13:54.525643+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:24.764906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.710319+0000","last_clean_scrub_stamp":"2024-03-11T22:13:24.764906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:25.188551+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":134,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526586+0000","last_change":"2024-03-11T22:12:44.474800+0000","last_active":"2024-03-11T22:13:54.526586+0000","last_peered":"2024-03-11T22:13:54.526586+0000","last_clean":"2024-03-11T22:13:54.526586+0000","last_became_active":"2024-03-11T22:10:35.418715+0000","last_became_peered":"2024-03-11T22:10:35.418715+0000","last_unstale":"2024-03-11T22:13:54.526586+0000","last_undegraded":"2024-03-11T22:13:54.526586+0000","last_fullsized":"2024-03-11T22:13:54.526586+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.474728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.474728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:57.548349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":230,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524373+0000","last_change":"2024-03-11T22:12:36.843050+0000","last_active":"2024-03-11T22:13:54.524373+0000","last_peered":"2024-03-11T22:13:54.524373+0000","last_clean":"2024-03-11T22:13:54.524373+0000","last_became_active":"2024-03-11T22:08:35.087046+0000","last_became_peered":"2024-03-11T22:08:35.087046+0000","last_unstale":"2024-03-11T22:13:54.524373+0000","last_undegraded":"2024-03-11T22:13:54.524373+0000","last_fullsized":"2024-03-11T22:13:54.524373+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:12:36.842912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:36.842912+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:04.592208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"151'13514","reported_seq":13706,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.565301+0000","last_change":"2024-03-11T22:12:25.097684+0000","last_active":"2024-03-11T22:13:55.565301+0000","last_peered":"2024-03-11T22:13:55.565301+0000","last_clean":"2024-03-11T22:13:55.565301+0000","last_became_active":"2024-03-11T22:10:36.410947+0000","last_became_peered":"2024-03-11T22:10:36.410947+0000","last_unstale":"2024-03-11T22:13:55.565301+0000","last_undegraded":"2024-03-11T22:13:55.565301+0000","last_fullsized":"2024-03-11T22:13:55.565301+0000","mapping_epoch":99,"log_start":"127'7348","ondisk_log_start":"127'7348","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'8252","last_scrub_stamp":"2024-03-11T22:12:25.097553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:25.097553+0000","objects_scrubbed":6851,"log_size":6166,"log_dups_size":0,"ondisk_log_size":6166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.958173+0000","scrub_duration":2502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98566144,"num_objects":1504,"num_object_clones":0,"num_object_copies":4512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13514,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.547810+0000","last_change":"2024-03-11T22:12:28.683756+0000","last_active":"2024-03-11T22:13:54.547810+0000","last_peered":"2024-03-11T22:13:54.547810+0000","last_clean":"2024-03-11T22:13:54.547810+0000","last_became_active":"2024-03-11T22:08:39.085496+0000","last_became_peered":"2024-03-11T22:08:39.085496+0000","last_unstale":"2024-03-11T22:13:54.547810+0000","last_undegraded":"2024-03-11T22:13:54.547810+0000","last_fullsized":"2024-03-11T22:13:54.547810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:28.683646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:28.683646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.964111+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526042+0000","last_change":"2024-03-11T22:13:19.781178+0000","last_active":"2024-03-11T22:13:54.526042+0000","last_peered":"2024-03-11T22:13:54.526042+0000","last_clean":"2024-03-11T22:13:54.526042+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:13:54.526042+0000","last_undegraded":"2024-03-11T22:13:54.526042+0000","last_fullsized":"2024-03-11T22:13:54.526042+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:19.776020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.776020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:38.461474+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526299+0000","last_change":"2024-03-11T22:12:34.391183+0000","last_active":"2024-03-11T22:13:54.526299+0000","last_peered":"2024-03-11T22:13:54.526299+0000","last_clean":"2024-03-11T22:13:54.526299+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:13:54.526299+0000","last_undegraded":"2024-03-11T22:13:54.526299+0000","last_fullsized":"2024-03-11T22:13:54.526299+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:34.391054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:34.391054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:57.548365+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":223,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.547701+0000","last_change":"2024-03-11T22:13:15.950715+0000","last_active":"2024-03-11T22:13:54.547701+0000","last_peered":"2024-03-11T22:13:54.547701+0000","last_clean":"2024-03-11T22:13:54.547701+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:13:54.547701+0000","last_undegraded":"2024-03-11T22:13:54.547701+0000","last_fullsized":"2024-03-11T22:13:54.547701+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:13:15.950564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:15.950564+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:28.253462+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":459,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525885+0000","last_change":"2024-03-11T22:13:38.573308+0000","last_active":"2024-03-11T22:13:54.525885+0000","last_peered":"2024-03-11T22:13:54.525885+0000","last_clean":"2024-03-11T22:13:54.525885+0000","last_became_active":"2024-03-11T22:13:38.572927+0000","last_became_peered":"2024-03-11T22:13:38.572927+0000","last_unstale":"2024-03-11T22:13:54.525885+0000","last_undegraded":"2024-03-11T22:13:54.525885+0000","last_fullsized":"2024-03-11T22:13:54.525885+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:13:34.052788+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:13:34.052788+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:01.710803+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"151'13284","reported_seq":13471,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.140462+0000","last_change":"2024-03-11T22:12:30.660272+0000","last_active":"2024-03-11T22:13:55.140462+0000","last_peered":"2024-03-11T22:13:55.140462+0000","last_clean":"2024-03-11T22:13:55.140462+0000","last_became_active":"2024-03-11T22:10:36.416476+0000","last_became_peered":"2024-03-11T22:10:36.416476+0000","last_unstale":"2024-03-11T22:13:55.140462+0000","last_undegraded":"2024-03-11T22:13:55.140462+0000","last_fullsized":"2024-03-11T22:13:55.140462+0000","mapping_epoch":99,"log_start":"127'7623","ondisk_log_start":"127'7623","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8438","last_scrub_stamp":"2024-03-11T22:12:30.660018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:30.660018+0000","objects_scrubbed":6392,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.099420+0000","scrub_duration":1794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97910784,"num_objects":1494,"num_object_clones":0,"num_object_copies":4482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13284,"num_write_kb":472896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.536643+0000","last_change":"2024-03-11T22:12:45.439711+0000","last_active":"2024-03-11T22:13:54.536643+0000","last_peered":"2024-03-11T22:13:54.536643+0000","last_clean":"2024-03-11T22:13:54.536643+0000","last_became_active":"2024-03-11T22:08:39.083738+0000","last_became_peered":"2024-03-11T22:08:39.083738+0000","last_unstale":"2024-03-11T22:13:54.536643+0000","last_undegraded":"2024-03-11T22:13:54.536643+0000","last_fullsized":"2024-03-11T22:13:54.536643+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.439584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":207,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526136+0000","last_change":"2024-03-11T22:11:42.128436+0000","last_active":"2024-03-11T22:13:54.526136+0000","last_peered":"2024-03-11T22:13:54.526136+0000","last_clean":"2024-03-11T22:13:54.526136+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:13:54.526136+0000","last_undegraded":"2024-03-11T22:13:54.526136+0000","last_fullsized":"2024-03-11T22:13:54.526136+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:11:42.128356+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:11:42.128356+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531310+0000","last_change":"2024-03-11T22:13:28.028641+0000","last_active":"2024-03-11T22:13:54.531310+0000","last_peered":"2024-03-11T22:13:54.531310+0000","last_clean":"2024-03-11T22:13:54.531310+0000","last_became_active":"2024-03-11T22:08:33.035469+0000","last_became_peered":"2024-03-11T22:08:33.035469+0000","last_unstale":"2024-03-11T22:13:54.531310+0000","last_undegraded":"2024-03-11T22:13:54.531310+0000","last_fullsized":"2024-03-11T22:13:54.531310+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:28.028531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:28.028531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:50.939614+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":312,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.522060+0000","last_change":"2024-03-11T22:13:17.830633+0000","last_active":"2024-03-11T22:13:54.522060+0000","last_peered":"2024-03-11T22:13:54.522060+0000","last_clean":"2024-03-11T22:13:54.522060+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:13:54.522060+0000","last_undegraded":"2024-03-11T22:13:54.522060+0000","last_fullsized":"2024-03-11T22:13:54.522060+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-11T22:13:17.830327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:17.830327+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:21.167864+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.551175+0000","last_change":"2024-03-11T22:11:14.224976+0000","last_active":"2024-03-11T22:13:54.551175+0000","last_peered":"2024-03-11T22:13:54.551175+0000","last_clean":"2024-03-11T22:13:54.551175+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:13:54.551175+0000","last_undegraded":"2024-03-11T22:13:54.551175+0000","last_fullsized":"2024-03-11T22:13:54.551175+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:14.224859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:14.224859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.963896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":459,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.546781+0000","last_change":"2024-03-11T22:13:38.573586+0000","last_active":"2024-03-11T22:13:54.546781+0000","last_peered":"2024-03-11T22:13:54.546781+0000","last_clean":"2024-03-11T22:13:54.546781+0000","last_became_active":"2024-03-11T22:13:38.573086+0000","last_became_peered":"2024-03-11T22:13:38.573086+0000","last_unstale":"2024-03-11T22:13:54.546781+0000","last_undegraded":"2024-03-11T22:13:54.546781+0000","last_fullsized":"2024-03-11T22:13:54.546781+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:11:34.382250+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.382250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:54.998990+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"151'13470","reported_seq":13621,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.632420+0000","last_change":"2024-03-11T22:13:29.879326+0000","last_active":"2024-03-11T22:13:55.632420+0000","last_peered":"2024-03-11T22:13:55.632420+0000","last_clean":"2024-03-11T22:13:55.632420+0000","last_became_active":"2024-03-11T22:10:36.412694+0000","last_became_peered":"2024-03-11T22:10:36.412694+0000","last_unstale":"2024-03-11T22:13:55.632420+0000","last_undegraded":"2024-03-11T22:13:55.632420+0000","last_fullsized":"2024-03-11T22:13:55.632420+0000","mapping_epoch":99,"log_start":"129'8161","ondisk_log_start":"129'8161","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'12216","last_scrub_stamp":"2024-03-11T22:13:29.879111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:13:29.879111+0000","objects_scrubbed":2790,"log_size":5309,"log_dups_size":0,"ondisk_log_size":5309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:49.251138+0000","scrub_duration":851,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99090432,"num_objects":1512,"num_object_clones":0,"num_object_copies":4536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13470,"num_write_kb":479424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":133,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525083+0000","last_change":"2024-03-11T22:13:22.005413+0000","last_active":"2024-03-11T22:13:54.525083+0000","last_peered":"2024-03-11T22:13:54.525083+0000","last_clean":"2024-03-11T22:13:54.525083+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:13:54.525083+0000","last_undegraded":"2024-03-11T22:13:54.525083+0000","last_fullsized":"2024-03-11T22:13:54.525083+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:22.005300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:22.005300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:25.411995+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525135+0000","last_change":"2024-03-11T22:13:19.750323+0000","last_active":"2024-03-11T22:13:54.525135+0000","last_peered":"2024-03-11T22:13:54.525135+0000","last_clean":"2024-03-11T22:13:54.525135+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:13:54.525135+0000","last_undegraded":"2024-03-11T22:13:54.525135+0000","last_fullsized":"2024-03-11T22:13:54.525135+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:19.748477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.748477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:32.843670+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.523636+0000","last_change":"2024-03-11T22:12:45.863561+0000","last_active":"2024-03-11T22:13:54.523636+0000","last_peered":"2024-03-11T22:13:54.523636+0000","last_clean":"2024-03-11T22:13:54.523636+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:13:54.523636+0000","last_undegraded":"2024-03-11T22:13:54.523636+0000","last_fullsized":"2024-03-11T22:13:54.523636+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.863440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.863440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:09.786177+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":226,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524406+0000","last_change":"2024-03-11T22:13:24.705841+0000","last_active":"2024-03-11T22:13:54.524406+0000","last_peered":"2024-03-11T22:13:54.524406+0000","last_clean":"2024-03-11T22:13:54.524406+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:13:54.524406+0000","last_undegraded":"2024-03-11T22:13:54.524406+0000","last_fullsized":"2024-03-11T22:13:54.524406+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-03-11T22:13:24.705697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:24.705697+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:29.566342+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.527489+0000","last_change":"2024-03-11T22:12:31.669715+0000","last_active":"2024-03-11T22:13:54.527489+0000","last_peered":"2024-03-11T22:13:54.527489+0000","last_clean":"2024-03-11T22:13:54.527489+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:13:54.527489+0000","last_undegraded":"2024-03-11T22:13:54.527489+0000","last_fullsized":"2024-03-11T22:13:54.527489+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:31.669555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.669555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.958226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"151'15399","reported_seq":15605,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.622137+0000","last_change":"2024-03-11T22:13:54.435445+0000","last_active":"2024-03-11T22:13:55.622137+0000","last_peered":"2024-03-11T22:13:55.622137+0000","last_clean":"2024-03-11T22:13:55.622137+0000","last_became_active":"2024-03-11T22:10:36.408867+0000","last_became_peered":"2024-03-11T22:10:36.408867+0000","last_unstale":"2024-03-11T22:13:55.622137+0000","last_undegraded":"2024-03-11T22:13:55.622137+0000","last_fullsized":"2024-03-11T22:13:55.622137+0000","mapping_epoch":99,"log_start":"137'10133","ondisk_log_start":"137'10133","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'15322","last_scrub_stamp":"2024-03-11T22:13:54.435291+0000","last_deep_scrub":"150'15322","last_deep_scrub_stamp":"2024-03-11T22:13:54.435291+0000","last_clean_scrub_stamp":"2024-03-11T22:13:54.435291+0000","objects_scrubbed":1948,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":22,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:01.062829+0000","scrub_duration":21325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91815936,"num_objects":1401,"num_object_clones":0,"num_object_copies":4203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13451,"num_write_kb":475264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":191,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526727+0000","last_change":"2024-03-11T22:13:34.478753+0000","last_active":"2024-03-11T22:13:54.526727+0000","last_peered":"2024-03-11T22:13:54.526727+0000","last_clean":"2024-03-11T22:13:54.526727+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:13:54.526727+0000","last_undegraded":"2024-03-11T22:13:54.526727+0000","last_fullsized":"2024-03-11T22:13:54.526727+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:34.478634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:34.478634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:02.370872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":459,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.550352+0000","last_change":"2024-03-11T22:13:35.093324+0000","last_active":"2024-03-11T22:13:54.550352+0000","last_peered":"2024-03-11T22:13:54.550352+0000","last_clean":"2024-03-11T22:13:54.550352+0000","last_became_active":"2024-03-11T22:13:33.516539+0000","last_became_peered":"2024-03-11T22:13:33.516539+0000","last_unstale":"2024-03-11T22:13:54.550352+0000","last_undegraded":"2024-03-11T22:13:54.550352+0000","last_fullsized":"2024-03-11T22:13:54.550352+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:13:35.093204+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:04.719504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531551+0000","last_change":"2024-03-11T22:13:33.460800+0000","last_active":"2024-03-11T22:13:54.531551+0000","last_peered":"2024-03-11T22:13:54.531551+0000","last_clean":"2024-03-11T22:13:54.531551+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:13:54.531551+0000","last_undegraded":"2024-03-11T22:13:54.531551+0000","last_fullsized":"2024-03-11T22:13:54.531551+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:33.460308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:13:33.460308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:54.540593+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":286,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525078+0000","last_change":"2024-03-11T22:13:29.093136+0000","last_active":"2024-03-11T22:13:54.525078+0000","last_peered":"2024-03-11T22:13:54.525078+0000","last_clean":"2024-03-11T22:13:54.525078+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:13:54.525078+0000","last_undegraded":"2024-03-11T22:13:54.525078+0000","last_fullsized":"2024-03-11T22:13:54.525078+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-11T22:13:29.093013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:29.093013+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:34.595204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":133,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524066+0000","last_change":"2024-03-11T22:12:33.715464+0000","last_active":"2024-03-11T22:13:54.524066+0000","last_peered":"2024-03-11T22:13:54.524066+0000","last_clean":"2024-03-11T22:13:54.524066+0000","last_became_active":"2024-03-11T22:10:35.412547+0000","last_became_peered":"2024-03-11T22:10:35.412547+0000","last_unstale":"2024-03-11T22:13:54.524066+0000","last_undegraded":"2024-03-11T22:13:54.524066+0000","last_fullsized":"2024-03-11T22:13:54.524066+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:33.715347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:12:33.715347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.958251+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":452,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.521570+0000","last_change":"2024-03-11T22:13:38.569788+0000","last_active":"2024-03-11T22:13:54.521570+0000","last_peered":"2024-03-11T22:13:54.521570+0000","last_clean":"2024-03-11T22:13:54.521570+0000","last_became_active":"2024-03-11T22:13:38.569265+0000","last_became_peered":"2024-03-11T22:13:38.569265+0000","last_unstale":"2024-03-11T22:13:54.521570+0000","last_undegraded":"2024-03-11T22:13:54.521570+0000","last_fullsized":"2024-03-11T22:13:54.521570+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:11:34.382250+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.382250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:54.706450+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"151'13445","reported_seq":13657,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.590364+0000","last_change":"2024-03-11T22:13:31.774638+0000","last_active":"2024-03-11T22:13:55.590364+0000","last_peered":"2024-03-11T22:13:55.590364+0000","last_clean":"2024-03-11T22:13:55.590364+0000","last_became_active":"2024-03-11T22:10:36.406423+0000","last_became_peered":"2024-03-11T22:10:36.406423+0000","last_unstale":"2024-03-11T22:13:55.590364+0000","last_undegraded":"2024-03-11T22:13:55.590364+0000","last_fullsized":"2024-03-11T22:13:55.590364+0000","mapping_epoch":99,"log_start":"127'7272","ondisk_log_start":"127'7272","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'12265","last_scrub_stamp":"2024-03-11T22:13:31.774415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:13:31.774415+0000","objects_scrubbed":2639,"log_size":6173,"log_dups_size":0,"ondisk_log_size":6173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:50.245474+0000","scrub_duration":969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93388800,"num_objects":1425,"num_object_clones":0,"num_object_copies":4275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13445,"num_write_kb":475840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":191,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.521780+0000","last_change":"2024-03-11T22:12:44.933303+0000","last_active":"2024-03-11T22:13:54.521780+0000","last_peered":"2024-03-11T22:13:54.521780+0000","last_clean":"2024-03-11T22:13:54.521780+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:13:54.521780+0000","last_undegraded":"2024-03-11T22:13:54.521780+0000","last_fullsized":"2024-03-11T22:13:54.521780+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.927967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.927967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:13.766656+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.547589+0000","last_change":"2024-03-11T22:12:19.685277+0000","last_active":"2024-03-11T22:13:54.547589+0000","last_peered":"2024-03-11T22:13:54.547589+0000","last_clean":"2024-03-11T22:13:54.547589+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:13:54.547589+0000","last_undegraded":"2024-03-11T22:13:54.547589+0000","last_fullsized":"2024-03-11T22:13:54.547589+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:19.684663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:19.684663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:54.999058+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":499,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525039+0000","last_change":"2024-03-11T22:11:34.857427+0000","last_active":"2024-03-11T22:13:54.525039+0000","last_peered":"2024-03-11T22:13:54.525039+0000","last_clean":"2024-03-11T22:13:54.525039+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:13:54.525039+0000","last_undegraded":"2024-03-11T22:13:54.525039+0000","last_fullsized":"2024-03-11T22:13:54.525039+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:11:34.857300+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.857300+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":289,"num_read_kb":284,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":209,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.547795+0000","last_change":"2024-03-11T22:11:06.292270+0000","last_active":"2024-03-11T22:13:54.547795+0000","last_peered":"2024-03-11T22:13:54.547795+0000","last_clean":"2024-03-11T22:13:54.547795+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:13:54.547795+0000","last_undegraded":"2024-03-11T22:13:54.547795+0000","last_fullsized":"2024-03-11T22:13:54.547795+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:11:06.292138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:06.292138+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.963931+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526932+0000","last_change":"2024-03-11T22:12:27.459283+0000","last_active":"2024-03-11T22:13:54.526932+0000","last_peered":"2024-03-11T22:13:54.526932+0000","last_clean":"2024-03-11T22:13:54.526932+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:13:54.526932+0000","last_undegraded":"2024-03-11T22:13:54.526932+0000","last_fullsized":"2024-03-11T22:13:54.526932+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:27.459154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:27.459154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":340,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524716+0000","last_change":"2024-03-11T22:13:38.846253+0000","last_active":"2024-03-11T22:13:54.524716+0000","last_peered":"2024-03-11T22:13:54.524716+0000","last_clean":"2024-03-11T22:13:54.524716+0000","last_became_active":"2024-03-11T22:13:38.845904+0000","last_became_peered":"2024-03-11T22:13:38.845904+0000","last_unstale":"2024-03-11T22:13:54.524716+0000","last_undegraded":"2024-03-11T22:13:54.524716+0000","last_fullsized":"2024-03-11T22:13:54.524716+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:11:34.382250+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.382250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:55.135628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"151'13176","reported_seq":13366,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.052249+0000","last_change":"2024-03-11T22:12:25.123093+0000","last_active":"2024-03-11T22:13:55.052249+0000","last_peered":"2024-03-11T22:13:55.052249+0000","last_clean":"2024-03-11T22:13:55.052249+0000","last_became_active":"2024-03-11T22:10:36.411289+0000","last_became_peered":"2024-03-11T22:10:36.411289+0000","last_unstale":"2024-03-11T22:13:55.052249+0000","last_undegraded":"2024-03-11T22:13:55.052249+0000","last_fullsized":"2024-03-11T22:13:55.052249+0000","mapping_epoch":99,"log_start":"129'7811","ondisk_log_start":"129'7811","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7985","last_scrub_stamp":"2024-03-11T22:12:25.122944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:25.122944+0000","objects_scrubbed":6655,"log_size":5365,"log_dups_size":0,"ondisk_log_size":5365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1760,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93585408,"num_objects":1428,"num_object_clones":0,"num_object_copies":4284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13176,"num_write_kb":467328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":191,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.528969+0000","last_change":"2024-03-11T22:13:20.800006+0000","last_active":"2024-03-11T22:13:54.528969+0000","last_peered":"2024-03-11T22:13:54.528969+0000","last_clean":"2024-03-11T22:13:54.528969+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:13:54.528969+0000","last_undegraded":"2024-03-11T22:13:54.528969+0000","last_fullsized":"2024-03-11T22:13:54.528969+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:20.799791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:20.799791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:35.396331+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.521979+0000","last_change":"2024-03-11T22:12:42.951642+0000","last_active":"2024-03-11T22:13:54.521979+0000","last_peered":"2024-03-11T22:13:54.521979+0000","last_clean":"2024-03-11T22:13:54.521979+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:13:54.521979+0000","last_undegraded":"2024-03-11T22:13:54.521979+0000","last_fullsized":"2024-03-11T22:13:54.521979+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.951379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.951379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:10.673379+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":227,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.547456+0000","last_change":"2024-03-11T22:13:27.025211+0000","last_active":"2024-03-11T22:13:54.547456+0000","last_peered":"2024-03-11T22:13:54.547456+0000","last_clean":"2024-03-11T22:13:54.547456+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:13:54.547456+0000","last_undegraded":"2024-03-11T22:13:54.547456+0000","last_fullsized":"2024-03-11T22:13:54.547456+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:13:27.025050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:27.025050+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:52.422159+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524382+0000","last_change":"2024-03-11T22:12:40.863550+0000","last_active":"2024-03-11T22:13:54.524382+0000","last_peered":"2024-03-11T22:13:54.524382+0000","last_clean":"2024-03-11T22:13:54.524382+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:13:54.524382+0000","last_undegraded":"2024-03-11T22:13:54.524382+0000","last_fullsized":"2024-03-11T22:13:54.524382+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:12:40.863445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.360193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":340,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524970+0000","last_change":"2024-03-11T22:13:38.576861+0000","last_active":"2024-03-11T22:13:54.524970+0000","last_peered":"2024-03-11T22:13:54.524970+0000","last_clean":"2024-03-11T22:13:54.524970+0000","last_became_active":"2024-03-11T22:13:38.576468+0000","last_became_peered":"2024-03-11T22:13:38.576468+0000","last_unstale":"2024-03-11T22:13:54.524970+0000","last_undegraded":"2024-03-11T22:13:54.524970+0000","last_fullsized":"2024-03-11T22:13:54.524970+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:11:34.382250+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.382250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:55.135607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"151'13209","reported_seq":13324,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.108028+0000","last_change":"2024-03-11T22:10:36.416445+0000","last_active":"2024-03-11T22:13:55.108028+0000","last_peered":"2024-03-11T22:13:55.108028+0000","last_clean":"2024-03-11T22:13:55.108028+0000","last_became_active":"2024-03-11T22:10:36.416083+0000","last_became_peered":"2024-03-11T22:10:36.416083+0000","last_unstale":"2024-03-11T22:13:55.108028+0000","last_undegraded":"2024-03-11T22:13:55.108028+0000","last_fullsized":"2024-03-11T22:13:55.108028+0000","mapping_epoch":99,"log_start":"131'8602","ondisk_log_start":"131'8602","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:10:35.397794+0000","objects_scrubbed":0,"log_size":4607,"log_dups_size":0,"ondisk_log_size":4607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:00.068207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97583104,"num_objects":1489,"num_object_clones":0,"num_object_copies":4467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13209,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":175,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.550266+0000","last_change":"2024-03-11T22:11:11.177052+0000","last_active":"2024-03-11T22:13:54.550266+0000","last_peered":"2024-03-11T22:13:54.550266+0000","last_clean":"2024-03-11T22:13:54.550266+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:13:54.550266+0000","last_undegraded":"2024-03-11T22:13:54.550266+0000","last_fullsized":"2024-03-11T22:13:54.550266+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:11.176930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:11.176930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.963828+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526531+0000","last_change":"2024-03-11T22:12:45.466625+0000","last_active":"2024-03-11T22:13:54.526531+0000","last_peered":"2024-03-11T22:13:54.526531+0000","last_clean":"2024-03-11T22:13:54.526531+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:13:54.526531+0000","last_undegraded":"2024-03-11T22:13:54.526531+0000","last_fullsized":"2024-03-11T22:13:54.526531+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.466508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.466508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:09.930146+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":206,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525537+0000","last_change":"2024-03-11T22:12:38.896205+0000","last_active":"2024-03-11T22:13:54.525537+0000","last_peered":"2024-03-11T22:13:54.525537+0000","last_clean":"2024-03-11T22:13:54.525537+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:13:54.525537+0000","last_undegraded":"2024-03-11T22:13:54.525537+0000","last_fullsized":"2024-03-11T22:13:54.525537+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:12:38.895763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.895763+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:08.358608+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":260,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.547810+0000","last_change":"2024-03-11T22:11:25.385148+0000","last_active":"2024-03-11T22:13:54.547810+0000","last_peered":"2024-03-11T22:13:54.547810+0000","last_clean":"2024-03-11T22:13:54.547810+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:13:54.547810+0000","last_undegraded":"2024-03-11T22:13:54.547810+0000","last_fullsized":"2024-03-11T22:13:54.547810+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:11:25.384965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.384965+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.964027+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":452,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.532527+0000","last_change":"2024-03-11T22:13:38.570269+0000","last_active":"2024-03-11T22:13:54.532527+0000","last_peered":"2024-03-11T22:13:54.532527+0000","last_clean":"2024-03-11T22:13:54.532527+0000","last_became_active":"2024-03-11T22:13:38.569867+0000","last_became_peered":"2024-03-11T22:13:38.569867+0000","last_unstale":"2024-03-11T22:13:54.532527+0000","last_undegraded":"2024-03-11T22:13:54.532527+0000","last_fullsized":"2024-03-11T22:13:54.532527+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:11:34.382250+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.382250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":488690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"151'13354","reported_seq":13539,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.490554+0000","last_change":"2024-03-11T22:12:35.256058+0000","last_active":"2024-03-11T22:13:55.490554+0000","last_peered":"2024-03-11T22:13:55.490554+0000","last_clean":"2024-03-11T22:13:55.490554+0000","last_became_active":"2024-03-11T22:10:36.410859+0000","last_became_peered":"2024-03-11T22:10:36.410859+0000","last_unstale":"2024-03-11T22:13:55.490554+0000","last_undegraded":"2024-03-11T22:13:55.490554+0000","last_fullsized":"2024-03-11T22:13:55.490554+0000","mapping_epoch":99,"log_start":"132'9011","ondisk_log_start":"132'9011","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8663","last_scrub_stamp":"2024-03-11T22:12:35.255739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.255739+0000","objects_scrubbed":6218,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:59.593220+0000","scrub_duration":2947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95158272,"num_objects":1452,"num_object_clones":0,"num_object_copies":4356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13354,"num_write_kb":473792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":133,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524671+0000","last_change":"2024-03-11T22:12:46.968879+0000","last_active":"2024-03-11T22:13:54.524671+0000","last_peered":"2024-03-11T22:13:54.524671+0000","last_clean":"2024-03-11T22:13:54.524671+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:13:54.524671+0000","last_undegraded":"2024-03-11T22:13:54.524671+0000","last_fullsized":"2024-03-11T22:13:54.524671+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.968807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.968807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:12.616431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.547902+0000","last_change":"2024-03-11T22:12:49.856525+0000","last_active":"2024-03-11T22:13:54.547902+0000","last_peered":"2024-03-11T22:13:54.547902+0000","last_clean":"2024-03-11T22:13:54.547902+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:13:54.547902+0000","last_undegraded":"2024-03-11T22:13:54.547902+0000","last_fullsized":"2024-03-11T22:13:54.547902+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:49.856381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.856381+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:57.055758+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531127+0000","last_change":"2024-03-11T22:13:27.028743+0000","last_active":"2024-03-11T22:13:54.531127+0000","last_peered":"2024-03-11T22:13:54.531127+0000","last_clean":"2024-03-11T22:13:54.531127+0000","last_became_active":"2024-03-11T22:08:35.053452+0000","last_became_peered":"2024-03-11T22:08:35.053452+0000","last_unstale":"2024-03-11T22:13:54.531127+0000","last_undegraded":"2024-03-11T22:13:54.531127+0000","last_fullsized":"2024-03-11T22:13:54.531127+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:13:27.028171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:27.028171+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:50.155527+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":260,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.535624+0000","last_change":"2024-03-11T22:12:31.264333+0000","last_active":"2024-03-11T22:13:54.535624+0000","last_peered":"2024-03-11T22:13:54.535624+0000","last_clean":"2024-03-11T22:13:54.535624+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:13:54.535624+0000","last_undegraded":"2024-03-11T22:13:54.535624+0000","last_fullsized":"2024-03-11T22:13:54.535624+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:12:31.264213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.264213+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":452,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531433+0000","last_change":"2024-03-11T22:13:38.565923+0000","last_active":"2024-03-11T22:13:54.531433+0000","last_peered":"2024-03-11T22:13:54.531433+0000","last_clean":"2024-03-11T22:13:54.531433+0000","last_became_active":"2024-03-11T22:13:38.565536+0000","last_became_peered":"2024-03-11T22:13:38.565536+0000","last_unstale":"2024-03-11T22:13:54.531433+0000","last_undegraded":"2024-03-11T22:13:54.531433+0000","last_fullsized":"2024-03-11T22:13:54.531433+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:11:34.382250+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.382250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:55.978048+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"151'13272","reported_seq":13456,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.504915+0000","last_change":"2024-03-11T22:12:35.900151+0000","last_active":"2024-03-11T22:13:55.504915+0000","last_peered":"2024-03-11T22:13:55.504915+0000","last_clean":"2024-03-11T22:13:55.504915+0000","last_became_active":"2024-03-11T22:10:36.414380+0000","last_became_peered":"2024-03-11T22:10:36.414380+0000","last_unstale":"2024-03-11T22:13:55.504915+0000","last_undegraded":"2024-03-11T22:13:55.504915+0000","last_fullsized":"2024-03-11T22:13:55.504915+0000","mapping_epoch":99,"log_start":"132'8944","ondisk_log_start":"132'8944","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8688","last_scrub_stamp":"2024-03-11T22:12:35.899750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.899750+0000","objects_scrubbed":6120,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:59.587355+0000","scrub_duration":2560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95289344,"num_objects":1454,"num_object_clones":0,"num_object_copies":4362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13272,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":175,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.530993+0000","last_change":"2024-03-11T22:11:35.248168+0000","last_active":"2024-03-11T22:13:54.530993+0000","last_peered":"2024-03-11T22:13:54.530993+0000","last_clean":"2024-03-11T22:13:54.530993+0000","last_became_active":"2024-03-11T22:08:39.078244+0000","last_became_peered":"2024-03-11T22:08:39.078244+0000","last_unstale":"2024-03-11T22:13:54.530993+0000","last_undegraded":"2024-03-11T22:13:54.530993+0000","last_fullsized":"2024-03-11T22:13:54.530993+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:35.248081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.248081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.537683+0000","last_change":"2024-03-11T22:13:26.481234+0000","last_active":"2024-03-11T22:13:54.537683+0000","last_peered":"2024-03-11T22:13:54.537683+0000","last_clean":"2024-03-11T22:13:54.537683+0000","last_became_active":"2024-03-11T22:08:33.036469+0000","last_became_peered":"2024-03-11T22:08:33.036469+0000","last_unstale":"2024-03-11T22:13:54.537683+0000","last_undegraded":"2024-03-11T22:13:54.537683+0000","last_fullsized":"2024-03-11T22:13:54.537683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:26.481122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:26.481122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:38.248860+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":226,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.535506+0000","last_change":"2024-03-11T22:12:39.447777+0000","last_active":"2024-03-11T22:13:54.535506+0000","last_peered":"2024-03-11T22:13:54.535506+0000","last_clean":"2024-03-11T22:13:54.535506+0000","last_became_active":"2024-03-11T22:08:35.085979+0000","last_became_peered":"2024-03-11T22:08:35.085979+0000","last_unstale":"2024-03-11T22:13:54.535506+0000","last_undegraded":"2024-03-11T22:13:54.535506+0000","last_fullsized":"2024-03-11T22:13:54.535506+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:12:39.447312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:39.447312+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:00.055885+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.530974+0000","last_change":"2024-03-11T22:11:40.165879+0000","last_active":"2024-03-11T22:13:54.530974+0000","last_peered":"2024-03-11T22:13:54.530974+0000","last_clean":"2024-03-11T22:13:54.530974+0000","last_became_active":"2024-03-11T22:08:37.076522+0000","last_became_peered":"2024-03-11T22:08:37.076522+0000","last_unstale":"2024-03-11T22:13:54.530974+0000","last_undegraded":"2024-03-11T22:13:54.530974+0000","last_fullsized":"2024-03-11T22:13:54.530974+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:40.165760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:40.165760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":452,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526028+0000","last_change":"2024-03-11T22:13:38.574325+0000","last_active":"2024-03-11T22:13:54.526028+0000","last_peered":"2024-03-11T22:13:54.526028+0000","last_clean":"2024-03-11T22:13:54.526028+0000","last_became_active":"2024-03-11T22:13:38.573794+0000","last_became_peered":"2024-03-11T22:13:38.573794+0000","last_unstale":"2024-03-11T22:13:54.526028+0000","last_undegraded":"2024-03-11T22:13:54.526028+0000","last_fullsized":"2024-03-11T22:13:54.526028+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:11:34.382250+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.382250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:55.553242+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"151'13275","reported_seq":13461,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.159456+0000","last_change":"2024-03-11T22:12:31.355260+0000","last_active":"2024-03-11T22:13:55.159456+0000","last_peered":"2024-03-11T22:13:55.159456+0000","last_clean":"2024-03-11T22:13:55.159456+0000","last_became_active":"2024-03-11T22:10:36.414372+0000","last_became_peered":"2024-03-11T22:10:36.414372+0000","last_unstale":"2024-03-11T22:13:55.159456+0000","last_undegraded":"2024-03-11T22:13:55.159456+0000","last_fullsized":"2024-03-11T22:13:55.159456+0000","mapping_epoch":99,"log_start":"127'7568","ondisk_log_start":"127'7568","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8481","last_scrub_stamp":"2024-03-11T22:12:31.354979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.354979+0000","objects_scrubbed":6285,"log_size":5707,"log_dups_size":0,"ondisk_log_size":5707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.099481+0000","scrub_duration":1504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95092736,"num_objects":1451,"num_object_clones":0,"num_object_copies":4353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13275,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526104+0000","last_change":"2024-03-11T22:12:42.469626+0000","last_active":"2024-03-11T22:13:54.526104+0000","last_peered":"2024-03-11T22:13:54.526104+0000","last_clean":"2024-03-11T22:13:54.526104+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:13:54.526104+0000","last_undegraded":"2024-03-11T22:13:54.526104+0000","last_fullsized":"2024-03-11T22:13:54.526104+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.469501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.469501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:57.548382+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":260,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531757+0000","last_change":"2024-03-11T22:13:35.126819+0000","last_active":"2024-03-11T22:13:54.531757+0000","last_peered":"2024-03-11T22:13:54.531757+0000","last_clean":"2024-03-11T22:13:54.531757+0000","last_became_active":"2024-03-11T22:08:37.077142+0000","last_became_peered":"2024-03-11T22:08:37.077142+0000","last_unstale":"2024-03-11T22:13:54.531757+0000","last_undegraded":"2024-03-11T22:13:54.531757+0000","last_fullsized":"2024-03-11T22:13:54.531757+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:13:35.126665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:13:35.126665+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:42.706921+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.535377+0000","last_change":"2024-03-11T22:12:24.278539+0000","last_active":"2024-03-11T22:13:54.535377+0000","last_peered":"2024-03-11T22:13:54.535377+0000","last_clean":"2024-03-11T22:13:54.535377+0000","last_became_active":"2024-03-11T22:08:33.037537+0000","last_became_peered":"2024-03-11T22:08:33.037537+0000","last_unstale":"2024-03-11T22:13:54.535377+0000","last_undegraded":"2024-03-11T22:13:54.535377+0000","last_fullsized":"2024-03-11T22:13:54.535377+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:24.278298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.278298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":216,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524557+0000","last_change":"2024-03-11T22:11:41.817430+0000","last_active":"2024-03-11T22:13:54.524557+0000","last_peered":"2024-03-11T22:13:54.524557+0000","last_clean":"2024-03-11T22:13:54.524557+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:13:54.524557+0000","last_undegraded":"2024-03-11T22:13:54.524557+0000","last_fullsized":"2024-03-11T22:13:54.524557+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:11:41.817285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:41.817285+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:54.706523+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"151'13653","reported_seq":13840,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.149866+0000","last_change":"2024-03-11T22:12:35.582515+0000","last_active":"2024-03-11T22:13:55.149866+0000","last_peered":"2024-03-11T22:13:55.149866+0000","last_clean":"2024-03-11T22:13:55.149866+0000","last_became_active":"2024-03-11T22:10:36.412847+0000","last_became_peered":"2024-03-11T22:10:36.412847+0000","last_unstale":"2024-03-11T22:13:55.149866+0000","last_undegraded":"2024-03-11T22:13:55.149866+0000","last_fullsized":"2024-03-11T22:13:55.149866+0000","mapping_epoch":99,"log_start":"127'7904","ondisk_log_start":"127'7904","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'9001","last_scrub_stamp":"2024-03-11T22:12:35.582375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.582375+0000","objects_scrubbed":6270,"log_size":5749,"log_dups_size":0,"ondisk_log_size":5749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:02.117832+0000","scrub_duration":1709,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103809048,"num_objects":1585,"num_object_clones":0,"num_object_copies":4755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13653,"num_write_kb":487553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":183,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.532724+0000","last_change":"2024-03-11T22:13:17.533805+0000","last_active":"2024-03-11T22:13:54.532724+0000","last_peered":"2024-03-11T22:13:54.532724+0000","last_clean":"2024-03-11T22:13:54.532724+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:13:54.532724+0000","last_undegraded":"2024-03-11T22:13:54.532724+0000","last_fullsized":"2024-03-11T22:13:54.532724+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:17.533607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:17.533607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:46.744148+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526898+0000","last_change":"2024-03-11T22:12:20.302654+0000","last_active":"2024-03-11T22:13:54.526898+0000","last_peered":"2024-03-11T22:13:54.526898+0000","last_clean":"2024-03-11T22:13:54.526898+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:13:54.526898+0000","last_undegraded":"2024-03-11T22:13:54.526898+0000","last_fullsized":"2024-03-11T22:13:54.526898+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:20.302574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:20.302574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.530916+0000","last_change":"2024-03-11T22:11:23.353503+0000","last_active":"2024-03-11T22:13:54.530916+0000","last_peered":"2024-03-11T22:13:54.530916+0000","last_clean":"2024-03-11T22:13:54.530916+0000","last_became_active":"2024-03-11T22:08:33.299778+0000","last_became_peered":"2024-03-11T22:08:33.299778+0000","last_unstale":"2024-03-11T22:13:54.530916+0000","last_undegraded":"2024-03-11T22:13:54.530916+0000","last_fullsized":"2024-03-11T22:13:54.530916+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:23.353379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:23.353379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":210,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531628+0000","last_change":"2024-03-11T22:12:24.093258+0000","last_active":"2024-03-11T22:13:54.531628+0000","last_peered":"2024-03-11T22:13:54.531628+0000","last_clean":"2024-03-11T22:13:54.531628+0000","last_became_active":"2024-03-11T22:08:35.053123+0000","last_became_peered":"2024-03-11T22:08:35.053123+0000","last_unstale":"2024-03-11T22:13:54.531628+0000","last_undegraded":"2024-03-11T22:13:54.531628+0000","last_fullsized":"2024-03-11T22:13:54.531628+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:12:24.093112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.093112+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":452,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.525494+0000","last_change":"2024-03-11T22:13:38.569874+0000","last_active":"2024-03-11T22:13:54.525494+0000","last_peered":"2024-03-11T22:13:54.525494+0000","last_clean":"2024-03-11T22:13:54.525494+0000","last_became_active":"2024-03-11T22:13:38.569606+0000","last_became_peered":"2024-03-11T22:13:38.569606+0000","last_unstale":"2024-03-11T22:13:54.525494+0000","last_undegraded":"2024-03-11T22:13:54.525494+0000","last_fullsized":"2024-03-11T22:13:54.525494+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:11:34.382250+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.382250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:55.553222+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":423697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"151'15475","reported_seq":15760,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.573257+0000","last_change":"2024-03-11T22:13:52.384647+0000","last_active":"2024-03-11T22:13:55.573257+0000","last_peered":"2024-03-11T22:13:55.573257+0000","last_clean":"2024-03-11T22:13:55.573257+0000","last_became_active":"2024-03-11T22:10:36.412847+0000","last_became_peered":"2024-03-11T22:10:36.412847+0000","last_unstale":"2024-03-11T22:13:55.573257+0000","last_undegraded":"2024-03-11T22:13:55.573257+0000","last_fullsized":"2024-03-11T22:13:55.573257+0000","mapping_epoch":99,"log_start":"132'9283","ondisk_log_start":"132'9283","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'15289","last_scrub_stamp":"2024-03-11T22:13:52.384437+0000","last_deep_scrub":"150'15289","last_deep_scrub_stamp":"2024-03-11T22:13:52.384437+0000","last_clean_scrub_stamp":"2024-03-11T22:13:52.384437+0000","objects_scrubbed":2194,"log_size":6192,"log_dups_size":0,"ondisk_log_size":6192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":24,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:05.504049+0000","scrub_duration":23610,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94699520,"num_objects":1445,"num_object_clones":0,"num_object_copies":4335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13281,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":191,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.534933+0000","last_change":"2024-03-11T22:12:50.861675+0000","last_active":"2024-03-11T22:13:54.534933+0000","last_peered":"2024-03-11T22:13:54.534933+0000","last_clean":"2024-03-11T22:13:54.534933+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:13:54.534933+0000","last_undegraded":"2024-03-11T22:13:54.534933+0000","last_fullsized":"2024-03-11T22:13:54.534933+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.861557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.861557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:11.522320+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":187,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.522198+0000","last_change":"2024-03-11T22:12:18.789167+0000","last_active":"2024-03-11T22:13:54.522198+0000","last_peered":"2024-03-11T22:13:54.522198+0000","last_clean":"2024-03-11T22:13:54.522198+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:13:54.522198+0000","last_undegraded":"2024-03-11T22:13:54.522198+0000","last_fullsized":"2024-03-11T22:13:54.522198+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:18.788609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:12:18.788609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:54.706557+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.524495+0000","last_change":"2024-03-11T22:12:42.910183+0000","last_active":"2024-03-11T22:13:54.524495+0000","last_peered":"2024-03-11T22:13:54.524495+0000","last_clean":"2024-03-11T22:13:54.524495+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:13:54.524495+0000","last_undegraded":"2024-03-11T22:13:54.524495+0000","last_fullsized":"2024-03-11T22:13:54.524495+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.910067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.910067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.099451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":211,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531641+0000","last_change":"2024-03-11T22:11:17.270935+0000","last_active":"2024-03-11T22:13:54.531641+0000","last_peered":"2024-03-11T22:13:54.531641+0000","last_clean":"2024-03-11T22:13:54.531641+0000","last_became_active":"2024-03-11T22:08:35.055770+0000","last_became_peered":"2024-03-11T22:08:35.055770+0000","last_unstale":"2024-03-11T22:13:54.531641+0000","last_undegraded":"2024-03-11T22:13:54.531641+0000","last_fullsized":"2024-03-11T22:13:54.531641+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-11T22:11:17.270813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:17.270813+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":452,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.529416+0000","last_change":"2024-03-11T22:13:38.575253+0000","last_active":"2024-03-11T22:13:54.529416+0000","last_peered":"2024-03-11T22:13:54.529416+0000","last_clean":"2024-03-11T22:13:54.529416+0000","last_became_active":"2024-03-11T22:13:38.571540+0000","last_became_peered":"2024-03-11T22:13:38.571540+0000","last_unstale":"2024-03-11T22:13:54.529416+0000","last_undegraded":"2024-03-11T22:13:54.529416+0000","last_fullsized":"2024-03-11T22:13:54.529416+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:11:34.382250+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:11:34.382250+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"151'13340","reported_seq":13495,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:55.585938+0000","last_change":"2024-03-11T22:13:19.688808+0000","last_active":"2024-03-11T22:13:55.585938+0000","last_peered":"2024-03-11T22:13:55.585938+0000","last_clean":"2024-03-11T22:13:55.585938+0000","last_became_active":"2024-03-11T22:10:36.408528+0000","last_became_peered":"2024-03-11T22:10:36.408528+0000","last_unstale":"2024-03-11T22:13:55.585938+0000","last_undegraded":"2024-03-11T22:13:55.585938+0000","last_fullsized":"2024-03-11T22:13:55.585938+0000","mapping_epoch":99,"log_start":"129'8059","ondisk_log_start":"129'8059","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'11496","last_scrub_stamp":"2024-03-11T22:13:19.688524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.688524+0000","objects_scrubbed":3273,"log_size":5281,"log_dups_size":0,"ondisk_log_size":5281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:23.402103+0000","scrub_duration":1754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90701824,"num_objects":1384,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13340,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":175,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.531822+0000","last_change":"2024-03-11T22:10:23.555240+0000","last_active":"2024-03-11T22:13:54.531822+0000","last_peered":"2024-03-11T22:13:54.531822+0000","last_clean":"2024-03-11T22:13:54.531822+0000","last_became_active":"2024-03-11T22:08:39.072422+0000","last_became_peered":"2024-03-11T22:08:39.072422+0000","last_unstale":"2024-03-11T22:13:54.531822+0000","last_undegraded":"2024-03-11T22:13:54.531822+0000","last_fullsized":"2024-03-11T22:13:54.531822+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:10:23.555159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:10:23.555159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":269,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.526776+0000","last_change":"2024-03-11T22:12:43.819554+0000","last_active":"2024-03-11T22:13:54.526776+0000","last_peered":"2024-03-11T22:13:54.526776+0000","last_clean":"2024-03-11T22:13:54.526776+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:13:54.526776+0000","last_undegraded":"2024-03-11T22:13:54.526776+0000","last_fullsized":"2024-03-11T22:13:54.526776+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:12:43.819269+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_clean_scrub_stamp":"2024-03-11T22:12:43.819269+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:13:58.958306+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.534928+0000","last_change":"2024-03-11T22:11:40.097948+0000","last_active":"2024-03-11T22:13:54.534928+0000","last_peered":"2024-03-11T22:13:54.534928+0000","last_clean":"2024-03-11T22:13:54.534928+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:13:54.534928+0000","last_undegraded":"2024-03-11T22:13:54.534928+0000","last_fullsized":"2024-03-11T22:13:54.534928+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:40.094297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:40.094297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":198,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-11T22:13:54.532017+0000","last_change":"2024-03-11T22:11:12.298926+0000","last_active":"2024-03-11T22:13:54.532017+0000","last_peered":"2024-03-11T22:13:54.532017+0000","last_clean":"2024-03-11T22:13:54.532017+0000","last_became_active":"2024-03-11T22:08:35.056698+0000","last_became_peered":"2024-03-11T22:08:35.056698+0000","last_unstale":"2024-03-11T22:13:54.532017+0000","last_undegraded":"2024-03-11T22:13:54.532017+0000","last_fullsized":"2024-03-11T22:13:54.532017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:11:12.298787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:12.298787+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1537933336,"num_objects":23468,"num_object_clones":0,"num_object_copies":70404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213470,"num_write_kb":7581953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4627509248,"data_stored":4627497032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84824,"ondisk_log_size":84824,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":289,"num_read_kb":284,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":483029,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":173,"num_read_kb":251,"num_write":146,"num_write_kb":1977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2039453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005125,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":601240,"kb_used_data":489552,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93143912,"statfs":{"total":95995035648,"available":95379365888,"internally_reserved":0,"allocated":501301248,"data_stored":500981857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":114358731},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332170,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":614408,"kb_used_data":502720,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93130744,"statfs":{"total":95995035648,"available":95365881856,"internally_reserved":0,"allocated":514785280,"data_stored":514482107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":693576,"kb_used_data":590204,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93051576,"statfs":{"total":95995035648,"available":95284813824,"internally_reserved":0,"allocated":604368896,"data_stored":604012584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738474,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896488,"kb_used_data":770908,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92848664,"statfs":{"total":95995035648,"available":95077031936,"internally_reserved":0,"allocated":789409792,"data_stored":789085574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,3,13,6,17,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":679956,"kb_used_data":580744,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93065196,"statfs":{"total":95995035648,"available":95298760704,"internally_reserved":0,"allocated":594681856,"data_stored":594378942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1036220,"kb_used_data":870320,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92708932,"statfs":{"total":95995035648,"available":94933946368,"internally_reserved":0,"allocated":891207680,"data_stored":890864486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920853,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":590636,"kb_used_data":487264,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93154516,"statfs":{"total":95995035648,"available":95390224384,"internally_reserved":0,"allocated":498958336,"data_stored":498630207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659214,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":491292,"kb_used_data":392080,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93253860,"statfs":{"total":95995035648,"available":95491952640,"internally_reserved":0,"allocated":401489920,"data_stored":401178261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":762187776,"data_stored":762183704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":585830400,"data_stored":585826328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":573833216,"data_stored":573833216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":848691200,"data_stored":848691200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":496435200,"data_stored":496435200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":388694016,"data_stored":388694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":492834816,"data_stored":492830744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":479002624,"data_stored":479002624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:13:59.311 INFO:tasks.thrashosds.thrasher:Setting 5.11 to [1, 0, 2] 2024-03-11T22:13:59.311 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.11', '1', '0', '2'] 2024-03-11T22:13:59.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pg-upmap 5.11 1 0 2 2024-03-11T22:13:59.393 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:59.423 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:59.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:59.452 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:59 smithi045 ceph-mon[39715]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:13:59.453 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:59 smithi045 ceph-mon[39715]: pgmap v211: 158 pgs: 158 active+clean; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 963 op/s 2024-03-11T22:13:59.453 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:59 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:13:59.453 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:13:59 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:13:59.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:59 smithi045 ceph-mon[30178]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:13:59.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:59 smithi045 ceph-mon[30178]: pgmap v211: 158 pgs: 158 active+clean; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 963 op/s 2024-03-11T22:13:59.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:59 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:13:59.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:13:59 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:13:59.524 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:59.553 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:59.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:59.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:59 smithi144 ceph-mon[33984]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:13:59.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:59 smithi144 ceph-mon[33984]: pgmap v211: 158 pgs: 158 active+clean; 1.4 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 963 op/s 2024-03-11T22:13:59.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:59 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:13:59.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:13:59 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:13:59.655 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:13:59.668 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:13:59.668 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:13:59.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:13:59.689 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:13:59.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:13:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:59.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:13:59.819 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:59.849 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:59.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:13:59.950 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:13:59.981 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:13:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:13:59.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:00.083 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:00.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:00.116 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:00.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:00.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:00.271 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:00.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:00.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:00 smithi045 ceph-mon[30178]: osdmap e153: 8 total, 8 up, 8 in 2024-03-11T22:14:00.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:00 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:14:00.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:00 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:14:00.279 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:14:00.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:00 smithi045 ceph-mon[39715]: osdmap e153: 8 total, 8 up, 8 in 2024-03-11T22:14:00.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:00 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:14:00.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:00 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:14:00.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:14:00.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:00.414 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:00.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:00.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:00.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T22:14:00.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:00.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:00.581 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:00.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:00.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:00 smithi144 ceph-mon[33984]: osdmap e153: 8 total, 8 up, 8 in 2024-03-11T22:14:00.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:00 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:14:00.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:00 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:14:00.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:14:00.682 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:00.708 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:00.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:00.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:00.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:00.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:00.858 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:00.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:00.959 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:00.987 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:00.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:01.094 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:01.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:01.140 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:01.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:01.239 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:01.241 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:01.271 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:01.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:01.313 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:01.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:01.383 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:01 smithi045 ceph-mon[39715]: pgmap v213: 158 pgs: 158 active+clean; 1.4 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 838 op/s 2024-03-11T22:14:01.383 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:01 smithi045 ceph-mon[39715]: osdmap e154: 8 total, 8 up, 8 in 2024-03-11T22:14:01.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:01 smithi045 ceph-mon[30178]: pgmap v213: 158 pgs: 158 active+clean; 1.4 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 838 op/s 2024-03-11T22:14:01.387 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:01 smithi045 ceph-mon[30178]: osdmap e154: 8 total, 8 up, 8 in 2024-03-11T22:14:01.426 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:01.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:01.527 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:01.553 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:01.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:01.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:01.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:01 smithi144 ceph-mon[33984]: pgmap v213: 158 pgs: 158 active+clean; 1.4 GiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 838 op/s 2024-03-11T22:14:01.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:01 smithi144 ceph-mon[33984]: osdmap e154: 8 total, 8 up, 8 in 2024-03-11T22:14:01.655 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:01.686 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:01.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:01.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:01.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:01.834 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:01.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:01.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:01.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:01.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:01.978 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:01.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:02.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:02.115 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:02.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:02.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:02.258 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:02.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:02.359 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:02.388 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:02.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:02.490 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:02.518 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:02.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:02.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:02.620 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:02.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:02.648 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:02.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:02.749 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:02.778 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:14:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:14:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:14:02.779 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:02.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:02.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:02.881 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:02.909 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:02.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:03.011 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:03.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:03.043 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:03.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:03.144 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:03.175 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:03.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:03.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:03.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:03.278 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:03 smithi045 ceph-mon[39715]: pgmap v215: 158 pgs: 158 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 887 op/s 2024-03-11T22:14:03.278 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:03 smithi045 ceph-mon[30178]: pgmap v215: 158 pgs: 158 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 887 op/s 2024-03-11T22:14:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:03.305 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:03.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:03.406 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:03.436 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:03.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:03.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:03.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:03.576 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:03.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:03.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:03 smithi144 ceph-mon[33984]: pgmap v215: 158 pgs: 158 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 887 op/s 2024-03-11T22:14:03.677 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:03.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:03.702 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:03.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:03.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:03.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:03.846 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:03.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:03.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:03.992 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:03.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:03.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:04.094 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:04.118 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:04.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:04.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:04.220 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:04.247 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:04.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:04.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:04.389 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:04.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:04.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:04.490 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:04.521 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:04.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:04.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:04 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:14:04.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:04 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:14:04.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:04 smithi144 ceph-mon[33984]: osdmap e155: 8 total, 8 up, 8 in 2024-03-11T22:14:04.623 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:04.627 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:04 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:14:04.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:04 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:14:04.628 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:04 smithi045 ceph-mon[39715]: osdmap e155: 8 total, 8 up, 8 in 2024-03-11T22:14:04.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:04 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:14:04.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:04 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:14:04.628 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:04 smithi045 ceph-mon[30178]: osdmap e155: 8 total, 8 up, 8 in 2024-03-11T22:14:04.649 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:04.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:04.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:04.792 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:04.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:04.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:04.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:04.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:04.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:05.043 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:05.072 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:05.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:05.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:05.174 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:05.201 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:05.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:05.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:05.344 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:05.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:05.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:05.446 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:05.476 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:05.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:05.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:05.528 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:05 smithi045 ceph-mon[39715]: pgmap v217: 158 pgs: 158 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 887 op/s 2024-03-11T22:14:05.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:05 smithi045 ceph-mon[30178]: pgmap v217: 158 pgs: 158 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 887 op/s 2024-03-11T22:14:05.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:05.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:05 smithi144 ceph-mon[33984]: pgmap v217: 158 pgs: 158 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 887 op/s 2024-03-11T22:14:05.619 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:05.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:05.722 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:05.751 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:05.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:05.852 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:05.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:05.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:05.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:06.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:06.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:06.046 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:06.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:06.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:14:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:14:06.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:06.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:06.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:06.301 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:06.333 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:06.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:06.434 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:06.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:06.463 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:06.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:06.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:06.613 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:06.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:06.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:06.661 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:06.715 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:06.771 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:06.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:06.873 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:06.902 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:06.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:06.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:07.003 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:07.033 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:07.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:07.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:07.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:07.172 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:07.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:07.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:07.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:07.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:07.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:07.343 INFO:teuthology.orchestra.run.smithi045.stderr:set 5.11 pg_upmap mapping to [1,0,2] 2024-03-11T22:14:07.364 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:14:07.416 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:07.443 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:07.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:07.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:07.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:07.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:07 smithi045 ceph-mon[39715]: pgmap v218: 158 pgs: 158 active+clean; 796 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-11T22:14:07.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:07 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1322283865' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:14:07.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:07 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:14:07.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:07 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1709785648' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.11", "id": [1, 0, 2]}]: dispatch 2024-03-11T22:14:07.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:07.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:07 smithi045 ceph-mon[30178]: pgmap v218: 158 pgs: 158 active+clean; 796 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-11T22:14:07.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:07 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1322283865' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:14:07.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:07 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:14:07.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:07 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1709785648' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.11", "id": [1, 0, 2]}]: dispatch 2024-03-11T22:14:07.545 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:07.588 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:07.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:07.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:07.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:07.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:07 smithi144 ceph-mon[33984]: pgmap v218: 158 pgs: 158 active+clean; 796 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-11T22:14:07.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:07 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1322283865' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:14:07.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:07 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:14:07.592 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:07 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1709785648' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.11", "id": [1, 0, 2]}]: dispatch 2024-03-11T22:14:07.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:07.730 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:07.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:07.736 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:14:07.737 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:14:07.737 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:14:07.737 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:14:07.746 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:14:07.791 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:14:07.832 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:07.832 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:14:07.860 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:07.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:07.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:07.870 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:14:07.875 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:14:07.875 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:14:07.875 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:14:07.875 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:14:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:14:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:14:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:14:07.876 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:14:07.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:08.001 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:08.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:08.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:08.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:08.141 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:08.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:08.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:08.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:08.288 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:08.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:08.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:08.415 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:08.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:08.520 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:08.549 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:08.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:08.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:08.651 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:08.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:08.678 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:08.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:08.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:08.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:14:08.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:14:08.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1709785648' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.11", "id": [1, 0, 2]}]': finished 2024-03-11T22:14:08.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[39715]: osdmap e156: 8 total, 8 up, 8 in 2024-03-11T22:14:08.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4196807011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:08.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[39715]: 3.18 scrub starts 2024-03-11T22:14:08.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[39715]: 3.18 scrub ok 2024-03-11T22:14:08.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:14:08.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:14:08.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1709785648' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.11", "id": [1, 0, 2]}]': finished 2024-03-11T22:14:08.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[30178]: osdmap e156: 8 total, 8 up, 8 in 2024-03-11T22:14:08.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4196807011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:08.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[30178]: 3.18 scrub starts 2024-03-11T22:14:08.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:08 smithi045 ceph-mon[30178]: 3.18 scrub ok 2024-03-11T22:14:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:08.819 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:08.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:08.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:08 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:14:08.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:08 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:14:08.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:08 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1709785648' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.11", "id": [1, 0, 2]}]': finished 2024-03-11T22:14:08.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:08 smithi144 ceph-mon[33984]: osdmap e156: 8 total, 8 up, 8 in 2024-03-11T22:14:08.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:08 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4196807011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:08.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:08 smithi144 ceph-mon[33984]: 3.18 scrub starts 2024-03-11T22:14:08.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:08 smithi144 ceph-mon[33984]: 3.18 scrub ok 2024-03-11T22:14:08.920 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:08.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:08.950 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:08.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:09.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:09.090 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:09.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:09.192 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:09.222 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:09.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:09.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:09.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:09.363 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:09.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:09.466 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:09.489 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:09.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:09.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:09.591 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:09.625 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:09.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:09.727 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:09.756 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:09.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:09.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:09.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:09 smithi045 ceph-mon[39715]: 5.12 scrub starts 2024-03-11T22:14:09.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:09 smithi045 ceph-mon[39715]: 5.12 scrub ok 2024-03-11T22:14:09.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:09 smithi045 ceph-mon[39715]: pgmap v220: 158 pgs: 158 active+clean; 796 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-11T22:14:09.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:09 smithi045 ceph-mon[39715]: osdmap e157: 8 total, 8 up, 8 in 2024-03-11T22:14:09.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:09 smithi045 ceph-mon[30178]: 5.12 scrub starts 2024-03-11T22:14:09.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:09 smithi045 ceph-mon[30178]: 5.12 scrub ok 2024-03-11T22:14:09.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:09 smithi045 ceph-mon[30178]: pgmap v220: 158 pgs: 158 active+clean; 796 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-11T22:14:09.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:09 smithi045 ceph-mon[30178]: osdmap e157: 8 total, 8 up, 8 in 2024-03-11T22:14:09.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:09 smithi144 ceph-mon[33984]: 5.12 scrub starts 2024-03-11T22:14:09.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:09 smithi144 ceph-mon[33984]: 5.12 scrub ok 2024-03-11T22:14:09.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:09 smithi144 ceph-mon[33984]: pgmap v220: 158 pgs: 158 active+clean; 796 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-11T22:14:09.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:09 smithi144 ceph-mon[33984]: osdmap e157: 8 total, 8 up, 8 in 2024-03-11T22:14:09.858 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:09.891 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:09.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:09.992 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:10.019 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:10.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:10.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:10.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:10.164 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:10.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:10.265 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:10.297 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:10.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:10.398 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:10.422 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:10.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:10.524 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:10.551 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:10.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:10.653 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:10.679 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:10.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:10.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:10.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[30178]: 6.e scrub starts 2024-03-11T22:14:10.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[30178]: 6.e scrub ok 2024-03-11T22:14:10.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:14:10.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:14:10.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[30178]: osdmap e158: 8 total, 8 up, 8 in 2024-03-11T22:14:10.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[30178]: 1.4 scrub starts 2024-03-11T22:14:10.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[30178]: 1.4 scrub ok 2024-03-11T22:14:10.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[30178]: 2.1 scrub starts 2024-03-11T22:14:10.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[30178]: 2.1 scrub ok 2024-03-11T22:14:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[39715]: 6.e scrub starts 2024-03-11T22:14:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[39715]: 6.e scrub ok 2024-03-11T22:14:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:14:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:14:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[39715]: osdmap e158: 8 total, 8 up, 8 in 2024-03-11T22:14:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[39715]: 1.4 scrub starts 2024-03-11T22:14:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[39715]: 1.4 scrub ok 2024-03-11T22:14:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[39715]: 2.1 scrub starts 2024-03-11T22:14:10.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:10 smithi045 ceph-mon[39715]: 2.1 scrub ok 2024-03-11T22:14:10.781 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:10.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:10.810 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:10.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:10.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:10 smithi144 ceph-mon[33984]: 6.e scrub starts 2024-03-11T22:14:10.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:10 smithi144 ceph-mon[33984]: 6.e scrub ok 2024-03-11T22:14:10.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:10 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:14:10.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:10 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:14:10.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:10 smithi144 ceph-mon[33984]: osdmap e158: 8 total, 8 up, 8 in 2024-03-11T22:14:10.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:10 smithi144 ceph-mon[33984]: 1.4 scrub starts 2024-03-11T22:14:10.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:10 smithi144 ceph-mon[33984]: 1.4 scrub ok 2024-03-11T22:14:10.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:10 smithi144 ceph-mon[33984]: 2.1 scrub starts 2024-03-11T22:14:10.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:10 smithi144 ceph-mon[33984]: 2.1 scrub ok 2024-03-11T22:14:10.912 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:10.941 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:10.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:10.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:11.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:11.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:11.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:11.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:11.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:11.208 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:11.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:11.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:11.353 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:11.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:11.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:11.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:11.500 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:11.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:11.601 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:11.636 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:11.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:11.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:11.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:11.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:11.778 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[39715]: 1.3 scrub starts 2024-03-11T22:14:11.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[39715]: 1.3 scrub ok 2024-03-11T22:14:11.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[39715]: pgmap v223: 158 pgs: 158 active+clean; 758 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T22:14:11.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[39715]: 1.9 scrub starts 2024-03-11T22:14:11.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[39715]: 1.9 scrub ok 2024-03-11T22:14:11.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[39715]: 7.7 scrub starts 2024-03-11T22:14:11.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[39715]: 3.11 scrub starts 2024-03-11T22:14:11.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[39715]: 3.11 scrub ok 2024-03-11T22:14:11.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[39715]: osdmap e159: 8 total, 8 up, 8 in 2024-03-11T22:14:11.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[30178]: 1.3 scrub starts 2024-03-11T22:14:11.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[30178]: 1.3 scrub ok 2024-03-11T22:14:11.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[30178]: pgmap v223: 158 pgs: 158 active+clean; 758 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T22:14:11.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[30178]: 1.9 scrub starts 2024-03-11T22:14:11.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[30178]: 1.9 scrub ok 2024-03-11T22:14:11.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[30178]: 7.7 scrub starts 2024-03-11T22:14:11.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[30178]: 3.11 scrub starts 2024-03-11T22:14:11.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[30178]: 3.11 scrub ok 2024-03-11T22:14:11.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:11 smithi045 ceph-mon[30178]: osdmap e159: 8 total, 8 up, 8 in 2024-03-11T22:14:11.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:11 smithi144 ceph-mon[33984]: 1.3 scrub starts 2024-03-11T22:14:11.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:11 smithi144 ceph-mon[33984]: 1.3 scrub ok 2024-03-11T22:14:11.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:11 smithi144 ceph-mon[33984]: pgmap v223: 158 pgs: 158 active+clean; 758 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T22:14:11.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:11 smithi144 ceph-mon[33984]: 1.9 scrub starts 2024-03-11T22:14:11.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:11 smithi144 ceph-mon[33984]: 1.9 scrub ok 2024-03-11T22:14:11.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:11 smithi144 ceph-mon[33984]: 7.7 scrub starts 2024-03-11T22:14:11.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:11 smithi144 ceph-mon[33984]: 3.11 scrub starts 2024-03-11T22:14:11.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:11 smithi144 ceph-mon[33984]: 3.11 scrub ok 2024-03-11T22:14:11.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:11 smithi144 ceph-mon[33984]: osdmap e159: 8 total, 8 up, 8 in 2024-03-11T22:14:11.862 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:11.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:11.899 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:11.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:12.000 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:12.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:12.026 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:12.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:12.126 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:12.165 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:12.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:12.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:12.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:12.305 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:12.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:12.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:12.446 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:12.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:12.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:12.548 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:12.579 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:12.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:12.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:12.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:12.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:12.779 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:14:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:14:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:14:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:12 smithi045 ceph-mon[39715]: 6.0 scrub starts 2024-03-11T22:14:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:12 smithi045 ceph-mon[39715]: 6.0 scrub ok 2024-03-11T22:14:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:12 smithi045 ceph-mon[39715]: 7.7 scrub ok 2024-03-11T22:14:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:12 smithi045 ceph-mon[39715]: 1.8 scrub starts 2024-03-11T22:14:12.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:12 smithi045 ceph-mon[39715]: 1.8 scrub ok 2024-03-11T22:14:12.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:12 smithi045 ceph-mon[30178]: 6.0 scrub starts 2024-03-11T22:14:12.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:12 smithi045 ceph-mon[30178]: 6.0 scrub ok 2024-03-11T22:14:12.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:12 smithi045 ceph-mon[30178]: 7.7 scrub ok 2024-03-11T22:14:12.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:12 smithi045 ceph-mon[30178]: 1.8 scrub starts 2024-03-11T22:14:12.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:12 smithi045 ceph-mon[30178]: 1.8 scrub ok 2024-03-11T22:14:12.818 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:12.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:12.842 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:12.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:12.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:12 smithi144 ceph-mon[33984]: 6.0 scrub starts 2024-03-11T22:14:12.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:12 smithi144 ceph-mon[33984]: 6.0 scrub ok 2024-03-11T22:14:12.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:12 smithi144 ceph-mon[33984]: 7.7 scrub ok 2024-03-11T22:14:12.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:12 smithi144 ceph-mon[33984]: 1.8 scrub starts 2024-03-11T22:14:12.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:12 smithi144 ceph-mon[33984]: 1.8 scrub ok 2024-03-11T22:14:12.944 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:12.972 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:12.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:12.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:13.074 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:13.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:13.099 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:13.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:13.200 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:13.228 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:13.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:13.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:13.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:13.376 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:13.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:13.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:13.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:13.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:13.524 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:13.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:13.625 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:13.653 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:13.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:13.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[30178]: 1.2 scrub starts 2024-03-11T22:14:13.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[30178]: 1.2 scrub ok 2024-03-11T22:14:13.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[30178]: pgmap v225: 158 pgs: 158 active+clean; 490 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-11T22:14:13.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[30178]: 1.a scrub starts 2024-03-11T22:14:13.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[30178]: 1.a scrub ok 2024-03-11T22:14:13.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[30178]: 3.b scrub starts 2024-03-11T22:14:13.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[30178]: 3.b scrub ok 2024-03-11T22:14:13.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[30178]: 1.7 scrub starts 2024-03-11T22:14:13.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[30178]: 1.7 scrub ok 2024-03-11T22:14:13.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[30178]: 1.5 scrub starts 2024-03-11T22:14:13.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[30178]: 1.5 scrub ok 2024-03-11T22:14:13.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:14:13.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[39715]: 1.2 scrub starts 2024-03-11T22:14:13.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[39715]: 1.2 scrub ok 2024-03-11T22:14:13.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[39715]: pgmap v225: 158 pgs: 158 active+clean; 490 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-11T22:14:13.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[39715]: 1.a scrub starts 2024-03-11T22:14:13.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[39715]: 1.a scrub ok 2024-03-11T22:14:13.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[39715]: 3.b scrub starts 2024-03-11T22:14:13.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[39715]: 3.b scrub ok 2024-03-11T22:14:13.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[39715]: 1.7 scrub starts 2024-03-11T22:14:13.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[39715]: 1.7 scrub ok 2024-03-11T22:14:13.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[39715]: 1.5 scrub starts 2024-03-11T22:14:13.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[39715]: 1.5 scrub ok 2024-03-11T22:14:13.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:13 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:14:13.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:13.810 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:13.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:13.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:13.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:13 smithi144 ceph-mon[33984]: 1.2 scrub starts 2024-03-11T22:14:13.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:13 smithi144 ceph-mon[33984]: 1.2 scrub ok 2024-03-11T22:14:13.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:13 smithi144 ceph-mon[33984]: pgmap v225: 158 pgs: 158 active+clean; 490 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-11T22:14:13.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:13 smithi144 ceph-mon[33984]: 1.a scrub starts 2024-03-11T22:14:13.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:13 smithi144 ceph-mon[33984]: 1.a scrub ok 2024-03-11T22:14:13.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:13 smithi144 ceph-mon[33984]: 3.b scrub starts 2024-03-11T22:14:13.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:13 smithi144 ceph-mon[33984]: 3.b scrub ok 2024-03-11T22:14:13.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:13 smithi144 ceph-mon[33984]: 1.7 scrub starts 2024-03-11T22:14:13.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:13 smithi144 ceph-mon[33984]: 1.7 scrub ok 2024-03-11T22:14:13.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:13 smithi144 ceph-mon[33984]: 1.5 scrub starts 2024-03-11T22:14:13.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:13 smithi144 ceph-mon[33984]: 1.5 scrub ok 2024-03-11T22:14:13.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:13 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:14:13.912 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:13.944 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:13.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:14.045 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:14.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:14.076 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:14.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:14.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:14.225 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:14.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:14.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:14.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:14.366 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:14.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:14.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:14.506 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:14.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:14.609 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:14.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:14.660 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:14.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:14.761 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:14.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:14 smithi045 ceph-mon[39715]: 4.1c scrub starts 2024-03-11T22:14:14.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:14 smithi045 ceph-mon[39715]: 4.1c scrub ok 2024-03-11T22:14:14.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:14 smithi045 ceph-mon[39715]: 1.6 scrub starts 2024-03-11T22:14:14.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:14 smithi045 ceph-mon[39715]: 1.6 scrub ok 2024-03-11T22:14:14.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:14 smithi045 ceph-mon[30178]: 4.1c scrub starts 2024-03-11T22:14:14.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:14 smithi045 ceph-mon[30178]: 4.1c scrub ok 2024-03-11T22:14:14.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:14 smithi045 ceph-mon[30178]: 1.6 scrub starts 2024-03-11T22:14:14.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:14 smithi045 ceph-mon[30178]: 1.6 scrub ok 2024-03-11T22:14:14.790 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:14.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:14.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:14.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:14 smithi144 ceph-mon[33984]: 4.1c scrub starts 2024-03-11T22:14:14.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:14 smithi144 ceph-mon[33984]: 4.1c scrub ok 2024-03-11T22:14:14.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:14 smithi144 ceph-mon[33984]: 1.6 scrub starts 2024-03-11T22:14:14.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:14 smithi144 ceph-mon[33984]: 1.6 scrub ok 2024-03-11T22:14:14.892 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:14.919 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:14.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:15.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:15.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:15.075 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:15.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:15.176 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:15.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:15.208 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:15.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:15.309 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:15.337 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:15.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:15.442 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:15.475 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:15.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:15.576 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:15.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: 2.0 scrub starts 2024-03-11T22:14:15.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: 2.0 scrub ok 2024-03-11T22:14:15.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: 6.8 scrub starts 2024-03-11T22:14:15.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: 6.8 scrub ok 2024-03-11T22:14:15.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: pgmap v226: 158 pgs: 158 active+clean; 490 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 810 op/s 2024-03-11T22:14:15.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:14:15.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:15.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:15.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: 5.17 scrub starts 2024-03-11T22:14:15.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: 5.17 scrub ok 2024-03-11T22:14:15.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: 3.a scrub starts 2024-03-11T22:14:15.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: 3.a scrub ok 2024-03-11T22:14:15.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: 4.18 scrub starts 2024-03-11T22:14:15.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[39715]: 4.18 scrub ok 2024-03-11T22:14:15.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: 2.0 scrub starts 2024-03-11T22:14:15.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: 2.0 scrub ok 2024-03-11T22:14:15.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: 6.8 scrub starts 2024-03-11T22:14:15.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: 6.8 scrub ok 2024-03-11T22:14:15.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: pgmap v226: 158 pgs: 158 active+clean; 490 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 810 op/s 2024-03-11T22:14:15.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:14:15.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:15.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:15.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: 5.17 scrub starts 2024-03-11T22:14:15.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: 5.17 scrub ok 2024-03-11T22:14:15.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: 3.a scrub starts 2024-03-11T22:14:15.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: 3.a scrub ok 2024-03-11T22:14:15.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: 4.18 scrub starts 2024-03-11T22:14:15.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:15 smithi045 ceph-mon[30178]: 4.18 scrub ok 2024-03-11T22:14:15.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:15.604 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:15.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:15.704 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:15.732 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:15.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:15.834 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:15.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: 2.0 scrub starts 2024-03-11T22:14:15.904 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: 2.0 scrub ok 2024-03-11T22:14:15.904 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: 6.8 scrub starts 2024-03-11T22:14:15.904 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: 6.8 scrub ok 2024-03-11T22:14:15.904 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: pgmap v226: 158 pgs: 158 active+clean; 490 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 810 op/s 2024-03-11T22:14:15.904 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:14:15.904 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:15.904 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:15.904 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: 5.17 scrub starts 2024-03-11T22:14:15.904 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: 5.17 scrub ok 2024-03-11T22:14:15.904 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: 3.a scrub starts 2024-03-11T22:14:15.904 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: 3.a scrub ok 2024-03-11T22:14:15.904 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: 4.18 scrub starts 2024-03-11T22:14:15.904 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-mon[33984]: 4.18 scrub ok 2024-03-11T22:14:15.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:15.906 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:15.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:16.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:16.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:16.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:16.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:16.152 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:16.181 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:16.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:16.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:16.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:16.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:16.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:16.329 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:14:16.329 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-03-11T22:14:16.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-03-11T22:14:16.329 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:14:16.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.330 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205570 7.6 7:6c76be4d:::benchmark_data_smithi045_83007_object87099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.330 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.334799+0000", 2024-03-11T22:14:16.330 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 30.991967533, 2024-03-11T22:14:16.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.254940903, 2024-03-11T22:14:16.330 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.330 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.330 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.330 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.330 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.330 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205570 2024-03-11T22:14:16.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.330 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334799+0000", 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334799+0000", 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334800+0000", 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9999999999999995e-07 2024-03-11T22:14:16.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334805+0000", 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1220000000000003e-06 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334818+0000", 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2605e-05 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334827+0000", 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6980000000000004e-06 2024-03-11T22:14:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.442726+0000", 2024-03-11T22:14:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.107898972 2024-03-11T22:14:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.442893+0000", 2024-03-11T22:14:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016715600000000001 2024-03-11T22:14:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.449827+0000", 2024-03-11T22:14:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0069343119999999998 2024-03-11T22:14:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.463002+0000", 2024-03-11T22:14:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013174859000000001 2024-03-11T22:14:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.463047+0000", 2024-03-11T22:14:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5019999999999999e-05 2024-03-11T22:14:16.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.589496+0000", 2024-03-11T22:14:16.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.126448703 2024-03-11T22:14:16.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.589674+0000", 2024-03-11T22:14:16.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000178324 2024-03-11T22:14:16.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.589740+0000", 2024-03-11T22:14:16.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6131999999999996e-05 2024-03-11T22:14:16.336 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205573 7.6 7:6e2605c0:::benchmark_data_smithi045_83007_object87102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.339661+0000", 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 30.987105695, 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26130369399999998, 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205573 2024-03-11T22:14:16.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339661+0000", 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339661+0000", 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339662+0000", 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5700000000000002e-07 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339666+0000", 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.715e-06 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339674+0000", 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1249999999999993e-06 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339679+0000", 2024-03-11T22:14:16.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1529999999999996e-06 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.463078+0000", 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.123399414 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.463200+0000", 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012128700000000001 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.467183+0000", 2024-03-11T22:14:16.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0039832870000000003 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.473615+0000", 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0064318769999999999 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.473658+0000", 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3689000000000001e-05 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.600799+0000", 2024-03-11T22:14:16.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.12714049199999999 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.600900+0000", 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000101374 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.600965+0000", 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4523999999999994e-05 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.343 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205765 7.6 7:61c98817:::benchmark_data_smithi045_83007_object87294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.343 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.659350+0000", 2024-03-11T22:14:16.343 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 30.667416987999999, 2024-03-11T22:14:16.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26587694699999997, 2024-03-11T22:14:16.343 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.343 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.343 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.343 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.343 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.344 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205765 2024-03-11T22:14:16.344 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.344 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.344 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.344 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.344 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659350+0000", 2024-03-11T22:14:16.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.345 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.345 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659350+0000", 2024-03-11T22:14:16.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.346 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.346 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659351+0000", 2024-03-11T22:14:16.346 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0299999999999998e-07 2024-03-11T22:14:16.346 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.346 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.346 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.346 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659355+0000", 2024-03-11T22:14:16.346 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0879999999999997e-06 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659368+0000", 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3246e-05 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659375+0000", 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2620000000000001e-06 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659433+0000", 2024-03-11T22:14:16.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7500000000000002e-05 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.660319+0000", 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00088605900000000002 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.660450+0000", 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000131118 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.663683+0000", 2024-03-11T22:14:16.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032326999999999998 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.667411+0000", 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037283849999999999 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925108+0000", 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25769741899999998 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925161+0000", 2024-03-11T22:14:16.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2249000000000001e-05 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925227+0000", 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6118000000000006e-05 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(osd.2.0:40363 7.e e150/99 7:7ad23d92:::benchmark_data_smithi045_83007_object117059:head v 150'14406, mlcod=150'14402)", 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.670957+0000", 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 30.655810272, 2024-03-11T22:14:16.350 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.254146123, 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.670957+0000", 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.670957+0000", 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.670957+0000", 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.97e-07 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.670960+0000", 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2030000000000001e-06 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.670965+0000", 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2259999999999996e-06 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.670969+0000", 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9079999999999997e-06 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.671761+0000", 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00079232900000000004 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.671779+0000", 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7638999999999998e-05 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925059+0000", 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25328029099999999 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925103+0000", 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.333e-05 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205777 7.6 7:6d0e94e9:::benchmark_data_smithi045_83007_object87306:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.674945+0000", 2024-03-11T22:14:16.354 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 30.651822226, 2024-03-11T22:14:16.355 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26027302000000002, 2024-03-11T22:14:16.355 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.355 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.355 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.355 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.355 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.355 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205777 2024-03-11T22:14:16.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.356 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.674945+0000", 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.674945+0000", 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.674945+0000", 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1399999999999997e-07 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.674948+0000", 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3780000000000001e-06 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.674955+0000", 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8730000000000001e-06 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.674961+0000", 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3290000000000001e-06 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678647+0000", 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003686553 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678780+0000", 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000133101 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678913+0000", 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013320899999999999 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.360 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:14:15 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:14:16.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925270+0000", 2024-03-11T22:14:16.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24635607000000001 2024-03-11T22:14:16.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.928772+0000", 2024-03-11T22:14:16.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035028960000000001 2024-03-11T22:14:16.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935100+0000", 2024-03-11T22:14:16.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0063271990000000004 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935156+0000", 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6638000000000002e-05 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935218+0000", 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1359999999999995e-05 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(osd.2.0:40365 7.e e150/99 7:7ad5e5b2:::benchmark_data_smithi045_83007_object116423:head v 150'14408, mlcod=150'14402)", 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.682871+0000", 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 30.64389538, 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25217707900000003, 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.682871+0000", 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.682871+0000", 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.682872+0000", 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8100000000000002e-07 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.682879+0000", 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6279999999999999e-06 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.682890+0000", 2024-03-11T22:14:16.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0725e-05 2024-03-11T22:14:16.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.365 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.365 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.682897+0000", 2024-03-11T22:14:16.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1999999999999997e-06 2024-03-11T22:14:16.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925128+0000", 2024-03-11T22:14:16.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.242230959 2024-03-11T22:14:16.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.367 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925146+0000", 2024-03-11T22:14:16.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8527999999999999e-05 2024-03-11T22:14:16.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.367 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935008+0000", 2024-03-11T22:14:16.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0098619789999999999 2024-03-11T22:14:16.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.367 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935049+0000", 2024-03-11T22:14:16.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0379000000000001e-05 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(osd.2.0:40367 7.e e150/99 7:7ad6d4df:::benchmark_data_smithi045_83007_object106301:head v 150'14409, mlcod=150'14402)", 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.685109+0000", 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 30.641657401, 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26332421700000003, 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685109+0000", 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685109+0000", 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685110+0000", 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1999999999999998e-07 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685117+0000", 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5400000000000001e-06 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685128+0000", 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1573000000000001e-05 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685136+0000", 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4739999999999999e-06 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.928850+0000", 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24371410600000001 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.928871+0000", 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1254e-05 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.948384+0000", 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.019512390000000001 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.948434+0000", 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9960000000000003e-05 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:205777 7.6 e150/99 ondisk, result = 0)", 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.685801+0000", 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 30.640966073000001, 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26263702300000002, 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685801+0000", 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685801+0000", 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685801+0000", 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.92e-07 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685806+0000", 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9679999999999997e-06 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.824171+0000", 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.138364602 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.824182+0000", 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0991e-05 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935084+0000", 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.110902444 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935093+0000", 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3829999999999992e-06 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.948438+0000", 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013344943 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.375 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.376 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.376 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206092 7.6 7:6f97db3b:::benchmark_data_smithi045_83007_object87621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.376 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.364084+0000", 2024-03-11T22:14:16.376 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 29.962682562000001, 2024-03-11T22:14:16.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26853602199999999, 2024-03-11T22:14:16.378 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.378 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.378 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.378 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.378 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.378 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206092 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.364084+0000", 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.364084+0000", 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.364085+0000", 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6499999999999998e-07 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.364089+0000", 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6509999999999999e-06 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.364100+0000", 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1148999999999999e-05 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.364108+0000", 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8499999999999994e-06 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.619709+0000", 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25560139399999998 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.619854+0000", 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014506 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.620091+0000", 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023680199999999999 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.623283+0000", 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031920569999999999 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.623326+0000", 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3340000000000002e-05 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632383+0000", 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090561749999999996 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632555+0000", 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017295 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632620+0000", 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4828999999999999e-05 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.383 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206102 7.6 7:6abe9cdb:::benchmark_data_smithi045_83007_object87631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.381022+0000", 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 29.945745207000002, 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25920192800000003, 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206102 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.384 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.381022+0000", 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.381022+0000", 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.381022+0000", 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4499999999999997e-07 2024-03-11T22:14:16.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.381025+0000", 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8459999999999999e-06 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.381031+0000", 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0630000000000002e-06 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.381036+0000", 2024-03-11T22:14:16.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8400000000000002e-06 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.623360+0000", 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24232437500000001 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.623487+0000", 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012716800000000001 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.623637+0000", 2024-03-11T22:14:16.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014924599999999999 2024-03-11T22:14:16.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.388 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632284+0000", 2024-03-11T22:14:16.388 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0086475739999999999 2024-03-11T22:14:16.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.637029+0000", 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0047448339999999999 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.640103+0000", 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030742759999999999 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.640160+0000", 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6295999999999998e-05 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.389 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.640224+0000", 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3965000000000002e-05 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206106 7.6 7:6eb77cd5:::benchmark_data_smithi045_83007_object87635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.386909+0000", 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 29.939857867000001, 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25651542799999999, 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.390 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206106 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.386909+0000", 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.386909+0000", 2024-03-11T22:14:16.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.386911+0000", 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.525e-06 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.386915+0000", 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3030000000000001e-06 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.386926+0000", 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0937e-05 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.386933+0000", 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5399999999999998e-06 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.626396+0000", 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.239463062 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.626515+0000", 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011841300000000001 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.626649+0000", 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000133819 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632331+0000", 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0056820280000000004 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.394 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.637035+0000", 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0047048860000000001 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.643259+0000", 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0062232880000000004 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.643311+0000", 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2222000000000002e-05 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.643424+0000", 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113405 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206109 7.6 7:639211c5:::benchmark_data_smithi045_83007_object87638:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.390165+0000", 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 29.936601606, 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26705835500000003, 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.396 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206109 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.390165+0000", 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.390165+0000", 2024-03-11T22:14:16.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.398 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.398 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.390166+0000", 2024-03-11T22:14:16.398 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1179999999999999e-06 2024-03-11T22:14:16.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.398 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.398 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.390171+0000", 2024-03-11T22:14:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5229999999999997e-06 2024-03-11T22:14:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.390182+0000", 2024-03-11T22:14:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0787e-05 2024-03-11T22:14:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.390189+0000", 2024-03-11T22:14:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2289999999999997e-06 2024-03-11T22:14:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.629400+0000", 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.239211216 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.629516+0000", 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011550899999999999 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.629641+0000", 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000125625 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632369+0000", 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027274249999999999 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.637041+0000", 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046722300000000003 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657111+0000", 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.020069908000000001 2024-03-11T22:14:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657163+0000", 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2206999999999997e-05 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657224+0000", 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0578000000000002e-05 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206113 7.6 7:6bba99fc:::benchmark_data_smithi045_83007_object87642:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.394088+0000", 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 29.932678818999999, 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26328315299999999, 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206113 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.403 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.394088+0000", 2024-03-11T22:14:16.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.404 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.404 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.394088+0000", 2024-03-11T22:14:16.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.404 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.404 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.394089+0000", 2024-03-11T22:14:16.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.389e-06 2024-03-11T22:14:16.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.394094+0000", 2024-03-11T22:14:16.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5789999999999997e-06 2024-03-11T22:14:16.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.394105+0000", 2024-03-11T22:14:16.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1455e-05 2024-03-11T22:14:16.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.394113+0000", 2024-03-11T22:14:16.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5650000000000004e-06 2024-03-11T22:14:16.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632727+0000", 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23861432799999999 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632903+0000", 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000176007 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.633016+0000", 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011265 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.640264+0000", 2024-03-11T22:14:16.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072476320000000004 2024-03-11T22:14:16.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.651405+0000", 2024-03-11T22:14:16.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011141751 2024-03-11T22:14:16.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657270+0000", 2024-03-11T22:14:16.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058646929999999998 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657319+0000", 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9270000000000001e-05 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657371+0000", 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1833999999999999e-05 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206115 7.6 7:6c309bcc:::benchmark_data_smithi045_83007_object87644:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.396871+0000", 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 29.929896279000001, 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26066324600000002, 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206115 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.396871+0000", 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.396871+0000", 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.396872+0000", 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.327e-06 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.396895+0000", 2024-03-11T22:14:16.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3008999999999999e-05 2024-03-11T22:14:16.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.396907+0000", 2024-03-11T22:14:16.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1812e-05 2024-03-11T22:14:16.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.396914+0000", 2024-03-11T22:14:16.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7670000000000003e-06 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.637102+0000", 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.240187228 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.637231+0000", 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012963100000000001 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.637349+0000", 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011807900000000001 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.640299+0000", 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029497719999999998 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.651411+0000", 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011112037 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657418+0000", 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060068689999999998 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657475+0000", 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6654999999999999e-05 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657534+0000", 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.906e-05 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206260 7.6 7:64cdc02f:::benchmark_data_smithi045_83007_object87789:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.716010+0000", 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 29.610757104000001, 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.35534831, 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206260 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.716010+0000", 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.716010+0000", 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.716010+0000", 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7700000000000005e-07 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.716013+0000", 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7829999999999999e-06 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.716019+0000", 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0190000000000003e-06 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.716024+0000", 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2209999999999996e-06 2024-03-11T22:14:16.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.929286+0000", 2024-03-11T22:14:16.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.213261531 2024-03-11T22:14:16.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.929393+0000", 2024-03-11T22:14:16.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010679300000000001 2024-03-11T22:14:16.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.929510+0000", 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116945 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.939058+0000", 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0095488250000000004 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.943333+0000", 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042741690000000004 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071236+0000", 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.127903724 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071295+0000", 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9200000000000002e-05 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071358+0000", 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2623e-05 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.421 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.422 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206278 7.6 7:606b27ef:::benchmark_data_smithi045_83007_object87807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.422 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.743193+0000", 2024-03-11T22:14:16.422 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 29.583574079000002, 2024-03-11T22:14:16.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32832548299999997, 2024-03-11T22:14:16.422 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.422 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.422 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.422 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206278 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.743193+0000", 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.743193+0000", 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.743195+0000", 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.73e-06 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.743200+0000", 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2750000000000003e-06 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.743216+0000", 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6545000000000002e-05 2024-03-11T22:14:16.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.743226+0000", 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2480000000000002e-06 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.933884+0000", 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19065871100000001 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.934001+0000", 2024-03-11T22:14:16.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116561 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.934132+0000", 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013122899999999999 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.943338+0000", 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0092062749999999999 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.943410+0000", 2024-03-11T22:14:16.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1379999999999998e-05 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071427+0000", 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.12801737199999999 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071480+0000", 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2843999999999998e-05 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071518+0000", 2024-03-11T22:14:16.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8312999999999997e-05 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206302 7.6 7:6f8f4a0d:::benchmark_data_smithi045_83007_object87831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.783991+0000", 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 29.542776270000001, 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.287799886, 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.428 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206302 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.783991+0000", 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.783991+0000", 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.429 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.783992+0000", 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1170000000000001e-06 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.783996+0000", 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3950000000000003e-06 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.784005+0000", 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0969999999999995e-06 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.784012+0000", 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1740000000000003e-06 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.939154+0000", 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15514196999999999 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.939302+0000", 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014793200000000001 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.939424+0000", 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012194899999999999 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.943452+0000", 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040274509999999996 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.943496+0000", 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3856999999999997e-05 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071598+0000", 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.128102418 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071737+0000", 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000139281 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071790+0000", 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3245000000000003e-05 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.433 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:207163 7.6 7:6ad72b27:::benchmark_data_smithi045_83007_object88692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:48.007054+0000", 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 28.319712685999999, 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.28364153800000003, 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 207163 2024-03-11T22:14:16.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.007054+0000", 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.007054+0000", 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.007055+0000", 2024-03-11T22:14:16.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6600000000000005e-07 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.007059+0000", 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1760000000000003e-06 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.007070+0000", 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1046000000000001e-05 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.007078+0000", 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.418e-06 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.167390+0000", 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16031189300000001 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.167479+0000", 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9093999999999998e-05 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.167545+0000", 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5897999999999998e-05 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.176768+0000", 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0092230179999999995 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.176786+0000", 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8175000000000001e-05 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.289974+0000", 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.113188146 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.290618+0000", 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00064369999999999998 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.290696+0000", 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8108000000000005e-05 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.439 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:207176 7.6 7:633abc52:::benchmark_data_smithi045_83007_object88705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:48.023879+0000", 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 28.302888136, 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26985240199999999, 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 207176 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.023879+0000", 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.023879+0000", 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.023880+0000", 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5850000000000001e-06 2024-03-11T22:14:16.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.023884+0000", 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7589999999999999e-06 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.023902+0000", 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7965000000000002e-05 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.023910+0000", 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5950000000000001e-06 2024-03-11T22:14:16.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.170249+0000", 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.146339513 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.170383+0000", 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000133657 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.170518+0000", 2024-03-11T22:14:16.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000135053 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.176823+0000", 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0063052079999999996 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.176903+0000", 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0298999999999998e-05 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.293591+0000", 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.116688002 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.293670+0000", 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8276999999999996e-05 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.293731+0000", 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1488999999999994e-05 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.445 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:207181 7.6 7:6f47bc4a:::benchmark_data_smithi045_83007_object88710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:48.028352+0000", 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 28.298414825999998, 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26552772899999999, 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 207181 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.446 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.028352+0000", 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.028352+0000", 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.028353+0000", 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.46e-06 2024-03-11T22:14:16.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:16.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.028358+0000", 2024-03-11T22:14:16.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9840000000000001e-06 2024-03-11T22:14:16.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:16.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.028373+0000", 2024-03-11T22:14:16.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4144e-05 2024-03-11T22:14:16.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:16.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.028380+0000", 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5850000000000002e-06 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.172534+0000", 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14415375999999999 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.172651+0000", 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117262 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:14:16.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.172793+0000", 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014129100000000001 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.176888+0000", 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.004095975 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.176910+0000", 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2014999999999998e-05 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:16.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.293777+0000", 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.116866316 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.293823+0000", 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6659000000000002e-05 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.293880+0000", 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6277999999999999e-05 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.451 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:16.452 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:16.452 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:14:16.452 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:16.480 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:16.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:16.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:16.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:14:16.582 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:16.614 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:16.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:16.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:16.761 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:16.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:16.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[39715]: 4.f scrub starts 2024-03-11T22:14:16.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[39715]: 4.f scrub ok 2024-03-11T22:14:16.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[39715]: 6.16 scrub starts 2024-03-11T22:14:16.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[39715]: 6.16 scrub ok 2024-03-11T22:14:16.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:14:16.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:14:16.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:16.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[39715]: 5.13 scrub starts 2024-03-11T22:14:16.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[39715]: 5.13 scrub ok 2024-03-11T22:14:16.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[39715]: osdmap e160: 8 total, 8 up, 8 in 2024-03-11T22:14:16.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[30178]: 4.f scrub starts 2024-03-11T22:14:16.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[30178]: 4.f scrub ok 2024-03-11T22:14:16.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[30178]: 6.16 scrub starts 2024-03-11T22:14:16.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[30178]: 6.16 scrub ok 2024-03-11T22:14:16.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:14:16.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:14:16.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:16.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[30178]: 5.13 scrub starts 2024-03-11T22:14:16.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[30178]: 5.13 scrub ok 2024-03-11T22:14:16.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:16 smithi045 ceph-mon[30178]: osdmap e160: 8 total, 8 up, 8 in 2024-03-11T22:14:16.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:16 smithi144 ceph-mon[33984]: 4.f scrub starts 2024-03-11T22:14:16.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:16 smithi144 ceph-mon[33984]: 4.f scrub ok 2024-03-11T22:14:16.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:16 smithi144 ceph-mon[33984]: 6.16 scrub starts 2024-03-11T22:14:16.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:16 smithi144 ceph-mon[33984]: 6.16 scrub ok 2024-03-11T22:14:16.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:14:16.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:14:16.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:16.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:16 smithi144 ceph-mon[33984]: 5.13 scrub starts 2024-03-11T22:14:16.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:16 smithi144 ceph-mon[33984]: 5.13 scrub ok 2024-03-11T22:14:16.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:16 smithi144 ceph-mon[33984]: osdmap e160: 8 total, 8 up, 8 in 2024-03-11T22:14:16.863 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:16.889 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:16.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:16.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:17.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:17.041 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:17.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:17.142 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:17.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:17.186 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:17.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:17.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:17.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:17.328 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:17.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:17.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:17.472 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:17.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:17.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:17.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:17.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[30178]: 7.8 scrub starts 2024-03-11T22:14:17.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[30178]: pgmap v228: 158 pgs: 158 active+clean; 200 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1.39k op/s 2024-03-11T22:14:17.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[30178]: 7.8 scrub ok 2024-03-11T22:14:17.560 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[30178]: 2.2 scrub starts 2024-03-11T22:14:17.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[30178]: 2.2 scrub ok 2024-03-11T22:14:17.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[30178]: 7.c scrub starts 2024-03-11T22:14:17.561 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[30178]: 7.c scrub ok 2024-03-11T22:14:17.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:17.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[39715]: 7.8 scrub starts 2024-03-11T22:14:17.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[39715]: pgmap v228: 158 pgs: 158 active+clean; 200 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1.39k op/s 2024-03-11T22:14:17.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[39715]: 7.8 scrub ok 2024-03-11T22:14:17.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[39715]: 2.2 scrub starts 2024-03-11T22:14:17.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[39715]: 2.2 scrub ok 2024-03-11T22:14:17.561 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[39715]: 7.c scrub starts 2024-03-11T22:14:17.562 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:17 smithi045 ceph-mon[39715]: 7.c scrub ok 2024-03-11T22:14:17.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:17.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T22:14:17.630 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:17.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:17.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:14:17.676 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:17.733 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:17.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:17.830 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:17.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:17.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:17.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:17 smithi144 ceph-mon[33984]: 7.8 scrub starts 2024-03-11T22:14:17.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:17 smithi144 ceph-mon[33984]: pgmap v228: 158 pgs: 158 active+clean; 200 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1.39k op/s 2024-03-11T22:14:17.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:17 smithi144 ceph-mon[33984]: 7.8 scrub ok 2024-03-11T22:14:17.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:17 smithi144 ceph-mon[33984]: 2.2 scrub starts 2024-03-11T22:14:17.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:17 smithi144 ceph-mon[33984]: 2.2 scrub ok 2024-03-11T22:14:17.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:17 smithi144 ceph-mon[33984]: 7.c scrub starts 2024-03-11T22:14:17.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:17 smithi144 ceph-mon[33984]: 7.c scrub ok 2024-03-11T22:14:17.932 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:17.959 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:17.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:18.060 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:18.087 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:18.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:18.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:18.189 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:18.217 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:18.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:18.318 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:18.349 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:18.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:18.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:18.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:18.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:18.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:18.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:18.596 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:18.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:18.631 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:18.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:18.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:18.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:18 smithi045 ceph-mon[39715]: 5.5 scrub starts 2024-03-11T22:14:18.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:18 smithi045 ceph-mon[39715]: 5.5 scrub ok 2024-03-11T22:14:18.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:18 smithi045 ceph-mon[30178]: 5.5 scrub starts 2024-03-11T22:14:18.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:18 smithi045 ceph-mon[30178]: 5.5 scrub ok 2024-03-11T22:14:18.786 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:18.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:18.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:18.787 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:14:18.787 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:14:18.787 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-11T22:14:18.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:14:18.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:18 smithi144 ceph-mon[33984]: 5.5 scrub starts 2024-03-11T22:14:18.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:18 smithi144 ceph-mon[33984]: 5.5 scrub ok 2024-03-11T22:14:18.881 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Removed 118468 objects 2024-03-11T22:14:18.882 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Clean up completed and total clean up time :125.235 2024-03-11T22:14:18.888 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:18.952 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:18.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:18.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:19.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:19.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:19.079 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:19.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:19.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:19.206 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:19.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:19.307 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:19.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:19.332 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:19.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:19.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:19.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:19.458 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:19.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:19.559 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:19.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:19.584 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:19.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:19.685 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:19.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:19.710 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:19.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:19.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:19 smithi045 ceph-mon[39715]: pgmap v229: 158 pgs: 158 active+clean; 200 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-03-11T22:14:19.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:19 smithi045 ceph-mon[39715]: 5.2 scrub starts 2024-03-11T22:14:19.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:19 smithi045 ceph-mon[39715]: 5.2 scrub ok 2024-03-11T22:14:19.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:19 smithi045 ceph-mon[39715]: 7.1 scrub starts 2024-03-11T22:14:19.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:19 smithi045 ceph-mon[39715]: 7.1 scrub ok 2024-03-11T22:14:19.729 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:19 smithi045 ceph-mon[30178]: pgmap v229: 158 pgs: 158 active+clean; 200 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-03-11T22:14:19.729 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:19 smithi045 ceph-mon[30178]: 5.2 scrub starts 2024-03-11T22:14:19.729 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:19 smithi045 ceph-mon[30178]: 5.2 scrub ok 2024-03-11T22:14:19.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:19 smithi045 ceph-mon[30178]: 7.1 scrub starts 2024-03-11T22:14:19.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:19 smithi045 ceph-mon[30178]: 7.1 scrub ok 2024-03-11T22:14:19.752 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:19.764 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:19.813 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:19.837 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:19.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:19.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:19 smithi144 ceph-mon[33984]: pgmap v229: 158 pgs: 158 active+clean; 200 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-03-11T22:14:19.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:19 smithi144 ceph-mon[33984]: 5.2 scrub starts 2024-03-11T22:14:19.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:19 smithi144 ceph-mon[33984]: 5.2 scrub ok 2024-03-11T22:14:19.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:19 smithi144 ceph-mon[33984]: 7.1 scrub starts 2024-03-11T22:14:19.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:19 smithi144 ceph-mon[33984]: 7.1 scrub ok 2024-03-11T22:14:19.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:19.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:19.963 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:19.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:20.064 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:20.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:20.089 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:20.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:20.190 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:20.214 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:20.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:20.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:20.340 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:20.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:20.441 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:20.466 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:20.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:20.567 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:20.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:20.592 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:20.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:20.692 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:20.717 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:20.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:20.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:20 smithi045 ceph-mon[39715]: 5.c scrub starts 2024-03-11T22:14:20.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:20 smithi045 ceph-mon[39715]: 5.c scrub ok 2024-03-11T22:14:20.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:20 smithi045 ceph-mon[30178]: 5.c scrub starts 2024-03-11T22:14:20.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:20 smithi045 ceph-mon[30178]: 5.c scrub ok 2024-03-11T22:14:20.818 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:20.841 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:20.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:20.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:20 smithi144 ceph-mon[33984]: 5.c scrub starts 2024-03-11T22:14:20.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:20 smithi144 ceph-mon[33984]: 5.c scrub ok 2024-03-11T22:14:20.943 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:20.968 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:20.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:21.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:21.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:21.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:21.196 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:21.223 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:21.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:21.324 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:21.349 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:21.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:21.450 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:21.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:21.475 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:21.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:21.556 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:14:21.556 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:14:21.556 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:14:21.557 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:14:21.557 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:14:21.557 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:14:21.557 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:14:21.557 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:14:21.557 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:14:21.557 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:14:21.557 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:14:21.557 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:14:21.557 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:14:21.557 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:14:21.557 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:14:21.557 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:14:21.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:21.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:21.601 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:21.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:21.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:21.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:21.727 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:21.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:21.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[30178]: 6.14 scrub starts 2024-03-11T22:14:21.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[30178]: 6.14 scrub ok 2024-03-11T22:14:21.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[30178]: pgmap v230: 158 pgs: 158 active+clean; 173 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 550 op/s 2024-03-11T22:14:21.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[30178]: 5.1f scrub starts 2024-03-11T22:14:21.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[30178]: 5.1f scrub ok 2024-03-11T22:14:21.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[30178]: osdmap e161: 8 total, 8 up, 8 in 2024-03-11T22:14:21.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3794353954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:21.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[39715]: 6.14 scrub starts 2024-03-11T22:14:21.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[39715]: 6.14 scrub ok 2024-03-11T22:14:21.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[39715]: pgmap v230: 158 pgs: 158 active+clean; 173 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 550 op/s 2024-03-11T22:14:21.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[39715]: 5.1f scrub starts 2024-03-11T22:14:21.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[39715]: 5.1f scrub ok 2024-03-11T22:14:21.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[39715]: osdmap e161: 8 total, 8 up, 8 in 2024-03-11T22:14:21.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:21 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3794353954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:21.829 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:21.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:21 smithi144 ceph-mon[33984]: 6.14 scrub starts 2024-03-11T22:14:21.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:21 smithi144 ceph-mon[33984]: 6.14 scrub ok 2024-03-11T22:14:21.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:21 smithi144 ceph-mon[33984]: pgmap v230: 158 pgs: 158 active+clean; 173 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 550 op/s 2024-03-11T22:14:21.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:21 smithi144 ceph-mon[33984]: 5.1f scrub starts 2024-03-11T22:14:21.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:21 smithi144 ceph-mon[33984]: 5.1f scrub ok 2024-03-11T22:14:21.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:21 smithi144 ceph-mon[33984]: osdmap e161: 8 total, 8 up, 8 in 2024-03-11T22:14:21.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:21 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3794353954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:21.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:21.855 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:21.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:21.956 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:21.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:21.983 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:21.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:22.084 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:22.109 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:22.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:22.211 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:22.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:22.237 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:22.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:22.338 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:22.365 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:22.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:22.466 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:22.490 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:22.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:22.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:22.592 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:22.622 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:22.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:22.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:22.750 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:22.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:22.779 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:14:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:14:22.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[30178]: 7.b scrub starts 2024-03-11T22:14:22.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[30178]: 7.b scrub ok 2024-03-11T22:14:22.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[30178]: 5.6 scrub starts 2024-03-11T22:14:22.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[30178]: 5.6 scrub ok 2024-03-11T22:14:22.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[30178]: 4.6 scrub starts 2024-03-11T22:14:22.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[30178]: 4.6 scrub ok 2024-03-11T22:14:22.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[30178]: 3.15 scrub starts 2024-03-11T22:14:22.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[30178]: 3.15 scrub ok 2024-03-11T22:14:22.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[39715]: 7.b scrub starts 2024-03-11T22:14:22.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[39715]: 7.b scrub ok 2024-03-11T22:14:22.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[39715]: 5.6 scrub starts 2024-03-11T22:14:22.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[39715]: 5.6 scrub ok 2024-03-11T22:14:22.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[39715]: 4.6 scrub starts 2024-03-11T22:14:22.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[39715]: 4.6 scrub ok 2024-03-11T22:14:22.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[39715]: 3.15 scrub starts 2024-03-11T22:14:22.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:22 smithi045 ceph-mon[39715]: 3.15 scrub ok 2024-03-11T22:14:22.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:22 smithi144 ceph-mon[33984]: 7.b scrub starts 2024-03-11T22:14:22.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:22 smithi144 ceph-mon[33984]: 7.b scrub ok 2024-03-11T22:14:22.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:22 smithi144 ceph-mon[33984]: 5.6 scrub starts 2024-03-11T22:14:22.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:22 smithi144 ceph-mon[33984]: 5.6 scrub ok 2024-03-11T22:14:22.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:22 smithi144 ceph-mon[33984]: 4.6 scrub starts 2024-03-11T22:14:22.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:22 smithi144 ceph-mon[33984]: 4.6 scrub ok 2024-03-11T22:14:22.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:22 smithi144 ceph-mon[33984]: 3.15 scrub starts 2024-03-11T22:14:22.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:22 smithi144 ceph-mon[33984]: 3.15 scrub ok 2024-03-11T22:14:22.852 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:22.878 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:22.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:22.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:22.980 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:23.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:23.007 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:23.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:23.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:23.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:23.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:23.216 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:23.238 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:23.264 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:23.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:23.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:23.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:23.391 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:23.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:23.492 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:23.518 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:23.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:23.620 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:23.648 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:23.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:23.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:23.775 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:23.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:23.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:23 smithi045 ceph-mon[30178]: pgmap v232: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 964 op/s 2024-03-11T22:14:23.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:23 smithi045 ceph-mon[30178]: 6.19 scrub starts 2024-03-11T22:14:23.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:23 smithi045 ceph-mon[30178]: 6.19 scrub ok 2024-03-11T22:14:23.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:23 smithi045 ceph-mon[39715]: pgmap v232: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 964 op/s 2024-03-11T22:14:23.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:23 smithi045 ceph-mon[39715]: 6.19 scrub starts 2024-03-11T22:14:23.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:23 smithi045 ceph-mon[39715]: 6.19 scrub ok 2024-03-11T22:14:23.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:23 smithi144 ceph-mon[33984]: pgmap v232: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 964 op/s 2024-03-11T22:14:23.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:23 smithi144 ceph-mon[33984]: 6.19 scrub starts 2024-03-11T22:14:23.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:23 smithi144 ceph-mon[33984]: 6.19 scrub ok 2024-03-11T22:14:23.878 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:23.908 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:23.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:23.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:24.009 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:24.039 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:24.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:24.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:24.140 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:24.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:24.170 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:24.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:24.271 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:24.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:24.299 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:24.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:24.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:24.426 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:24.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:24.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:24.528 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:24.555 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:24.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:24.657 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:24.685 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:24.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:24.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:24 smithi045 ceph-mon[39715]: 5.1e scrub starts 2024-03-11T22:14:24.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:24 smithi045 ceph-mon[39715]: 5.1e scrub ok 2024-03-11T22:14:24.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:24 smithi045 ceph-mon[39715]: 5.4 scrub starts 2024-03-11T22:14:24.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:24 smithi045 ceph-mon[39715]: 5.4 scrub ok 2024-03-11T22:14:24.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:24 smithi045 ceph-mon[39715]: 6.13 scrub starts 2024-03-11T22:14:24.735 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:24 smithi045 ceph-mon[39715]: 6.13 scrub ok 2024-03-11T22:14:24.735 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:24 smithi045 ceph-mon[30178]: 5.1e scrub starts 2024-03-11T22:14:24.735 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:24 smithi045 ceph-mon[30178]: 5.1e scrub ok 2024-03-11T22:14:24.735 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:24 smithi045 ceph-mon[30178]: 5.4 scrub starts 2024-03-11T22:14:24.735 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:24 smithi045 ceph-mon[30178]: 5.4 scrub ok 2024-03-11T22:14:24.735 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:24 smithi045 ceph-mon[30178]: 6.13 scrub starts 2024-03-11T22:14:24.735 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:24 smithi045 ceph-mon[30178]: 6.13 scrub ok 2024-03-11T22:14:24.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:24.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:24.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:24.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:24.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:24 smithi144 ceph-mon[33984]: 5.1e scrub starts 2024-03-11T22:14:24.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:24 smithi144 ceph-mon[33984]: 5.1e scrub ok 2024-03-11T22:14:24.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:24 smithi144 ceph-mon[33984]: 5.4 scrub starts 2024-03-11T22:14:24.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:24 smithi144 ceph-mon[33984]: 5.4 scrub ok 2024-03-11T22:14:24.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:24 smithi144 ceph-mon[33984]: 6.13 scrub starts 2024-03-11T22:14:24.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:24 smithi144 ceph-mon[33984]: 6.13 scrub ok 2024-03-11T22:14:24.912 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:24.940 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:24.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:25.041 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:25.069 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:25.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:25.170 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:25.197 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:25.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:25.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:25.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:14:25.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:25.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:25.323 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:25.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:25.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:25.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:25.449 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:25.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:25.530 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:14:25.551 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:25.579 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:25.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:25.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:25.705 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:25.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:25.710 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[39715]: 3.19 scrub starts 2024-03-11T22:14:25.710 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[39715]: 3.19 scrub ok 2024-03-11T22:14:25.710 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[39715]: 5.8 scrub starts 2024-03-11T22:14:25.710 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[39715]: 5.8 scrub ok 2024-03-11T22:14:25.711 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[39715]: pgmap v233: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 399 op/s 2024-03-11T22:14:25.711 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2186064604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:14:25.711 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[39715]: 4.c scrub starts 2024-03-11T22:14:25.711 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[39715]: 4.c scrub ok 2024-03-11T22:14:25.711 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[39715]: 7.d scrub starts 2024-03-11T22:14:25.711 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[39715]: 7.d scrub ok 2024-03-11T22:14:25.711 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[30178]: 3.19 scrub starts 2024-03-11T22:14:25.711 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[30178]: 3.19 scrub ok 2024-03-11T22:14:25.712 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[30178]: 5.8 scrub starts 2024-03-11T22:14:25.712 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[30178]: 5.8 scrub ok 2024-03-11T22:14:25.712 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[30178]: pgmap v233: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 399 op/s 2024-03-11T22:14:25.712 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2186064604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:14:25.712 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[30178]: 4.c scrub starts 2024-03-11T22:14:25.712 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[30178]: 4.c scrub ok 2024-03-11T22:14:25.712 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[30178]: 7.d scrub starts 2024-03-11T22:14:25.712 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:25 smithi045 ceph-mon[30178]: 7.d scrub ok 2024-03-11T22:14:25.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:25.832 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:25.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:25.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:25 smithi144 ceph-mon[33984]: 3.19 scrub starts 2024-03-11T22:14:25.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:25 smithi144 ceph-mon[33984]: 3.19 scrub ok 2024-03-11T22:14:25.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:25 smithi144 ceph-mon[33984]: 5.8 scrub starts 2024-03-11T22:14:25.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:25 smithi144 ceph-mon[33984]: 5.8 scrub ok 2024-03-11T22:14:25.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:25 smithi144 ceph-mon[33984]: pgmap v233: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 399 op/s 2024-03-11T22:14:25.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:25 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2186064604' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:14:25.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:25 smithi144 ceph-mon[33984]: 4.c scrub starts 2024-03-11T22:14:25.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:25 smithi144 ceph-mon[33984]: 4.c scrub ok 2024-03-11T22:14:25.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:25 smithi144 ceph-mon[33984]: 7.d scrub starts 2024-03-11T22:14:25.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:25 smithi144 ceph-mon[33984]: 7.d scrub ok 2024-03-11T22:14:25.933 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:25.961 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:25.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:26.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:26.089 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:26.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:26.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:26.191 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:26.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:26.219 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:26.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:26.320 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:26.342 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:14:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:14:26.344 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:26.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:26.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:26.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:26.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:26.572 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:26.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:26.600 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:26.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:26.607 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:14:26.608 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:14:26.701 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:26.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:14:26.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:14:26.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2186064604' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:14:26.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[30178]: osdmap e162: 8 total, 8 up, 8 in 2024-03-11T22:14:26.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[30178]: 3.9 scrub starts 2024-03-11T22:14:26.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[30178]: 3.9 scrub ok 2024-03-11T22:14:26.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[30178]: 4.a scrub starts 2024-03-11T22:14:26.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[30178]: 4.a scrub ok 2024-03-11T22:14:26.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[30178]: 5.14 scrub starts 2024-03-11T22:14:26.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[30178]: 5.14 scrub ok 2024-03-11T22:14:26.726 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[30178]: osdmap e163: 8 total, 8 up, 8 in 2024-03-11T22:14:26.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:14:26.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:14:26.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2186064604' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:14:26.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[39715]: osdmap e162: 8 total, 8 up, 8 in 2024-03-11T22:14:26.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[39715]: 3.9 scrub starts 2024-03-11T22:14:26.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[39715]: 3.9 scrub ok 2024-03-11T22:14:26.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[39715]: 4.a scrub starts 2024-03-11T22:14:26.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[39715]: 4.a scrub ok 2024-03-11T22:14:26.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[39715]: 5.14 scrub starts 2024-03-11T22:14:26.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[39715]: 5.14 scrub ok 2024-03-11T22:14:26.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:26 smithi045 ceph-mon[39715]: osdmap e163: 8 total, 8 up, 8 in 2024-03-11T22:14:26.729 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:26.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:26.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:26.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:26.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:26 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:14:26.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:26 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:14:26.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:26 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2186064604' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:14:26.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:26 smithi144 ceph-mon[33984]: osdmap e162: 8 total, 8 up, 8 in 2024-03-11T22:14:26.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:26 smithi144 ceph-mon[33984]: 3.9 scrub starts 2024-03-11T22:14:26.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:26 smithi144 ceph-mon[33984]: 3.9 scrub ok 2024-03-11T22:14:26.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:26 smithi144 ceph-mon[33984]: 4.a scrub starts 2024-03-11T22:14:26.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:26 smithi144 ceph-mon[33984]: 4.a scrub ok 2024-03-11T22:14:26.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:26 smithi144 ceph-mon[33984]: 5.14 scrub starts 2024-03-11T22:14:26.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:26 smithi144 ceph-mon[33984]: 5.14 scrub ok 2024-03-11T22:14:26.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:26 smithi144 ceph-mon[33984]: osdmap e163: 8 total, 8 up, 8 in 2024-03-11T22:14:26.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:26.859 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:26.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:26.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:26.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:26.986 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:26.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:27.087 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:27.116 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:27.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:27.217 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:27.244 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:27.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:27.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:27.372 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:27.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:27.474 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:27.502 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:27.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:27.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:27.628 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:27.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:27.639 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:27.639 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[39715]: 5.11 scrub starts 2024-03-11T22:14:27.639 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[39715]: 5.11 scrub ok 2024-03-11T22:14:27.639 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[39715]: 4.16 scrub starts 2024-03-11T22:14:27.639 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[39715]: 4.16 scrub ok 2024-03-11T22:14:27.639 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[39715]: pgmap v236: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 461 op/s 2024-03-11T22:14:27.639 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[39715]: 5.f scrub starts 2024-03-11T22:14:27.639 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[39715]: 5.f scrub ok 2024-03-11T22:14:27.639 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:27.640 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[30178]: 5.11 scrub starts 2024-03-11T22:14:27.640 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[30178]: 5.11 scrub ok 2024-03-11T22:14:27.640 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[30178]: 4.16 scrub starts 2024-03-11T22:14:27.640 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[30178]: 4.16 scrub ok 2024-03-11T22:14:27.640 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[30178]: pgmap v236: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 461 op/s 2024-03-11T22:14:27.640 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[30178]: 5.f scrub starts 2024-03-11T22:14:27.640 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:27 smithi045 ceph-mon[30178]: 5.f scrub ok 2024-03-11T22:14:27.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:27.753 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:27.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:27.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:27.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:27 smithi144 ceph-mon[33984]: 5.11 scrub starts 2024-03-11T22:14:27.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:27 smithi144 ceph-mon[33984]: 5.11 scrub ok 2024-03-11T22:14:27.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:27 smithi144 ceph-mon[33984]: 4.16 scrub starts 2024-03-11T22:14:27.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:27 smithi144 ceph-mon[33984]: 4.16 scrub ok 2024-03-11T22:14:27.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:27 smithi144 ceph-mon[33984]: pgmap v236: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 461 op/s 2024-03-11T22:14:27.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:27 smithi144 ceph-mon[33984]: 5.f scrub starts 2024-03-11T22:14:27.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:27 smithi144 ceph-mon[33984]: 5.f scrub ok 2024-03-11T22:14:27.854 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:27.871 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:14:27.874 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:14:27.882 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:27.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:27.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:28.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:28.010 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:28.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:28.111 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:28.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:28.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:28.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:28.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T22:14:28.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:28.264 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:28.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:28.365 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:28.391 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:28.393 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:28.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:28.495 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:28.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:28.523 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:28.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:28.624 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:28.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:28.648 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:28.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:28.749 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:28.777 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:28.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:28.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[39715]: 5.1d scrub starts 2024-03-11T22:14:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[39715]: 5.1d scrub ok 2024-03-11T22:14:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[39715]: 5.3 scrub starts 2024-03-11T22:14:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[39715]: 5.3 scrub ok 2024-03-11T22:14:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[39715]: 7.0 scrub starts 2024-03-11T22:14:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[39715]: 7.0 scrub ok 2024-03-11T22:14:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[39715]: 4.1d scrub starts 2024-03-11T22:14:28.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[39715]: 4.1d scrub ok 2024-03-11T22:14:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[30178]: 5.1d scrub starts 2024-03-11T22:14:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[30178]: 5.1d scrub ok 2024-03-11T22:14:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[30178]: 5.3 scrub starts 2024-03-11T22:14:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[30178]: 5.3 scrub ok 2024-03-11T22:14:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[30178]: 7.0 scrub starts 2024-03-11T22:14:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[30178]: 7.0 scrub ok 2024-03-11T22:14:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[30178]: 4.1d scrub starts 2024-03-11T22:14:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:28 smithi045 ceph-mon[30178]: 4.1d scrub ok 2024-03-11T22:14:28.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:28 smithi144 ceph-mon[33984]: 5.1d scrub starts 2024-03-11T22:14:28.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:28 smithi144 ceph-mon[33984]: 5.1d scrub ok 2024-03-11T22:14:28.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:28 smithi144 ceph-mon[33984]: 5.3 scrub starts 2024-03-11T22:14:28.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:28 smithi144 ceph-mon[33984]: 5.3 scrub ok 2024-03-11T22:14:28.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:28 smithi144 ceph-mon[33984]: 7.0 scrub starts 2024-03-11T22:14:28.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:28 smithi144 ceph-mon[33984]: 7.0 scrub ok 2024-03-11T22:14:28.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:28 smithi144 ceph-mon[33984]: 4.1d scrub starts 2024-03-11T22:14:28.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:28 smithi144 ceph-mon[33984]: 4.1d scrub ok 2024-03-11T22:14:28.852 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-03-11T22:14:26.151400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1032,"num_read_kb":784,"num_write":237374,"num_write_kb":7581998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86406,"ondisk_log_size":86406,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1002600,"kb_used_data":9052,"kb_used_omap":12,"kb_used_meta":993459,"kb_avail":748958616,"statfs":{"total":767960285184,"available":766933622784,"internally_reserved":0,"allocated":9269248,"data_stored":6353380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":1017302579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-180813848,"num_objects":-2760,"num_object_clones":0,"num_object_copies":-8280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2760,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2760,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001225"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":195,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137787+0000","last_change":"2024-03-11T22:11:30.350663+0000","last_active":"2024-03-11T22:14:21.137787+0000","last_peered":"2024-03-11T22:14:21.137787+0000","last_clean":"2024-03-11T22:14:21.137787+0000","last_became_active":"2024-03-11T22:08:39.074695+0000","last_became_peered":"2024-03-11T22:08:39.074695+0000","last_unstale":"2024-03-11T22:14:21.137787+0000","last_undegraded":"2024-03-11T22:14:21.137787+0000","last_fullsized":"2024-03-11T22:14:21.137787+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:30.350540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.350540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":207,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336271+0000","last_change":"2024-03-11T22:12:45.929428+0000","last_active":"2024-03-11T22:14:21.336271+0000","last_peered":"2024-03-11T22:14:21.336271+0000","last_clean":"2024-03-11T22:14:21.336271+0000","last_became_active":"2024-03-11T22:08:37.073775+0000","last_became_peered":"2024-03-11T22:08:37.073775+0000","last_unstale":"2024-03-11T22:14:21.336271+0000","last_undegraded":"2024-03-11T22:14:21.336271+0000","last_fullsized":"2024-03-11T22:14:21.336271+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.928689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.928689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140085+0000","last_change":"2024-03-11T22:11:45.340779+0000","last_active":"2024-03-11T22:14:21.140085+0000","last_peered":"2024-03-11T22:14:21.140085+0000","last_clean":"2024-03-11T22:14:21.140085+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:14:21.140085+0000","last_undegraded":"2024-03-11T22:14:21.140085+0000","last_fullsized":"2024-03-11T22:14:21.140085+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:45.340657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:45.340657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":227,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137625+0000","last_change":"2024-03-11T22:14:15.059747+0000","last_active":"2024-03-11T22:14:21.137625+0000","last_peered":"2024-03-11T22:14:21.137625+0000","last_clean":"2024-03-11T22:14:21.137625+0000","last_became_active":"2024-03-11T22:08:35.053179+0000","last_became_peered":"2024-03-11T22:08:35.053179+0000","last_unstale":"2024-03-11T22:14:21.137625+0000","last_undegraded":"2024-03-11T22:14:21.137625+0000","last_fullsized":"2024-03-11T22:14:21.137625+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:14:15.059599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:15.059599+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:28.271172+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138323+0000","last_change":"2024-03-11T22:12:50.533192+0000","last_active":"2024-03-11T22:14:21.138323+0000","last_peered":"2024-03-11T22:14:21.138323+0000","last_clean":"2024-03-11T22:14:21.138323+0000","last_became_active":"2024-03-11T22:08:39.079045+0000","last_became_peered":"2024-03-11T22:08:39.079045+0000","last_unstale":"2024-03-11T22:14:21.138323+0000","last_undegraded":"2024-03-11T22:14:21.138323+0000","last_fullsized":"2024-03-11T22:14:21.138323+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.533067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.533067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":207,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137742+0000","last_change":"2024-03-11T22:11:43.150997+0000","last_active":"2024-03-11T22:14:21.137742+0000","last_peered":"2024-03-11T22:14:21.137742+0000","last_clean":"2024-03-11T22:14:21.137742+0000","last_became_active":"2024-03-11T22:08:37.081861+0000","last_became_peered":"2024-03-11T22:08:37.081861+0000","last_unstale":"2024-03-11T22:14:21.137742+0000","last_undegraded":"2024-03-11T22:14:21.137742+0000","last_fullsized":"2024-03-11T22:14:21.137742+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:43.150872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:43.150872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":229,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138658+0000","last_change":"2024-03-11T22:13:19.463087+0000","last_active":"2024-03-11T22:14:21.138658+0000","last_peered":"2024-03-11T22:14:21.138658+0000","last_clean":"2024-03-11T22:14:21.138658+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:14:21.138658+0000","last_undegraded":"2024-03-11T22:14:21.138658+0000","last_fullsized":"2024-03-11T22:14:21.138658+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:13:19.462955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.462955+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:30.805293+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":236,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.139761+0000","last_change":"2024-03-11T22:12:49.510144+0000","last_active":"2024-03-11T22:14:21.139761+0000","last_peered":"2024-03-11T22:14:21.139761+0000","last_clean":"2024-03-11T22:14:21.139761+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:14:21.139761+0000","last_undegraded":"2024-03-11T22:14:21.139761+0000","last_fullsized":"2024-03-11T22:14:21.139761+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:12:49.509860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.509860+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138663+0000","last_change":"2024-03-11T22:13:25.042445+0000","last_active":"2024-03-11T22:14:21.138663+0000","last_peered":"2024-03-11T22:14:21.138663+0000","last_clean":"2024-03-11T22:14:21.138663+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:14:21.138663+0000","last_undegraded":"2024-03-11T22:14:21.138663+0000","last_fullsized":"2024-03-11T22:14:21.138663+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:25.042307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:25.042307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:43.208699+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":207,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336488+0000","last_change":"2024-03-11T22:12:19.337740+0000","last_active":"2024-03-11T22:14:21.336488+0000","last_peered":"2024-03-11T22:14:21.336488+0000","last_clean":"2024-03-11T22:14:21.336488+0000","last_became_active":"2024-03-11T22:08:37.081996+0000","last_became_peered":"2024-03-11T22:08:37.081996+0000","last_unstale":"2024-03-11T22:14:21.336488+0000","last_undegraded":"2024-03-11T22:14:21.336488+0000","last_fullsized":"2024-03-11T22:14:21.336488+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:19.337662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:12:19.337662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":223,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335950+0000","last_change":"2024-03-11T22:12:47.432839+0000","last_active":"2024-03-11T22:14:21.335950+0000","last_peered":"2024-03-11T22:14:21.335950+0000","last_clean":"2024-03-11T22:14:21.335950+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:14:21.335950+0000","last_undegraded":"2024-03-11T22:14:21.335950+0000","last_fullsized":"2024-03-11T22:14:21.335950+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:47.432769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:47.432769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":227,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137977+0000","last_change":"2024-03-11T22:11:45.169864+0000","last_active":"2024-03-11T22:14:21.137977+0000","last_peered":"2024-03-11T22:14:21.137977+0000","last_clean":"2024-03-11T22:14:21.137977+0000","last_became_active":"2024-03-11T22:08:35.060802+0000","last_became_peered":"2024-03-11T22:08:35.060802+0000","last_unstale":"2024-03-11T22:14:21.137977+0000","last_undegraded":"2024-03-11T22:14:21.137977+0000","last_fullsized":"2024-03-11T22:14:21.137977+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:11:45.169706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:45.169706+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":211,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:22.631535+0000","last_change":"2024-03-11T22:14:22.631560+0000","last_active":"2024-03-11T22:14:22.631535+0000","last_peered":"2024-03-11T22:14:22.631535+0000","last_clean":"2024-03-11T22:14:22.631535+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:14:22.631535+0000","last_undegraded":"2024-03-11T22:14:22.631535+0000","last_fullsized":"2024-03-11T22:14:22.631535+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:22.631456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:22.631456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:48.639261+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":145,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138096+0000","last_change":"2024-03-11T22:13:36.039706+0000","last_active":"2024-03-11T22:14:21.138096+0000","last_peered":"2024-03-11T22:14:21.138096+0000","last_clean":"2024-03-11T22:14:21.138096+0000","last_became_active":"2024-03-11T22:10:35.410624+0000","last_became_peered":"2024-03-11T22:10:35.410624+0000","last_unstale":"2024-03-11T22:14:21.138096+0000","last_undegraded":"2024-03-11T22:14:21.138096+0000","last_fullsized":"2024-03-11T22:14:21.138096+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:36.039591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.270166+0000","last_clean_scrub_stamp":"2024-03-11T22:13:36.039591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:46.694507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335867+0000","last_change":"2024-03-11T22:12:24.388572+0000","last_active":"2024-03-11T22:14:21.335867+0000","last_peered":"2024-03-11T22:14:21.335867+0000","last_clean":"2024-03-11T22:14:21.335867+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:14:21.335867+0000","last_undegraded":"2024-03-11T22:14:21.335867+0000","last_fullsized":"2024-03-11T22:14:21.335867+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:24.388497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.388497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"95'7","reported_seq":223,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138163+0000","last_change":"2024-03-11T22:13:32.041334+0000","last_active":"2024-03-11T22:14:21.138163+0000","last_peered":"2024-03-11T22:14:21.138163+0000","last_clean":"2024-03-11T22:14:21.138163+0000","last_became_active":"2024-03-11T22:08:35.060777+0000","last_became_peered":"2024-03-11T22:08:35.060777+0000","last_unstale":"2024-03-11T22:14:21.138163+0000","last_undegraded":"2024-03-11T22:14:21.138163+0000","last_fullsized":"2024-03-11T22:14:21.138163+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'7","last_scrub_stamp":"2024-03-11T22:13:32.041219+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:13:32.041219+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:57.879414+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138580+0000","last_change":"2024-03-11T22:12:32.044430+0000","last_active":"2024-03-11T22:14:21.138580+0000","last_peered":"2024-03-11T22:14:21.138580+0000","last_clean":"2024-03-11T22:14:21.138580+0000","last_became_active":"2024-03-11T22:08:39.075609+0000","last_became_peered":"2024-03-11T22:08:39.075609+0000","last_unstale":"2024-03-11T22:14:21.138580+0000","last_undegraded":"2024-03-11T22:14:21.138580+0000","last_fullsized":"2024-03-11T22:14:21.138580+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:32.044320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:32.044320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":207,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336091+0000","last_change":"2024-03-11T22:12:30.962293+0000","last_active":"2024-03-11T22:14:21.336091+0000","last_peered":"2024-03-11T22:14:21.336091+0000","last_clean":"2024-03-11T22:14:21.336091+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:14:21.336091+0000","last_undegraded":"2024-03-11T22:14:21.336091+0000","last_fullsized":"2024-03-11T22:14:21.336091+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:30.962171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:30.962171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.139913+0000","last_change":"2024-03-11T22:12:22.649189+0000","last_active":"2024-03-11T22:14:21.139913+0000","last_peered":"2024-03-11T22:14:21.139913+0000","last_clean":"2024-03-11T22:14:21.139913+0000","last_became_active":"2024-03-11T22:08:33.035624+0000","last_became_peered":"2024-03-11T22:08:33.035624+0000","last_unstale":"2024-03-11T22:14:21.139913+0000","last_undegraded":"2024-03-11T22:14:21.139913+0000","last_fullsized":"2024-03-11T22:14:21.139913+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:22.649073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:22.649073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":254,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.529244+0000","last_change":"2024-03-11T22:14:12.989218+0000","last_active":"2024-03-11T22:14:25.529244+0000","last_peered":"2024-03-11T22:14:25.529244+0000","last_clean":"2024-03-11T22:14:25.529244+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:14:25.529244+0000","last_undegraded":"2024-03-11T22:14:25.529244+0000","last_fullsized":"2024-03-11T22:14:25.529244+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:14:12.989078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:12.989078+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:19.385123+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.139740+0000","last_change":"2024-03-11T22:12:44.464428+0000","last_active":"2024-03-11T22:14:21.139740+0000","last_peered":"2024-03-11T22:14:21.139740+0000","last_clean":"2024-03-11T22:14:21.139740+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:14:21.139740+0000","last_undegraded":"2024-03-11T22:14:21.139740+0000","last_fullsized":"2024-03-11T22:14:21.139740+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.464308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.464308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":199,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138405+0000","last_change":"2024-03-11T22:11:27.391809+0000","last_active":"2024-03-11T22:14:21.138405+0000","last_peered":"2024-03-11T22:14:21.138405+0000","last_clean":"2024-03-11T22:14:21.138405+0000","last_became_active":"2024-03-11T22:08:37.080014+0000","last_became_peered":"2024-03-11T22:08:37.080014+0000","last_unstale":"2024-03-11T22:14:21.138405+0000","last_undegraded":"2024-03-11T22:14:21.138405+0000","last_fullsized":"2024-03-11T22:14:21.138405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:27.391727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:27.391727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137980+0000","last_change":"2024-03-11T22:13:22.058907+0000","last_active":"2024-03-11T22:14:21.137980+0000","last_peered":"2024-03-11T22:14:21.137980+0000","last_clean":"2024-03-11T22:14:21.137980+0000","last_became_active":"2024-03-11T22:08:33.299189+0000","last_became_peered":"2024-03-11T22:08:33.299189+0000","last_unstale":"2024-03-11T22:14:21.137980+0000","last_undegraded":"2024-03-11T22:14:21.137980+0000","last_fullsized":"2024-03-11T22:14:21.137980+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:22.058795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:22.058795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:25.056228+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":234,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137882+0000","last_change":"2024-03-11T22:12:28.292078+0000","last_active":"2024-03-11T22:14:21.137882+0000","last_peered":"2024-03-11T22:14:21.137882+0000","last_clean":"2024-03-11T22:14:21.137882+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:14:21.137882+0000","last_undegraded":"2024-03-11T22:14:21.137882+0000","last_fullsized":"2024-03-11T22:14:21.137882+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:12:28.291919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:28.291919+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":164,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335753+0000","last_change":"2024-03-11T22:13:21.532369+0000","last_active":"2024-03-11T22:14:21.335753+0000","last_peered":"2024-03-11T22:14:21.335753+0000","last_clean":"2024-03-11T22:14:21.335753+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:14:21.335753+0000","last_undegraded":"2024-03-11T22:14:21.335753+0000","last_fullsized":"2024-03-11T22:14:21.335753+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:13:21.532280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:21.532280+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:45.199055+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":305,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336594+0000","last_change":"2024-03-11T22:14:21.037744+0000","last_active":"2024-03-11T22:14:21.336594+0000","last_peered":"2024-03-11T22:14:21.336594+0000","last_clean":"2024-03-11T22:14:21.336594+0000","last_became_active":"2024-03-11T22:08:37.078588+0000","last_became_peered":"2024-03-11T22:08:37.078588+0000","last_unstale":"2024-03-11T22:14:21.336594+0000","last_undegraded":"2024-03-11T22:14:21.336594+0000","last_fullsized":"2024-03-11T22:14:21.336594+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:14:21.037625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.037625+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:38.139047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":224,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:23.995881+0000","last_change":"2024-03-11T22:14:23.995912+0000","last_active":"2024-03-11T22:14:23.995881+0000","last_peered":"2024-03-11T22:14:23.995881+0000","last_clean":"2024-03-11T22:14:23.995881+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:14:23.995881+0000","last_undegraded":"2024-03-11T22:14:23.995881+0000","last_fullsized":"2024-03-11T22:14:23.995881+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:14:23.995781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.995781+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:33.646778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":226,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140548+0000","last_change":"2024-03-11T22:13:16.935454+0000","last_active":"2024-03-11T22:14:21.140548+0000","last_peered":"2024-03-11T22:14:21.140548+0000","last_clean":"2024-03-11T22:14:21.140548+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:14:21.140548+0000","last_undegraded":"2024-03-11T22:14:21.140548+0000","last_fullsized":"2024-03-11T22:14:21.140548+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:13:16.935159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:16.935159+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:36.018341+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":211,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336262+0000","last_change":"2024-03-11T22:13:30.794816+0000","last_active":"2024-03-11T22:14:21.336262+0000","last_peered":"2024-03-11T22:14:21.336262+0000","last_clean":"2024-03-11T22:14:21.336262+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:14:21.336262+0000","last_undegraded":"2024-03-11T22:14:21.336262+0000","last_fullsized":"2024-03-11T22:14:21.336262+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:30.794739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:30.794739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:42.625116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:23.034921+0000","last_change":"2024-03-11T22:14:23.034943+0000","last_active":"2024-03-11T22:14:23.034921+0000","last_peered":"2024-03-11T22:14:23.034921+0000","last_clean":"2024-03-11T22:14:23.034921+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:14:23.034921+0000","last_undegraded":"2024-03-11T22:14:23.034921+0000","last_fullsized":"2024-03-11T22:14:23.034921+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.034850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.034850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:37.593852+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":223,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137678+0000","last_change":"2024-03-11T22:14:07.822714+0000","last_active":"2024-03-11T22:14:21.137678+0000","last_peered":"2024-03-11T22:14:21.137678+0000","last_clean":"2024-03-11T22:14:21.137678+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:14:21.137678+0000","last_undegraded":"2024-03-11T22:14:21.137678+0000","last_fullsized":"2024-03-11T22:14:21.137678+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:07.822036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:07.822036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:12.568652+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":238,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336455+0000","last_change":"2024-03-11T22:13:17.923943+0000","last_active":"2024-03-11T22:14:21.336455+0000","last_peered":"2024-03-11T22:14:21.336455+0000","last_clean":"2024-03-11T22:14:21.336455+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:14:21.336455+0000","last_undegraded":"2024-03-11T22:14:21.336455+0000","last_fullsized":"2024-03-11T22:14:21.336455+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:13:17.923797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:17.923797+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:38.893975+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":155,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.529411+0000","last_change":"2024-03-11T22:13:27.763590+0000","last_active":"2024-03-11T22:14:25.529411+0000","last_peered":"2024-03-11T22:14:25.529411+0000","last_clean":"2024-03-11T22:14:25.529411+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:14:25.529411+0000","last_undegraded":"2024-03-11T22:14:25.529411+0000","last_fullsized":"2024-03-11T22:14:25.529411+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:27.763466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:27.763466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:48.983317+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":19,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335732+0000","last_change":"2024-03-11T22:14:08.443136+0000","last_active":"2024-03-11T22:14:21.335732+0000","last_peered":"2024-03-11T22:14:21.335732+0000","last_clean":"2024-03-11T22:14:21.335732+0000","last_became_active":"2024-03-11T22:14:08.442596+0000","last_became_peered":"2024-03-11T22:14:08.442596+0000","last_unstale":"2024-03-11T22:14:21.335732+0000","last_undegraded":"2024-03-11T22:14:21.335732+0000","last_fullsized":"2024-03-11T22:14:21.335732+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.941275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.941275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140203+0000","last_change":"2024-03-11T22:13:32.033557+0000","last_active":"2024-03-11T22:14:21.140203+0000","last_peered":"2024-03-11T22:14:21.140203+0000","last_clean":"2024-03-11T22:14:21.140203+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:14:21.140203+0000","last_undegraded":"2024-03-11T22:14:21.140203+0000","last_fullsized":"2024-03-11T22:14:21.140203+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:32.033444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:32.033444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:32.453949+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":237,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137703+0000","last_change":"2024-03-11T22:13:31.491151+0000","last_active":"2024-03-11T22:14:21.137703+0000","last_peered":"2024-03-11T22:14:21.137703+0000","last_clean":"2024-03-11T22:14:21.137703+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:14:21.137703+0000","last_undegraded":"2024-03-11T22:14:21.137703+0000","last_fullsized":"2024-03-11T22:14:21.137703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-11T22:13:31.490878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:31.490878+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:50.850169+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":211,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:23.930169+0000","last_change":"2024-03-11T22:14:23.930193+0000","last_active":"2024-03-11T22:14:23.930169+0000","last_peered":"2024-03-11T22:14:23.930169+0000","last_clean":"2024-03-11T22:14:23.930169+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:14:23.930169+0000","last_undegraded":"2024-03-11T22:14:23.930169+0000","last_fullsized":"2024-03-11T22:14:23.930169+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.930091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.930091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:52.770218+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":145,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137563+0000","last_change":"2024-03-11T22:11:42.175942+0000","last_active":"2024-03-11T22:14:21.137563+0000","last_peered":"2024-03-11T22:14:21.137563+0000","last_clean":"2024-03-11T22:14:21.137563+0000","last_became_active":"2024-03-11T22:10:35.415813+0000","last_became_peered":"2024-03-11T22:10:35.415813+0000","last_unstale":"2024-03-11T22:14:21.137563+0000","last_undegraded":"2024-03-11T22:14:21.137563+0000","last_fullsized":"2024-03-11T22:14:21.137563+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:11:42.175852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":223,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336945+0000","last_change":"2024-03-11T22:13:29.550985+0000","last_active":"2024-03-11T22:14:21.336945+0000","last_peered":"2024-03-11T22:14:21.336945+0000","last_clean":"2024-03-11T22:14:21.336945+0000","last_became_active":"2024-03-11T22:08:33.039349+0000","last_became_peered":"2024-03-11T22:08:33.039349+0000","last_unstale":"2024-03-11T22:14:21.336945+0000","last_undegraded":"2024-03-11T22:14:21.336945+0000","last_fullsized":"2024-03-11T22:14:21.336945+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:29.550889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:29.550889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:57.436176+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":230,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.139719+0000","last_change":"2024-03-11T22:13:18.486183+0000","last_active":"2024-03-11T22:14:21.139719+0000","last_peered":"2024-03-11T22:14:21.139719+0000","last_clean":"2024-03-11T22:14:21.139719+0000","last_became_active":"2024-03-11T22:08:35.085928+0000","last_became_peered":"2024-03-11T22:08:35.085928+0000","last_unstale":"2024-03-11T22:14:21.139719+0000","last_undegraded":"2024-03-11T22:14:21.139719+0000","last_fullsized":"2024-03-11T22:14:21.139719+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:13:18.486039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:18.486039+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:24.736759+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138315+0000","last_change":"2024-03-11T22:12:46.829082+0000","last_active":"2024-03-11T22:14:21.138315+0000","last_peered":"2024-03-11T22:14:21.138315+0000","last_clean":"2024-03-11T22:14:21.138315+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:14:21.138315+0000","last_undegraded":"2024-03-11T22:14:21.138315+0000","last_fullsized":"2024-03-11T22:14:21.138315+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.828942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.828942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137937+0000","last_change":"2024-03-11T22:14:15.858384+0000","last_active":"2024-03-11T22:14:21.137937+0000","last_peered":"2024-03-11T22:14:21.137937+0000","last_clean":"2024-03-11T22:14:21.137937+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:14:21.137937+0000","last_undegraded":"2024-03-11T22:14:21.137937+0000","last_fullsized":"2024-03-11T22:14:21.137937+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:15.857082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:14:15.857082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:29.241026+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":154,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.857039+0000","last_change":"2024-03-11T22:14:21.857065+0000","last_active":"2024-03-11T22:14:21.857039+0000","last_peered":"2024-03-11T22:14:21.857039+0000","last_clean":"2024-03-11T22:14:21.857039+0000","last_became_active":"2024-03-11T22:10:35.414121+0000","last_became_peered":"2024-03-11T22:10:35.414121+0000","last_unstale":"2024-03-11T22:14:21.857039+0000","last_undegraded":"2024-03-11T22:14:21.857039+0000","last_fullsized":"2024-03-11T22:14:21.857039+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:21.856963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.856963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:24.600199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":235,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336492+0000","last_change":"2024-03-11T22:13:31.796744+0000","last_active":"2024-03-11T22:14:21.336492+0000","last_peered":"2024-03-11T22:14:21.336492+0000","last_clean":"2024-03-11T22:14:21.336492+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:14:21.336492+0000","last_undegraded":"2024-03-11T22:14:21.336492+0000","last_fullsized":"2024-03-11T22:14:21.336492+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:13:31.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:31.796611+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:49.930017+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.139105+0000","last_change":"2024-03-11T22:12:46.492608+0000","last_active":"2024-03-11T22:14:21.139105+0000","last_peered":"2024-03-11T22:14:21.139105+0000","last_clean":"2024-03-11T22:14:21.139105+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:14:21.139105+0000","last_undegraded":"2024-03-11T22:14:21.139105+0000","last_fullsized":"2024-03-11T22:14:21.139105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.492494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.492494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336106+0000","last_change":"2024-03-11T22:14:07.844362+0000","last_active":"2024-03-11T22:14:21.336106+0000","last_peered":"2024-03-11T22:14:21.336106+0000","last_clean":"2024-03-11T22:14:21.336106+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:14:21.336106+0000","last_undegraded":"2024-03-11T22:14:21.336106+0000","last_fullsized":"2024-03-11T22:14:21.336106+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:07.844240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:07.844240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:27.085453+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":221,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138205+0000","last_change":"2024-03-11T22:12:18.154605+0000","last_active":"2024-03-11T22:14:21.138205+0000","last_peered":"2024-03-11T22:14:21.138205+0000","last_clean":"2024-03-11T22:14:21.138205+0000","last_became_active":"2024-03-11T22:08:33.036083+0000","last_became_peered":"2024-03-11T22:08:33.036083+0000","last_unstale":"2024-03-11T22:14:21.138205+0000","last_undegraded":"2024-03-11T22:14:21.138205+0000","last_fullsized":"2024-03-11T22:14:21.138205+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:18.154515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:18.154515+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":222,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138261+0000","last_change":"2024-03-11T22:11:19.294807+0000","last_active":"2024-03-11T22:14:21.138261+0000","last_peered":"2024-03-11T22:14:21.138261+0000","last_clean":"2024-03-11T22:14:21.138261+0000","last_became_active":"2024-03-11T22:08:35.054726+0000","last_became_peered":"2024-03-11T22:08:35.054726+0000","last_unstale":"2024-03-11T22:14:21.138261+0000","last_undegraded":"2024-03-11T22:14:21.138261+0000","last_fullsized":"2024-03-11T22:14:21.138261+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:11:19.294671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:19.294671+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140326+0000","last_change":"2024-03-11T22:14:14.969800+0000","last_active":"2024-03-11T22:14:21.140326+0000","last_peered":"2024-03-11T22:14:21.140326+0000","last_clean":"2024-03-11T22:14:21.140326+0000","last_became_active":"2024-03-11T22:08:39.082938+0000","last_became_peered":"2024-03-11T22:08:39.082938+0000","last_unstale":"2024-03-11T22:14:21.140326+0000","last_undegraded":"2024-03-11T22:14:21.140326+0000","last_fullsized":"2024-03-11T22:14:21.140326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.969572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.969572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:44.759439+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":294,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336813+0000","last_change":"2024-03-11T22:13:28.520411+0000","last_active":"2024-03-11T22:14:21.336813+0000","last_peered":"2024-03-11T22:14:21.336813+0000","last_clean":"2024-03-11T22:14:21.336813+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:14:21.336813+0000","last_undegraded":"2024-03-11T22:14:21.336813+0000","last_fullsized":"2024-03-11T22:14:21.336813+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:13:28.520283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:13:28.520283+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:34.669903+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":146,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138235+0000","last_change":"2024-03-11T22:11:39.174355+0000","last_active":"2024-03-11T22:14:21.138235+0000","last_peered":"2024-03-11T22:14:21.138235+0000","last_clean":"2024-03-11T22:14:21.138235+0000","last_became_active":"2024-03-11T22:10:35.407845+0000","last_became_peered":"2024-03-11T22:10:35.407845+0000","last_unstale":"2024-03-11T22:14:21.138235+0000","last_undegraded":"2024-03-11T22:14:21.138235+0000","last_fullsized":"2024-03-11T22:14:21.138235+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:39.174103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:39.174103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":226,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138194+0000","last_change":"2024-03-11T22:12:20.295562+0000","last_active":"2024-03-11T22:14:21.138194+0000","last_peered":"2024-03-11T22:14:21.138194+0000","last_clean":"2024-03-11T22:14:21.138194+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:14:21.138194+0000","last_undegraded":"2024-03-11T22:14:21.138194+0000","last_fullsized":"2024-03-11T22:14:21.138194+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:12:20.295427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:20.295427+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":195,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138686+0000","last_change":"2024-03-11T22:11:31.308182+0000","last_active":"2024-03-11T22:14:21.138686+0000","last_peered":"2024-03-11T22:14:21.138686+0000","last_clean":"2024-03-11T22:14:21.138686+0000","last_became_active":"2024-03-11T22:08:39.072089+0000","last_became_peered":"2024-03-11T22:08:39.072089+0000","last_unstale":"2024-03-11T22:14:21.138686+0000","last_undegraded":"2024-03-11T22:14:21.138686+0000","last_fullsized":"2024-03-11T22:14:21.138686+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:31.308103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:31.308103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":289,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138882+0000","last_change":"2024-03-11T22:11:44.093057+0000","last_active":"2024-03-11T22:14:21.138882+0000","last_peered":"2024-03-11T22:14:21.138882+0000","last_clean":"2024-03-11T22:14:21.138882+0000","last_became_active":"2024-03-11T22:08:37.071202+0000","last_became_peered":"2024-03-11T22:08:37.071202+0000","last_unstale":"2024-03-11T22:14:21.138882+0000","last_undegraded":"2024-03-11T22:14:21.138882+0000","last_fullsized":"2024-03-11T22:14:21.138882+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:11:44.092784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:44.092784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138175+0000","last_change":"2024-03-11T22:12:29.629914+0000","last_active":"2024-03-11T22:14:21.138175+0000","last_peered":"2024-03-11T22:14:21.138175+0000","last_clean":"2024-03-11T22:14:21.138175+0000","last_became_active":"2024-03-11T22:08:33.032850+0000","last_became_peered":"2024-03-11T22:08:33.032850+0000","last_unstale":"2024-03-11T22:14:21.138175+0000","last_undegraded":"2024-03-11T22:14:21.138175+0000","last_fullsized":"2024-03-11T22:14:21.138175+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:29.629778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:29.629778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":235,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336867+0000","last_change":"2024-03-11T22:13:30.515940+0000","last_active":"2024-03-11T22:14:21.336867+0000","last_peered":"2024-03-11T22:14:21.336867+0000","last_clean":"2024-03-11T22:14:21.336867+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:14:21.336867+0000","last_undegraded":"2024-03-11T22:14:21.336867+0000","last_fullsized":"2024-03-11T22:14:21.336867+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:13:30.515836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:30.515836+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:33.944436+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":221,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.529508+0000","last_change":"2024-03-11T22:14:19.987755+0000","last_active":"2024-03-11T22:14:25.529508+0000","last_peered":"2024-03-11T22:14:25.529508+0000","last_clean":"2024-03-11T22:14:25.529508+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:14:25.529508+0000","last_undegraded":"2024-03-11T22:14:25.529508+0000","last_fullsized":"2024-03-11T22:14:25.529508+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:19.987672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:19.987672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:44.507245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140123+0000","last_change":"2024-03-11T22:14:14.850354+0000","last_active":"2024-03-11T22:14:21.140123+0000","last_peered":"2024-03-11T22:14:21.140123+0000","last_clean":"2024-03-11T22:14:21.140123+0000","last_became_active":"2024-03-11T22:08:37.069611+0000","last_became_peered":"2024-03-11T22:08:37.069611+0000","last_unstale":"2024-03-11T22:14:21.140123+0000","last_undegraded":"2024-03-11T22:14:21.140123+0000","last_fullsized":"2024-03-11T22:14:21.140123+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.850128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.850128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:39.356529+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":146,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138015+0000","last_change":"2024-03-11T22:14:11.092405+0000","last_active":"2024-03-11T22:14:21.138015+0000","last_peered":"2024-03-11T22:14:21.138015+0000","last_clean":"2024-03-11T22:14:21.138015+0000","last_became_active":"2024-03-11T22:10:35.415644+0000","last_became_peered":"2024-03-11T22:10:35.415644+0000","last_unstale":"2024-03-11T22:14:21.138015+0000","last_undegraded":"2024-03-11T22:14:21.138015+0000","last_fullsized":"2024-03-11T22:14:21.138015+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:11.092328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:11.092328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:21.188140+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"93'13","reported_seq":231,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140647+0000","last_change":"2024-03-11T22:11:35.407557+0000","last_active":"2024-03-11T22:14:21.140647+0000","last_peered":"2024-03-11T22:14:21.140647+0000","last_clean":"2024-03-11T22:14:21.140647+0000","last_became_active":"2024-03-11T22:08:35.085787+0000","last_became_peered":"2024-03-11T22:08:35.085787+0000","last_unstale":"2024-03-11T22:14:21.140647+0000","last_undegraded":"2024-03-11T22:14:21.140647+0000","last_fullsized":"2024-03-11T22:14:21.140647+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:11:35.407301+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.407301+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":153,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336691+0000","last_change":"2024-03-11T22:13:37.172811+0000","last_active":"2024-03-11T22:14:21.336691+0000","last_peered":"2024-03-11T22:14:21.336691+0000","last_clean":"2024-03-11T22:14:21.336691+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:14:21.336691+0000","last_undegraded":"2024-03-11T22:14:21.336691+0000","last_fullsized":"2024-03-11T22:14:21.336691+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:13:37.172700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:00.602583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336962+0000","last_change":"2024-03-11T22:13:25.550514+0000","last_active":"2024-03-11T22:14:21.336962+0000","last_peered":"2024-03-11T22:14:21.336962+0000","last_clean":"2024-03-11T22:14:21.336962+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:14:21.336962+0000","last_undegraded":"2024-03-11T22:14:21.336962+0000","last_fullsized":"2024-03-11T22:14:21.336962+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:25.550429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:07.195289+0000","last_clean_scrub_stamp":"2024-03-11T22:13:25.550429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:50.191276+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":223,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.334980+0000","last_change":"2024-03-11T22:13:28.086496+0000","last_active":"2024-03-11T22:14:21.334980+0000","last_peered":"2024-03-11T22:14:21.334980+0000","last_clean":"2024-03-11T22:14:21.334980+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:14:21.334980+0000","last_undegraded":"2024-03-11T22:14:21.334980+0000","last_fullsized":"2024-03-11T22:14:21.334980+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:28.086387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:13:28.086387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:54.604819+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":220,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140214+0000","last_change":"2024-03-11T22:12:21.278151+0000","last_active":"2024-03-11T22:14:21.140214+0000","last_peered":"2024-03-11T22:14:21.140214+0000","last_clean":"2024-03-11T22:14:21.140214+0000","last_became_active":"2024-03-11T22:08:35.085914+0000","last_became_peered":"2024-03-11T22:08:35.085914+0000","last_unstale":"2024-03-11T22:14:21.140214+0000","last_undegraded":"2024-03-11T22:14:21.140214+0000","last_fullsized":"2024-03-11T22:14:21.140214+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:12:21.270671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:21.270671+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"160'17108","reported_seq":17474,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335525+0000","last_change":"2024-03-11T22:14:20.940951+0000","last_active":"2024-03-11T22:14:21.335525+0000","last_peered":"2024-03-11T22:14:21.335525+0000","last_clean":"2024-03-11T22:14:21.335525+0000","last_became_active":"2024-03-11T22:10:36.410723+0000","last_became_peered":"2024-03-11T22:10:36.410723+0000","last_unstale":"2024-03-11T22:14:21.335525+0000","last_undegraded":"2024-03-11T22:14:21.335525+0000","last_fullsized":"2024-03-11T22:14:21.335525+0000","mapping_epoch":99,"log_start":"137'12221","ondisk_log_start":"137'12221","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'17108","last_scrub_stamp":"2024-03-11T22:14:20.940846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:14:20.940846+0000","objects_scrubbed":0,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:47.496650+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14628,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335804+0000","last_change":"2024-03-11T22:12:38.434933+0000","last_active":"2024-03-11T22:14:21.335804+0000","last_peered":"2024-03-11T22:14:21.335804+0000","last_clean":"2024-03-11T22:14:21.335804+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:14:21.335804+0000","last_undegraded":"2024-03-11T22:14:21.335804+0000","last_fullsized":"2024-03-11T22:14:21.335804+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:38.434796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.434796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138558+0000","last_change":"2024-03-11T22:13:24.079957+0000","last_active":"2024-03-11T22:14:21.138558+0000","last_peered":"2024-03-11T22:14:21.138558+0000","last_clean":"2024-03-11T22:14:21.138558+0000","last_became_active":"2024-03-11T22:10:35.412565+0000","last_became_peered":"2024-03-11T22:10:35.412565+0000","last_unstale":"2024-03-11T22:14:21.138558+0000","last_undegraded":"2024-03-11T22:14:21.138558+0000","last_fullsized":"2024-03-11T22:14:21.138558+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:13:24.079822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:24.079822+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:44.565938+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"91'22","reported_seq":251,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335349+0000","last_change":"2024-03-11T22:13:18.458453+0000","last_active":"2024-03-11T22:14:21.335349+0000","last_peered":"2024-03-11T22:14:21.335349+0000","last_clean":"2024-03-11T22:14:21.335349+0000","last_became_active":"2024-03-11T22:08:35.085916+0000","last_became_peered":"2024-03-11T22:08:35.085916+0000","last_unstale":"2024-03-11T22:14:21.335349+0000","last_undegraded":"2024-03-11T22:14:21.335349+0000","last_fullsized":"2024-03-11T22:14:21.335349+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'22","last_scrub_stamp":"2024-03-11T22:13:18.458313+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:13:18.458313+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:29.608130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"99'9","reported_seq":225,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335266+0000","last_change":"2024-03-11T22:13:18.957027+0000","last_active":"2024-03-11T22:14:21.335266+0000","last_peered":"2024-03-11T22:14:21.335266+0000","last_clean":"2024-03-11T22:14:21.335266+0000","last_became_active":"2024-03-11T22:10:35.413089+0000","last_became_peered":"2024-03-11T22:10:35.413089+0000","last_unstale":"2024-03-11T22:14:21.335266+0000","last_undegraded":"2024-03-11T22:14:21.335266+0000","last_fullsized":"2024-03-11T22:14:21.335266+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'9","last_scrub_stamp":"2024-03-11T22:13:18.956933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:13:18.956933+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:33.528867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"160'14768","reported_seq":14973,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.139014+0000","last_change":"2024-03-11T22:12:38.052754+0000","last_active":"2024-03-11T22:14:21.139014+0000","last_peered":"2024-03-11T22:14:21.139014+0000","last_clean":"2024-03-11T22:14:21.139014+0000","last_became_active":"2024-03-11T22:10:36.411631+0000","last_became_peered":"2024-03-11T22:10:36.411631+0000","last_unstale":"2024-03-11T22:14:21.139014+0000","last_undegraded":"2024-03-11T22:14:21.139014+0000","last_fullsized":"2024-03-11T22:14:21.139014+0000","mapping_epoch":99,"log_start":"138'10458","ondisk_log_start":"138'10458","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'8787","last_scrub_stamp":"2024-03-11T22:12:38.052573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.052573+0000","objects_scrubbed":6049,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2625,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14768,"num_write_kb":472576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":205,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138423+0000","last_change":"2024-03-11T22:12:36.449430+0000","last_active":"2024-03-11T22:14:21.138423+0000","last_peered":"2024-03-11T22:14:21.138423+0000","last_clean":"2024-03-11T22:14:21.138423+0000","last_became_active":"2024-03-11T22:08:39.083141+0000","last_became_peered":"2024-03-11T22:08:39.083141+0000","last_unstale":"2024-03-11T22:14:21.138423+0000","last_undegraded":"2024-03-11T22:14:21.138423+0000","last_fullsized":"2024-03-11T22:14:21.138423+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:12:36.449307+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:12:36.449307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":146,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138411+0000","last_change":"2024-03-11T22:12:29.074360+0000","last_active":"2024-03-11T22:14:21.138411+0000","last_peered":"2024-03-11T22:14:21.138411+0000","last_clean":"2024-03-11T22:14:21.138411+0000","last_became_active":"2024-03-11T22:10:35.413337+0000","last_became_peered":"2024-03-11T22:10:35.413337+0000","last_unstale":"2024-03-11T22:14:21.138411+0000","last_undegraded":"2024-03-11T22:14:21.138411+0000","last_fullsized":"2024-03-11T22:14:21.138411+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:29.074256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:29.074256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":244,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138066+0000","last_change":"2024-03-11T22:11:21.379985+0000","last_active":"2024-03-11T22:14:21.138066+0000","last_peered":"2024-03-11T22:14:21.138066+0000","last_clean":"2024-03-11T22:14:21.138066+0000","last_became_active":"2024-03-11T22:08:54.607153+0000","last_became_peered":"2024-03-11T22:08:54.607153+0000","last_unstale":"2024-03-11T22:14:21.138066+0000","last_undegraded":"2024-03-11T22:14:21.138066+0000","last_fullsized":"2024-03-11T22:14:21.138066+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:11:21.379862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:21.379862+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":225,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.528176+0000","last_change":"2024-03-11T22:14:24.051145+0000","last_active":"2024-03-11T22:14:25.528176+0000","last_peered":"2024-03-11T22:14:25.528176+0000","last_clean":"2024-03-11T22:14:25.528176+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:14:25.528176+0000","last_undegraded":"2024-03-11T22:14:25.528176+0000","last_fullsized":"2024-03-11T22:14:25.528176+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:24.051020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:14:24.051020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:48.994239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"160'14794","reported_seq":14969,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138754+0000","last_change":"2024-03-11T22:13:25.207275+0000","last_active":"2024-03-11T22:14:21.138754+0000","last_peered":"2024-03-11T22:14:21.138754+0000","last_clean":"2024-03-11T22:14:21.138754+0000","last_became_active":"2024-03-11T22:10:36.412662+0000","last_became_peered":"2024-03-11T22:10:36.412662+0000","last_unstale":"2024-03-11T22:14:21.138754+0000","last_undegraded":"2024-03-11T22:14:21.138754+0000","last_fullsized":"2024-03-11T22:14:21.138754+0000","mapping_epoch":99,"log_start":"135'9593","ondisk_log_start":"135'9593","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'11754","last_scrub_stamp":"2024-03-11T22:13:25.206897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:13:25.206897+0000","objects_scrubbed":3073,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:52.018247+0000","scrub_duration":1174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14794,"num_write_kb":473408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":163,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.528453+0000","last_change":"2024-03-11T22:14:14.005962+0000","last_active":"2024-03-11T22:14:25.528453+0000","last_peered":"2024-03-11T22:14:25.528453+0000","last_clean":"2024-03-11T22:14:25.528453+0000","last_became_active":"2024-03-11T22:10:35.410893+0000","last_became_peered":"2024-03-11T22:10:35.410893+0000","last_unstale":"2024-03-11T22:14:25.528453+0000","last_undegraded":"2024-03-11T22:14:25.528453+0000","last_fullsized":"2024-03-11T22:14:25.528453+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.005795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.005795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:32.664197+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":217,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.528606+0000","last_change":"2024-03-11T22:13:24.767308+0000","last_active":"2024-03-11T22:14:25.528606+0000","last_peered":"2024-03-11T22:14:25.528606+0000","last_clean":"2024-03-11T22:14:25.528606+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:14:25.528606+0000","last_undegraded":"2024-03-11T22:14:25.528606+0000","last_fullsized":"2024-03-11T22:14:25.528606+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:24.764906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.710319+0000","last_clean_scrub_stamp":"2024-03-11T22:13:24.764906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":154,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336858+0000","last_change":"2024-03-11T22:12:44.474800+0000","last_active":"2024-03-11T22:14:21.336858+0000","last_peered":"2024-03-11T22:14:21.336858+0000","last_clean":"2024-03-11T22:14:21.336858+0000","last_became_active":"2024-03-11T22:10:35.418715+0000","last_became_peered":"2024-03-11T22:10:35.418715+0000","last_unstale":"2024-03-11T22:14:21.336858+0000","last_undegraded":"2024-03-11T22:14:21.336858+0000","last_fullsized":"2024-03-11T22:14:21.336858+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.474728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.474728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":250,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335617+0000","last_change":"2024-03-11T22:12:36.843050+0000","last_active":"2024-03-11T22:14:21.335617+0000","last_peered":"2024-03-11T22:14:21.335617+0000","last_clean":"2024-03-11T22:14:21.335617+0000","last_became_active":"2024-03-11T22:08:35.087046+0000","last_became_peered":"2024-03-11T22:08:35.087046+0000","last_unstale":"2024-03-11T22:14:21.335617+0000","last_undegraded":"2024-03-11T22:14:21.335617+0000","last_fullsized":"2024-03-11T22:14:21.335617+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:12:36.842912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:36.842912+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"160'15018","reported_seq":15241,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.528549+0000","last_change":"2024-03-11T22:14:16.227959+0000","last_active":"2024-03-11T22:14:25.528549+0000","last_peered":"2024-03-11T22:14:25.528549+0000","last_clean":"2024-03-11T22:14:25.528549+0000","last_became_active":"2024-03-11T22:10:36.410947+0000","last_became_peered":"2024-03-11T22:10:36.410947+0000","last_unstale":"2024-03-11T22:14:25.528549+0000","last_undegraded":"2024-03-11T22:14:25.528549+0000","last_fullsized":"2024-03-11T22:14:25.528549+0000","mapping_epoch":99,"log_start":"132'8972","ondisk_log_start":"132'8972","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'14831","last_scrub_stamp":"2024-03-11T22:14:16.227599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:14:16.227599+0000","objects_scrubbed":191,"log_size":6046,"log_dups_size":0,"ondisk_log_size":6046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:33.714002+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15018,"num_write_kb":480576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138668+0000","last_change":"2024-03-11T22:12:28.683756+0000","last_active":"2024-03-11T22:14:21.138668+0000","last_peered":"2024-03-11T22:14:21.138668+0000","last_clean":"2024-03-11T22:14:21.138668+0000","last_became_active":"2024-03-11T22:08:39.085496+0000","last_became_peered":"2024-03-11T22:08:39.085496+0000","last_unstale":"2024-03-11T22:14:21.138668+0000","last_undegraded":"2024-03-11T22:14:21.138668+0000","last_fullsized":"2024-03-11T22:14:21.138668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:28.683646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:28.683646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":217,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.529218+0000","last_change":"2024-03-11T22:13:19.781178+0000","last_active":"2024-03-11T22:14:25.529218+0000","last_peered":"2024-03-11T22:14:25.529218+0000","last_clean":"2024-03-11T22:14:25.529218+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:14:25.529218+0000","last_undegraded":"2024-03-11T22:14:25.529218+0000","last_fullsized":"2024-03-11T22:14:25.529218+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:19.776020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.776020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:38.461474+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336017+0000","last_change":"2024-03-11T22:12:34.391183+0000","last_active":"2024-03-11T22:14:21.336017+0000","last_peered":"2024-03-11T22:14:21.336017+0000","last_clean":"2024-03-11T22:14:21.336017+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:14:21.336017+0000","last_undegraded":"2024-03-11T22:14:21.336017+0000","last_fullsized":"2024-03-11T22:14:21.336017+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:34.391054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:34.391054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":243,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140126+0000","last_change":"2024-03-11T22:13:15.950715+0000","last_active":"2024-03-11T22:14:21.140126+0000","last_peered":"2024-03-11T22:14:21.140126+0000","last_clean":"2024-03-11T22:14:21.140126+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:14:21.140126+0000","last_undegraded":"2024-03-11T22:14:21.140126+0000","last_fullsized":"2024-03-11T22:14:21.140126+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:13:15.950564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:15.950564+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:28.253462+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":479,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335439+0000","last_change":"2024-03-11T22:13:38.573308+0000","last_active":"2024-03-11T22:14:21.335439+0000","last_peered":"2024-03-11T22:14:21.335439+0000","last_clean":"2024-03-11T22:14:21.335439+0000","last_became_active":"2024-03-11T22:13:38.572927+0000","last_became_peered":"2024-03-11T22:13:38.572927+0000","last_unstale":"2024-03-11T22:14:21.335439+0000","last_undegraded":"2024-03-11T22:14:21.335439+0000","last_fullsized":"2024-03-11T22:14:21.335439+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:13:34.052788+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:13:34.052788+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:01.710803+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"160'14778","reported_seq":14997,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336133+0000","last_change":"2024-03-11T22:14:11.124037+0000","last_active":"2024-03-11T22:14:21.336133+0000","last_peered":"2024-03-11T22:14:21.336133+0000","last_clean":"2024-03-11T22:14:21.336133+0000","last_became_active":"2024-03-11T22:10:36.416476+0000","last_became_peered":"2024-03-11T22:10:36.416476+0000","last_unstale":"2024-03-11T22:14:21.336133+0000","last_undegraded":"2024-03-11T22:14:21.336133+0000","last_fullsized":"2024-03-11T22:14:21.336133+0000","mapping_epoch":99,"log_start":"132'8923","ondisk_log_start":"132'8923","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'14357","last_scrub_stamp":"2024-03-11T22:14:11.123803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:14:11.123803+0000","objects_scrubbed":423,"log_size":5855,"log_dups_size":0,"ondisk_log_size":5855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:23.330826+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14778,"num_write_kb":472896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138993+0000","last_change":"2024-03-11T22:12:45.439711+0000","last_active":"2024-03-11T22:14:21.138993+0000","last_peered":"2024-03-11T22:14:21.138993+0000","last_clean":"2024-03-11T22:14:21.138993+0000","last_became_active":"2024-03-11T22:08:39.083738+0000","last_became_peered":"2024-03-11T22:08:39.083738+0000","last_unstale":"2024-03-11T22:14:21.138993+0000","last_undegraded":"2024-03-11T22:14:21.138993+0000","last_fullsized":"2024-03-11T22:14:21.138993+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.439584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":235,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335642+0000","last_change":"2024-03-11T22:14:16.662735+0000","last_active":"2024-03-11T22:14:21.335642+0000","last_peered":"2024-03-11T22:14:21.335642+0000","last_clean":"2024-03-11T22:14:21.335642+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:14:21.335642+0000","last_undegraded":"2024-03-11T22:14:21.335642+0000","last_fullsized":"2024-03-11T22:14:21.335642+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:14:16.662596+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:14:16.662596+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:45.012386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138001+0000","last_change":"2024-03-11T22:13:28.028641+0000","last_active":"2024-03-11T22:14:21.138001+0000","last_peered":"2024-03-11T22:14:21.138001+0000","last_clean":"2024-03-11T22:14:21.138001+0000","last_became_active":"2024-03-11T22:08:33.035469+0000","last_became_peered":"2024-03-11T22:08:33.035469+0000","last_unstale":"2024-03-11T22:14:21.138001+0000","last_undegraded":"2024-03-11T22:14:21.138001+0000","last_fullsized":"2024-03-11T22:14:21.138001+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:28.028531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:28.028531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:50.939614+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":338,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336218+0000","last_change":"2024-03-11T22:13:17.830633+0000","last_active":"2024-03-11T22:14:21.336218+0000","last_peered":"2024-03-11T22:14:21.336218+0000","last_clean":"2024-03-11T22:14:21.336218+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:14:21.336218+0000","last_undegraded":"2024-03-11T22:14:21.336218+0000","last_fullsized":"2024-03-11T22:14:21.336218+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-11T22:13:17.830327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:17.830327+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140501+0000","last_change":"2024-03-11T22:14:16.998358+0000","last_active":"2024-03-11T22:14:21.140501+0000","last_peered":"2024-03-11T22:14:21.140501+0000","last_clean":"2024-03-11T22:14:21.140501+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:14:21.140501+0000","last_undegraded":"2024-03-11T22:14:21.140501+0000","last_fullsized":"2024-03-11T22:14:21.140501+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:16.998238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:16.998238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:18.606617+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":487,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138117+0000","last_change":"2024-03-11T22:14:11.947176+0000","last_active":"2024-03-11T22:14:21.138117+0000","last_peered":"2024-03-11T22:14:21.138117+0000","last_clean":"2024-03-11T22:14:21.138117+0000","last_became_active":"2024-03-11T22:13:38.573086+0000","last_became_peered":"2024-03-11T22:13:38.573086+0000","last_unstale":"2024-03-11T22:14:21.138117+0000","last_undegraded":"2024-03-11T22:14:21.138117+0000","last_fullsized":"2024-03-11T22:14:21.138117+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:11.947049+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:11.947049+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:15.319123+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"160'14982","reported_seq":15153,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.139506+0000","last_change":"2024-03-11T22:13:29.879326+0000","last_active":"2024-03-11T22:14:21.139506+0000","last_peered":"2024-03-11T22:14:21.139506+0000","last_clean":"2024-03-11T22:14:21.139506+0000","last_became_active":"2024-03-11T22:10:36.412694+0000","last_became_peered":"2024-03-11T22:10:36.412694+0000","last_unstale":"2024-03-11T22:14:21.139506+0000","last_undegraded":"2024-03-11T22:14:21.139506+0000","last_fullsized":"2024-03-11T22:14:21.139506+0000","mapping_epoch":99,"log_start":"135'9761","ondisk_log_start":"135'9761","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'12216","last_scrub_stamp":"2024-03-11T22:13:29.879111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:13:29.879111+0000","objects_scrubbed":2790,"log_size":5221,"log_dups_size":0,"ondisk_log_size":5221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:49.251138+0000","scrub_duration":851,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14982,"num_write_kb":479424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":153,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336614+0000","last_change":"2024-03-11T22:13:22.005413+0000","last_active":"2024-03-11T22:14:21.336614+0000","last_peered":"2024-03-11T22:14:21.336614+0000","last_clean":"2024-03-11T22:14:21.336614+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:14:21.336614+0000","last_undegraded":"2024-03-11T22:14:21.336614+0000","last_fullsized":"2024-03-11T22:14:21.336614+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:22.005300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:22.005300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:25.411995+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":223,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335377+0000","last_change":"2024-03-11T22:13:19.750323+0000","last_active":"2024-03-11T22:14:21.335377+0000","last_peered":"2024-03-11T22:14:21.335377+0000","last_clean":"2024-03-11T22:14:21.335377+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:14:21.335377+0000","last_undegraded":"2024-03-11T22:14:21.335377+0000","last_fullsized":"2024-03-11T22:14:21.335377+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:19.748477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.748477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:32.843670+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":242,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.528938+0000","last_change":"2024-03-11T22:14:09.871684+0000","last_active":"2024-03-11T22:14:25.528938+0000","last_peered":"2024-03-11T22:14:25.528938+0000","last_clean":"2024-03-11T22:14:25.528938+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:14:25.528938+0000","last_undegraded":"2024-03-11T22:14:25.528938+0000","last_fullsized":"2024-03-11T22:14:25.528938+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:09.871543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:14:09.871543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:16.454261+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":246,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335397+0000","last_change":"2024-03-11T22:13:24.705841+0000","last_active":"2024-03-11T22:14:21.335397+0000","last_peered":"2024-03-11T22:14:21.335397+0000","last_clean":"2024-03-11T22:14:21.335397+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:14:21.335397+0000","last_undegraded":"2024-03-11T22:14:21.335397+0000","last_fullsized":"2024-03-11T22:14:21.335397+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-03-11T22:13:24.705697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:24.705697+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:29.566342+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":225,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.529791+0000","last_change":"2024-03-11T22:14:21.033755+0000","last_active":"2024-03-11T22:14:25.529791+0000","last_peered":"2024-03-11T22:14:25.529791+0000","last_clean":"2024-03-11T22:14:25.529791+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:14:25.529791+0000","last_undegraded":"2024-03-11T22:14:25.529791+0000","last_fullsized":"2024-03-11T22:14:25.529791+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:21.033664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.033664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:38.113619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"160'16800","reported_seq":17026,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140590+0000","last_change":"2024-03-11T22:13:54.435445+0000","last_active":"2024-03-11T22:14:21.140590+0000","last_peered":"2024-03-11T22:14:21.140590+0000","last_clean":"2024-03-11T22:14:21.140590+0000","last_became_active":"2024-03-11T22:10:36.408867+0000","last_became_peered":"2024-03-11T22:10:36.408867+0000","last_unstale":"2024-03-11T22:14:21.140590+0000","last_undegraded":"2024-03-11T22:14:21.140590+0000","last_fullsized":"2024-03-11T22:14:21.140590+0000","mapping_epoch":99,"log_start":"141'11564","ondisk_log_start":"141'11564","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'15322","last_scrub_stamp":"2024-03-11T22:13:54.435291+0000","last_deep_scrub":"150'15322","last_deep_scrub_stamp":"2024-03-11T22:13:54.435291+0000","last_clean_scrub_stamp":"2024-03-11T22:13:54.435291+0000","objects_scrubbed":1948,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":22,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:01.062829+0000","scrub_duration":21325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14852,"num_write_kb":475264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":211,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336379+0000","last_change":"2024-03-11T22:13:34.478753+0000","last_active":"2024-03-11T22:14:21.336379+0000","last_peered":"2024-03-11T22:14:21.336379+0000","last_clean":"2024-03-11T22:14:21.336379+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:14:21.336379+0000","last_undegraded":"2024-03-11T22:14:21.336379+0000","last_fullsized":"2024-03-11T22:14:21.336379+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:34.478634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:34.478634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:02.370872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":479,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138505+0000","last_change":"2024-03-11T22:13:35.093324+0000","last_active":"2024-03-11T22:14:21.138505+0000","last_peered":"2024-03-11T22:14:21.138505+0000","last_clean":"2024-03-11T22:14:21.138505+0000","last_became_active":"2024-03-11T22:13:33.516539+0000","last_became_peered":"2024-03-11T22:13:33.516539+0000","last_unstale":"2024-03-11T22:14:21.138505+0000","last_undegraded":"2024-03-11T22:14:21.138505+0000","last_fullsized":"2024-03-11T22:14:21.138505+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:13:35.093204+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:04.719504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":223,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137881+0000","last_change":"2024-03-11T22:13:33.460800+0000","last_active":"2024-03-11T22:14:21.137881+0000","last_peered":"2024-03-11T22:14:21.137881+0000","last_clean":"2024-03-11T22:14:21.137881+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:14:21.137881+0000","last_undegraded":"2024-03-11T22:14:21.137881+0000","last_fullsized":"2024-03-11T22:14:21.137881+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:33.460308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:13:33.460308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:54.540593+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":312,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336608+0000","last_change":"2024-03-11T22:13:29.093136+0000","last_active":"2024-03-11T22:14:21.336608+0000","last_peered":"2024-03-11T22:14:21.336608+0000","last_clean":"2024-03-11T22:14:21.336608+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:14:21.336608+0000","last_undegraded":"2024-03-11T22:14:21.336608+0000","last_fullsized":"2024-03-11T22:14:21.336608+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-11T22:13:29.093013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:29.093013+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:34.595204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":163,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.529596+0000","last_change":"2024-03-11T22:14:23.041388+0000","last_active":"2024-03-11T22:14:25.529596+0000","last_peered":"2024-03-11T22:14:25.529596+0000","last_clean":"2024-03-11T22:14:25.529596+0000","last_became_active":"2024-03-11T22:10:35.412547+0000","last_became_peered":"2024-03-11T22:10:35.412547+0000","last_unstale":"2024-03-11T22:14:25.529596+0000","last_undegraded":"2024-03-11T22:14:25.529596+0000","last_fullsized":"2024-03-11T22:14:25.529596+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.041267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.041267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:29.148588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":480,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335722+0000","last_change":"2024-03-11T22:14:09.860538+0000","last_active":"2024-03-11T22:14:21.335722+0000","last_peered":"2024-03-11T22:14:21.335722+0000","last_clean":"2024-03-11T22:14:21.335722+0000","last_became_active":"2024-03-11T22:13:38.569265+0000","last_became_peered":"2024-03-11T22:13:38.569265+0000","last_unstale":"2024-03-11T22:14:21.335722+0000","last_undegraded":"2024-03-11T22:14:21.335722+0000","last_fullsized":"2024-03-11T22:14:21.335722+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:09.860200+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:09.860200+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:12.360322+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"160'14870","reported_seq":15105,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.529714+0000","last_change":"2024-03-11T22:13:31.774638+0000","last_active":"2024-03-11T22:14:25.529714+0000","last_peered":"2024-03-11T22:14:25.529714+0000","last_clean":"2024-03-11T22:14:25.529714+0000","last_became_active":"2024-03-11T22:10:36.406423+0000","last_became_peered":"2024-03-11T22:10:36.406423+0000","last_unstale":"2024-03-11T22:14:25.529714+0000","last_undegraded":"2024-03-11T22:14:25.529714+0000","last_fullsized":"2024-03-11T22:14:25.529714+0000","mapping_epoch":99,"log_start":"131'8801","ondisk_log_start":"131'8801","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'12265","last_scrub_stamp":"2024-03-11T22:13:31.774415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:13:31.774415+0000","objects_scrubbed":2639,"log_size":6069,"log_dups_size":0,"ondisk_log_size":6069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:50.245474+0000","scrub_duration":969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14870,"num_write_kb":475840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":211,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336017+0000","last_change":"2024-03-11T22:12:44.933303+0000","last_active":"2024-03-11T22:14:21.336017+0000","last_peered":"2024-03-11T22:14:21.336017+0000","last_clean":"2024-03-11T22:14:21.336017+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:14:21.336017+0000","last_undegraded":"2024-03-11T22:14:21.336017+0000","last_fullsized":"2024-03-11T22:14:21.336017+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.927967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.927967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138476+0000","last_change":"2024-03-11T22:12:19.685277+0000","last_active":"2024-03-11T22:14:21.138476+0000","last_peered":"2024-03-11T22:14:21.138476+0000","last_clean":"2024-03-11T22:14:21.138476+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:14:21.138476+0000","last_undegraded":"2024-03-11T22:14:21.138476+0000","last_fullsized":"2024-03-11T22:14:21.138476+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:19.684663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:19.684663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":557,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:24.033911+0000","last_change":"2024-03-11T22:14:13.848646+0000","last_active":"2024-03-11T22:14:24.033911+0000","last_peered":"2024-03-11T22:14:24.033911+0000","last_clean":"2024-03-11T22:14:24.033911+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:14:24.033911+0000","last_undegraded":"2024-03-11T22:14:24.033911+0000","last_fullsized":"2024-03-11T22:14:24.033911+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:14:13.848518+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:14:13.848518+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:36.660519+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":319,"num_read_kb":314,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":237,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140422+0000","last_change":"2024-03-11T22:14:21.065394+0000","last_active":"2024-03-11T22:14:21.140422+0000","last_peered":"2024-03-11T22:14:21.140422+0000","last_clean":"2024-03-11T22:14:21.140422+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:14:21.140422+0000","last_undegraded":"2024-03-11T22:14:21.140422+0000","last_fullsized":"2024-03-11T22:14:21.140422+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:14:21.065306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.065306+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:37.928690+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":207,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336685+0000","last_change":"2024-03-11T22:12:27.459283+0000","last_active":"2024-03-11T22:14:21.336685+0000","last_peered":"2024-03-11T22:14:21.336685+0000","last_clean":"2024-03-11T22:14:21.336685+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:14:21.336685+0000","last_undegraded":"2024-03-11T22:14:21.336685+0000","last_fullsized":"2024-03-11T22:14:21.336685+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:27.459154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:27.459154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":368,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336013+0000","last_change":"2024-03-11T22:14:13.958991+0000","last_active":"2024-03-11T22:14:21.336013+0000","last_peered":"2024-03-11T22:14:21.336013+0000","last_clean":"2024-03-11T22:14:21.336013+0000","last_became_active":"2024-03-11T22:13:38.845904+0000","last_became_peered":"2024-03-11T22:13:38.845904+0000","last_unstale":"2024-03-11T22:14:21.336013+0000","last_undegraded":"2024-03-11T22:14:21.336013+0000","last_fullsized":"2024-03-11T22:14:21.336013+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:13.958859+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:13.958859+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:39.139563+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"160'14604","reported_seq":14815,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335201+0000","last_change":"2024-03-11T22:12:25.123093+0000","last_active":"2024-03-11T22:14:21.335201+0000","last_peered":"2024-03-11T22:14:21.335201+0000","last_clean":"2024-03-11T22:14:21.335201+0000","last_became_active":"2024-03-11T22:10:36.411289+0000","last_became_peered":"2024-03-11T22:10:36.411289+0000","last_unstale":"2024-03-11T22:14:21.335201+0000","last_undegraded":"2024-03-11T22:14:21.335201+0000","last_fullsized":"2024-03-11T22:14:21.335201+0000","mapping_epoch":99,"log_start":"133'9211","ondisk_log_start":"133'9211","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7985","last_scrub_stamp":"2024-03-11T22:12:25.122944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:25.122944+0000","objects_scrubbed":6655,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1760,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14604,"num_write_kb":467328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":211,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336161+0000","last_change":"2024-03-11T22:13:20.800006+0000","last_active":"2024-03-11T22:14:21.336161+0000","last_peered":"2024-03-11T22:14:21.336161+0000","last_clean":"2024-03-11T22:14:21.336161+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:14:21.336161+0000","last_undegraded":"2024-03-11T22:14:21.336161+0000","last_fullsized":"2024-03-11T22:14:21.336161+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:20.799791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:20.799791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:35.396331+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336139+0000","last_change":"2024-03-11T22:12:42.951642+0000","last_active":"2024-03-11T22:14:21.336139+0000","last_peered":"2024-03-11T22:14:21.336139+0000","last_clean":"2024-03-11T22:14:21.336139+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:14:21.336139+0000","last_undegraded":"2024-03-11T22:14:21.336139+0000","last_fullsized":"2024-03-11T22:14:21.336139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.951379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.951379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":247,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.139829+0000","last_change":"2024-03-11T22:13:27.025211+0000","last_active":"2024-03-11T22:14:21.139829+0000","last_peered":"2024-03-11T22:14:21.139829+0000","last_clean":"2024-03-11T22:14:21.139829+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:14:21.139829+0000","last_undegraded":"2024-03-11T22:14:21.139829+0000","last_fullsized":"2024-03-11T22:14:21.139829+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:13:27.025050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:27.025050+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:52.422159+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":223,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335640+0000","last_change":"2024-03-11T22:14:18.961752+0000","last_active":"2024-03-11T22:14:21.335640+0000","last_peered":"2024-03-11T22:14:21.335640+0000","last_clean":"2024-03-11T22:14:21.335640+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:14:21.335640+0000","last_undegraded":"2024-03-11T22:14:21.335640+0000","last_fullsized":"2024-03-11T22:14:21.335640+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:18.961673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:14:18.961673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:45.145191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":368,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336355+0000","last_change":"2024-03-11T22:14:12.921931+0000","last_active":"2024-03-11T22:14:21.336355+0000","last_peered":"2024-03-11T22:14:21.336355+0000","last_clean":"2024-03-11T22:14:21.336355+0000","last_became_active":"2024-03-11T22:13:38.576468+0000","last_became_peered":"2024-03-11T22:13:38.576468+0000","last_unstale":"2024-03-11T22:14:21.336355+0000","last_undegraded":"2024-03-11T22:14:21.336355+0000","last_fullsized":"2024-03-11T22:14:21.336355+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:12.921840+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:12.921840+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:29.307630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"160'14698","reported_seq":14841,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137625+0000","last_change":"2024-03-11T22:14:18.980951+0000","last_active":"2024-03-11T22:14:21.137625+0000","last_peered":"2024-03-11T22:14:21.137625+0000","last_clean":"2024-03-11T22:14:21.137625+0000","last_became_active":"2024-03-11T22:10:36.416083+0000","last_became_peered":"2024-03-11T22:10:36.416083+0000","last_unstale":"2024-03-11T22:14:21.137625+0000","last_undegraded":"2024-03-11T22:14:21.137625+0000","last_fullsized":"2024-03-11T22:14:21.137625+0000","mapping_epoch":99,"log_start":"137'10002","ondisk_log_start":"137'10002","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'14698","last_scrub_stamp":"2024-03-11T22:14:18.980850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:14:18.980850+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:22.115035+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14698,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138409+0000","last_change":"2024-03-11T22:14:10.990148+0000","last_active":"2024-03-11T22:14:21.138409+0000","last_peered":"2024-03-11T22:14:21.138409+0000","last_clean":"2024-03-11T22:14:21.138409+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:14:21.138409+0000","last_undegraded":"2024-03-11T22:14:21.138409+0000","last_fullsized":"2024-03-11T22:14:21.138409+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:10.990032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:10.990032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:13.968549+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":223,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336766+0000","last_change":"2024-03-11T22:12:45.466625+0000","last_active":"2024-03-11T22:14:21.336766+0000","last_peered":"2024-03-11T22:14:21.336766+0000","last_clean":"2024-03-11T22:14:21.336766+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:14:21.336766+0000","last_undegraded":"2024-03-11T22:14:21.336766+0000","last_fullsized":"2024-03-11T22:14:21.336766+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.466508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.466508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":226,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336315+0000","last_change":"2024-03-11T22:12:38.896205+0000","last_active":"2024-03-11T22:14:21.336315+0000","last_peered":"2024-03-11T22:14:21.336315+0000","last_clean":"2024-03-11T22:14:21.336315+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:14:21.336315+0000","last_undegraded":"2024-03-11T22:14:21.336315+0000","last_fullsized":"2024-03-11T22:14:21.336315+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:12:38.895763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.895763+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140080+0000","last_change":"2024-03-11T22:11:25.385148+0000","last_active":"2024-03-11T22:14:21.140080+0000","last_peered":"2024-03-11T22:14:21.140080+0000","last_clean":"2024-03-11T22:14:21.140080+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:14:21.140080+0000","last_undegraded":"2024-03-11T22:14:21.140080+0000","last_fullsized":"2024-03-11T22:14:21.140080+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:11:25.384965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:25.384965+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":480,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137957+0000","last_change":"2024-03-11T22:14:09.832992+0000","last_active":"2024-03-11T22:14:21.137957+0000","last_peered":"2024-03-11T22:14:21.137957+0000","last_clean":"2024-03-11T22:14:21.137957+0000","last_became_active":"2024-03-11T22:13:38.569867+0000","last_became_peered":"2024-03-11T22:13:38.569867+0000","last_unstale":"2024-03-11T22:14:21.137957+0000","last_undegraded":"2024-03-11T22:14:21.137957+0000","last_fullsized":"2024-03-11T22:14:21.137957+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:09.832674+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:09.832674+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:27.974504+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"160'14806","reported_seq":15011,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137700+0000","last_change":"2024-03-11T22:12:35.256058+0000","last_active":"2024-03-11T22:14:21.137700+0000","last_peered":"2024-03-11T22:14:21.137700+0000","last_clean":"2024-03-11T22:14:21.137700+0000","last_became_active":"2024-03-11T22:10:36.410859+0000","last_became_peered":"2024-03-11T22:10:36.410859+0000","last_unstale":"2024-03-11T22:14:21.137700+0000","last_undegraded":"2024-03-11T22:14:21.137700+0000","last_fullsized":"2024-03-11T22:14:21.137700+0000","mapping_epoch":99,"log_start":"139'10511","ondisk_log_start":"139'10511","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8663","last_scrub_stamp":"2024-03-11T22:12:35.255739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.255739+0000","objects_scrubbed":6218,"log_size":4295,"log_dups_size":0,"ondisk_log_size":4295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14806,"num_write_kb":473792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":153,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336380+0000","last_change":"2024-03-11T22:12:46.968879+0000","last_active":"2024-03-11T22:14:21.336380+0000","last_peered":"2024-03-11T22:14:21.336380+0000","last_clean":"2024-03-11T22:14:21.336380+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:14:21.336380+0000","last_undegraded":"2024-03-11T22:14:21.336380+0000","last_fullsized":"2024-03-11T22:14:21.336380+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.968807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.968807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":216,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140426+0000","last_change":"2024-03-11T22:12:49.856525+0000","last_active":"2024-03-11T22:14:21.140426+0000","last_peered":"2024-03-11T22:14:21.140426+0000","last_clean":"2024-03-11T22:14:21.140426+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:14:21.140426+0000","last_undegraded":"2024-03-11T22:14:21.140426+0000","last_fullsized":"2024-03-11T22:14:21.140426+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:49.856381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.856381+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":232,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137947+0000","last_change":"2024-03-11T22:13:27.028743+0000","last_active":"2024-03-11T22:14:21.137947+0000","last_peered":"2024-03-11T22:14:21.137947+0000","last_clean":"2024-03-11T22:14:21.137947+0000","last_became_active":"2024-03-11T22:08:35.053452+0000","last_became_peered":"2024-03-11T22:08:35.053452+0000","last_unstale":"2024-03-11T22:14:21.137947+0000","last_undegraded":"2024-03-11T22:14:21.137947+0000","last_fullsized":"2024-03-11T22:14:21.137947+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:13:27.028171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:27.028171+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:50.155527+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138532+0000","last_change":"2024-03-11T22:12:31.264333+0000","last_active":"2024-03-11T22:14:21.138532+0000","last_peered":"2024-03-11T22:14:21.138532+0000","last_clean":"2024-03-11T22:14:21.138532+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:14:21.138532+0000","last_undegraded":"2024-03-11T22:14:21.138532+0000","last_fullsized":"2024-03-11T22:14:21.138532+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:12:31.264213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.264213+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":480,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138251+0000","last_change":"2024-03-11T22:14:13.029495+0000","last_active":"2024-03-11T22:14:21.138251+0000","last_peered":"2024-03-11T22:14:21.138251+0000","last_clean":"2024-03-11T22:14:21.138251+0000","last_became_active":"2024-03-11T22:13:38.565536+0000","last_became_peered":"2024-03-11T22:13:38.565536+0000","last_unstale":"2024-03-11T22:14:21.138251+0000","last_undegraded":"2024-03-11T22:14:21.138251+0000","last_fullsized":"2024-03-11T22:14:21.138251+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:13.029360+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:13.029360+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:13.611287+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"160'14726","reported_seq":14930,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138534+0000","last_change":"2024-03-11T22:12:35.900151+0000","last_active":"2024-03-11T22:14:21.138534+0000","last_peered":"2024-03-11T22:14:21.138534+0000","last_clean":"2024-03-11T22:14:21.138534+0000","last_became_active":"2024-03-11T22:10:36.414380+0000","last_became_peered":"2024-03-11T22:10:36.414380+0000","last_unstale":"2024-03-11T22:14:21.138534+0000","last_undegraded":"2024-03-11T22:14:21.138534+0000","last_fullsized":"2024-03-11T22:14:21.138534+0000","mapping_epoch":99,"log_start":"138'10344","ondisk_log_start":"138'10344","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8688","last_scrub_stamp":"2024-03-11T22:12:35.899750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:12:35.899750+0000","objects_scrubbed":6120,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14726,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":195,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137844+0000","last_change":"2024-03-11T22:11:35.248168+0000","last_active":"2024-03-11T22:14:21.137844+0000","last_peered":"2024-03-11T22:14:21.137844+0000","last_clean":"2024-03-11T22:14:21.137844+0000","last_became_active":"2024-03-11T22:08:39.078244+0000","last_became_peered":"2024-03-11T22:08:39.078244+0000","last_unstale":"2024-03-11T22:14:21.137844+0000","last_undegraded":"2024-03-11T22:14:21.137844+0000","last_fullsized":"2024-03-11T22:14:21.137844+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:35.248081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.248081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":223,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.139915+0000","last_change":"2024-03-11T22:13:26.481234+0000","last_active":"2024-03-11T22:14:21.139915+0000","last_peered":"2024-03-11T22:14:21.139915+0000","last_clean":"2024-03-11T22:14:21.139915+0000","last_became_active":"2024-03-11T22:08:33.036469+0000","last_became_peered":"2024-03-11T22:08:33.036469+0000","last_unstale":"2024-03-11T22:14:21.139915+0000","last_undegraded":"2024-03-11T22:14:21.139915+0000","last_fullsized":"2024-03-11T22:14:21.139915+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:26.481122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:26.481122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:38.248860+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":246,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138038+0000","last_change":"2024-03-11T22:12:39.447777+0000","last_active":"2024-03-11T22:14:21.138038+0000","last_peered":"2024-03-11T22:14:21.138038+0000","last_clean":"2024-03-11T22:14:21.138038+0000","last_became_active":"2024-03-11T22:08:35.085979+0000","last_became_peered":"2024-03-11T22:08:35.085979+0000","last_unstale":"2024-03-11T22:14:21.138038+0000","last_undegraded":"2024-03-11T22:14:21.138038+0000","last_fullsized":"2024-03-11T22:14:21.138038+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:12:39.447312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:39.447312+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":207,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137764+0000","last_change":"2024-03-11T22:11:40.165879+0000","last_active":"2024-03-11T22:14:21.137764+0000","last_peered":"2024-03-11T22:14:21.137764+0000","last_clean":"2024-03-11T22:14:21.137764+0000","last_became_active":"2024-03-11T22:08:37.076522+0000","last_became_peered":"2024-03-11T22:08:37.076522+0000","last_unstale":"2024-03-11T22:14:21.137764+0000","last_undegraded":"2024-03-11T22:14:21.137764+0000","last_fullsized":"2024-03-11T22:14:21.137764+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:40.165760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:40.165760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":480,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335523+0000","last_change":"2024-03-11T22:14:12.712999+0000","last_active":"2024-03-11T22:14:21.335523+0000","last_peered":"2024-03-11T22:14:21.335523+0000","last_clean":"2024-03-11T22:14:21.335523+0000","last_became_active":"2024-03-11T22:13:38.573794+0000","last_became_peered":"2024-03-11T22:13:38.573794+0000","last_unstale":"2024-03-11T22:14:21.335523+0000","last_undegraded":"2024-03-11T22:14:21.335523+0000","last_fullsized":"2024-03-11T22:14:21.335523+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:12.712874+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:12.712874+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:37.734676+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"160'14726","reported_seq":14941,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335940+0000","last_change":"2024-03-11T22:14:16.944229+0000","last_active":"2024-03-11T22:14:21.335940+0000","last_peered":"2024-03-11T22:14:21.335940+0000","last_clean":"2024-03-11T22:14:21.335940+0000","last_became_active":"2024-03-11T22:10:36.414372+0000","last_became_peered":"2024-03-11T22:10:36.414372+0000","last_unstale":"2024-03-11T22:14:21.335940+0000","last_undegraded":"2024-03-11T22:14:21.335940+0000","last_fullsized":"2024-03-11T22:14:21.335940+0000","mapping_epoch":99,"log_start":"132'8868","ondisk_log_start":"132'8868","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'14600","last_scrub_stamp":"2024-03-11T22:14:16.944060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:14:16.944060+0000","objects_scrubbed":127,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:44.358442+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14726,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335942+0000","last_change":"2024-03-11T22:12:42.469626+0000","last_active":"2024-03-11T22:14:21.335942+0000","last_peered":"2024-03-11T22:14:21.335942+0000","last_clean":"2024-03-11T22:14:21.335942+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:14:21.335942+0000","last_undegraded":"2024-03-11T22:14:21.335942+0000","last_fullsized":"2024-03-11T22:14:21.335942+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.469501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.469501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":286,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138661+0000","last_change":"2024-03-11T22:13:35.126819+0000","last_active":"2024-03-11T22:14:21.138661+0000","last_peered":"2024-03-11T22:14:21.138661+0000","last_clean":"2024-03-11T22:14:21.138661+0000","last_became_active":"2024-03-11T22:08:37.077142+0000","last_became_peered":"2024-03-11T22:08:37.077142+0000","last_unstale":"2024-03-11T22:14:21.138661+0000","last_undegraded":"2024-03-11T22:14:21.138661+0000","last_fullsized":"2024-03-11T22:14:21.138661+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:13:35.126665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:13:35.126665+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:42.706921+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137778+0000","last_change":"2024-03-11T22:12:24.278539+0000","last_active":"2024-03-11T22:14:21.137778+0000","last_peered":"2024-03-11T22:14:21.137778+0000","last_clean":"2024-03-11T22:14:21.137778+0000","last_became_active":"2024-03-11T22:08:33.037537+0000","last_became_peered":"2024-03-11T22:08:33.037537+0000","last_unstale":"2024-03-11T22:14:21.137778+0000","last_undegraded":"2024-03-11T22:14:21.137778+0000","last_fullsized":"2024-03-11T22:14:21.137778+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:24.278298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.278298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":244,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335510+0000","last_change":"2024-03-11T22:14:14.861720+0000","last_active":"2024-03-11T22:14:21.335510+0000","last_peered":"2024-03-11T22:14:21.335510+0000","last_clean":"2024-03-11T22:14:21.335510+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:14:21.335510+0000","last_undegraded":"2024-03-11T22:14:21.335510+0000","last_fullsized":"2024-03-11T22:14:21.335510+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:14:14.861573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.861573+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:23.560000+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"160'15238","reported_seq":15453,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:24.974663+0000","last_change":"2024-03-11T22:14:24.974712+0000","last_active":"2024-03-11T22:14:24.974663+0000","last_peered":"2024-03-11T22:14:24.974663+0000","last_clean":"2024-03-11T22:14:24.974663+0000","last_became_active":"2024-03-11T22:10:36.412847+0000","last_became_peered":"2024-03-11T22:10:36.412847+0000","last_unstale":"2024-03-11T22:14:24.974663+0000","last_undegraded":"2024-03-11T22:14:24.974663+0000","last_fullsized":"2024-03-11T22:14:24.974663+0000","mapping_epoch":99,"log_start":"132'9404","ondisk_log_start":"132'9404","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"160'15238","last_scrub_stamp":"2024-03-11T22:14:24.974565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:14:24.974565+0000","objects_scrubbed":0,"log_size":5834,"log_dups_size":0,"ondisk_log_size":5834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:44.380512+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15238,"num_write_kb":487553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138080+0000","last_change":"2024-03-11T22:13:17.533805+0000","last_active":"2024-03-11T22:14:21.138080+0000","last_peered":"2024-03-11T22:14:21.138080+0000","last_clean":"2024-03-11T22:14:21.138080+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:14:21.138080+0000","last_undegraded":"2024-03-11T22:14:21.138080+0000","last_fullsized":"2024-03-11T22:14:21.138080+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:17.533607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:17.533607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:46.744148+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":207,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336665+0000","last_change":"2024-03-11T22:12:20.302654+0000","last_active":"2024-03-11T22:14:21.336665+0000","last_peered":"2024-03-11T22:14:21.336665+0000","last_clean":"2024-03-11T22:14:21.336665+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:14:21.336665+0000","last_undegraded":"2024-03-11T22:14:21.336665+0000","last_fullsized":"2024-03-11T22:14:21.336665+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:20.302574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:20.302574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":207,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137772+0000","last_change":"2024-03-11T22:11:23.353503+0000","last_active":"2024-03-11T22:14:21.137772+0000","last_peered":"2024-03-11T22:14:21.137772+0000","last_clean":"2024-03-11T22:14:21.137772+0000","last_became_active":"2024-03-11T22:08:33.299778+0000","last_became_peered":"2024-03-11T22:08:33.299778+0000","last_unstale":"2024-03-11T22:14:21.137772+0000","last_undegraded":"2024-03-11T22:14:21.137772+0000","last_fullsized":"2024-03-11T22:14:21.137772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:23.353379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:11:23.353379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":230,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138508+0000","last_change":"2024-03-11T22:12:24.093258+0000","last_active":"2024-03-11T22:14:21.138508+0000","last_peered":"2024-03-11T22:14:21.138508+0000","last_clean":"2024-03-11T22:14:21.138508+0000","last_became_active":"2024-03-11T22:08:35.053123+0000","last_became_peered":"2024-03-11T22:08:35.053123+0000","last_unstale":"2024-03-11T22:14:21.138508+0000","last_undegraded":"2024-03-11T22:14:21.138508+0000","last_fullsized":"2024-03-11T22:14:21.138508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:12:24.093112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.093112+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":480,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335037+0000","last_change":"2024-03-11T22:14:11.673622+0000","last_active":"2024-03-11T22:14:21.335037+0000","last_peered":"2024-03-11T22:14:21.335037+0000","last_clean":"2024-03-11T22:14:21.335037+0000","last_became_active":"2024-03-11T22:13:38.569606+0000","last_became_peered":"2024-03-11T22:13:38.569606+0000","last_unstale":"2024-03-11T22:14:21.335037+0000","last_undegraded":"2024-03-11T22:14:21.335037+0000","last_fullsized":"2024-03-11T22:14:21.335037+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:11.673454+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:11.673454+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:21.642510+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"160'16920","reported_seq":17227,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.529969+0000","last_change":"2024-03-11T22:13:52.384647+0000","last_active":"2024-03-11T22:14:25.529969+0000","last_peered":"2024-03-11T22:14:25.529969+0000","last_clean":"2024-03-11T22:14:25.529969+0000","last_became_active":"2024-03-11T22:10:36.412847+0000","last_became_peered":"2024-03-11T22:10:36.412847+0000","last_unstale":"2024-03-11T22:14:25.529969+0000","last_undegraded":"2024-03-11T22:14:25.529969+0000","last_fullsized":"2024-03-11T22:14:25.529969+0000","mapping_epoch":99,"log_start":"140'10897","ondisk_log_start":"140'10897","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'15289","last_scrub_stamp":"2024-03-11T22:13:52.384437+0000","last_deep_scrub":"150'15289","last_deep_scrub_stamp":"2024-03-11T22:13:52.384437+0000","last_clean_scrub_stamp":"2024-03-11T22:13:52.384437+0000","objects_scrubbed":2194,"log_size":6023,"log_dups_size":0,"ondisk_log_size":6023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":24,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:05.504049+0000","scrub_duration":23610,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14726,"num_write_kb":471232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":213,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.528818+0000","last_change":"2024-03-11T22:12:50.861675+0000","last_active":"2024-03-11T22:14:25.528818+0000","last_peered":"2024-03-11T22:14:25.528818+0000","last_clean":"2024-03-11T22:14:25.528818+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:14:25.528818+0000","last_undegraded":"2024-03-11T22:14:25.528818+0000","last_fullsized":"2024-03-11T22:14:25.528818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.861557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.861557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":215,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.336354+0000","last_change":"2024-03-11T22:14:18.859904+0000","last_active":"2024-03-11T22:14:21.336354+0000","last_peered":"2024-03-11T22:14:21.336354+0000","last_clean":"2024-03-11T22:14:21.336354+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:14:21.336354+0000","last_undegraded":"2024-03-11T22:14:21.336354+0000","last_fullsized":"2024-03-11T22:14:21.336354+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:18.859546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:14:18.859546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:41.507002+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":231,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.335722+0000","last_change":"2024-03-11T22:14:14.990482+0000","last_active":"2024-03-11T22:14:21.335722+0000","last_peered":"2024-03-11T22:14:21.335722+0000","last_clean":"2024-03-11T22:14:21.335722+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:14:21.335722+0000","last_undegraded":"2024-03-11T22:14:21.335722+0000","last_fullsized":"2024-03-11T22:14:21.335722+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.990368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.990368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:15.948091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":231,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138622+0000","last_change":"2024-03-11T22:11:17.270935+0000","last_active":"2024-03-11T22:14:21.138622+0000","last_peered":"2024-03-11T22:14:21.138622+0000","last_clean":"2024-03-11T22:14:21.138622+0000","last_became_active":"2024-03-11T22:08:35.055770+0000","last_became_peered":"2024-03-11T22:08:35.055770+0000","last_unstale":"2024-03-11T22:14:21.138622+0000","last_undegraded":"2024-03-11T22:14:21.138622+0000","last_fullsized":"2024-03-11T22:14:21.138622+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-11T22:11:17.270813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:17.270813+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":480,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137873+0000","last_change":"2024-03-11T22:14:10.906814+0000","last_active":"2024-03-11T22:14:21.137873+0000","last_peered":"2024-03-11T22:14:21.137873+0000","last_clean":"2024-03-11T22:14:21.137873+0000","last_became_active":"2024-03-11T22:13:38.571540+0000","last_became_peered":"2024-03-11T22:13:38.571540+0000","last_unstale":"2024-03-11T22:14:21.137873+0000","last_undegraded":"2024-03-11T22:14:21.137873+0000","last_fullsized":"2024-03-11T22:14:21.137873+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:10.906610+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:10.906610+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:13.783665+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"160'14724","reported_seq":14900,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.140042+0000","last_change":"2024-03-11T22:13:19.688808+0000","last_active":"2024-03-11T22:14:21.140042+0000","last_peered":"2024-03-11T22:14:21.140042+0000","last_clean":"2024-03-11T22:14:21.140042+0000","last_became_active":"2024-03-11T22:10:36.408528+0000","last_became_peered":"2024-03-11T22:10:36.408528+0000","last_unstale":"2024-03-11T22:14:21.140042+0000","last_undegraded":"2024-03-11T22:14:21.140042+0000","last_fullsized":"2024-03-11T22:14:21.140042+0000","mapping_epoch":99,"log_start":"134'9544","ondisk_log_start":"134'9544","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'11496","last_scrub_stamp":"2024-03-11T22:13:19.688524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:35.397794+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.688524+0000","objects_scrubbed":3273,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:14:23.402103+0000","scrub_duration":1754,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14724,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.138680+0000","last_change":"2024-03-11T22:14:09.099561+0000","last_active":"2024-03-11T22:14:21.138680+0000","last_peered":"2024-03-11T22:14:21.138680+0000","last_clean":"2024-03-11T22:14:21.138680+0000","last_became_active":"2024-03-11T22:08:39.072422+0000","last_became_peered":"2024-03-11T22:08:39.072422+0000","last_unstale":"2024-03-11T22:14:21.138680+0000","last_undegraded":"2024-03-11T22:14:21.138680+0000","last_fullsized":"2024-03-11T22:14:21.138680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:09.099443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:09.099443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:25.679020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":297,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-03-11T22:14:25.528978+0000","last_change":"2024-03-11T22:12:43.819554+0000","last_active":"2024-03-11T22:14:25.528978+0000","last_peered":"2024-03-11T22:14:25.528978+0000","last_clean":"2024-03-11T22:14:25.528978+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:14:25.528978+0000","last_undegraded":"2024-03-11T22:14:25.528978+0000","last_fullsized":"2024-03-11T22:14:25.528978+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:12:43.819269+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_clean_scrub_stamp":"2024-03-11T22:12:43.819269+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":223,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:21.137824+0000","last_change":"2024-03-11T22:14:12.801583+0000","last_active":"2024-03-11T22:14:21.137824+0000","last_peered":"2024-03-11T22:14:21.137824+0000","last_clean":"2024-03-11T22:14:21.137824+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:14:21.137824+0000","last_undegraded":"2024-03-11T22:14:21.137824+0000","last_fullsized":"2024-03-11T22:14:21.137824+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:12.801470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:12.801470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:25.460969+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":226,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-03-11T22:14:24.847740+0000","last_change":"2024-03-11T22:14:24.847801+0000","last_active":"2024-03-11T22:14:24.847740+0000","last_peered":"2024-03-11T22:14:24.847740+0000","last_clean":"2024-03-11T22:14:24.847740+0000","last_became_active":"2024-03-11T22:08:35.056698+0000","last_became_peered":"2024-03-11T22:08:35.056698+0000","last_unstale":"2024-03-11T22:14:24.847740+0000","last_undegraded":"2024-03-11T22:14:24.847740+0000","last_fullsized":"2024-03-11T22:14:24.847740+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:14:24.847623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:24.847623+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:32.898695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":236938,"num_write_kb":7581953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84486,"ondisk_log_size":84486,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":319,"num_read_kb":314,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005131,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108764,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636388,"statfs":{"total":95995035648,"available":95883661312,"internally_reserved":0,"allocated":1265664,"data_stored":900749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":110098891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332176,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108428,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636724,"statfs":{"total":95995035648,"available":95884005376,"internally_reserved":0,"allocated":917504,"data_stored":572879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":122360,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93622792,"statfs":{"total":95995035648,"available":95869739008,"internally_reserved":0,"allocated":966656,"data_stored":573075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738480,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":154264,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93590888,"statfs":{"total":95995035648,"available":95837069312,"internally_reserved":0,"allocated":933888,"data_stored":572401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411429,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":136456,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93608696,"statfs":{"total":95995035648,"available":95855304704,"internally_reserved":0,"allocated":1179648,"data_stored":835393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":159024,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93586128,"statfs":{"total":95995035648,"available":95832195072,"internally_reserved":0,"allocated":1548288,"data_stored":1163753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":161282503},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920859,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104292,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93640860,"statfs":{"total":95995035648,"available":95888240640,"internally_reserved":0,"allocated":942080,"data_stored":572610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105839046},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659220,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109012,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636140,"statfs":{"total":95995035648,"available":95883407360,"internally_reserved":0,"allocated":1515520,"data_stored":1162520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:14:28.859 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-11T22:14:28.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-03-11T22:14:28.879 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:28.906 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:28.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:28.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:29.007 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:29.034 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:29.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:29.136 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:29.163 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:29.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:29.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:29.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:29.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:29.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:29.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:29.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:14:29.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:29.418 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:29.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:29.520 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:29.547 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:29.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:29.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:29.599 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:29.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:29.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:29.674 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:29.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:29.729 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[30178]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:14:29.729 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[30178]: 3.1f scrub starts 2024-03-11T22:14:29.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[30178]: 3.1f scrub ok 2024-03-11T22:14:29.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[30178]: pgmap v237: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-11T22:14:29.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:14:29.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:14:29.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[30178]: osdmap e164: 8 total, 8 up, 8 in 2024-03-11T22:14:29.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[30178]: 5.1b scrub starts 2024-03-11T22:14:29.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[30178]: 5.1b scrub ok 2024-03-11T22:14:29.730 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[39715]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:14:29.730 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[39715]: 3.1f scrub starts 2024-03-11T22:14:29.730 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[39715]: 3.1f scrub ok 2024-03-11T22:14:29.730 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[39715]: pgmap v237: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-11T22:14:29.730 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:14:29.731 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:14:29.731 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[39715]: osdmap e164: 8 total, 8 up, 8 in 2024-03-11T22:14:29.731 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[39715]: 5.1b scrub starts 2024-03-11T22:14:29.731 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:29 smithi045 ceph-mon[39715]: 5.1b scrub ok 2024-03-11T22:14:29.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:29.801 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:29.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:29.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:29 smithi144 ceph-mon[33984]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:14:29.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:29 smithi144 ceph-mon[33984]: 3.1f scrub starts 2024-03-11T22:14:29.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:29 smithi144 ceph-mon[33984]: 3.1f scrub ok 2024-03-11T22:14:29.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:29 smithi144 ceph-mon[33984]: pgmap v237: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-11T22:14:29.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:29 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:14:29.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:29 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:14:29.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:29 smithi144 ceph-mon[33984]: osdmap e164: 8 total, 8 up, 8 in 2024-03-11T22:14:29.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:29 smithi144 ceph-mon[33984]: 5.1b scrub starts 2024-03-11T22:14:29.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:29 smithi144 ceph-mon[33984]: 5.1b scrub ok 2024-03-11T22:14:29.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:29.925 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:29.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:29.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:30.027 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:30.056 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:30.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:30.157 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:30.167 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:30.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:30.186 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:30.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:30.287 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:30.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:30.316 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:30.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:30.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:30.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:30.441 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:30.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:30.542 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:30.573 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:30.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:30.654 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[30178]: 4.2 scrub starts 2024-03-11T22:14:30.654 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[30178]: 4.2 scrub ok 2024-03-11T22:14:30.654 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[30178]: 5.d scrub starts 2024-03-11T22:14:30.654 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[30178]: 5.d scrub ok 2024-03-11T22:14:30.654 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:30.654 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:14:30.654 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[30178]: 5.7 scrub starts 2024-03-11T22:14:30.654 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[30178]: 5.7 scrub ok 2024-03-11T22:14:30.654 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[39715]: 4.2 scrub starts 2024-03-11T22:14:30.654 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[39715]: 4.2 scrub ok 2024-03-11T22:14:30.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[39715]: 5.d scrub starts 2024-03-11T22:14:30.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[39715]: 5.d scrub ok 2024-03-11T22:14:30.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:30.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:14:30.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[39715]: 5.7 scrub starts 2024-03-11T22:14:30.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:30 smithi045 ceph-mon[39715]: 5.7 scrub ok 2024-03-11T22:14:30.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:30.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:30.700 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:30.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:30.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:30.825 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:30.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:30.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:30 smithi144 ceph-mon[33984]: 4.2 scrub starts 2024-03-11T22:14:30.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:30 smithi144 ceph-mon[33984]: 4.2 scrub ok 2024-03-11T22:14:30.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:30 smithi144 ceph-mon[33984]: 5.d scrub starts 2024-03-11T22:14:30.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:30 smithi144 ceph-mon[33984]: 5.d scrub ok 2024-03-11T22:14:30.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:30.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:14:30.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:30 smithi144 ceph-mon[33984]: 5.7 scrub starts 2024-03-11T22:14:30.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:30 smithi144 ceph-mon[33984]: 5.7 scrub ok 2024-03-11T22:14:30.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:30.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:30.950 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:30.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:31.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:31.072 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:14:31.072 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:14:31.072 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:14:31.072 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:14:31.074 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:31.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:31.086 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:14:31.106 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:14:31.127 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:14:31.151 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:14:31.152 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:14:31.152 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:14:31.152 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:14:31.153 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:14:31.153 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:14:31.153 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:14:31.153 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:14:31.153 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:14:31.176 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:31.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:31.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:31.303 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:31.331 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:31.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:31.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:31.433 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:31.460 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:31.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:31.562 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:31.591 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:31.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:31.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:31.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:31.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:31 smithi045 ceph-mon[39715]: pgmap v239: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:14:31.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:31 smithi045 ceph-mon[39715]: 3.1c deep-scrub starts 2024-03-11T22:14:31.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:31 smithi045 ceph-mon[39715]: 3.1c deep-scrub ok 2024-03-11T22:14:31.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:31 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3356866921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:31.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:31 smithi045 ceph-mon[39715]: osdmap e165: 8 total, 8 up, 8 in 2024-03-11T22:14:31.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:31 smithi045 ceph-mon[30178]: pgmap v239: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:14:31.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:31 smithi045 ceph-mon[30178]: 3.1c deep-scrub starts 2024-03-11T22:14:31.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:31 smithi045 ceph-mon[30178]: 3.1c deep-scrub ok 2024-03-11T22:14:31.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:31 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3356866921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:31.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:31 smithi045 ceph-mon[30178]: osdmap e165: 8 total, 8 up, 8 in 2024-03-11T22:14:31.818 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:31.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:31 smithi144 ceph-mon[33984]: pgmap v239: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:14:31.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:31 smithi144 ceph-mon[33984]: 3.1c deep-scrub starts 2024-03-11T22:14:31.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:31 smithi144 ceph-mon[33984]: 3.1c deep-scrub ok 2024-03-11T22:14:31.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:31 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3356866921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:31.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:31 smithi144 ceph-mon[33984]: osdmap e165: 8 total, 8 up, 8 in 2024-03-11T22:14:31.847 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:31.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:31.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:31.949 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:31.977 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:31.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:32.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:32.107 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:32.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:32.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:32.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:32.233 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:32.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:32.269 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:32.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:32.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:32.359 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:32.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:32.460 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:32.488 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:32.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:32.590 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:32.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:32.619 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:32.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:32.683 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:14:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:14:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:14:32.720 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:32.748 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:32.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:32.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:32.875 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:32.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:32.976 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:33.004 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:33.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:33.105 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:33.133 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:33.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:33.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:33.235 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:33.264 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:33.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:33.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:33.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:33.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:33.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:33.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:33.520 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:33.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:33.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:33.622 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:33.646 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:33.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:33.748 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:33.776 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:33.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:33.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[39715]: 3.1b scrub starts 2024-03-11T22:14:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[39715]: 3.1b scrub ok 2024-03-11T22:14:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[39715]: pgmap v241: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-11T22:14:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[39715]: 4.13 scrub starts 2024-03-11T22:14:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[39715]: 4.13 scrub ok 2024-03-11T22:14:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:14:33.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:14:33.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[30178]: 3.1b scrub starts 2024-03-11T22:14:33.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[30178]: 3.1b scrub ok 2024-03-11T22:14:33.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[30178]: pgmap v241: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-11T22:14:33.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[30178]: 4.13 scrub starts 2024-03-11T22:14:33.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[30178]: 4.13 scrub ok 2024-03-11T22:14:33.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:14:33.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:33 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:14:33.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:33 smithi144 ceph-mon[33984]: 3.1b scrub starts 2024-03-11T22:14:33.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:33 smithi144 ceph-mon[33984]: 3.1b scrub ok 2024-03-11T22:14:33.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:33 smithi144 ceph-mon[33984]: pgmap v241: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-11T22:14:33.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:33 smithi144 ceph-mon[33984]: 4.13 scrub starts 2024-03-11T22:14:33.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:33 smithi144 ceph-mon[33984]: 4.13 scrub ok 2024-03-11T22:14:33.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:33 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:14:33.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:33 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:14:33.878 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:33.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:33.904 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:33.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:34.005 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:34.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:34.035 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:34.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:34.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:34.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:34.160 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:34.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:34.262 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:34.290 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:34.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:34.392 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:34.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:34.421 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:34.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:34.522 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:34.550 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:34.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:34.652 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:34.679 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:34.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:34.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:14:34.738 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[39715]: 3.12 scrub starts 2024-03-11T22:14:34.738 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[39715]: 3.12 scrub ok 2024-03-11T22:14:34.738 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:14:34.738 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:14:34.738 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[39715]: 6.3 scrub starts 2024-03-11T22:14:34.738 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[39715]: 6.3 scrub ok 2024-03-11T22:14:34.738 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[39715]: osdmap e166: 8 total, 8 up, 8 in 2024-03-11T22:14:34.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[30178]: 3.12 scrub starts 2024-03-11T22:14:34.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[30178]: 3.12 scrub ok 2024-03-11T22:14:34.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:14:34.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:14:34.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[30178]: 6.3 scrub starts 2024-03-11T22:14:34.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[30178]: 6.3 scrub ok 2024-03-11T22:14:34.739 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:34 smithi045 ceph-mon[30178]: osdmap e166: 8 total, 8 up, 8 in 2024-03-11T22:14:34.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:34.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:34.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:34.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:34.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:34 smithi144 ceph-mon[33984]: 3.12 scrub starts 2024-03-11T22:14:34.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:34 smithi144 ceph-mon[33984]: 3.12 scrub ok 2024-03-11T22:14:34.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:34 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:14:34.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:34 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:14:34.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:34 smithi144 ceph-mon[33984]: 6.3 scrub starts 2024-03-11T22:14:34.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:34 smithi144 ceph-mon[33984]: 6.3 scrub ok 2024-03-11T22:14:34.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:34 smithi144 ceph-mon[33984]: osdmap e166: 8 total, 8 up, 8 in 2024-03-11T22:14:34.907 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:34.935 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:34.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:35.036 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:35.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:35.065 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:35.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:35.166 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:35.194 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:35.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:35.295 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:35.323 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:35.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:35.332 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:14:35.332 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:14:35.332 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-03-11T22:14:35.332 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-03-11T22:14:35.332 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:14:35.425 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:35.452 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:35.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:35.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:35.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:35.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:35.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:35.589 INFO:teuthology.orchestra.run.smithi045.stderr:set pool 1 pg_num to 21 2024-03-11T22:14:35.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:35.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:35.704 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:35.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:35.805 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:35.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:35.835 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:35.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:35.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:35 smithi144 ceph-mon[33984]: 6.5 deep-scrub starts 2024-03-11T22:14:35.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:35 smithi144 ceph-mon[33984]: 6.5 deep-scrub ok 2024-03-11T22:14:35.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:35 smithi144 ceph-mon[33984]: pgmap v243: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:14:35.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:35 smithi144 ceph-mon[33984]: 3.d scrub starts 2024-03-11T22:14:35.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:35 smithi144 ceph-mon[33984]: 3.d scrub ok 2024-03-11T22:14:35.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:35 smithi144 ceph-mon[33984]: 5.1c scrub starts 2024-03-11T22:14:35.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:35 smithi144 ceph-mon[33984]: 5.1c scrub ok 2024-03-11T22:14:35.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:35 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3720940887' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T22:14:35.936 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:35.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:35.961 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:35.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[39715]: 6.5 deep-scrub starts 2024-03-11T22:14:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[39715]: 6.5 deep-scrub ok 2024-03-11T22:14:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[39715]: pgmap v243: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:14:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[39715]: 3.d scrub starts 2024-03-11T22:14:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[39715]: 3.d scrub ok 2024-03-11T22:14:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[39715]: 5.1c scrub starts 2024-03-11T22:14:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[39715]: 5.1c scrub ok 2024-03-11T22:14:36.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3720940887' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T22:14:36.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[30178]: 6.5 deep-scrub starts 2024-03-11T22:14:36.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[30178]: 6.5 deep-scrub ok 2024-03-11T22:14:36.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[30178]: pgmap v243: 158 pgs: 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:14:36.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[30178]: 3.d scrub starts 2024-03-11T22:14:36.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[30178]: 3.d scrub ok 2024-03-11T22:14:36.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[30178]: 5.1c scrub starts 2024-03-11T22:14:36.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[30178]: 5.1c scrub ok 2024-03-11T22:14:36.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:35 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3720940887' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T22:14:36.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:36.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:36.088 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:36.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:36.189 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:36.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:36.215 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:36.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:36.262 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:14:35 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:14:36.316 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:36.341 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:36.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:36.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:36.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:36.468 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:36.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:36.569 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:36.579 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-11T22:14:36.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-11T22:14:36.592 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:36.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:36.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:36.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:36.718 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:36.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:36.784 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:36.819 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:36.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:36 smithi144 ceph-mon[33984]: 3.1 scrub starts 2024-03-11T22:14:36.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:36 smithi144 ceph-mon[33984]: 3.1 scrub ok 2024-03-11T22:14:36.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:36 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3720940887' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-11T22:14:36.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:36 smithi144 ceph-mon[33984]: osdmap e167: 8 total, 8 up, 8 in 2024-03-11T22:14:36.844 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:36 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T22:14:36.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:36.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:36.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:36.889 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:36 smithi045 ceph-mon[39715]: 3.1 scrub starts 2024-03-11T22:14:36.889 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:36 smithi045 ceph-mon[39715]: 3.1 scrub ok 2024-03-11T22:14:36.889 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:36 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3720940887' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-11T22:14:36.889 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:36 smithi045 ceph-mon[39715]: osdmap e167: 8 total, 8 up, 8 in 2024-03-11T22:14:36.889 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:36 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T22:14:36.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:36 smithi045 ceph-mon[30178]: 3.1 scrub starts 2024-03-11T22:14:36.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:36 smithi045 ceph-mon[30178]: 3.1 scrub ok 2024-03-11T22:14:36.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:36 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3720940887' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-11T22:14:36.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:36 smithi045 ceph-mon[30178]: osdmap e167: 8 total, 8 up, 8 in 2024-03-11T22:14:36.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:36 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T22:14:36.945 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:36.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:36.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:36.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:37.071 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:37.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:37.096 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:37.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:37.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:37.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:37.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:37.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:37.322 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:37.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:37.347 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:37.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:37.448 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:37.473 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:37.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:37.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:37.599 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:37.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:37.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:37.724 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:37.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:37.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[30178]: 6.9 scrub starts 2024-03-11T22:14:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[30178]: 6.9 scrub ok 2024-03-11T22:14:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[30178]: 3.4 scrub starts 2024-03-11T22:14:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[30178]: 3.4 scrub ok 2024-03-11T22:14:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[30178]: 5.9 scrub starts 2024-03-11T22:14:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[30178]: 5.9 scrub ok 2024-03-11T22:14:37.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[30178]: pgmap v245: 158 pgs: 158 active+clean; 582 KiB data, 980 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:14:37.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T22:14:37.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[30178]: osdmap e168: 8 total, 8 up, 8 in 2024-03-11T22:14:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[39715]: 6.9 scrub starts 2024-03-11T22:14:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[39715]: 6.9 scrub ok 2024-03-11T22:14:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[39715]: 3.4 scrub starts 2024-03-11T22:14:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[39715]: 3.4 scrub ok 2024-03-11T22:14:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[39715]: 5.9 scrub starts 2024-03-11T22:14:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[39715]: 5.9 scrub ok 2024-03-11T22:14:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[39715]: pgmap v245: 158 pgs: 158 active+clean; 582 KiB data, 980 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:14:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T22:14:37.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:37 smithi045 ceph-mon[39715]: osdmap e168: 8 total, 8 up, 8 in 2024-03-11T22:14:37.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:37 smithi144 ceph-mon[33984]: 6.9 scrub starts 2024-03-11T22:14:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:37 smithi144 ceph-mon[33984]: 6.9 scrub ok 2024-03-11T22:14:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:37 smithi144 ceph-mon[33984]: 3.4 scrub starts 2024-03-11T22:14:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:37 smithi144 ceph-mon[33984]: 3.4 scrub ok 2024-03-11T22:14:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:37 smithi144 ceph-mon[33984]: 5.9 scrub starts 2024-03-11T22:14:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:37 smithi144 ceph-mon[33984]: 5.9 scrub ok 2024-03-11T22:14:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:37 smithi144 ceph-mon[33984]: pgmap v245: 158 pgs: 158 active+clean; 582 KiB data, 980 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:14:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:37 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T22:14:37.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:37 smithi144 ceph-mon[33984]: osdmap e168: 8 total, 8 up, 8 in 2024-03-11T22:14:37.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:37.851 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:37.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:37.953 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:37.977 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:37.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:37.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:38.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T22:14:38.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:38.087 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:38.106 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:38.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:38.165 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:14:38.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:38.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:38.232 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:38.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:38.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:38.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:38.359 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:38.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:38.460 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:38.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:38.486 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:38.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:38.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:38.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:38.612 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:38.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:38.691 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[30178]: 4.5 scrub starts 2024-03-11T22:14:38.691 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[30178]: 4.5 scrub ok 2024-03-11T22:14:38.691 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[30178]: 3.c scrub starts 2024-03-11T22:14:38.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[30178]: 3.c scrub ok 2024-03-11T22:14:38.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1492920865' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:14:38.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[30178]: 3.8 scrub starts 2024-03-11T22:14:38.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[30178]: 3.8 scrub ok 2024-03-11T22:14:38.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:14:38.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1492920865' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:14:38.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[30178]: osdmap e169: 8 total, 8 up, 8 in 2024-03-11T22:14:38.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T22:14:38.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[39715]: 4.5 scrub starts 2024-03-11T22:14:38.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[39715]: 4.5 scrub ok 2024-03-11T22:14:38.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[39715]: 3.c scrub starts 2024-03-11T22:14:38.692 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[39715]: 3.c scrub ok 2024-03-11T22:14:38.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1492920865' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:14:38.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[39715]: 3.8 scrub starts 2024-03-11T22:14:38.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[39715]: 3.8 scrub ok 2024-03-11T22:14:38.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:14:38.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1492920865' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:14:38.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[39715]: osdmap e169: 8 total, 8 up, 8 in 2024-03-11T22:14:38.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:38 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T22:14:38.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:38.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:38.738 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:38.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:38.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:38.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:38 smithi144 ceph-mon[33984]: 4.5 scrub starts 2024-03-11T22:14:38.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:38 smithi144 ceph-mon[33984]: 4.5 scrub ok 2024-03-11T22:14:38.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:38 smithi144 ceph-mon[33984]: 3.c scrub starts 2024-03-11T22:14:38.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:38 smithi144 ceph-mon[33984]: 3.c scrub ok 2024-03-11T22:14:38.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:38 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1492920865' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:14:38.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:38 smithi144 ceph-mon[33984]: 3.8 scrub starts 2024-03-11T22:14:38.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:38 smithi144 ceph-mon[33984]: 3.8 scrub ok 2024-03-11T22:14:38.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:38 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:14:38.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:38 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1492920865' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:14:38.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:38 smithi144 ceph-mon[33984]: osdmap e169: 8 total, 8 up, 8 in 2024-03-11T22:14:38.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:38 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T22:14:38.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:38.863 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:38.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:38.964 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:38.991 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:38.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:39.092 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:39.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:39.116 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:39.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:39.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:39.241 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:39.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:39.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:39.368 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:39.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:39.470 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:39.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:39.498 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:39.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:39.599 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:39.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:39.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:39.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:39.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:39.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:39.751 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:39.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:39.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:39 smithi045 ceph-mon[39715]: 6.4 scrub starts 2024-03-11T22:14:39.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:39 smithi045 ceph-mon[39715]: 6.4 scrub ok 2024-03-11T22:14:39.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:39 smithi045 ceph-mon[39715]: pgmap v248: 168 pgs: 10 unknown, 158 active+clean; 582 KiB data, 980 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:14:39.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:39 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-11T22:14:39.779 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:39 smithi045 ceph-mon[39715]: osdmap e170: 8 total, 8 up, 8 in 2024-03-11T22:14:39.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:39 smithi045 ceph-mon[30178]: 6.4 scrub starts 2024-03-11T22:14:39.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:39 smithi045 ceph-mon[30178]: 6.4 scrub ok 2024-03-11T22:14:39.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:39 smithi045 ceph-mon[30178]: pgmap v248: 168 pgs: 10 unknown, 158 active+clean; 582 KiB data, 980 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:14:39.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:39 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-11T22:14:39.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:39 smithi045 ceph-mon[30178]: osdmap e170: 8 total, 8 up, 8 in 2024-03-11T22:14:39.852 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:39.881 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:39.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:39.893 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:39.983 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:40.010 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:40.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:40.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:39 smithi144 ceph-mon[33984]: 6.4 scrub starts 2024-03-11T22:14:40.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:39 smithi144 ceph-mon[33984]: 6.4 scrub ok 2024-03-11T22:14:40.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:39 smithi144 ceph-mon[33984]: pgmap v248: 168 pgs: 10 unknown, 158 active+clean; 582 KiB data, 980 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:14:40.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:39 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-11T22:14:40.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:39 smithi144 ceph-mon[33984]: osdmap e170: 8 total, 8 up, 8 in 2024-03-11T22:14:40.112 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:40.139 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:40.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:40.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:40.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:40.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:40.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:40.367 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:40.392 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:40.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:40.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:40.520 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:40.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:40.622 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:40.649 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:40.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:40.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:40.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:40.775 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:40.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:40.877 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:40.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:40.907 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:40.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:41.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:41.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:41.033 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:41.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:41.134 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:41.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:41.169 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:41.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:41.270 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:41.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:41.295 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:41.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:41.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:41.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:41.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:41.467 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:41 smithi045 ceph-mon[39715]: pgmap v251: 168 pgs: 10 unknown, 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail 2024-03-11T22:14:41.467 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:41 smithi045 ceph-mon[39715]: osdmap e171: 8 total, 8 up, 8 in 2024-03-11T22:14:41.467 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:41 smithi045 ceph-mon[30178]: pgmap v251: 168 pgs: 10 unknown, 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail 2024-03-11T22:14:41.467 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:41 smithi045 ceph-mon[30178]: osdmap e171: 8 total, 8 up, 8 in 2024-03-11T22:14:41.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:14:41.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:41.545 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:41.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:41.580 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:14:41.580 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:14:41.580 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-11T22:14:41.580 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2024-03-11T22:14:41.581 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 4 0 2024-03-11T22:14:41.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:41 smithi144 ceph-mon[33984]: pgmap v251: 168 pgs: 10 unknown, 158 active+clean; 582 KiB data, 979 MiB used, 714 GiB / 715 GiB avail 2024-03-11T22:14:41.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:41 smithi144 ceph-mon[33984]: osdmap e171: 8 total, 8 up, 8 in 2024-03-11T22:14:41.646 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:41.675 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:41.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:41.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:41.802 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:41.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:41.903 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:41.931 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:41.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:42.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:42.058 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:42.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:42.159 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:42.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:42 smithi045 ceph-mon[30178]: osdmap e172: 8 total, 8 up, 8 in 2024-03-11T22:14:42.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:42 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/886690467' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:14:42.188 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'unique_pool_0' removed 2024-03-11T22:14:42.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:42.194 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:42.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:42.295 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:42.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:42.324 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:42.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:42.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:42.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:42.450 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:42.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:42 smithi045 ceph-mon[39715]: osdmap e172: 8 total, 8 up, 8 in 2024-03-11T22:14:42.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:42 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/886690467' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:14:42.551 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:42.577 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:42.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:42.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:42 smithi144 ceph-mon[33984]: osdmap e172: 8 total, 8 up, 8 in 2024-03-11T22:14:42.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:42 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/886690467' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:14:42.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:42.701 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:42.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:42.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:42.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:42.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:42.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:42.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:42.952 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:42.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:42.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:43.026 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:14:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:14:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:14:43.053 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:43.055 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T22:14:43.055 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T22:14:43.056 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-11T22:14:43.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-11T22:14:43.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:43.082 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:43.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:43.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:43.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:43.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:43.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:43.279 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:43 smithi045 ceph-mon[39715]: pgmap v253: 168 pgs: 9 remapped+peering, 1 unknown, 158 active+clean; 609 KiB data, 978 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 5 op/s; 20 KiB/s, 0 objects/s recovering 2024-03-11T22:14:43.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:43 smithi045 ceph-mon[39715]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-11T22:14:43.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:43 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/886690467' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:14:43.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:43 smithi045 ceph-mon[39715]: osdmap e173: 8 total, 8 up, 8 in 2024-03-11T22:14:43.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:43 smithi045 ceph-mon[30178]: pgmap v253: 168 pgs: 9 remapped+peering, 1 unknown, 158 active+clean; 609 KiB data, 978 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 5 op/s; 20 KiB/s, 0 objects/s recovering 2024-03-11T22:14:43.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:43 smithi045 ceph-mon[30178]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-11T22:14:43.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:43 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/886690467' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:14:43.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:43 smithi045 ceph-mon[30178]: osdmap e173: 8 total, 8 up, 8 in 2024-03-11T22:14:43.319 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:43.344 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:43.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:43.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:43.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:43.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:43.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:43.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:43.577 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:14:43.577 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-03-11T22:14:43.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-03-11T22:14:43.577 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:14:43.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:06:14.230307+0000", 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 509.34760738799997, 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.86020543199999999, 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14158", 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3509006862", 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 1 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230307+0000", 2024-03-11T22:14:43.579 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.579 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230307+0000", 2024-03-11T22:14:43.579 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.579 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230309+0000", 2024-03-11T22:14:43.579 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.714e-06 2024-03-11T22:14:43.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230315+0000", 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.353e-06 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230326+0000", 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0332e-05 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230336+0000", 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0094e-05 2024-03-11T22:14:43.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230394+0000", 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.842e-05 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:14.230403+0000", 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4230000000000005e-06 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.089270+0000", 2024-03-11T22:14:43.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.85886745099999995 2024-03-11T22:14:43.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.582 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.089427+0000", 2024-03-11T22:14:43.582 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000157069 2024-03-11T22:14:43.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,2", 2024-03-11T22:14:43.582 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.089631+0000", 2024-03-11T22:14:43.582 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000203649 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.090353+0000", 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00072196000000000001 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.090390+0000", 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7076999999999998e-05 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.090410+0000", 2024-03-11T22:14:43.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0588e-05 2024-03-11T22:14:43.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.090463+0000", 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2340999999999999e-05 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:06:15.090513+0000", 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9960999999999998e-05 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24326.0:4 3.0 3:00000000::::head [pgnls start_epoch 68] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e68)", 2024-03-11T22:14:43.585 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:08:33.043301+0000", 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 370.53461379800001, 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25786676200000003, 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24326", 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2228576527", 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 4 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043301+0000", 2024-03-11T22:14:43.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043301+0000", 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043302+0000", 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1209999999999999e-06 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043306+0000", 2024-03-11T22:14:43.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3769999999999998e-06 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043314+0000", 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0700000000000007e-06 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043324+0000", 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5529999999999998e-06 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043371+0000", 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7068999999999999e-05 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043380+0000", 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5229999999999995e-06 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.300840+0000", 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.257460668 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.300879+0000", 2024-03-11T22:14:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8714000000000002e-05 2024-03-11T22:14:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.301168+0000", 2024-03-11T22:14:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028866699999999999 2024-03-11T22:14:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.590 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24403.0:109 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e98)", 2024-03-11T22:14:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:34.741322+0000", 2024-03-11T22:14:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 248.83659231199999, 2024-03-11T22:14:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.67899541299999999, 2024-03-11T22:14:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:14:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24403", 2024-03-11T22:14:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.144:0/840743586", 2024-03-11T22:14:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 109 2024-03-11T22:14:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741322+0000", 2024-03-11T22:14:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741322+0000", 2024-03-11T22:14:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741324+0000", 2024-03-11T22:14:43.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6279999999999999e-06 2024-03-11T22:14:43.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.594 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.594 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741331+0000", 2024-03-11T22:14:43.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8449999999999997e-06 2024-03-11T22:14:43.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741340+0000", 2024-03-11T22:14:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6190000000000006e-06 2024-03-11T22:14:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741351+0000", 2024-03-11T22:14:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0372e-05 2024-03-11T22:14:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741431+0000", 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0368e-05 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741444+0000", 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2971e-05 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:35.420168+0000", 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.67872427000000002 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:35.420267+0000", 2024-03-11T22:14:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8834999999999998e-05 2024-03-11T22:14:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:35.420318+0000", 2024-03-11T22:14:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0504999999999998e-05 2024-03-11T22:14:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:41626 7.b 7:d90bee90:::benchmark_data_smithi045_83007_object41625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:11:03.868731+0000", 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 219.709183983, 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.112628617, 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 41626 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868731+0000", 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868731+0000", 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868732+0000", 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.793e-06 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868758+0000", 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5933000000000001e-05 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868778+0000", 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9499000000000001e-05 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868787+0000", 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4630000000000007e-06 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868863+0000", 2024-03-11T22:14:43.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5364999999999994e-05 2024-03-11T22:14:43.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.868957+0000", 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4256e-05 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.869046+0000", 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8976999999999999e-05 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872439+0000", 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033925940000000001 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.603 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.876086+0000", 2024-03-11T22:14:43.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003647127 2024-03-11T22:14:43.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.981251+0000", 2024-03-11T22:14:43.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105165353 2024-03-11T22:14:43.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.981310+0000", 2024-03-11T22:14:43.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9333000000000003e-05 2024-03-11T22:14:43.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.606 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.606 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.606 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.981359+0000", 2024-03-11T22:14:43.606 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8924000000000001e-05 2024-03-11T22:14:43.606 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.606 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:41631 7.b 7:d85749de:::benchmark_data_smithi045_83007_object41630:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:11:03.872638+0000", 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 219.705276599, 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.113385656, 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 41631 2024-03-11T22:14:43.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872638+0000", 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872638+0000", 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872640+0000", 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.556e-06 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872665+0000", 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5123999999999999e-05 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872680+0000", 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5719999999999999e-05 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.609 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872689+0000", 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.123e-06 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872753+0000", 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4731000000000001e-05 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.872926+0000", 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017301600000000001 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.874603+0000", 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016767819999999999 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.876181+0000", 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015781809999999999 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.877689+0000", 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001507944 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.985762+0000", 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10807291400000001 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.985940+0000", 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017804499999999999 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:03.986024+0000", 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.352e-05 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.612 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:104387 7.b 7:df0bad5e:::benchmark_data_smithi045_83007_object104386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:11:59.150464+0000", 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.427450534, 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15740042900000001, 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 104387 2024-03-11T22:14:43.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.150464+0000", 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.150464+0000", 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.150466+0000", 2024-03-11T22:14:43.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.61e-06 2024-03-11T22:14:43.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.150490+0000", 2024-03-11T22:14:43.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3969999999999999e-05 2024-03-11T22:14:43.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.150503+0000", 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3179999999999999e-05 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.150512+0000", 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0469999999999999e-06 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.157740+0000", 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072285049999999997 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.157883+0000", 2024-03-11T22:14:43.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014308800000000001 2024-03-11T22:14:43.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:14:43.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.158034+0000", 2024-03-11T22:14:43.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015051800000000001 2024-03-11T22:14:43.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.163788+0000", 2024-03-11T22:14:43.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0057538609999999999 2024-03-11T22:14:43.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307382+0000", 2024-03-11T22:14:43.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14359450400000001 2024-03-11T22:14:43.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307583+0000", 2024-03-11T22:14:43.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020014100000000001 2024-03-11T22:14:43.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307740+0000", 2024-03-11T22:14:43.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000157353 2024-03-11T22:14:43.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307865+0000", 2024-03-11T22:14:43.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000124652 2024-03-11T22:14:43.619 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.619 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.619 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:104397 7.b 7:d952fc32:::benchmark_data_smithi045_83007_object104396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:11:59.155411+0000", 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.422503586, 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15261744899999999, 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 104397 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.155411+0000", 2024-03-11T22:14:43.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.155411+0000", 2024-03-11T22:14:43.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.155413+0000", 2024-03-11T22:14:43.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6640000000000001e-06 2024-03-11T22:14:43.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.155437+0000", 2024-03-11T22:14:43.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4524000000000001e-05 2024-03-11T22:14:43.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.155452+0000", 2024-03-11T22:14:43.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4314999999999999e-05 2024-03-11T22:14:43.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.155460+0000", 2024-03-11T22:14:43.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5080000000000005e-06 2024-03-11T22:14:43.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.160887+0000", 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0054268509999999999 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.161013+0000", 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012621400000000001 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.161174+0000", 2024-03-11T22:14:43.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016122099999999999 2024-03-11T22:14:43.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.163920+0000", 2024-03-11T22:14:43.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002745482 2024-03-11T22:14:43.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307532+0000", 2024-03-11T22:14:43.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14361226299999999 2024-03-11T22:14:43.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307914+0000", 2024-03-11T22:14:43.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00038154600000000002 2024-03-11T22:14:43.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307973+0000", 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9379999999999997e-05 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.308028+0000", 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5481e-05 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.627 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:104387 7.b e124/99 ondisk, result = 0)", 2024-03-11T22:14:43.628 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:11:59.172734+0000", 2024-03-11T22:14:43.628 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 164.405180675, 2024-03-11T22:14:43.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13475810999999999, 2024-03-11T22:14:43.628 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.629 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:14:43.629 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.172734+0000", 2024-03-11T22:14:43.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.172734+0000", 2024-03-11T22:14:43.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.172735+0000", 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4000000000000001e-07 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.172739+0000", 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6079999999999998e-06 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.252931+0000", 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.080191585999999995 2024-03-11T22:14:43.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.252947+0000", 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6714e-05 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307373+0000", 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.054425306999999999 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307378+0000", 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2619999999999998e-06 2024-03-11T22:14:43.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307492+0000", 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113993 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:105476 7.b 7:d7bdd260:::benchmark_data_smithi045_83007_object105475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:00.338837+0000", 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.23907772800001, 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13040384299999999, 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.632 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 105476 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.338837+0000", 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.338837+0000", 2024-03-11T22:14:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.338838+0000", 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.226e-06 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.338860+0000", 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1418000000000001e-05 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.338869+0000", 2024-03-11T22:14:43.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6390000000000004e-06 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.338876+0000", 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7599999999999997e-06 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.423952+0000", 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.085076499999999999 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.424062+0000", 2024-03-11T22:14:43.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000109219 2024-03-11T22:14:43.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:14:43.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.424207+0000", 2024-03-11T22:14:43.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014562200000000001 2024-03-11T22:14:43.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.430914+0000", 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0067071889999999997 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.430958+0000", 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3109999999999999e-05 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.469051+0000", 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038093082 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.469167+0000", 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116786 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.469241+0000", 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3292000000000002e-05 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:105477 7.b 7:d80a290b:::benchmark_data_smithi045_83007_object105476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:14:43.638 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:00.339000+0000", 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.23891462700001, 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.130386109, 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 105477 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.339000+0000", 2024-03-11T22:14:43.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.339000+0000", 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.339001+0000", 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.44e-07 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.339020+0000", 2024-03-11T22:14:43.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9505999999999999e-05 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.339028+0000", 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8939999999999993e-06 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.339034+0000", 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7790000000000001e-06 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.426890+0000", 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.087855723999999996 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.427025+0000", 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013506600000000001 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.428229+0000", 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00120411 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.431049+0000", 2024-03-11T22:14:43.643 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028202689999999998 2024-03-11T22:14:43.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.643 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.643 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.431110+0000", 2024-03-11T22:14:43.643 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0888999999999999e-05 2024-03-11T22:14:43.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.643 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.643 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.469251+0000", 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038140652999999997 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.469336+0000", 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5028e-05 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.469386+0000", 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0547000000000002e-05 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:105612 7.b 7:d114f01c:::benchmark_data_smithi045_83007_object105611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:00.480208+0000", 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 163.09770642500001, 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.111537861, 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 105612 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.646 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.646 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.480208+0000", 2024-03-11T22:14:43.646 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.646 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.646 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.480208+0000", 2024-03-11T22:14:43.647 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.647 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.647 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.480210+0000", 2024-03-11T22:14:43.647 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.99e-06 2024-03-11T22:14:43.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.647 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.647 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.480230+0000", 2024-03-11T22:14:43.647 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9840999999999999e-05 2024-03-11T22:14:43.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.480239+0000", 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3710000000000006e-06 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.480246+0000", 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5579999999999997e-06 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.482332+0000", 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020858809999999999 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.482472+0000", 2024-03-11T22:14:43.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014006899999999999 2024-03-11T22:14:43.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:14:43.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.482652+0000", 2024-03-11T22:14:43.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018014699999999999 2024-03-11T22:14:43.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.570376+0000", 2024-03-11T22:14:43.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.087724332000000002 2024-03-11T22:14:43.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.570716+0000", 2024-03-11T22:14:43.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00033996299999999998 2024-03-11T22:14:43.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.651 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.591495+0000", 2024-03-11T22:14:43.651 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.020778909000000002 2024-03-11T22:14:43.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.651 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.651 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.591676+0000", 2024-03-11T22:14:43.651 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000180667 2024-03-11T22:14:43.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.651 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.651 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.591746+0000", 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0133000000000001e-05 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:106016 7.b 7:da0a578d:::benchmark_data_smithi045_83007_object106015:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:00.947814+0000", 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 162.630100328, 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.109945313, 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:43.652 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:43.653 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 106016 2024-03-11T22:14:43.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.653 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.653 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.653 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.947814+0000", 2024-03-11T22:14:43.653 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.653 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.947814+0000", 2024-03-11T22:14:43.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.654 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.947815+0000", 2024-03-11T22:14:43.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0399999999999996e-07 2024-03-11T22:14:43.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.654 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.947852+0000", 2024-03-11T22:14:43.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6859000000000001e-05 2024-03-11T22:14:43.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.655 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.655 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.947861+0000", 2024-03-11T22:14:43.655 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1470000000000007e-06 2024-03-11T22:14:43.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.655 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.655 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.947868+0000", 2024-03-11T22:14:43.655 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0670000000000003e-06 2024-03-11T22:14:43.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.656 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:43 smithi144 ceph-mon[33984]: pgmap v253: 168 pgs: 9 remapped+peering, 1 unknown, 158 active+clean; 609 KiB data, 978 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 5 op/s; 20 KiB/s, 0 objects/s recovering 2024-03-11T22:14:43.656 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:43 smithi144 ceph-mon[33984]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-11T22:14:43.656 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:43 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/886690467' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:14:43.656 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:43 smithi144 ceph-mon[33984]: osdmap e173: 8 total, 8 up, 8 in 2024-03-11T22:14:43.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.657 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.953610+0000", 2024-03-11T22:14:43.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0057424479999999998 2024-03-11T22:14:43.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.658 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.658 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.953764+0000", 2024-03-11T22:14:43.658 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015364299999999999 2024-03-11T22:14:43.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.658 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:14:43.658 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.954066+0000", 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00030176999999999998 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.960863+0000", 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0067974250000000002 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:01.055330+0000", 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.094467050999999996 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.659 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:01.057318+0000", 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019879429999999998 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:01.057607+0000", 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000288378 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:01.057760+0000", 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015297800000000001 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.660 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.661 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.661 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.661 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:106023 7.b 7:dc8d55d8:::benchmark_data_smithi045_83007_object106022:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:14:43.661 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:00.947881+0000", 2024-03-11T22:14:43.661 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 162.630033978, 2024-03-11T22:14:43.661 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11042603300000001, 2024-03-11T22:14:43.662 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.662 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.662 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.662 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:43.662 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:43.662 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 106023 2024-03-11T22:14:43.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.947881+0000", 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.947881+0000", 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.947881+0000", 2024-03-11T22:14:43.663 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0200000000000002e-07 2024-03-11T22:14:43.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.664 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.664 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.665 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.947902+0000", 2024-03-11T22:14:43.665 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0988e-05 2024-03-11T22:14:43.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.665 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.665 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.947909+0000", 2024-03-11T22:14:43.665 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.34e-06 2024-03-11T22:14:43.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.666 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.666 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.947915+0000", 2024-03-11T22:14:43.666 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7080000000000002e-06 2024-03-11T22:14:43.666 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.666 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.666 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.956952+0000", 2024-03-11T22:14:43.666 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090369969999999997 2024-03-11T22:14:43.666 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.666 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.666 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.957167+0000", 2024-03-11T22:14:43.666 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021477099999999999 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.957342+0000", 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000175217 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.960868+0000", 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035262620000000001 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:01.055944+0000", 2024-03-11T22:14:43.667 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.095075173999999998 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:01.057947+0000", 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002003473 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:01.058147+0000", 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000200044 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:01.058307+0000", 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000159557 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:107158 7.b 7:dc298b22:::benchmark_data_smithi045_83007_object107157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:02.232541+0000", 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 161.345373932, 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.12231336199999999, 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:43.669 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 107158 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232541+0000", 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232541+0000", 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.670 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232542+0000", 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7090000000000001e-06 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232565+0000", 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2538e-05 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232579+0000", 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3643e-05 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.671 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232587+0000", 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0940000000000008e-06 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232692+0000", 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000105221 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232820+0000", 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012834899999999999 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.673 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:14:43.673 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232970+0000", 2024-03-11T22:14:43.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014940099999999999 2024-03-11T22:14:43.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:43.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:43.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.676 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.353477+0000", 2024-03-11T22:14:43.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.120507527 2024-03-11T22:14:43.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.676 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.353546+0000", 2024-03-11T22:14:43.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8837999999999994e-05 2024-03-11T22:14:43.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.676 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.354700+0000", 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001153938 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.354777+0000", 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6934000000000006e-05 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.354854+0000", 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7169999999999995e-05 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:107161 7.b 7:dec0fd31:::benchmark_data_smithi045_83007_object107160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:02.232611+0000", 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 161.34530411200001, 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.122408901, 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 107161 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232611+0000", 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232611+0000", 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232611+0000", 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4199999999999996e-07 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232630+0000", 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.91e-05 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232638+0000", 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.498e-06 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.232644+0000", 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9259999999999997e-06 2024-03-11T22:14:43.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.237109+0000", 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044651179999999997 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.237236+0000", 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000126842 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.237383+0000", 2024-03-11T22:14:43.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000147062 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.353532+0000", 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.116149688 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.353551+0000", 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8576999999999998e-05 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.354904+0000", 2024-03-11T22:14:43.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001353489 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.354961+0000", 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6360999999999998e-05 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.355019+0000", 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8697999999999999e-05 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:107442 7.b 7:d1965d5c:::benchmark_data_smithi045_83007_object107441:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:02.572237+0000", 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 161.00567810499999, 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.12934298299999999, 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 107442 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.684 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.572237+0000", 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.572237+0000", 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.572238+0000", 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5370000000000001e-06 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.572262+0000", 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4124000000000001e-05 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.572274+0000", 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1953999999999999e-05 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.572283+0000", 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4849999999999992e-06 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.691256+0000", 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11897300199999999 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.691383+0000", 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012762599999999999 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.691540+0000", 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000157174 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.701355+0000", 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0098143090000000002 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.701410+0000", 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5498000000000003e-05 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.701453+0000", 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.32e-05 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.701515+0000", 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1812999999999999e-05 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.701579+0000", 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4261e-05 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.1.0:9157 7.b e124/99 ondisk, result = 0)", 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:02.572538+0000", 2024-03-11T22:14:43.689 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 161.005376633, 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11327952299999999, 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.572538+0000", 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.572538+0000", 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.572539+0000", 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0900000000000001e-07 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.572547+0000", 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.994e-06 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.572561+0000", 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4252e-05 2024-03-11T22:14:43.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.572568+0000", 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6959999999999999e-06 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.621843+0000", 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.049275476999999998 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.621850+0000", 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5590000000000002e-06 2024-03-11T22:14:43.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.685818+0000", 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.063967836 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.1.0:9160 7.b e124/99 ondisk, result = 0)", 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:02.580549+0000", 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 160.997365414, 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.112553639, 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.693 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.580549+0000", 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.580549+0000", 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.694 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.580549+0000", 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3900000000000001e-07 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.580551+0000", 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8700000000000001e-06 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.580559+0000", 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7160000000000003e-06 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.580563+0000", 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1110000000000001e-06 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.686147+0000", 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105584283 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.686154+0000", 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0169999999999999e-06 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.693103+0000", 2024-03-11T22:14:43.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0069484029999999997 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:107458 7.b 7:d7a40f63:::benchmark_data_smithi045_83007_object107457:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:02.590955+0000", 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 160.98695947900001, 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11077390199999999, 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:14:43.697 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 107458 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.590955+0000", 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.590955+0000", 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.590957+0000", 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7e-06 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.590978+0000", 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1233000000000002e-05 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.590990+0000", 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2109999999999999e-05 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.590998+0000", 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4959999999999999e-06 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.694117+0000", 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103119396 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.694247+0000", 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000129787 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.694410+0000", 2024-03-11T22:14:43.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016319699999999999 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.701398+0000", 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0069883059999999997 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.701416+0000", 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7261999999999998e-05 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.701624+0000", 2024-03-11T22:14:43.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020814799999999999 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.701678+0000", 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4364000000000003e-05 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:02.701729+0000", 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0902999999999997e-05 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.14790.0:229683 7.d e158/99 7:b3eb5a94:::benchmark_data_smithi045_83007_object111212:head v 158'14757, mlcod=158'14751)", 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:14:11.148214+0000", 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 32.429700502999999, 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14398571299999999, 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148214+0000", 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:14:43.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148214+0000", 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148214+0000", 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5699999999999999e-07 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148218+0000", 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1250000000000003e-06 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:14:43.704 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148230+0000", 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1420000000000001e-05 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148238+0000", 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1769999999999998e-06 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148704+0000", 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000466155 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:14:43.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148733+0000", 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8277e-05 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.292055+0000", 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14332220200000001 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.292200+0000", 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0001451 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.706 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.707 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:14:43.707 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:14:43.707 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:14:43.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:43.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:43.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:43.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:43.907 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:43.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:43.936 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:43.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:44.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:44.061 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:44.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:44.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:44.163 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:44.191 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:44.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:44.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:44.293 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:44.320 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:44.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:44.422 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:44.440 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:44 smithi045 ceph-mon[39715]: osdmap e174: 8 total, 8 up, 8 in 2024-03-11T22:14:44.440 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:44 smithi045 ceph-mon[30178]: osdmap e174: 8 total, 8 up, 8 in 2024-03-11T22:14:44.450 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:44.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:44.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:44.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:44.576 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:44.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:44.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:44.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:44 smithi144 ceph-mon[33984]: osdmap e174: 8 total, 8 up, 8 in 2024-03-11T22:14:44.680 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:44.708 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:44.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:44.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:44.810 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:44.837 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:44.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:44.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:44.939 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:44.967 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:44.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:45.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:45.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:45.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:45.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:45.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:45.222 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:45.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:45.322 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:45.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:45.351 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:45.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:45.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T22:14:45.452 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:45.480 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:45.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:45.544 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:45.564 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:45.570 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:45.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:45 smithi045 ceph-mon[30178]: pgmap v256: 152 pgs: 9 remapped+peering, 1 unknown, 142 active+clean; 609 KiB data, 978 MiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 3 op/s; 20 KiB/s, 0 objects/s recovering 2024-03-11T22:14:45.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:45.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:14:45.582 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:45 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3564336731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:45.582 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:45.591 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:14:45.608 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:45.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:45.611 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:14:45.631 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:14:45.650 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:14:45.670 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:14:45.691 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:14:45.710 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:45.711 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:14:45.731 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:14:45.732 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:14:45.733 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:14:45.733 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:14:45.733 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:14:45.733 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:14:45.733 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:14:45.733 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:14:45.733 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:14:45.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:45.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:45.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:45.835 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:45.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:45 smithi045 ceph-mon[39715]: pgmap v256: 152 pgs: 9 remapped+peering, 1 unknown, 142 active+clean; 609 KiB data, 978 MiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 3 op/s; 20 KiB/s, 0 objects/s recovering 2024-03-11T22:14:45.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:45.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:14:45.841 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:45 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3564336731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:45.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:45 smithi144 ceph-mon[33984]: pgmap v256: 152 pgs: 9 remapped+peering, 1 unknown, 142 active+clean; 609 KiB data, 978 MiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 3 op/s; 20 KiB/s, 0 objects/s recovering 2024-03-11T22:14:45.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:14:45.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:14:45.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:45 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3564336731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:45.864 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:45.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:45.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:45.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:45.990 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:45.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:46.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:46.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:46.117 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:46.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:46.218 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:46.245 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:46.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:46.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:46.343 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:14:45 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:14:46.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:46.372 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:46.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:46.473 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:46.502 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:46.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:46.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:46.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:46.633 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:46.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:46.734 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:46.763 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:46.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:46.865 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:46.892 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:46.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:46.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:47.018 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:47.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:47.119 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:47.146 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:47.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:47.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:47.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:47.272 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:47.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:47.373 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:47.399 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:47.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:47.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:47.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:47 smithi045 ceph-mon[39715]: pgmap v257: 152 pgs: 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s; 66 KiB/s, 0 objects/s recovering 2024-03-11T22:14:47.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:47.480 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:47 smithi045 ceph-mon[30178]: pgmap v257: 152 pgs: 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s; 66 KiB/s, 0 objects/s recovering 2024-03-11T22:14:47.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:47.526 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:47.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:47.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:47.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:47 smithi144 ceph-mon[33984]: pgmap v257: 152 pgs: 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s; 66 KiB/s, 0 objects/s recovering 2024-03-11T22:14:47.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:47.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:47.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:47.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:47.753 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:47.777 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:47.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:47.878 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:47.903 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:47.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:47.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:48.005 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:48.034 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:48.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:48.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:48.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:48.163 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:48.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:48.264 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:48.292 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:48.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:48.394 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:48.413 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:48 smithi045 ceph-mon[39715]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-11T22:14:48.413 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:48 smithi045 ceph-mon[30178]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-11T22:14:48.419 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:48.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:48.521 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:48.549 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:48.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:48.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:48 smithi144 ceph-mon[33984]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-11T22:14:48.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:48.676 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:48.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:48.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:48.755 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:48.778 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:48.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:48.807 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:48.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:48.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:48.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:48.932 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:48.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:49.033 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:49.061 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:49.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:49.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:49.163 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:49.187 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:49.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:49.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:49.312 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:49.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:49.399 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:49 smithi045 ceph-mon[39715]: pgmap v258: 152 pgs: 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 585 B/s rd, 0 op/s; 46 KiB/s, 0 objects/s recovering 2024-03-11T22:14:49.400 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:49 smithi045 ceph-mon[30178]: pgmap v258: 152 pgs: 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 585 B/s rd, 0 op/s; 46 KiB/s, 0 objects/s recovering 2024-03-11T22:14:49.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:49.438 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:49.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:49.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:49.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:49.564 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:49.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:49.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:49 smithi144 ceph-mon[33984]: pgmap v258: 152 pgs: 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 585 B/s rd, 0 op/s; 46 KiB/s, 0 objects/s recovering 2024-03-11T22:14:49.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:49.689 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:49.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:49.790 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:49.815 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:49.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:49.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:14:49.916 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:49.944 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:49.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:50.046 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:50.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:50.076 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:50.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:50.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:50.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:50.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:50.306 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:50.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:50.330 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:50.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:50.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:50.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:50.456 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:50.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:50.557 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:50.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:50.587 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:50.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:50.688 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:50.714 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:50.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:50.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:50.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:50.840 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:50.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:50.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:50.942 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:50.969 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:50.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:51.071 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:51.099 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:51.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:51.201 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:51.222 INFO:teuthology.orchestra.run.smithi045.stderr:set osd.4 primary-affinity to 0 (802) 2024-03-11T22:14:51.229 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'unique_pool_1' created 2024-03-11T22:14:51.234 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:51.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:51.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:51.361 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:51.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:51.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:51.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:51.486 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:51.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:51.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:51 smithi045 ceph-mon[39715]: pgmap v259: 152 pgs: 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 40 KiB/s, 0 objects/s recovering 2024-03-11T22:14:51.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:51 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4286158967' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-11T22:14:51.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:51 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-11T22:14:51.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:51 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3532413483' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-03-11T22:14:51.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:51 smithi045 ceph-mon[30178]: pgmap v259: 152 pgs: 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 40 KiB/s, 0 objects/s recovering 2024-03-11T22:14:51.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:51 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4286158967' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-11T22:14:51.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:51 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-11T22:14:51.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:51 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3532413483' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-03-11T22:14:51.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:51.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:51 smithi144 ceph-mon[33984]: pgmap v259: 152 pgs: 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 40 KiB/s, 0 objects/s recovering 2024-03-11T22:14:51.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:51 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4286158967' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-11T22:14:51.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:51 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-11T22:14:51.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:51 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3532413483' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-03-11T22:14:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:51.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:51.712 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:51.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:51.742 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:51.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:51.843 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:51.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:51.869 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:51.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:51.970 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:51.998 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:51.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:52.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:52.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:52.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:52.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:52.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:52.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:52.258 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:52.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:52.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:52.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:52.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:52.486 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:52.515 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:52.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:52.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:52 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-11T22:14:52.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:52 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3532413483' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-03-11T22:14:52.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:52 smithi045 ceph-mon[30178]: osdmap e175: 8 total, 8 up, 8 in 2024-03-11T22:14:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:52 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-11T22:14:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:52 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3532413483' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-03-11T22:14:52.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:52 smithi045 ceph-mon[39715]: osdmap e175: 8 total, 8 up, 8 in 2024-03-11T22:14:52.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:52 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-11T22:14:52.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:52 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3532413483' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-03-11T22:14:52.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:52 smithi144 ceph-mon[33984]: osdmap e175: 8 total, 8 up, 8 in 2024-03-11T22:14:52.616 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:52.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:52.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:52.743 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:52.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-11T22:14:52.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:52.767 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:52.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:52.869 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:52.884 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:14:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:14:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:14:52.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:52.897 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:52.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:52.998 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:53.023 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:53.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:53.124 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:53.152 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:53.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:53.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:53.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:53.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:53.317 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:53.381 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:53.411 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:53.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:53.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:53 smithi045 ceph-mon[39715]: pgmap v261: 168 pgs: 16 unknown, 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 36 KiB/s, 0 objects/s recovering 2024-03-11T22:14:53.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:53 smithi045 ceph-mon[39715]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:14:53.430 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:53 smithi045 ceph-mon[39715]: osdmap e176: 8 total, 8 up, 8 in 2024-03-11T22:14:53.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:53 smithi045 ceph-mon[30178]: pgmap v261: 168 pgs: 16 unknown, 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 36 KiB/s, 0 objects/s recovering 2024-03-11T22:14:53.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:53 smithi045 ceph-mon[30178]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:14:53.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:53 smithi045 ceph-mon[30178]: osdmap e176: 8 total, 8 up, 8 in 2024-03-11T22:14:53.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:53.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:53.536 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:53.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:53.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:53 smithi144 ceph-mon[33984]: pgmap v261: 168 pgs: 16 unknown, 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 36 KiB/s, 0 objects/s recovering 2024-03-11T22:14:53.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:53 smithi144 ceph-mon[33984]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:14:53.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:53 smithi144 ceph-mon[33984]: osdmap e176: 8 total, 8 up, 8 in 2024-03-11T22:14:53.637 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:53.666 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:53.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:53.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:53.792 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:53.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:53.795 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:53.895 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:53.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:53.923 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:53.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:54.024 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:54.052 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:54.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:54.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:54.154 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:54.182 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:54.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:54.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:54.236 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:14:54.285 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:54.313 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:54.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:54.331 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:54 smithi045 ceph-mon[30178]: osdmap e177: 8 total, 8 up, 8 in 2024-03-11T22:14:54.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:54 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1688852931' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:14:54.332 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:54 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:14:54.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:54 smithi045 ceph-mon[39715]: osdmap e177: 8 total, 8 up, 8 in 2024-03-11T22:14:54.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:54 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1688852931' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:14:54.332 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:54 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:14:54.415 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:54.442 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:54.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:54.543 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:54.569 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:54.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:54.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:54 smithi144 ceph-mon[33984]: osdmap e177: 8 total, 8 up, 8 in 2024-03-11T22:14:54.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:54 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1688852931' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:14:54.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:54 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:14:54.670 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:54.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:54.699 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:54.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:54.800 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:54.825 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:54.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:54.927 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:54.956 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:54.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:55.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:55.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:55.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:55.184 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:55.213 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:55.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:55.314 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:55.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:55.343 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:55.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:55.444 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:55.472 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:55.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:55.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:55.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:55 smithi045 ceph-mon[39715]: pgmap v264: 168 pgs: 16 unknown, 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:14:55.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:55 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:14:55.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:55 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:14:55.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:55 smithi045 ceph-mon[39715]: osdmap e178: 8 total, 8 up, 8 in 2024-03-11T22:14:55.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:55 smithi045 ceph-mon[30178]: pgmap v264: 168 pgs: 16 unknown, 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:14:55.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:55 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:14:55.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:55 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:14:55.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:55 smithi045 ceph-mon[30178]: osdmap e178: 8 total, 8 up, 8 in 2024-03-11T22:14:55.574 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:55.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:55 smithi144 ceph-mon[33984]: pgmap v264: 168 pgs: 16 unknown, 152 active+clean; 609 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:14:55.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:55 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:14:55.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:55 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:14:55.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:55 smithi144 ceph-mon[33984]: osdmap e178: 8 total, 8 up, 8 in 2024-03-11T22:14:55.600 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:55.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:55.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:55.729 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:55.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:55.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:55.858 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:55.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:55.959 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:55.987 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:55.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:56.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:56.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:56.114 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:56.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:56.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:56.241 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:56.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:56.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:56.344 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:14:55 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:14:56.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:56.367 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:56.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:56.468 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:56.496 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:56.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:56.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:56.622 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:56.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:56.705 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:14:56.705 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:14:56.724 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:56.752 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:56.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:56.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:56.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:14:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:56.879 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:56.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:56.932 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:14:56.952 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:14:56.968 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:14:56.968 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:14:56.973 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:14:56.981 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:56.993 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:14:57.008 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:57.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:57.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:57.014 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:14:57.035 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:14:57.057 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:14:57.079 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:14:57.080 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:14:57.080 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:14:57.080 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:14:57.080 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:14:57.080 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:14:57.080 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:14:57.080 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:14:57.080 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:14:57.111 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:57.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:57.139 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:57.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:57.240 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:57.266 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:57.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:57.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:57.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:57.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:57 smithi045 ceph-mon[30178]: pgmap v266: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s wr, 1 op/s; 16 KiB/s, 0 objects/s recovering 2024-03-11T22:14:57.345 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:57 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/870164038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:57.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:57.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:57 smithi045 ceph-mon[39715]: pgmap v266: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s wr, 1 op/s; 16 KiB/s, 0 objects/s recovering 2024-03-11T22:14:57.345 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:57 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/870164038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:57.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:57.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:57.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:57.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:57.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:57.518 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:57.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:57.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:14:57.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:57 smithi144 ceph-mon[33984]: pgmap v266: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 22 KiB/s wr, 1 op/s; 16 KiB/s, 0 objects/s recovering 2024-03-11T22:14:57.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:57 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/870164038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:14:57.619 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:57.647 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:57.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:57.749 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:14:57.776 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:57.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:57.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:14:57.878 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:57.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:57.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:58.006 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:58.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:58.036 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:58.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:58.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:58.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:58.164 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:58.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:58.264 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:58.293 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:58.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:58.394 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:14:58.422 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:58.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:14:58.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T22:14:58.524 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:14:58.551 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:58.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:14:58.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:58.677 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:58.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:58.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:14:58.779 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:58.807 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:58.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:58.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:58.934 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:58.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:58.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:59.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:14:59.060 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:59.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:14:59.163 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:59.192 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:59.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:59.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:59.317 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:59.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:59.418 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:14:59.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:59.447 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:59.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:14:59.464 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:14:59 smithi045 ceph-mon[39715]: pgmap v267: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s wr, 1 op/s; 13 KiB/s, 0 objects/s recovering 2024-03-11T22:14:59.464 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:14:59 smithi045 ceph-mon[30178]: pgmap v267: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s wr, 1 op/s; 13 KiB/s, 0 objects/s recovering 2024-03-11T22:14:59.482 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:14:59.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:59.573 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:59.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:59.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:14:59 smithi144 ceph-mon[33984]: pgmap v267: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s wr, 1 op/s; 13 KiB/s, 0 objects/s recovering 2024-03-11T22:14:59.675 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:14:59.703 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:14:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:59.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:14:59.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:14:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:59.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:14:59.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:14:59.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:14:59.955 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:14:59.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:00.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:00.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:00.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:00.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:00.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:00.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:00.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:00.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:00.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:00.332 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:00.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:00.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:00.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:00.458 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:00.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:00.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:00.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:00.583 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:00.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:00.601 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-11T22:15:00.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:00.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:00.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:00.810 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:00.838 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:00.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:00.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:00.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:00.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:15:00.843 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:00 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/967347368' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:15:00.889 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:00.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:15:00.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:00 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/967347368' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:15:00.890 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:00.890 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:15:00.890 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:00 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/967347368' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:15:00.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:00.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:15:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:00.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:00.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:01.066 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:01.095 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:01.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:01.197 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:01.225 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:01.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:01.299 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:01.327 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:01.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:01.355 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:01.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:01.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:01.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:01.481 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:01.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:01.582 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:01.607 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:01.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:01.708 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:01.736 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:01.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:01.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:01.864 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:01.866 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:01.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:01.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:01 smithi045 ceph-mon[39715]: pgmap v268: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s wr, 1 op/s; 9.9 KiB/s, 0 objects/s recovering 2024-03-11T22:15:01.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:01 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/967347368' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:15:01.983 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:01 smithi045 ceph-mon[39715]: osdmap e179: 8 total, 8 up, 8 in 2024-03-11T22:15:01.983 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:01 smithi045 ceph-mon[30178]: pgmap v268: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s wr, 1 op/s; 9.9 KiB/s, 0 objects/s recovering 2024-03-11T22:15:01.983 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:01 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/967347368' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:15:01.983 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:01 smithi045 ceph-mon[30178]: osdmap e179: 8 total, 8 up, 8 in 2024-03-11T22:15:01.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:02.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:02.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:02.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:02.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:01 smithi144 ceph-mon[33984]: pgmap v268: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 14 KiB/s wr, 1 op/s; 9.9 KiB/s, 0 objects/s recovering 2024-03-11T22:15:02.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:01 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/967347368' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:15:02.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:01 smithi144 ceph-mon[33984]: osdmap e179: 8 total, 8 up, 8 in 2024-03-11T22:15:02.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:02.133 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:02.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:02.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:02.239 DEBUG:teuthology.orchestra.run.smithi045:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-11T22:15:02.240 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-03-11T22:15:02.240 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-03-11T22:15:02.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:15:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:02.259 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:02.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:02.260 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T22:15:02.260 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T22:15:02.308 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:hints = 1 2024-03-11T22:15:02.308 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T22:15:02.308 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Object prefix: benchmark_data_smithi045_115206 2024-03-11T22:15:02.308 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:15:02.308 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 0 0 0 0 0 0 - 0 2024-03-11T22:15:02.361 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:02.394 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:02.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:02.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:02.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:02.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:02.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:02.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:02.694 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:02.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:02.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:02.779 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:02 smithi045 ceph-mon[30178]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:15:02.780 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:15:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:15:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:15:02.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:02 smithi045 ceph-mon[39715]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:15:02.796 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:02.826 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:02.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:02.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:02.968 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:02.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:03.069 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:03.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:02 smithi144 ceph-mon[33984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:15:03.102 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:03.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:03.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:03.203 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:03.239 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:03.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:03.308 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 1 16 2987 2971 185.678 185.688 0.00215766 0.00531115 2024-03-11T22:15:03.340 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:03.378 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:03.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:03.480 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:03.513 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:03.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:03.615 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:03.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:03.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:03.662 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:15:03.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:03 smithi045 ceph-mon[39715]: pgmap v270: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 895 B/s rd, 14 KiB/s wr, 1 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-03-11T22:15:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:03 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/267008494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:15:03.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:03 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:15:03.686 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:03 smithi045 ceph-mon[30178]: pgmap v270: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 895 B/s rd, 14 KiB/s wr, 1 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-03-11T22:15:03.686 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:03 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/267008494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:15:03.686 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:03 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:15:03.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:03.795 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:03.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:03.898 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:03.931 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:03.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:03.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:04.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:04.075 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:04.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:04.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:04.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:03 smithi144 ceph-mon[33984]: pgmap v270: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 895 B/s rd, 14 KiB/s wr, 1 op/s; 9.8 KiB/s, 0 objects/s recovering 2024-03-11T22:15:04.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:03 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/267008494' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:15:04.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:03 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:15:04.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:04.217 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:04.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:04.309 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 2 16 6099 6083 190.082 194.5 0.00352749 0.00524699 2024-03-11T22:15:04.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:04.371 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:04.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:04.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:04.473 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:04.507 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:04.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:04.609 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:04.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:04.645 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:04.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:04.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:04.790 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:04.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:04.891 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:04.924 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:04.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:05.026 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:05.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:15:05.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:15:05.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[39715]: osdmap e180: 8 total, 8 up, 8 in 2024-03-11T22:15:05.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[39715]: 1.0 scrub starts 2024-03-11T22:15:05.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[39715]: 1.0 scrub ok 2024-03-11T22:15:05.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[39715]: 1.b scrub starts 2024-03-11T22:15:05.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[39715]: 1.b scrub ok 2024-03-11T22:15:05.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:15:05.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:15:05.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[30178]: osdmap e180: 8 total, 8 up, 8 in 2024-03-11T22:15:05.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[30178]: 1.0 scrub starts 2024-03-11T22:15:05.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[30178]: 1.0 scrub ok 2024-03-11T22:15:05.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[30178]: 1.b scrub starts 2024-03-11T22:15:05.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:04 smithi045 ceph-mon[30178]: 1.b scrub ok 2024-03-11T22:15:05.066 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:05.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:05.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:04 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:15:05.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:04 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:15:05.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:04 smithi144 ceph-mon[33984]: osdmap e180: 8 total, 8 up, 8 in 2024-03-11T22:15:05.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:04 smithi144 ceph-mon[33984]: 1.0 scrub starts 2024-03-11T22:15:05.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:04 smithi144 ceph-mon[33984]: 1.0 scrub ok 2024-03-11T22:15:05.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:04 smithi144 ceph-mon[33984]: 1.b scrub starts 2024-03-11T22:15:05.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:04 smithi144 ceph-mon[33984]: 1.b scrub ok 2024-03-11T22:15:05.168 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:05.202 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:05.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:05.304 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:05.309 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 3 16 9212 9196 191.543 194.562 0.00293402 0.00520988 2024-03-11T22:15:05.337 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:05.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:05.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:05.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:05.485 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:05.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:05.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:05.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:05.628 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:05.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:05.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:05.769 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:05.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:05.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:05.910 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:05.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:05.972 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:05 smithi144 ceph-mon[33984]: 1.5 scrub starts 2024-03-11T22:15:05.972 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:05 smithi144 ceph-mon[33984]: 1.5 scrub ok 2024-03-11T22:15:05.972 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:05 smithi144 ceph-mon[33984]: pgmap v272: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:15:05.972 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:05 smithi144 ceph-mon[33984]: 1.2 scrub starts 2024-03-11T22:15:05.972 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:05 smithi144 ceph-mon[33984]: 1.2 scrub ok 2024-03-11T22:15:05.973 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:05 smithi144 ceph-mon[33984]: 1.c scrub starts 2024-03-11T22:15:05.973 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:05 smithi144 ceph-mon[33984]: 1.c scrub ok 2024-03-11T22:15:05.973 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:05 smithi144 ceph-mon[33984]: 1.13 scrub starts 2024-03-11T22:15:05.973 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:05 smithi144 ceph-mon[33984]: 1.13 scrub ok 2024-03-11T22:15:06.012 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:06.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[39715]: 1.5 scrub starts 2024-03-11T22:15:06.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[39715]: 1.5 scrub ok 2024-03-11T22:15:06.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[39715]: pgmap v272: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:15:06.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[39715]: 1.2 scrub starts 2024-03-11T22:15:06.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[39715]: 1.2 scrub ok 2024-03-11T22:15:06.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[39715]: 1.c scrub starts 2024-03-11T22:15:06.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[39715]: 1.c scrub ok 2024-03-11T22:15:06.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[39715]: 1.13 scrub starts 2024-03-11T22:15:06.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[39715]: 1.13 scrub ok 2024-03-11T22:15:06.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[30178]: 1.5 scrub starts 2024-03-11T22:15:06.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[30178]: 1.5 scrub ok 2024-03-11T22:15:06.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[30178]: pgmap v272: 168 pgs: 168 active+clean; 635 KiB data, 982 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:15:06.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[30178]: 1.2 scrub starts 2024-03-11T22:15:06.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[30178]: 1.2 scrub ok 2024-03-11T22:15:06.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[30178]: 1.c scrub starts 2024-03-11T22:15:06.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[30178]: 1.c scrub ok 2024-03-11T22:15:06.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[30178]: 1.13 scrub starts 2024-03-11T22:15:06.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:05 smithi045 ceph-mon[30178]: 1.13 scrub ok 2024-03-11T22:15:06.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:06.047 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:06.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:06.148 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:06.176 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:06.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:06.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:06.309 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 4 16 11982 11966 186.935 173.125 0.00657782 0.00534044 2024-03-11T22:15:06.313 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:06.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:06.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:06.343 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:15:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:15:06.415 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:06.440 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:06.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:06.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:06.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:06.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:06.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:06.723 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:06.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:06.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:06.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:06.866 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:06.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:06.969 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:07.003 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:07.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:07.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:07.010 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[30178]: 4.d scrub starts 2024-03-11T22:15:07.010 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[30178]: 4.d scrub ok 2024-03-11T22:15:07.010 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[30178]: 1.12 scrub starts 2024-03-11T22:15:07.010 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[30178]: 1.12 scrub ok 2024-03-11T22:15:07.010 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[30178]: 1.4 scrub starts 2024-03-11T22:15:07.010 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[30178]: 1.4 scrub ok 2024-03-11T22:15:07.010 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[30178]: 1.3 scrub starts 2024-03-11T22:15:07.010 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[30178]: 1.3 scrub ok 2024-03-11T22:15:07.010 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[39715]: 4.d scrub starts 2024-03-11T22:15:07.011 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[39715]: 4.d scrub ok 2024-03-11T22:15:07.011 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[39715]: 1.12 scrub starts 2024-03-11T22:15:07.011 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[39715]: 1.12 scrub ok 2024-03-11T22:15:07.011 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[39715]: 1.4 scrub starts 2024-03-11T22:15:07.011 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[39715]: 1.4 scrub ok 2024-03-11T22:15:07.011 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[39715]: 1.3 scrub starts 2024-03-11T22:15:07.011 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:06 smithi045 ceph-mon[39715]: 1.3 scrub ok 2024-03-11T22:15:07.038 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:07.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:06 smithi144 ceph-mon[33984]: 4.d scrub starts 2024-03-11T22:15:07.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:06 smithi144 ceph-mon[33984]: 4.d scrub ok 2024-03-11T22:15:07.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:06 smithi144 ceph-mon[33984]: 1.12 scrub starts 2024-03-11T22:15:07.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:06 smithi144 ceph-mon[33984]: 1.12 scrub ok 2024-03-11T22:15:07.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:06 smithi144 ceph-mon[33984]: 1.4 scrub starts 2024-03-11T22:15:07.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:06 smithi144 ceph-mon[33984]: 1.4 scrub ok 2024-03-11T22:15:07.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:06 smithi144 ceph-mon[33984]: 1.3 scrub starts 2024-03-11T22:15:07.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:06 smithi144 ceph-mon[33984]: 1.3 scrub ok 2024-03-11T22:15:07.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:07.132 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:15:07.132 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:15:07.132 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-03-11T22:15:07.133 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-03-11T22:15:07.133 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:15:07.152 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:07.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:07.254 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:07.286 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:07.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:07.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:07.309 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 5 16 14373 14357 179.433 149.438 0.00566172 0.00556555 2024-03-11T22:15:07.388 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:07.421 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:07.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:07.523 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:07.554 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:07.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:07.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:07.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:07.700 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:07.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[39715]: 1.f scrub starts 2024-03-11T22:15:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[39715]: 1.f scrub ok 2024-03-11T22:15:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[39715]: pgmap v273: 168 pgs: 168 active+clean; 599 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T22:15:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[39715]: 1.1 scrub starts 2024-03-11T22:15:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[39715]: 1.1 scrub ok 2024-03-11T22:15:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[39715]: 1.11 scrub starts 2024-03-11T22:15:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[39715]: 1.11 scrub ok 2024-03-11T22:15:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[39715]: 1.d scrub starts 2024-03-11T22:15:07.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[39715]: 1.d scrub ok 2024-03-11T22:15:07.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:07.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[30178]: 1.f scrub starts 2024-03-11T22:15:07.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[30178]: 1.f scrub ok 2024-03-11T22:15:07.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[30178]: pgmap v273: 168 pgs: 168 active+clean; 599 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T22:15:07.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[30178]: 1.1 scrub starts 2024-03-11T22:15:07.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[30178]: 1.1 scrub ok 2024-03-11T22:15:07.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[30178]: 1.11 scrub starts 2024-03-11T22:15:07.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[30178]: 1.11 scrub ok 2024-03-11T22:15:07.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[30178]: 1.d scrub starts 2024-03-11T22:15:07.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:07 smithi045 ceph-mon[30178]: 1.d scrub ok 2024-03-11T22:15:07.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:07.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:07.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:07.945 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:07.979 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:07.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:07.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:08.081 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:08.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:08.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:07 smithi144 ceph-mon[33984]: 1.f scrub starts 2024-03-11T22:15:08.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:07 smithi144 ceph-mon[33984]: 1.f scrub ok 2024-03-11T22:15:08.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:07 smithi144 ceph-mon[33984]: pgmap v273: 168 pgs: 168 active+clean; 599 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 0 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T22:15:08.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:07 smithi144 ceph-mon[33984]: 1.1 scrub starts 2024-03-11T22:15:08.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:07 smithi144 ceph-mon[33984]: 1.1 scrub ok 2024-03-11T22:15:08.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:07 smithi144 ceph-mon[33984]: 1.11 scrub starts 2024-03-11T22:15:08.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:07 smithi144 ceph-mon[33984]: 1.11 scrub ok 2024-03-11T22:15:08.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:07 smithi144 ceph-mon[33984]: 1.d scrub starts 2024-03-11T22:15:08.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:07 smithi144 ceph-mon[33984]: 1.d scrub ok 2024-03-11T22:15:08.118 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:08.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:08.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:08.260 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:08.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:08.309 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 6 15 17111 17096 178.057 171.188 0.00912479 0.00561014 2024-03-11T22:15:08.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:08.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:08.388 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:08.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:08.489 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:08.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T22:15:08.522 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:08.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:08.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:08.624 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:08.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:08.680 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:08.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:08.780 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:08.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:15:08.833 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:08.833 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:15:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:08.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:08.882 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:15:08.926 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:15:08.935 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:08.969 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:08.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:08.971 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:15:08.971 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[39715]: 1.14 scrub starts 2024-03-11T22:15:08.972 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[39715]: 1.14 scrub ok 2024-03-11T22:15:08.972 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[39715]: 1.6 scrub starts 2024-03-11T22:15:08.972 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[39715]: 1.6 scrub ok 2024-03-11T22:15:08.972 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:15:08.972 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:15:08.972 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[39715]: 1.10 scrub starts 2024-03-11T22:15:08.972 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[39715]: 1.10 scrub ok 2024-03-11T22:15:08.972 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[39715]: osdmap e181: 8 total, 8 up, 8 in 2024-03-11T22:15:08.972 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[39715]: 3.e scrub starts 2024-03-11T22:15:08.972 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[39715]: 3.e scrub ok 2024-03-11T22:15:08.972 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[30178]: 1.14 scrub starts 2024-03-11T22:15:08.973 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[30178]: 1.14 scrub ok 2024-03-11T22:15:08.973 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[30178]: 1.6 scrub starts 2024-03-11T22:15:08.973 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[30178]: 1.6 scrub ok 2024-03-11T22:15:08.973 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:15:08.973 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:15:08.973 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[30178]: 1.10 scrub starts 2024-03-11T22:15:08.973 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[30178]: 1.10 scrub ok 2024-03-11T22:15:08.973 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[30178]: osdmap e181: 8 total, 8 up, 8 in 2024-03-11T22:15:08.973 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[30178]: 3.e scrub starts 2024-03-11T22:15:08.973 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:08 smithi045 ceph-mon[30178]: 3.e scrub ok 2024-03-11T22:15:09.021 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:15:09.070 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:09.071 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:15:09.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:08 smithi144 ceph-mon[33984]: 1.14 scrub starts 2024-03-11T22:15:09.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:08 smithi144 ceph-mon[33984]: 1.14 scrub ok 2024-03-11T22:15:09.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:08 smithi144 ceph-mon[33984]: 1.6 scrub starts 2024-03-11T22:15:09.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:08 smithi144 ceph-mon[33984]: 1.6 scrub ok 2024-03-11T22:15:09.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:08 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:15:09.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:08 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:15:09.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:08 smithi144 ceph-mon[33984]: 1.10 scrub starts 2024-03-11T22:15:09.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:08 smithi144 ceph-mon[33984]: 1.10 scrub ok 2024-03-11T22:15:09.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:08 smithi144 ceph-mon[33984]: osdmap e181: 8 total, 8 up, 8 in 2024-03-11T22:15:09.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:08 smithi144 ceph-mon[33984]: 3.e scrub starts 2024-03-11T22:15:09.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:08 smithi144 ceph-mon[33984]: 3.e scrub ok 2024-03-11T22:15:09.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:09.116 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:09.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:09.117 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:15:09.163 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:15:09.167 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:15:09.167 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:15:09.167 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:15:09.168 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:15:09.168 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:15:09.168 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:15:09.168 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:15:09.168 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:15:09.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:09.264 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:09.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:09.310 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 7 15 19111 19096 170.476 125 0.00504202 0.0058597 2024-03-11T22:15:09.366 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:09.401 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:09.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:09.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:09.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:09.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:09.546 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:09.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:09.647 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:09.680 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:09.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:09.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:09.717 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[30178]: 3.13 scrub starts 2024-03-11T22:15:09.717 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[30178]: 3.13 scrub ok 2024-03-11T22:15:09.717 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[30178]: pgmap v275: 168 pgs: 168 active+clean; 599 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-11T22:15:09.717 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[30178]: 1.7 scrub starts 2024-03-11T22:15:09.717 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[30178]: 1.7 scrub ok 2024-03-11T22:15:09.717 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/887535875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:15:09.717 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[30178]: 3.6 scrub starts 2024-03-11T22:15:09.717 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[30178]: 3.6 scrub ok 2024-03-11T22:15:09.717 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[30178]: 1.e scrub starts 2024-03-11T22:15:09.717 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[30178]: 1.e scrub ok 2024-03-11T22:15:09.717 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[30178]: 5.19 scrub starts 2024-03-11T22:15:09.717 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[30178]: 5.19 scrub ok 2024-03-11T22:15:09.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:09.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:09.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:09.929 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:09.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:09.962 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:09.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[39715]: 3.13 scrub starts 2024-03-11T22:15:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[39715]: 3.13 scrub ok 2024-03-11T22:15:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[39715]: pgmap v275: 168 pgs: 168 active+clean; 599 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-11T22:15:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[39715]: 1.7 scrub starts 2024-03-11T22:15:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[39715]: 1.7 scrub ok 2024-03-11T22:15:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/887535875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:15:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[39715]: 3.6 scrub starts 2024-03-11T22:15:10.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[39715]: 3.6 scrub ok 2024-03-11T22:15:10.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[39715]: 1.e scrub starts 2024-03-11T22:15:10.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[39715]: 1.e scrub ok 2024-03-11T22:15:10.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[39715]: 5.19 scrub starts 2024-03-11T22:15:10.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:09 smithi045 ceph-mon[39715]: 5.19 scrub ok 2024-03-11T22:15:10.063 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:10.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:09 smithi144 ceph-mon[33984]: 3.13 scrub starts 2024-03-11T22:15:10.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:09 smithi144 ceph-mon[33984]: 3.13 scrub ok 2024-03-11T22:15:10.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:09 smithi144 ceph-mon[33984]: pgmap v275: 168 pgs: 168 active+clean; 599 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-11T22:15:10.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:09 smithi144 ceph-mon[33984]: 1.7 scrub starts 2024-03-11T22:15:10.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:09 smithi144 ceph-mon[33984]: 1.7 scrub ok 2024-03-11T22:15:10.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:09 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/887535875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:15:10.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:09 smithi144 ceph-mon[33984]: 3.6 scrub starts 2024-03-11T22:15:10.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:09 smithi144 ceph-mon[33984]: 3.6 scrub ok 2024-03-11T22:15:10.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:09 smithi144 ceph-mon[33984]: 1.e scrub starts 2024-03-11T22:15:10.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:09 smithi144 ceph-mon[33984]: 1.e scrub ok 2024-03-11T22:15:10.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:09 smithi144 ceph-mon[33984]: 5.19 scrub starts 2024-03-11T22:15:10.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:09 smithi144 ceph-mon[33984]: 5.19 scrub ok 2024-03-11T22:15:10.096 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:10.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:10.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:10.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:10.246 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:10.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:10.249 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:10.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:10.310 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 8 16 21651 21635 168.986 158.688 0.00322612 0.0059118 2024-03-11T22:15:10.351 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:10.382 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:10.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:10.484 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:10.518 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:10.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:10.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:10.621 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:10.655 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:10.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:10.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:10.796 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:10.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:10.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:10.898 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:10.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:10.932 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:10.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:10 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:15:11.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:10 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:15:11.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:10 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:15:11.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:10 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:15:11.033 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:11.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:11.068 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:11.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:11.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:10 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:15:11.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:10 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:15:11.169 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:11.201 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:11.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:11.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:11.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:11.311 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 9 16 24049 24033 166.847 149.875 0.00681741 0.00598603 2024-03-11T22:15:11.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:11.344 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:11.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:11.445 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:11.475 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:11.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:11.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:11.616 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:11.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:11.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:11.718 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:11.757 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:11.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:11.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:11.859 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:11.888 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:11.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:11.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:11.916 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:11.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:12.022 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:12.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:12.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[30178]: pgmap v276: 168 pgs: 168 active+clean; 751 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-03-11T22:15:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:15:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:15:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[30178]: osdmap e182: 8 total, 8 up, 8 in 2024-03-11T22:15:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[30178]: 6.10 scrub starts 2024-03-11T22:15:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[30178]: 6.10 scrub ok 2024-03-11T22:15:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[30178]: osdmap e183: 8 total, 8 up, 8 in 2024-03-11T22:15:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[30178]: 4.4 scrub starts 2024-03-11T22:15:12.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[30178]: 4.4 scrub ok 2024-03-11T22:15:12.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[39715]: pgmap v276: 168 pgs: 168 active+clean; 751 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-03-11T22:15:12.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:15:12.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:15:12.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[39715]: osdmap e182: 8 total, 8 up, 8 in 2024-03-11T22:15:12.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[39715]: 6.10 scrub starts 2024-03-11T22:15:12.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[39715]: 6.10 scrub ok 2024-03-11T22:15:12.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[39715]: osdmap e183: 8 total, 8 up, 8 in 2024-03-11T22:15:12.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[39715]: 4.4 scrub starts 2024-03-11T22:15:12.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:11 smithi045 ceph-mon[39715]: 4.4 scrub ok 2024-03-11T22:15:12.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:11 smithi144 ceph-mon[33984]: pgmap v276: 168 pgs: 168 active+clean; 751 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-03-11T22:15:12.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:11 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:15:12.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:11 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:15:12.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:11 smithi144 ceph-mon[33984]: osdmap e182: 8 total, 8 up, 8 in 2024-03-11T22:15:12.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:11 smithi144 ceph-mon[33984]: 6.10 scrub starts 2024-03-11T22:15:12.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:11 smithi144 ceph-mon[33984]: 6.10 scrub ok 2024-03-11T22:15:12.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:11 smithi144 ceph-mon[33984]: osdmap e183: 8 total, 8 up, 8 in 2024-03-11T22:15:12.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:11 smithi144 ceph-mon[33984]: 4.4 scrub starts 2024-03-11T22:15:12.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:11 smithi144 ceph-mon[33984]: 4.4 scrub ok 2024-03-11T22:15:12.123 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:12.155 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:12.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:12.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:12.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:15:12.257 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:12.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:12.282 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:12.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:12.311 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 10 16 26072 26056 162.806 126.438 0.00195616 0.00606353 2024-03-11T22:15:12.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:12.428 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:12.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:12.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:12.529 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:12.562 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:12.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:12.664 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:12.688 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:12.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:12.789 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:12.823 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:12.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:12.925 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:12.961 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:12.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:13.029 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:15:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:15:13.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[39715]: 4.9 scrub starts 2024-03-11T22:15:13.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[39715]: 4.9 scrub ok 2024-03-11T22:15:13.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[39715]: 4.b scrub starts 2024-03-11T22:15:13.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[39715]: 4.b scrub ok 2024-03-11T22:15:13.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[39715]: pgmap v279: 168 pgs: 168 active+clean; 1.3 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-03-11T22:15:13.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[39715]: 6.2 scrub starts 2024-03-11T22:15:13.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[39715]: 6.2 scrub ok 2024-03-11T22:15:13.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[30178]: 4.9 scrub starts 2024-03-11T22:15:13.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[30178]: 4.9 scrub ok 2024-03-11T22:15:13.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[30178]: 4.b scrub starts 2024-03-11T22:15:13.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[30178]: 4.b scrub ok 2024-03-11T22:15:13.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[30178]: pgmap v279: 168 pgs: 168 active+clean; 1.3 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-03-11T22:15:13.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[30178]: 6.2 scrub starts 2024-03-11T22:15:13.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:12 smithi045 ceph-mon[30178]: 6.2 scrub ok 2024-03-11T22:15:13.063 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:13.093 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:13.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:12 smithi144 ceph-mon[33984]: 4.9 scrub starts 2024-03-11T22:15:13.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:12 smithi144 ceph-mon[33984]: 4.9 scrub ok 2024-03-11T22:15:13.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:12 smithi144 ceph-mon[33984]: 4.b scrub starts 2024-03-11T22:15:13.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:12 smithi144 ceph-mon[33984]: 4.b scrub ok 2024-03-11T22:15:13.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:12 smithi144 ceph-mon[33984]: pgmap v279: 168 pgs: 168 active+clean; 1.3 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-03-11T22:15:13.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:12 smithi144 ceph-mon[33984]: 6.2 scrub starts 2024-03-11T22:15:13.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:12 smithi144 ceph-mon[33984]: 6.2 scrub ok 2024-03-11T22:15:13.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:13.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:13.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:13.236 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:13.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:13.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:13.311 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 11 16 27879 27863 158.272 112.938 0.00434577 0.00631306 2024-03-11T22:15:13.338 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:13.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:13.362 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:13.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:13.463 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:13.494 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:13.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:13.596 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:13.635 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:13.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:13.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:13.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:13.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:13.761 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:13.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:13.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:13.901 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:13.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:13.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:14.003 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:14.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:14.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:14.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:14.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:13 smithi045 ceph-mon[39715]: 6.17 scrub starts 2024-03-11T22:15:14.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:13 smithi045 ceph-mon[39715]: 6.17 scrub ok 2024-03-11T22:15:14.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:13 smithi045 ceph-mon[39715]: 6.1a scrub starts 2024-03-11T22:15:14.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:13 smithi045 ceph-mon[39715]: 6.1a scrub ok 2024-03-11T22:15:14.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:13 smithi045 ceph-mon[39715]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:15:14.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:13 smithi045 ceph-mon[30178]: 6.17 scrub starts 2024-03-11T22:15:14.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:13 smithi045 ceph-mon[30178]: 6.17 scrub ok 2024-03-11T22:15:14.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:13 smithi045 ceph-mon[30178]: 6.1a scrub starts 2024-03-11T22:15:14.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:13 smithi045 ceph-mon[30178]: 6.1a scrub ok 2024-03-11T22:15:14.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:13 smithi045 ceph-mon[30178]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:15:14.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:13 smithi144 ceph-mon[33984]: 6.17 scrub starts 2024-03-11T22:15:14.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:13 smithi144 ceph-mon[33984]: 6.17 scrub ok 2024-03-11T22:15:14.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:13 smithi144 ceph-mon[33984]: 6.1a scrub starts 2024-03-11T22:15:14.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:13 smithi144 ceph-mon[33984]: 6.1a scrub ok 2024-03-11T22:15:14.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:13 smithi144 ceph-mon[33984]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:15:14.128 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:14.153 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:14.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:14.254 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:14.287 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:14.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:14.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:14.312 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 12 16 29192 29176 151.922 82.0625 0.00854817 0.00657667 2024-03-11T22:15:14.383 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:15:14.387 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:15:14.389 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:14.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:14.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:14.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:14.560 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:14.589 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:14.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:14.690 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:14.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:14.715 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:14.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:14.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:14.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:14.842 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:14.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:14.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:14.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:14 smithi045 ceph-mon[39715]: pgmap v280: 168 pgs: 168 active+clean; 1.3 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-03-11T22:15:14.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:14 smithi045 ceph-mon[39715]: 5.1 scrub starts 2024-03-11T22:15:14.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:14 smithi045 ceph-mon[39715]: 5.1 scrub ok 2024-03-11T22:15:14.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:14.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:15:14.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:14 smithi045 ceph-mon[30178]: pgmap v280: 168 pgs: 168 active+clean; 1.3 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-03-11T22:15:14.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:14 smithi045 ceph-mon[30178]: 5.1 scrub starts 2024-03-11T22:15:14.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:14 smithi045 ceph-mon[30178]: 5.1 scrub ok 2024-03-11T22:15:14.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:14.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:15:14.982 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:14.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:15.084 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:15.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:14 smithi144 ceph-mon[33984]: pgmap v280: 168 pgs: 168 active+clean; 1.3 GiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s 2024-03-11T22:15:15.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:14 smithi144 ceph-mon[33984]: 5.1 scrub starts 2024-03-11T22:15:15.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:14 smithi144 ceph-mon[33984]: 5.1 scrub ok 2024-03-11T22:15:15.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:15.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:15:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:15.116 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:15.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:15.217 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:15.250 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:15.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:15.312 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 13 16 30653 30637 147.26 91.3125 0.00548781 0.00678497 2024-03-11T22:15:15.351 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:15.391 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:15.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:15.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:15.517 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:15.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:15.618 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:15.653 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:15.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:15.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:15.795 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:15.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:15.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:15.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:15.921 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:15.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:16.022 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:16.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:15 smithi045 ceph-mon[30178]: 6.d scrub starts 2024-03-11T22:15:16.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:15 smithi045 ceph-mon[30178]: 6.d scrub ok 2024-03-11T22:15:16.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:15 smithi045 ceph-mon[30178]: 3.17 scrub starts 2024-03-11T22:15:16.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:15 smithi045 ceph-mon[30178]: 3.17 scrub ok 2024-03-11T22:15:16.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:15 smithi045 ceph-mon[30178]: 4.7 scrub starts 2024-03-11T22:15:16.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:15 smithi045 ceph-mon[30178]: 4.7 scrub ok 2024-03-11T22:15:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:15 smithi045 ceph-mon[39715]: 6.d scrub starts 2024-03-11T22:15:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:15 smithi045 ceph-mon[39715]: 6.d scrub ok 2024-03-11T22:15:16.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:15 smithi045 ceph-mon[39715]: 3.17 scrub starts 2024-03-11T22:15:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:15 smithi045 ceph-mon[39715]: 3.17 scrub ok 2024-03-11T22:15:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:15 smithi045 ceph-mon[39715]: 4.7 scrub starts 2024-03-11T22:15:16.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:15 smithi045 ceph-mon[39715]: 4.7 scrub ok 2024-03-11T22:15:16.060 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:16.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:16.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:16.093 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:15:15 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:15:16.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:15 smithi144 ceph-mon[33984]: 6.d scrub starts 2024-03-11T22:15:16.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:15 smithi144 ceph-mon[33984]: 6.d scrub ok 2024-03-11T22:15:16.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:15 smithi144 ceph-mon[33984]: 3.17 scrub starts 2024-03-11T22:15:16.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:15 smithi144 ceph-mon[33984]: 3.17 scrub ok 2024-03-11T22:15:16.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:15 smithi144 ceph-mon[33984]: 4.7 scrub starts 2024-03-11T22:15:16.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:15 smithi144 ceph-mon[33984]: 4.7 scrub ok 2024-03-11T22:15:16.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:16.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:16.187 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:16.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:16.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:16.312 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 14 15 31927 31912 142.428 79.6875 0.0052759 0.00701347 2024-03-11T22:15:16.337 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:16.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:16.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:16.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:16.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:16.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:16.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:16.588 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:16.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:16.619 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:16.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:16.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:16.757 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:16.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:16.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:16.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:16.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:16.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:16.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:17.001 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[39715]: 3.1d scrub starts 2024-03-11T22:15:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[39715]: 3.1d scrub ok 2024-03-11T22:15:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:15:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[39715]: 3.f scrub starts 2024-03-11T22:15:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[39715]: 3.f scrub ok 2024-03-11T22:15:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[39715]: osdmap e184: 8 total, 8 up, 8 in 2024-03-11T22:15:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[39715]: pgmap v282: 168 pgs: 168 active+clean; 1.9 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 196 MiB/s wr, 3.14k op/s 2024-03-11T22:15:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[30178]: 3.1d scrub starts 2024-03-11T22:15:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[30178]: 3.1d scrub ok 2024-03-11T22:15:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:15:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[30178]: 3.f scrub starts 2024-03-11T22:15:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[30178]: 3.f scrub ok 2024-03-11T22:15:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[30178]: osdmap e184: 8 total, 8 up, 8 in 2024-03-11T22:15:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[30178]: pgmap v282: 168 pgs: 168 active+clean; 1.9 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 196 MiB/s wr, 3.14k op/s 2024-03-11T22:15:17.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:17.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:17.033 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:17.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:17.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:16 smithi144 ceph-mon[33984]: 3.1d scrub starts 2024-03-11T22:15:17.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:16 smithi144 ceph-mon[33984]: 3.1d scrub ok 2024-03-11T22:15:17.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:15:17.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:16 smithi144 ceph-mon[33984]: 3.f scrub starts 2024-03-11T22:15:17.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:16 smithi144 ceph-mon[33984]: 3.f scrub ok 2024-03-11T22:15:17.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:16 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:17.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:16 smithi144 ceph-mon[33984]: osdmap e184: 8 total, 8 up, 8 in 2024-03-11T22:15:17.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:16 smithi144 ceph-mon[33984]: pgmap v282: 168 pgs: 168 active+clean; 1.9 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 196 MiB/s wr, 3.14k op/s 2024-03-11T22:15:17.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:17.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:17.134 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:17.168 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:17.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:17.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:17.270 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:17.312 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:17.313 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 15 15 33303 33288 138.666 86 0.0149146 0.00720518 2024-03-11T22:15:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:17.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:17.414 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:17.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:17.442 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:17.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:17.543 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:17.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:17.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:17.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:17.723 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:17.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:17.824 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:17.856 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:17.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:17.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:17.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:17.999 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:17.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:18.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:17 smithi045 ceph-mon[39715]: 5.18 scrub starts 2024-03-11T22:15:18.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:17 smithi045 ceph-mon[39715]: 5.18 scrub ok 2024-03-11T22:15:18.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:17 smithi045 ceph-mon[30178]: 5.18 scrub starts 2024-03-11T22:15:18.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:17 smithi045 ceph-mon[30178]: 5.18 scrub ok 2024-03-11T22:15:18.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:17 smithi144 ceph-mon[33984]: 5.18 scrub starts 2024-03-11T22:15:18.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:17 smithi144 ceph-mon[33984]: 5.18 scrub ok 2024-03-11T22:15:18.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:18.138 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:18.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:18.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:18.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:18.280 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:18.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:18.312 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 16 15 34989 34974 136.585 105.375 0.00543645 0.00731563 2024-03-11T22:15:18.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:18.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:18.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:18.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:18.523 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:18.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:18.557 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:18.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:18.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:18.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:18.699 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:18.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:18.800 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:18.835 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:18.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:18.896 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-03-11T22:15:12.164372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1440291164,"num_objects":22166,"num_object_clones":0,"num_object_copies":66498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22166,"num_whiteouts":0,"num_read":1059,"num_read_kb":811,"num_write":22404,"num_write_kb":1405997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25189,"ondisk_log_size":25189,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4979620,"kb_used_data":3869468,"kb_used_omap":12,"kb_used_meta":1110067,"kb_avail":744981596,"statfs":{"total":767960285184,"available":762861154304,"internally_reserved":0,"allocated":3962335232,"data_stored":3959594132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":1136709171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":812204504,"num_objects":12397,"num_object_clones":0,"num_object_copies":37191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12397,"num_whiteouts":0,"num_read":-5,"num_read_kb":-9,"num_write":12389,"num_write_kb":793298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001839"},"pg_stats":[{"pgid":"8.0","version":"181'1344","reported_seq":1366,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.594701+0000","last_change":"2024-03-11T22:14:52.240741+0000","last_active":"2024-03-11T22:15:10.594701+0000","last_peered":"2024-03-11T22:15:10.594701+0000","last_clean":"2024-03-11T22:15:10.594701+0000","last_became_active":"2024-03-11T22:14:52.240283+0000","last_became_peered":"2024-03-11T22:14:52.240283+0000","last_unstale":"2024-03-11T22:15:10.594701+0000","last_undegraded":"2024-03-11T22:15:10.594701+0000","last_fullsized":"2024-03-11T22:15:10.594701+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1344,"log_dups_size":0,"ondisk_log_size":1344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:01.057390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":1344,"num_object_clones":0,"num_object_copies":4032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1344,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":56,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.720022+0000","last_change":"2024-03-11T22:14:52.237967+0000","last_active":"2024-03-11T22:15:07.720022+0000","last_peered":"2024-03-11T22:15:07.720022+0000","last_clean":"2024-03-11T22:15:07.720022+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:15:07.720022+0000","last_undegraded":"2024-03-11T22:15:07.720022+0000","last_fullsized":"2024-03-11T22:15:07.720022+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:14:24.847623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:24.847623+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:30.799272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718597+0000","last_change":"2024-03-11T22:14:12.801583+0000","last_active":"2024-03-11T22:15:07.718597+0000","last_peered":"2024-03-11T22:15:07.718597+0000","last_clean":"2024-03-11T22:15:07.718597+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:15:07.718597+0000","last_undegraded":"2024-03-11T22:15:07.718597+0000","last_fullsized":"2024-03-11T22:15:07.718597+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:12.801470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:12.801470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:25.460969+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":352,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.711122+0000","last_change":"2024-03-11T22:14:29.061747+0000","last_active":"2024-03-11T22:15:07.711122+0000","last_peered":"2024-03-11T22:15:07.711122+0000","last_clean":"2024-03-11T22:15:07.711122+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:15:07.711122+0000","last_undegraded":"2024-03-11T22:15:07.711122+0000","last_fullsized":"2024-03-11T22:15:07.711122+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:14:29.061624+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_clean_scrub_stamp":"2024-03-11T22:14:29.061624+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:46.082482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":22,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.713518+0000","last_change":"2024-03-11T22:14:52.244107+0000","last_active":"2024-03-11T22:15:07.713518+0000","last_peered":"2024-03-11T22:15:07.713518+0000","last_clean":"2024-03-11T22:15:07.713518+0000","last_became_active":"2024-03-11T22:14:52.243690+0000","last_became_peered":"2024-03-11T22:14:52.243690+0000","last_unstale":"2024-03-11T22:15:07.713518+0000","last_undegraded":"2024-03-11T22:15:07.713518+0000","last_fullsized":"2024-03-11T22:15:07.713518+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:09.099443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:09.099443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:38.958957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":523,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718883+0000","last_change":"2024-03-11T22:14:10.906814+0000","last_active":"2024-03-11T22:15:07.718883+0000","last_peered":"2024-03-11T22:15:07.718883+0000","last_clean":"2024-03-11T22:15:07.718883+0000","last_became_active":"2024-03-11T22:13:38.571540+0000","last_became_peered":"2024-03-11T22:13:38.571540+0000","last_unstale":"2024-03-11T22:15:07.718883+0000","last_undegraded":"2024-03-11T22:15:07.718883+0000","last_fullsized":"2024-03-11T22:15:07.718883+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:10.906610+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:10.906610+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:19.084750+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"181'1420","reported_seq":1442,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.406260+0000","last_change":"2024-03-11T22:14:52.245149+0000","last_active":"2024-03-11T22:15:10.406260+0000","last_peered":"2024-03-11T22:15:10.406260+0000","last_clean":"2024-03-11T22:15:10.406260+0000","last_became_active":"2024-03-11T22:14:52.244799+0000","last_became_peered":"2024-03-11T22:14:52.244799+0000","last_unstale":"2024-03-11T22:15:10.406260+0000","last_undegraded":"2024-03-11T22:15:10.406260+0000","last_fullsized":"2024-03-11T22:15:10.406260+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1420,"log_dups_size":0,"ondisk_log_size":1420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:09.738242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93061120,"num_objects":1420,"num_object_clones":0,"num_object_copies":4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1420,"num_write_kb":90880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":77,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.710947+0000","last_change":"2024-03-11T22:15:05.001672+0000","last_active":"2024-03-11T22:15:07.710947+0000","last_peered":"2024-03-11T22:15:07.710947+0000","last_clean":"2024-03-11T22:15:07.710947+0000","last_became_active":"2024-03-11T22:14:52.239649+0000","last_became_peered":"2024-03-11T22:14:52.239649+0000","last_unstale":"2024-03-11T22:15:07.710947+0000","last_undegraded":"2024-03-11T22:15:07.710947+0000","last_fullsized":"2024-03-11T22:15:07.710947+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-11T22:15:05.001522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.001522+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.788107+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":272,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718182+0000","last_change":"2024-03-11T22:14:14.990482+0000","last_active":"2024-03-11T22:15:07.718182+0000","last_peered":"2024-03-11T22:15:07.718182+0000","last_clean":"2024-03-11T22:15:07.718182+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:15:07.718182+0000","last_undegraded":"2024-03-11T22:15:07.718182+0000","last_fullsized":"2024-03-11T22:15:07.718182+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.990368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.990368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:15.948091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.712802+0000","last_change":"2024-03-11T22:14:18.859904+0000","last_active":"2024-03-11T22:15:07.712802+0000","last_peered":"2024-03-11T22:15:07.712802+0000","last_clean":"2024-03-11T22:15:07.712802+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:15:07.712802+0000","last_undegraded":"2024-03-11T22:15:07.712802+0000","last_fullsized":"2024-03-11T22:15:07.712802+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:18.859546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:14:18.859546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:41.507002+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":251,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.711383+0000","last_change":"2024-03-11T22:12:50.861675+0000","last_active":"2024-03-11T22:15:07.711383+0000","last_peered":"2024-03-11T22:15:07.711383+0000","last_clean":"2024-03-11T22:15:07.711383+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:15:07.711383+0000","last_undegraded":"2024-03-11T22:15:07.711383+0000","last_fullsized":"2024-03-11T22:15:07.711383+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.861557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.861557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":523,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716165+0000","last_change":"2024-03-11T22:14:11.673622+0000","last_active":"2024-03-11T22:15:07.716165+0000","last_peered":"2024-03-11T22:15:07.716165+0000","last_clean":"2024-03-11T22:15:07.716165+0000","last_became_active":"2024-03-11T22:13:38.569606+0000","last_became_peered":"2024-03-11T22:13:38.569606+0000","last_unstale":"2024-03-11T22:15:07.716165+0000","last_undegraded":"2024-03-11T22:15:07.716165+0000","last_fullsized":"2024-03-11T22:15:07.716165+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:11.673454+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:11.673454+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:37.168404+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"181'1404","reported_seq":1426,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.668349+0000","last_change":"2024-03-11T22:14:52.243433+0000","last_active":"2024-03-11T22:15:10.668349+0000","last_peered":"2024-03-11T22:15:10.668349+0000","last_clean":"2024-03-11T22:15:10.668349+0000","last_became_active":"2024-03-11T22:14:52.243038+0000","last_became_peered":"2024-03-11T22:14:52.243038+0000","last_unstale":"2024-03-11T22:15:10.668349+0000","last_undegraded":"2024-03-11T22:15:10.668349+0000","last_fullsized":"2024-03-11T22:15:10.668349+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1404,"log_dups_size":0,"ondisk_log_size":1404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:52.893231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92012544,"num_objects":1404,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1404,"num_write_kb":89856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":60,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718518+0000","last_change":"2024-03-11T22:14:52.240819+0000","last_active":"2024-03-11T22:15:07.718518+0000","last_peered":"2024-03-11T22:15:07.718518+0000","last_clean":"2024-03-11T22:15:07.718518+0000","last_became_active":"2024-03-11T22:14:52.240437+0000","last_became_peered":"2024-03-11T22:14:52.240437+0000","last_unstale":"2024-03-11T22:15:07.718518+0000","last_undegraded":"2024-03-11T22:15:07.718518+0000","last_fullsized":"2024-03-11T22:15:07.718518+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:12:24.093112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:24.093112+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":23,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.711724+0000","last_change":"2024-03-11T22:14:52.242036+0000","last_active":"2024-03-11T22:15:07.711724+0000","last_peered":"2024-03-11T22:15:07.711724+0000","last_clean":"2024-03-11T22:15:07.711724+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:15:07.711724+0000","last_undegraded":"2024-03-11T22:15:07.711724+0000","last_fullsized":"2024-03-11T22:15:07.711724+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:25.892396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:25.892396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:35.013158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717124+0000","last_change":"2024-03-11T22:14:26.672389+0000","last_active":"2024-03-11T22:15:07.717124+0000","last_peered":"2024-03-11T22:15:07.717124+0000","last_clean":"2024-03-11T22:15:07.717124+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:15:07.717124+0000","last_undegraded":"2024-03-11T22:15:07.717124+0000","last_fullsized":"2024-03-11T22:15:07.717124+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:26.672153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.672153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:47.573533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":243,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718938+0000","last_change":"2024-03-11T22:13:17.533805+0000","last_active":"2024-03-11T22:15:07.718938+0000","last_peered":"2024-03-11T22:15:07.718938+0000","last_clean":"2024-03-11T22:15:07.718938+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:15:07.718938+0000","last_undegraded":"2024-03-11T22:15:07.718938+0000","last_fullsized":"2024-03-11T22:15:07.718938+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:17.533607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:17.533607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"93'130","reported_seq":544,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.715497+0000","last_change":"2024-03-11T22:15:04.424215+0000","last_active":"2024-03-11T22:15:07.715497+0000","last_peered":"2024-03-11T22:15:07.715497+0000","last_clean":"2024-03-11T22:15:07.715497+0000","last_became_active":"2024-03-11T22:14:43.194793+0000","last_became_peered":"2024-03-11T22:14:43.194793+0000","last_unstale":"2024-03-11T22:15:07.715497+0000","last_undegraded":"2024-03-11T22:15:07.715497+0000","last_fullsized":"2024-03-11T22:15:07.715497+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:04.424114+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:04.424114+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:32.665794+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3","version":"181'1364","reported_seq":1386,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.422064+0000","last_change":"2024-03-11T22:14:52.245584+0000","last_active":"2024-03-11T22:15:10.422064+0000","last_peered":"2024-03-11T22:15:10.422064+0000","last_clean":"2024-03-11T22:15:10.422064+0000","last_became_active":"2024-03-11T22:14:52.245192+0000","last_became_peered":"2024-03-11T22:14:52.245192+0000","last_unstale":"2024-03-11T22:15:10.422064+0000","last_undegraded":"2024-03-11T22:15:10.422064+0000","last_fullsized":"2024-03-11T22:15:10.422064+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1364,"log_dups_size":0,"ondisk_log_size":1364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:11.735324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89391104,"num_objects":1364,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1364,"num_write_kb":87296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":284,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.714482+0000","last_change":"2024-03-11T22:14:14.861720+0000","last_active":"2024-03-11T22:15:07.714482+0000","last_peered":"2024-03-11T22:15:07.714482+0000","last_clean":"2024-03-11T22:15:07.714482+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:15:07.714482+0000","last_undegraded":"2024-03-11T22:15:07.714482+0000","last_fullsized":"2024-03-11T22:15:07.714482+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:14:14.861573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.861573+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:23.560000+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.721364+0000","last_change":"2024-03-11T22:14:38.013139+0000","last_active":"2024-03-11T22:15:07.721364+0000","last_peered":"2024-03-11T22:15:07.721364+0000","last_clean":"2024-03-11T22:15:07.721364+0000","last_became_active":"2024-03-11T22:08:33.037537+0000","last_became_peered":"2024-03-11T22:08:33.037537+0000","last_unstale":"2024-03-11T22:15:07.721364+0000","last_undegraded":"2024-03-11T22:15:07.721364+0000","last_fullsized":"2024-03-11T22:15:07.721364+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:38.013058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:38.013058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:44.722852+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":52,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716916+0000","last_change":"2024-03-11T22:14:52.248753+0000","last_active":"2024-03-11T22:15:07.716916+0000","last_peered":"2024-03-11T22:15:07.716916+0000","last_clean":"2024-03-11T22:15:07.716916+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:15:07.716916+0000","last_undegraded":"2024-03-11T22:15:07.716916+0000","last_fullsized":"2024-03-11T22:15:07.716916+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:13:35.126665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:13:35.126665+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":243,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716746+0000","last_change":"2024-03-11T22:12:42.469626+0000","last_active":"2024-03-11T22:15:07.716746+0000","last_peered":"2024-03-11T22:15:07.716746+0000","last_clean":"2024-03-11T22:15:07.716746+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:15:07.716746+0000","last_undegraded":"2024-03-11T22:15:07.716746+0000","last_fullsized":"2024-03-11T22:15:07.716746+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:42.469501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:12:42.469501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":523,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716788+0000","last_change":"2024-03-11T22:14:12.712999+0000","last_active":"2024-03-11T22:15:07.716788+0000","last_peered":"2024-03-11T22:15:07.716788+0000","last_clean":"2024-03-11T22:15:07.716788+0000","last_became_active":"2024-03-11T22:13:38.573794+0000","last_became_peered":"2024-03-11T22:13:38.573794+0000","last_unstale":"2024-03-11T22:15:07.716788+0000","last_undegraded":"2024-03-11T22:15:07.716788+0000","last_fullsized":"2024-03-11T22:15:07.716788+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:12.712874+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:12.712874+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:40.396091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"181'1399","reported_seq":1421,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.660759+0000","last_change":"2024-03-11T22:14:52.236231+0000","last_active":"2024-03-11T22:15:10.660759+0000","last_peered":"2024-03-11T22:15:10.660759+0000","last_clean":"2024-03-11T22:15:10.660759+0000","last_became_active":"2024-03-11T22:14:52.235927+0000","last_became_peered":"2024-03-11T22:14:52.235927+0000","last_unstale":"2024-03-11T22:15:10.660759+0000","last_undegraded":"2024-03-11T22:15:10.660759+0000","last_fullsized":"2024-03-11T22:15:10.660759+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1399,"log_dups_size":0,"ondisk_log_size":1399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:56.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91684864,"num_objects":1399,"num_object_clones":0,"num_object_copies":4197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1399,"num_write_kb":89536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":22,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.719188+0000","last_change":"2024-03-11T22:14:52.233618+0000","last_active":"2024-03-11T22:15:07.719188+0000","last_peered":"2024-03-11T22:15:07.719188+0000","last_clean":"2024-03-11T22:15:07.719188+0000","last_became_active":"2024-03-11T22:14:52.233157+0000","last_became_peered":"2024-03-11T22:14:52.233157+0000","last_unstale":"2024-03-11T22:15:07.719188+0000","last_undegraded":"2024-03-11T22:15:07.719188+0000","last_fullsized":"2024-03-11T22:15:07.719188+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:40.165760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:40.165760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":287,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.721499+0000","last_change":"2024-03-11T22:12:39.447777+0000","last_active":"2024-03-11T22:15:07.721499+0000","last_peered":"2024-03-11T22:15:07.721499+0000","last_clean":"2024-03-11T22:15:07.721499+0000","last_became_active":"2024-03-11T22:08:35.085979+0000","last_became_peered":"2024-03-11T22:08:35.085979+0000","last_unstale":"2024-03-11T22:15:07.721499+0000","last_undegraded":"2024-03-11T22:15:07.721499+0000","last_fullsized":"2024-03-11T22:15:07.721499+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:12:39.447312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:39.447312+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.723720+0000","last_change":"2024-03-11T22:13:26.481234+0000","last_active":"2024-03-11T22:15:07.723720+0000","last_peered":"2024-03-11T22:15:07.723720+0000","last_clean":"2024-03-11T22:15:07.723720+0000","last_became_active":"2024-03-11T22:08:33.036469+0000","last_became_peered":"2024-03-11T22:08:33.036469+0000","last_unstale":"2024-03-11T22:15:07.723720+0000","last_undegraded":"2024-03-11T22:15:07.723720+0000","last_fullsized":"2024-03-11T22:15:07.723720+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:26.481122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:26.481122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":22,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718566+0000","last_change":"2024-03-11T22:14:52.232961+0000","last_active":"2024-03-11T22:15:07.718566+0000","last_peered":"2024-03-11T22:15:07.718566+0000","last_clean":"2024-03-11T22:15:07.718566+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:15:07.718566+0000","last_undegraded":"2024-03-11T22:15:07.718566+0000","last_fullsized":"2024-03-11T22:15:07.718566+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:35.248081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:35.248081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":468,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.711157+0000","last_change":"2024-03-11T22:15:04.043741+0000","last_active":"2024-03-11T22:15:07.711157+0000","last_peered":"2024-03-11T22:15:07.711157+0000","last_clean":"2024-03-11T22:15:07.711157+0000","last_became_active":"2024-03-11T22:14:52.240971+0000","last_became_peered":"2024-03-11T22:14:52.240971+0000","last_unstale":"2024-03-11T22:15:07.711157+0000","last_undegraded":"2024-03-11T22:15:07.711157+0000","last_fullsized":"2024-03-11T22:15:07.711157+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:04.043590+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:04.043590+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:13.992528+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"181'1347","reported_seq":1369,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.146272+0000","last_change":"2024-03-11T22:14:52.243501+0000","last_active":"2024-03-11T22:15:10.146272+0000","last_peered":"2024-03-11T22:15:10.146272+0000","last_clean":"2024-03-11T22:15:10.146272+0000","last_became_active":"2024-03-11T22:14:52.243195+0000","last_became_peered":"2024-03-11T22:14:52.243195+0000","last_unstale":"2024-03-11T22:15:10.146272+0000","last_undegraded":"2024-03-11T22:15:10.146272+0000","last_fullsized":"2024-03-11T22:15:10.146272+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1347,"log_dups_size":0,"ondisk_log_size":1347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:55.466333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88276992,"num_objects":1347,"num_object_clones":0,"num_object_copies":4041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1347,"num_write_kb":86208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":336,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.721640+0000","last_change":"2024-03-11T22:12:31.264333+0000","last_active":"2024-03-11T22:15:07.721640+0000","last_peered":"2024-03-11T22:15:07.721640+0000","last_clean":"2024-03-11T22:15:07.721640+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:15:07.721640+0000","last_undegraded":"2024-03-11T22:15:07.721640+0000","last_fullsized":"2024-03-11T22:15:07.721640+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:12:31.264213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.264213+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":62,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.715911+0000","last_change":"2024-03-11T22:14:52.248658+0000","last_active":"2024-03-11T22:15:07.715911+0000","last_peered":"2024-03-11T22:15:07.715911+0000","last_clean":"2024-03-11T22:15:07.715911+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:15:07.715911+0000","last_undegraded":"2024-03-11T22:15:07.715911+0000","last_fullsized":"2024-03-11T22:15:07.715911+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:13:27.028171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:27.028171+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":265,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:08.820563+0000","last_change":"2024-03-11T22:15:08.820614+0000","last_active":"2024-03-11T22:15:08.820563+0000","last_peered":"2024-03-11T22:15:08.820563+0000","last_clean":"2024-03-11T22:15:08.820563+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:15:08.820563+0000","last_undegraded":"2024-03-11T22:15:08.820563+0000","last_fullsized":"2024-03-11T22:15:08.820563+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:15:08.820296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:08.820296+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:20.880401+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":202,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718151+0000","last_change":"2024-03-11T22:14:33.134910+0000","last_active":"2024-03-11T22:15:07.718151+0000","last_peered":"2024-03-11T22:15:07.718151+0000","last_clean":"2024-03-11T22:15:07.718151+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:15:07.718151+0000","last_undegraded":"2024-03-11T22:15:07.718151+0000","last_fullsized":"2024-03-11T22:15:07.718151+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:33.134806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:14:33.134806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:53.275391+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":536,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718760+0000","last_change":"2024-03-11T22:15:06.243411+0000","last_active":"2024-03-11T22:15:07.718760+0000","last_peered":"2024-03-11T22:15:07.718760+0000","last_clean":"2024-03-11T22:15:07.718760+0000","last_became_active":"2024-03-11T22:14:38.164924+0000","last_became_peered":"2024-03-11T22:14:38.164924+0000","last_unstale":"2024-03-11T22:15:07.718760+0000","last_undegraded":"2024-03-11T22:15:07.718760+0000","last_fullsized":"2024-03-11T22:15:07.718760+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.243267+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.243267+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:13.005193+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"181'1384","reported_seq":1406,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.646265+0000","last_change":"2024-03-11T22:14:52.242543+0000","last_active":"2024-03-11T22:15:10.646265+0000","last_peered":"2024-03-11T22:15:10.646265+0000","last_clean":"2024-03-11T22:15:10.646265+0000","last_became_active":"2024-03-11T22:14:52.240224+0000","last_became_peered":"2024-03-11T22:14:52.240224+0000","last_unstale":"2024-03-11T22:15:10.646265+0000","last_undegraded":"2024-03-11T22:15:10.646265+0000","last_fullsized":"2024-03-11T22:15:10.646265+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1384,"log_dups_size":0,"ondisk_log_size":1384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:53.025545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90701824,"num_objects":1384,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1384,"num_write_kb":88576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":344,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.719136+0000","last_change":"2024-03-11T22:14:27.105480+0000","last_active":"2024-03-11T22:15:07.719136+0000","last_peered":"2024-03-11T22:15:07.719136+0000","last_clean":"2024-03-11T22:15:07.719136+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:15:07.719136+0000","last_undegraded":"2024-03-11T22:15:07.719136+0000","last_fullsized":"2024-03-11T22:15:07.719136+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:14:27.105394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:27.105394+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:56.745563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":275,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.713655+0000","last_change":"2024-03-11T22:14:28.936592+0000","last_active":"2024-03-11T22:15:07.713655+0000","last_peered":"2024-03-11T22:15:07.713655+0000","last_clean":"2024-03-11T22:15:07.713655+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:15:07.713655+0000","last_undegraded":"2024-03-11T22:15:07.713655+0000","last_fullsized":"2024-03-11T22:15:07.713655+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:14:28.936501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:28.936501+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:32.315105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717078+0000","last_change":"2024-03-11T22:12:45.466625+0000","last_active":"2024-03-11T22:15:07.717078+0000","last_peered":"2024-03-11T22:15:07.717078+0000","last_clean":"2024-03-11T22:15:07.717078+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:15:07.717078+0000","last_undegraded":"2024-03-11T22:15:07.717078+0000","last_fullsized":"2024-03-11T22:15:07.717078+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.466508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.466508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":244,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716027+0000","last_change":"2024-03-11T22:14:10.990148+0000","last_active":"2024-03-11T22:15:07.716027+0000","last_peered":"2024-03-11T22:15:07.716027+0000","last_clean":"2024-03-11T22:15:07.716027+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:15:07.716027+0000","last_undegraded":"2024-03-11T22:15:07.716027+0000","last_fullsized":"2024-03-11T22:15:07.716027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:10.990032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:10.990032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:13.968549+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":424,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:08.189142+0000","last_change":"2024-03-11T22:15:08.189185+0000","last_active":"2024-03-11T22:15:08.189142+0000","last_peered":"2024-03-11T22:15:08.189142+0000","last_clean":"2024-03-11T22:15:08.189142+0000","last_became_active":"2024-03-11T22:14:38.164708+0000","last_became_peered":"2024-03-11T22:14:38.164708+0000","last_unstale":"2024-03-11T22:15:08.189142+0000","last_undegraded":"2024-03-11T22:15:08.189142+0000","last_fullsized":"2024-03-11T22:15:08.189142+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:08.189043+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:08.189043+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:10.087716+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.f","version":"181'1389","reported_seq":1411,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.582518+0000","last_change":"2024-03-11T22:14:52.238556+0000","last_active":"2024-03-11T22:15:10.582518+0000","last_peered":"2024-03-11T22:15:10.582518+0000","last_clean":"2024-03-11T22:15:10.582518+0000","last_became_active":"2024-03-11T22:14:52.238241+0000","last_became_peered":"2024-03-11T22:14:52.238241+0000","last_unstale":"2024-03-11T22:15:10.582518+0000","last_undegraded":"2024-03-11T22:15:10.582518+0000","last_fullsized":"2024-03-11T22:15:10.582518+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1389,"log_dups_size":0,"ondisk_log_size":1389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:08.580556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91029504,"num_objects":1389,"num_object_clones":0,"num_object_copies":4167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1389,"num_write_kb":88896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718124+0000","last_change":"2024-03-11T22:14:18.961752+0000","last_active":"2024-03-11T22:15:07.718124+0000","last_peered":"2024-03-11T22:15:07.718124+0000","last_clean":"2024-03-11T22:15:07.718124+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:15:07.718124+0000","last_undegraded":"2024-03-11T22:15:07.718124+0000","last_fullsized":"2024-03-11T22:15:07.718124+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:18.961673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:14:18.961673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:45.145191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":288,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718534+0000","last_change":"2024-03-11T22:13:27.025211+0000","last_active":"2024-03-11T22:15:07.718534+0000","last_peered":"2024-03-11T22:15:07.718534+0000","last_clean":"2024-03-11T22:15:07.718534+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:15:07.718534+0000","last_undegraded":"2024-03-11T22:15:07.718534+0000","last_fullsized":"2024-03-11T22:15:07.718534+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:13:27.025050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:27.025050+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.712563+0000","last_change":"2024-03-11T22:14:36.079416+0000","last_active":"2024-03-11T22:15:07.712563+0000","last_peered":"2024-03-11T22:15:07.712563+0000","last_clean":"2024-03-11T22:15:07.712563+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:15:07.712563+0000","last_undegraded":"2024-03-11T22:15:07.712563+0000","last_fullsized":"2024-03-11T22:15:07.712563+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:36.079334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:36.079334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:41.253472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":252,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.713109+0000","last_change":"2024-03-11T22:13:20.800006+0000","last_active":"2024-03-11T22:15:07.713109+0000","last_peered":"2024-03-11T22:15:07.713109+0000","last_clean":"2024-03-11T22:15:07.713109+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:15:07.713109+0000","last_undegraded":"2024-03-11T22:15:07.713109+0000","last_fullsized":"2024-03-11T22:15:07.713109+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:20.799791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:20.799791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":424,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718121+0000","last_change":"2024-03-11T22:15:07.207510+0000","last_active":"2024-03-11T22:15:07.718121+0000","last_peered":"2024-03-11T22:15:07.718121+0000","last_clean":"2024-03-11T22:15:07.718121+0000","last_became_active":"2024-03-11T22:14:38.166870+0000","last_became_peered":"2024-03-11T22:14:38.166870+0000","last_unstale":"2024-03-11T22:15:07.718121+0000","last_undegraded":"2024-03-11T22:15:07.718121+0000","last_fullsized":"2024-03-11T22:15:07.718121+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:07.207365+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:07.207365+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:28.797642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"181'1364","reported_seq":1386,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.423706+0000","last_change":"2024-03-11T22:14:52.247032+0000","last_active":"2024-03-11T22:15:10.423706+0000","last_peered":"2024-03-11T22:15:10.423706+0000","last_clean":"2024-03-11T22:15:10.423706+0000","last_became_active":"2024-03-11T22:14:52.246614+0000","last_became_peered":"2024-03-11T22:14:52.246614+0000","last_unstale":"2024-03-11T22:15:10.423706+0000","last_undegraded":"2024-03-11T22:15:10.423706+0000","last_fullsized":"2024-03-11T22:15:10.423706+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1364,"log_dups_size":0,"ondisk_log_size":1364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:54.213666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89391104,"num_objects":1364,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1364,"num_write_kb":87296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":256,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717395+0000","last_change":"2024-03-11T22:14:29.697272+0000","last_active":"2024-03-11T22:15:07.717395+0000","last_peered":"2024-03-11T22:15:07.717395+0000","last_clean":"2024-03-11T22:15:07.717395+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:15:07.717395+0000","last_undegraded":"2024-03-11T22:15:07.717395+0000","last_fullsized":"2024-03-11T22:15:07.717395+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:29.697144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:29.697144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:37.479317+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718511+0000","last_change":"2024-03-11T22:14:21.065394+0000","last_active":"2024-03-11T22:15:07.718511+0000","last_peered":"2024-03-11T22:15:07.718511+0000","last_clean":"2024-03-11T22:15:07.718511+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:15:07.718511+0000","last_undegraded":"2024-03-11T22:15:07.718511+0000","last_fullsized":"2024-03-11T22:15:07.718511+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:14:21.065306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.065306+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:37.928690+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":643,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:09.110397+0000","last_change":"2024-03-11T22:14:13.848646+0000","last_active":"2024-03-11T22:15:09.110397+0000","last_peered":"2024-03-11T22:15:09.110397+0000","last_clean":"2024-03-11T22:15:09.110397+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:15:09.110397+0000","last_undegraded":"2024-03-11T22:15:09.110397+0000","last_fullsized":"2024-03-11T22:15:09.110397+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:14:13.848518+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:14:13.848518+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:36.660519+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":364,"num_read_kb":359,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.721929+0000","last_change":"2024-03-11T22:14:35.025497+0000","last_active":"2024-03-11T22:15:07.721929+0000","last_peered":"2024-03-11T22:15:07.721929+0000","last_clean":"2024-03-11T22:15:07.721929+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:15:07.721929+0000","last_undegraded":"2024-03-11T22:15:07.721929+0000","last_fullsized":"2024-03-11T22:15:07.721929+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:35.025383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:35.025383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:52.119736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":260,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.712333+0000","last_change":"2024-03-11T22:14:38.073584+0000","last_active":"2024-03-11T22:15:07.712333+0000","last_peered":"2024-03-11T22:15:07.712333+0000","last_clean":"2024-03-11T22:15:07.712333+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:15:07.712333+0000","last_undegraded":"2024-03-11T22:15:07.712333+0000","last_fullsized":"2024-03-11T22:15:07.712333+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:38.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:38.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:40.723809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":536,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.715724+0000","last_change":"2024-03-11T22:15:06.523755+0000","last_active":"2024-03-11T22:15:07.715724+0000","last_peered":"2024-03-11T22:15:07.715724+0000","last_clean":"2024-03-11T22:15:07.715724+0000","last_became_active":"2024-03-11T22:14:38.164101+0000","last_became_peered":"2024-03-11T22:14:38.164101+0000","last_unstale":"2024-03-11T22:15:07.715724+0000","last_undegraded":"2024-03-11T22:15:07.715724+0000","last_fullsized":"2024-03-11T22:15:07.715724+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.523513+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.523513+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:09.939331+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"181'1295","reported_seq":1316,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.181629+0000","last_change":"2024-03-11T22:14:52.240542+0000","last_active":"2024-03-11T22:15:10.181629+0000","last_peered":"2024-03-11T22:15:10.181629+0000","last_clean":"2024-03-11T22:15:10.181629+0000","last_became_active":"2024-03-11T22:14:52.240210+0000","last_became_peered":"2024-03-11T22:14:52.240210+0000","last_unstale":"2024-03-11T22:15:10.181629+0000","last_undegraded":"2024-03-11T22:15:10.181629+0000","last_fullsized":"2024-03-11T22:15:10.181629+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:06.873414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84869120,"num_objects":1295,"num_object_clones":0,"num_object_copies":3885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1295,"num_write_kb":82880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":202,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.711205+0000","last_change":"2024-03-11T22:14:23.041388+0000","last_active":"2024-03-11T22:15:07.711205+0000","last_peered":"2024-03-11T22:15:07.711205+0000","last_clean":"2024-03-11T22:15:07.711205+0000","last_became_active":"2024-03-11T22:10:35.412547+0000","last_became_peered":"2024-03-11T22:10:35.412547+0000","last_unstale":"2024-03-11T22:15:07.711205+0000","last_undegraded":"2024-03-11T22:15:07.711205+0000","last_fullsized":"2024-03-11T22:15:07.711205+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.041267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.041267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:29.148588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":370,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718565+0000","last_change":"2024-03-11T22:14:37.137570+0000","last_active":"2024-03-11T22:15:07.718565+0000","last_peered":"2024-03-11T22:15:07.718565+0000","last_clean":"2024-03-11T22:15:07.718565+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:15:07.718565+0000","last_undegraded":"2024-03-11T22:15:07.718565+0000","last_fullsized":"2024-03-11T22:15:07.718565+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-11T22:14:37.137464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:37.137464+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:49.689233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718239+0000","last_change":"2024-03-11T22:13:33.460800+0000","last_active":"2024-03-11T22:15:07.718239+0000","last_peered":"2024-03-11T22:15:07.718239+0000","last_clean":"2024-03-11T22:15:07.718239+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:15:07.718239+0000","last_undegraded":"2024-03-11T22:15:07.718239+0000","last_fullsized":"2024-03-11T22:15:07.718239+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:33.460308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:13:33.460308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":536,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716135+0000","last_change":"2024-03-11T22:15:03.739785+0000","last_active":"2024-03-11T22:15:07.716135+0000","last_peered":"2024-03-11T22:15:07.716135+0000","last_clean":"2024-03-11T22:15:07.716135+0000","last_became_active":"2024-03-11T22:14:38.165091+0000","last_became_peered":"2024-03-11T22:14:38.165091+0000","last_unstale":"2024-03-11T22:15:07.716135+0000","last_undegraded":"2024-03-11T22:15:07.716135+0000","last_fullsized":"2024-03-11T22:15:07.716135+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:03.739634+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:15:03.739634+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:26.259765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":252,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716664+0000","last_change":"2024-03-11T22:13:34.478753+0000","last_active":"2024-03-11T22:15:07.716664+0000","last_peered":"2024-03-11T22:15:07.716664+0000","last_clean":"2024-03-11T22:15:07.716664+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:15:07.716664+0000","last_undegraded":"2024-03-11T22:15:07.716664+0000","last_fullsized":"2024-03-11T22:15:07.716664+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:34.478634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:34.478634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"181'1335","reported_seq":1357,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.585708+0000","last_change":"2024-03-11T22:14:52.240355+0000","last_active":"2024-03-11T22:15:10.585708+0000","last_peered":"2024-03-11T22:15:10.585708+0000","last_clean":"2024-03-11T22:15:10.585708+0000","last_became_active":"2024-03-11T22:14:52.238824+0000","last_became_peered":"2024-03-11T22:14:52.238824+0000","last_unstale":"2024-03-11T22:15:10.585708+0000","last_undegraded":"2024-03-11T22:15:10.585708+0000","last_fullsized":"2024-03-11T22:15:10.585708+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1335,"log_dups_size":0,"ondisk_log_size":1335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:07.988297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87490560,"num_objects":1335,"num_object_clones":0,"num_object_copies":4005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1335,"num_write_kb":85440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.710838+0000","last_change":"2024-03-11T22:14:21.033755+0000","last_active":"2024-03-11T22:15:07.710838+0000","last_peered":"2024-03-11T22:15:07.710838+0000","last_clean":"2024-03-11T22:15:07.710838+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:15:07.710838+0000","last_undegraded":"2024-03-11T22:15:07.710838+0000","last_fullsized":"2024-03-11T22:15:07.710838+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:21.033664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.033664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:38.113619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":287,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.714791+0000","last_change":"2024-03-11T22:13:24.705841+0000","last_active":"2024-03-11T22:15:07.714791+0000","last_peered":"2024-03-11T22:15:07.714791+0000","last_clean":"2024-03-11T22:15:07.714791+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:15:07.714791+0000","last_undegraded":"2024-03-11T22:15:07.714791+0000","last_fullsized":"2024-03-11T22:15:07.714791+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-03-11T22:13:24.705697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:24.705697+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":281,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.711879+0000","last_change":"2024-03-11T22:14:09.871684+0000","last_active":"2024-03-11T22:15:07.711879+0000","last_peered":"2024-03-11T22:15:07.711879+0000","last_clean":"2024-03-11T22:15:07.711879+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:15:07.711879+0000","last_undegraded":"2024-03-11T22:15:07.711879+0000","last_fullsized":"2024-03-11T22:15:07.711879+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:09.871543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:14:09.871543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:16.454261+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.712373+0000","last_change":"2024-03-11T22:13:19.750323+0000","last_active":"2024-03-11T22:15:07.712373+0000","last_peered":"2024-03-11T22:15:07.712373+0000","last_clean":"2024-03-11T22:15:07.712373+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:15:07.712373+0000","last_undegraded":"2024-03-11T22:15:07.712373+0000","last_fullsized":"2024-03-11T22:15:07.712373+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:19.748477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.748477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":202,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718455+0000","last_change":"2024-03-11T22:14:34.131682+0000","last_active":"2024-03-11T22:15:07.718455+0000","last_peered":"2024-03-11T22:15:07.718455+0000","last_clean":"2024-03-11T22:15:07.718455+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:15:07.718455+0000","last_undegraded":"2024-03-11T22:15:07.718455+0000","last_fullsized":"2024-03-11T22:15:07.718455+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.131530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:02.487163+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":543,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717819+0000","last_change":"2024-03-11T22:15:04.771002+0000","last_active":"2024-03-11T22:15:07.717819+0000","last_peered":"2024-03-11T22:15:07.717819+0000","last_clean":"2024-03-11T22:15:07.717819+0000","last_became_active":"2024-03-11T22:14:38.166129+0000","last_became_peered":"2024-03-11T22:14:38.166129+0000","last_unstale":"2024-03-11T22:15:07.717819+0000","last_undegraded":"2024-03-11T22:15:07.717819+0000","last_fullsized":"2024-03-11T22:15:07.717819+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:04.770858+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:04.770858+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:30.991906+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"181'1371","reported_seq":1393,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.141238+0000","last_change":"2024-03-11T22:14:52.243442+0000","last_active":"2024-03-11T22:15:10.141238+0000","last_peered":"2024-03-11T22:15:10.141238+0000","last_clean":"2024-03-11T22:15:10.141238+0000","last_became_active":"2024-03-11T22:14:52.243066+0000","last_became_peered":"2024-03-11T22:14:52.243066+0000","last_unstale":"2024-03-11T22:15:10.141238+0000","last_undegraded":"2024-03-11T22:15:10.141238+0000","last_fullsized":"2024-03-11T22:15:10.141238+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1371,"log_dups_size":0,"ondisk_log_size":1371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:11.959240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89849856,"num_objects":1371,"num_object_clones":0,"num_object_copies":4113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1371,"num_write_kb":87744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":256,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716129+0000","last_change":"2024-03-11T22:14:16.998358+0000","last_active":"2024-03-11T22:15:07.716129+0000","last_peered":"2024-03-11T22:15:07.716129+0000","last_clean":"2024-03-11T22:15:07.716129+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:15:07.716129+0000","last_undegraded":"2024-03-11T22:15:07.716129+0000","last_fullsized":"2024-03-11T22:15:07.716129+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:16.998238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:16.998238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:18.606617+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":388,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.712691+0000","last_change":"2024-03-11T22:13:17.830633+0000","last_active":"2024-03-11T22:15:07.712691+0000","last_peered":"2024-03-11T22:15:07.712691+0000","last_clean":"2024-03-11T22:15:07.712691+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:15:07.712691+0000","last_undegraded":"2024-03-11T22:15:07.712691+0000","last_fullsized":"2024-03-11T22:15:07.712691+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-11T22:13:17.830327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:17.830327+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718412+0000","last_change":"2024-03-11T22:14:52.241367+0000","last_active":"2024-03-11T22:15:07.718412+0000","last_peered":"2024-03-11T22:15:07.718412+0000","last_clean":"2024-03-11T22:15:07.718412+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:15:07.718412+0000","last_undegraded":"2024-03-11T22:15:07.718412+0000","last_fullsized":"2024-03-11T22:15:07.718412+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:28.028531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:28.028531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":276,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716976+0000","last_change":"2024-03-11T22:14:16.662735+0000","last_active":"2024-03-11T22:15:07.716976+0000","last_peered":"2024-03-11T22:15:07.716976+0000","last_clean":"2024-03-11T22:15:07.716976+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:15:07.716976+0000","last_undegraded":"2024-03-11T22:15:07.716976+0000","last_fullsized":"2024-03-11T22:15:07.716976+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:14:16.662596+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:14:16.662596+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:45.012386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":244,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.722996+0000","last_change":"2024-03-11T22:12:45.439711+0000","last_active":"2024-03-11T22:15:07.722996+0000","last_peered":"2024-03-11T22:15:07.722996+0000","last_clean":"2024-03-11T22:15:07.722996+0000","last_became_active":"2024-03-11T22:08:39.083738+0000","last_became_peered":"2024-03-11T22:08:39.083738+0000","last_unstale":"2024-03-11T22:15:07.722996+0000","last_undegraded":"2024-03-11T22:15:07.722996+0000","last_fullsized":"2024-03-11T22:15:07.722996+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.439584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":535,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.715849+0000","last_change":"2024-03-11T22:15:06.673117+0000","last_active":"2024-03-11T22:15:07.715849+0000","last_peered":"2024-03-11T22:15:07.715849+0000","last_clean":"2024-03-11T22:15:07.715849+0000","last_became_active":"2024-03-11T22:14:38.164859+0000","last_became_peered":"2024-03-11T22:14:38.164859+0000","last_unstale":"2024-03-11T22:15:07.715849+0000","last_undegraded":"2024-03-11T22:15:07.715849+0000","last_fullsized":"2024-03-11T22:15:07.715849+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.672938+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.672938+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:25.674338+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"181'1374","reported_seq":1396,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.408140+0000","last_change":"2024-03-11T22:14:52.242768+0000","last_active":"2024-03-11T22:15:10.408140+0000","last_peered":"2024-03-11T22:15:10.408140+0000","last_clean":"2024-03-11T22:15:10.408140+0000","last_became_active":"2024-03-11T22:14:52.242455+0000","last_became_peered":"2024-03-11T22:14:52.242455+0000","last_unstale":"2024-03-11T22:15:10.408140+0000","last_undegraded":"2024-03-11T22:15:10.408140+0000","last_fullsized":"2024-03-11T22:15:10.408140+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1374,"log_dups_size":0,"ondisk_log_size":1374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:15.907365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90046464,"num_objects":1374,"num_object_clones":0,"num_object_copies":4122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1374,"num_write_kb":87936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":283,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718956+0000","last_change":"2024-03-11T22:13:15.950715+0000","last_active":"2024-03-11T22:15:07.718956+0000","last_peered":"2024-03-11T22:15:07.718956+0000","last_clean":"2024-03-11T22:15:07.718956+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:15:07.718956+0000","last_undegraded":"2024-03-11T22:15:07.718956+0000","last_fullsized":"2024-03-11T22:15:07.718956+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:13:15.950564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:15.950564+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716595+0000","last_change":"2024-03-11T22:14:37.615454+0000","last_active":"2024-03-11T22:15:07.716595+0000","last_peered":"2024-03-11T22:15:07.716595+0000","last_clean":"2024-03-11T22:15:07.716595+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:15:07.716595+0000","last_undegraded":"2024-03-11T22:15:07.716595+0000","last_fullsized":"2024-03-11T22:15:07.716595+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:37.615375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:37.615375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:04.760537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":256,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.709960+0000","last_change":"2024-03-11T22:13:19.781178+0000","last_active":"2024-03-11T22:15:07.709960+0000","last_peered":"2024-03-11T22:15:07.709960+0000","last_clean":"2024-03-11T22:15:07.709960+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:15:07.709960+0000","last_undegraded":"2024-03-11T22:15:07.709960+0000","last_fullsized":"2024-03-11T22:15:07.709960+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:19.776020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.776020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":252,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.721359+0000","last_change":"2024-03-11T22:14:36.008065+0000","last_active":"2024-03-11T22:15:07.721359+0000","last_peered":"2024-03-11T22:15:07.721359+0000","last_clean":"2024-03-11T22:15:07.721359+0000","last_became_active":"2024-03-11T22:08:39.085496+0000","last_became_peered":"2024-03-11T22:08:39.085496+0000","last_unstale":"2024-03-11T22:15:07.721359+0000","last_undegraded":"2024-03-11T22:15:07.721359+0000","last_fullsized":"2024-03-11T22:15:07.721359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:36.007981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:36.007981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:55.604651+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"93'130","reported_seq":351,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:09.205207+0000","last_change":"2024-03-11T22:15:09.205252+0000","last_active":"2024-03-11T22:15:09.205207+0000","last_peered":"2024-03-11T22:15:09.205207+0000","last_clean":"2024-03-11T22:15:09.205207+0000","last_became_active":"2024-03-11T22:14:52.236711+0000","last_became_peered":"2024-03-11T22:14:52.236711+0000","last_unstale":"2024-03-11T22:15:09.205207+0000","last_undegraded":"2024-03-11T22:15:09.205207+0000","last_fullsized":"2024-03-11T22:15:09.205207+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:09.205116+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:09.205116+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:10.406518+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"181'1491","reported_seq":1513,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.655171+0000","last_change":"2024-03-11T22:14:52.243118+0000","last_active":"2024-03-11T22:15:10.655171+0000","last_peered":"2024-03-11T22:15:10.655171+0000","last_clean":"2024-03-11T22:15:10.655171+0000","last_became_active":"2024-03-11T22:14:52.242807+0000","last_became_peered":"2024-03-11T22:14:52.242807+0000","last_unstale":"2024-03-11T22:15:10.655171+0000","last_undegraded":"2024-03-11T22:15:10.655171+0000","last_fullsized":"2024-03-11T22:15:10.655171+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1491,"log_dups_size":0,"ondisk_log_size":1491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:19.483709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97714176,"num_objects":1491,"num_object_clones":0,"num_object_copies":4473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1491,"num_write_kb":95424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":299,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718039+0000","last_change":"2024-03-11T22:14:26.007003+0000","last_active":"2024-03-11T22:15:07.718039+0000","last_peered":"2024-03-11T22:15:07.718039+0000","last_clean":"2024-03-11T22:15:07.718039+0000","last_became_active":"2024-03-11T22:08:35.087046+0000","last_became_peered":"2024-03-11T22:08:35.087046+0000","last_unstale":"2024-03-11T22:15:07.718039+0000","last_undegraded":"2024-03-11T22:15:07.718039+0000","last_fullsized":"2024-03-11T22:15:07.718039+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:14:26.006872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.006872+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:52.717574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":203,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718402+0000","last_change":"2024-03-11T22:14:34.649271+0000","last_active":"2024-03-11T22:15:07.718402+0000","last_peered":"2024-03-11T22:15:07.718402+0000","last_clean":"2024-03-11T22:15:07.718402+0000","last_became_active":"2024-03-11T22:10:35.418715+0000","last_became_peered":"2024-03-11T22:10:35.418715+0000","last_unstale":"2024-03-11T22:15:07.718402+0000","last_undegraded":"2024-03-11T22:15:07.718402+0000","last_fullsized":"2024-03-11T22:15:07.718402+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.649172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.649172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:51.298499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.710614+0000","last_change":"2024-03-11T22:13:24.767308+0000","last_active":"2024-03-11T22:15:07.710614+0000","last_peered":"2024-03-11T22:15:07.710614+0000","last_clean":"2024-03-11T22:15:07.710614+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:15:07.710614+0000","last_undegraded":"2024-03-11T22:15:07.710614+0000","last_fullsized":"2024-03-11T22:15:07.710614+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:24.764906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.710319+0000","last_clean_scrub_stamp":"2024-03-11T22:13:24.764906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":202,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.710095+0000","last_change":"2024-03-11T22:14:14.005962+0000","last_active":"2024-03-11T22:15:07.710095+0000","last_peered":"2024-03-11T22:15:07.710095+0000","last_clean":"2024-03-11T22:15:07.710095+0000","last_became_active":"2024-03-11T22:10:35.410893+0000","last_became_peered":"2024-03-11T22:10:35.410893+0000","last_unstale":"2024-03-11T22:15:07.710095+0000","last_undegraded":"2024-03-11T22:15:07.710095+0000","last_fullsized":"2024-03-11T22:15:07.710095+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.005795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.005795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:32.664197+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"93'130","reported_seq":432,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718533+0000","last_change":"2024-03-11T22:15:06.157130+0000","last_active":"2024-03-11T22:15:07.718533+0000","last_peered":"2024-03-11T22:15:07.718533+0000","last_clean":"2024-03-11T22:15:07.718533+0000","last_became_active":"2024-03-11T22:14:43.195433+0000","last_became_peered":"2024-03-11T22:14:43.195433+0000","last_unstale":"2024-03-11T22:15:07.718533+0000","last_undegraded":"2024-03-11T22:15:07.718533+0000","last_fullsized":"2024-03-11T22:15:07.718533+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.157005+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.157005+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:06.486166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"181'1343","reported_seq":1365,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.174602+0000","last_change":"2024-03-11T22:14:52.240970+0000","last_active":"2024-03-11T22:15:10.174602+0000","last_peered":"2024-03-11T22:15:10.174602+0000","last_clean":"2024-03-11T22:15:10.174602+0000","last_became_active":"2024-03-11T22:14:52.240614+0000","last_became_peered":"2024-03-11T22:14:52.240614+0000","last_unstale":"2024-03-11T22:15:10.174602+0000","last_undegraded":"2024-03-11T22:15:10.174602+0000","last_fullsized":"2024-03-11T22:15:10.174602+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1343,"log_dups_size":0,"ondisk_log_size":1343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:17.716033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88014848,"num_objects":1343,"num_object_clones":0,"num_object_copies":4029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1343,"num_write_kb":85952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.709846+0000","last_change":"2024-03-11T22:14:24.051145+0000","last_active":"2024-03-11T22:15:07.709846+0000","last_peered":"2024-03-11T22:15:07.709846+0000","last_clean":"2024-03-11T22:15:07.709846+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:15:07.709846+0000","last_undegraded":"2024-03-11T22:15:07.709846+0000","last_fullsized":"2024-03-11T22:15:07.709846+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:24.051020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:14:24.051020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:48.994239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":71,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716166+0000","last_change":"2024-03-11T22:14:52.248333+0000","last_active":"2024-03-11T22:15:07.716166+0000","last_peered":"2024-03-11T22:15:07.716166+0000","last_clean":"2024-03-11T22:15:07.716166+0000","last_became_active":"2024-03-11T22:14:52.247400+0000","last_became_peered":"2024-03-11T22:14:52.247400+0000","last_unstale":"2024-03-11T22:15:07.716166+0000","last_undegraded":"2024-03-11T22:15:07.716166+0000","last_fullsized":"2024-03-11T22:15:07.716166+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:11:21.379862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:21.379862+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:08.478653+0000","last_change":"2024-03-11T22:15:08.478721+0000","last_active":"2024-03-11T22:15:08.478653+0000","last_peered":"2024-03-11T22:15:08.478653+0000","last_clean":"2024-03-11T22:15:08.478653+0000","last_became_active":"2024-03-11T22:14:52.239329+0000","last_became_peered":"2024-03-11T22:14:52.239329+0000","last_unstale":"2024-03-11T22:15:08.478653+0000","last_undegraded":"2024-03-11T22:15:08.478653+0000","last_fullsized":"2024-03-11T22:15:08.478653+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:08.477912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:08.477912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:13.992163+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":245,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718862+0000","last_change":"2024-03-11T22:12:36.449430+0000","last_active":"2024-03-11T22:15:07.718862+0000","last_peered":"2024-03-11T22:15:07.718862+0000","last_clean":"2024-03-11T22:15:07.718862+0000","last_became_active":"2024-03-11T22:08:39.083141+0000","last_became_peered":"2024-03-11T22:08:39.083141+0000","last_unstale":"2024-03-11T22:15:07.718862+0000","last_undegraded":"2024-03-11T22:15:07.718862+0000","last_fullsized":"2024-03-11T22:15:07.718862+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:12:36.449307+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:12:36.449307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"93'130","reported_seq":550,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718633+0000","last_change":"2024-03-11T22:15:05.249082+0000","last_active":"2024-03-11T22:15:07.718633+0000","last_peered":"2024-03-11T22:15:07.718633+0000","last_clean":"2024-03-11T22:15:07.718633+0000","last_became_active":"2024-03-11T22:14:43.196117+0000","last_became_peered":"2024-03-11T22:14:43.196117+0000","last_unstale":"2024-03-11T22:15:07.718633+0000","last_undegraded":"2024-03-11T22:15:07.718633+0000","last_fullsized":"2024-03-11T22:15:07.718633+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:05.248940+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.248940+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:24.043494+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"181'1344","reported_seq":1366,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:10.120744+0000","last_change":"2024-03-11T22:14:52.249598+0000","last_active":"2024-03-11T22:15:10.120744+0000","last_peered":"2024-03-11T22:15:10.120744+0000","last_clean":"2024-03-11T22:15:10.120744+0000","last_became_active":"2024-03-11T22:14:52.248793+0000","last_became_peered":"2024-03-11T22:14:52.248793+0000","last_unstale":"2024-03-11T22:15:10.120744+0000","last_undegraded":"2024-03-11T22:15:10.120744+0000","last_fullsized":"2024-03-11T22:15:10.120744+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":1344,"log_dups_size":0,"ondisk_log_size":1344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:16.661255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":1344,"num_object_clones":0,"num_object_copies":4032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1344,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"99'9","reported_seq":283,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718214+0000","last_change":"2024-03-11T22:14:36.152155+0000","last_active":"2024-03-11T22:15:07.718214+0000","last_peered":"2024-03-11T22:15:07.718214+0000","last_clean":"2024-03-11T22:15:07.718214+0000","last_became_active":"2024-03-11T22:10:35.413089+0000","last_became_peered":"2024-03-11T22:10:35.413089+0000","last_unstale":"2024-03-11T22:15:07.718214+0000","last_undegraded":"2024-03-11T22:15:07.718214+0000","last_fullsized":"2024-03-11T22:15:07.718214+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'9","last_scrub_stamp":"2024-03-11T22:14:36.152067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:36.152067+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:59.182225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"91'22","reported_seq":292,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.715969+0000","last_change":"2024-03-11T22:13:18.458453+0000","last_active":"2024-03-11T22:15:07.715969+0000","last_peered":"2024-03-11T22:15:07.715969+0000","last_clean":"2024-03-11T22:15:07.715969+0000","last_became_active":"2024-03-11T22:08:35.085916+0000","last_became_peered":"2024-03-11T22:08:35.085916+0000","last_unstale":"2024-03-11T22:15:07.715969+0000","last_undegraded":"2024-03-11T22:15:07.715969+0000","last_fullsized":"2024-03-11T22:15:07.715969+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'22","last_scrub_stamp":"2024-03-11T22:13:18.458313+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:13:18.458313+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":32,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716647+0000","last_change":"2024-03-11T22:14:52.239801+0000","last_active":"2024-03-11T22:15:07.716647+0000","last_peered":"2024-03-11T22:15:07.716647+0000","last_clean":"2024-03-11T22:15:07.716647+0000","last_became_active":"2024-03-11T22:14:52.239442+0000","last_became_peered":"2024-03-11T22:14:52.239442+0000","last_unstale":"2024-03-11T22:15:07.716647+0000","last_undegraded":"2024-03-11T22:15:07.716647+0000","last_fullsized":"2024-03-11T22:15:07.716647+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:13:24.079822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:24.079822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":243,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717159+0000","last_change":"2024-03-11T22:12:38.434933+0000","last_active":"2024-03-11T22:15:07.717159+0000","last_peered":"2024-03-11T22:15:07.717159+0000","last_clean":"2024-03-11T22:15:07.717159+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:15:07.717159+0000","last_undegraded":"2024-03-11T22:15:07.717159+0000","last_fullsized":"2024-03-11T22:15:07.717159+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:38.434796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:38.434796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"93'130","reported_seq":535,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.712925+0000","last_change":"2024-03-11T22:15:07.482011+0000","last_active":"2024-03-11T22:15:07.712925+0000","last_peered":"2024-03-11T22:15:07.712925+0000","last_clean":"2024-03-11T22:15:07.712925+0000","last_became_active":"2024-03-11T22:14:52.240083+0000","last_became_peered":"2024-03-11T22:14:52.240083+0000","last_unstale":"2024-03-11T22:15:07.712925+0000","last_undegraded":"2024-03-11T22:15:07.712925+0000","last_fullsized":"2024-03-11T22:15:07.712925+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:07.481870+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:07.481870+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:26.692915+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":260,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.724018+0000","last_change":"2024-03-11T22:12:21.278151+0000","last_active":"2024-03-11T22:15:07.724018+0000","last_peered":"2024-03-11T22:15:07.724018+0000","last_clean":"2024-03-11T22:15:07.724018+0000","last_became_active":"2024-03-11T22:08:35.085914+0000","last_became_peered":"2024-03-11T22:08:35.085914+0000","last_unstale":"2024-03-11T22:15:07.724018+0000","last_undegraded":"2024-03-11T22:15:07.724018+0000","last_fullsized":"2024-03-11T22:15:07.724018+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:12:21.270671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:21.270671+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717556+0000","last_change":"2024-03-11T22:13:28.086496+0000","last_active":"2024-03-11T22:15:07.717556+0000","last_peered":"2024-03-11T22:15:07.717556+0000","last_clean":"2024-03-11T22:15:07.717556+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:15:07.717556+0000","last_undegraded":"2024-03-11T22:15:07.717556+0000","last_fullsized":"2024-03-11T22:15:07.717556+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:28.086387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:13:28.086387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717060+0000","last_change":"2024-03-11T22:13:25.550514+0000","last_active":"2024-03-11T22:15:07.717060+0000","last_peered":"2024-03-11T22:15:07.717060+0000","last_clean":"2024-03-11T22:15:07.717060+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:15:07.717060+0000","last_undegraded":"2024-03-11T22:15:07.717060+0000","last_fullsized":"2024-03-11T22:15:07.717060+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:25.550429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:07.195289+0000","last_clean_scrub_stamp":"2024-03-11T22:13:25.550429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":193,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718560+0000","last_change":"2024-03-11T22:13:37.172811+0000","last_active":"2024-03-11T22:15:07.718560+0000","last_peered":"2024-03-11T22:15:07.718560+0000","last_clean":"2024-03-11T22:15:07.718560+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:15:07.718560+0000","last_undegraded":"2024-03-11T22:15:07.718560+0000","last_fullsized":"2024-03-11T22:15:07.718560+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:13:37.172700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"93'130","reported_seq":545,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716686+0000","last_change":"2024-03-11T22:15:05.687130+0000","last_active":"2024-03-11T22:15:07.716686+0000","last_peered":"2024-03-11T22:15:07.716686+0000","last_clean":"2024-03-11T22:15:07.716686+0000","last_became_active":"2024-03-11T22:14:43.194413+0000","last_became_peered":"2024-03-11T22:14:43.194413+0000","last_unstale":"2024-03-11T22:15:07.716686+0000","last_undegraded":"2024-03-11T22:15:07.716686+0000","last_fullsized":"2024-03-11T22:15:07.716686+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:05.687010+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.687010+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:20.099917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"93'130","reported_seq":544,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.715401+0000","last_change":"2024-03-11T22:15:05.403493+0000","last_active":"2024-03-11T22:15:07.715401+0000","last_peered":"2024-03-11T22:15:07.715401+0000","last_clean":"2024-03-11T22:15:07.715401+0000","last_became_active":"2024-03-11T22:14:43.198216+0000","last_became_peered":"2024-03-11T22:14:43.198216+0000","last_unstale":"2024-03-11T22:15:07.715401+0000","last_undegraded":"2024-03-11T22:15:07.715401+0000","last_fullsized":"2024-03-11T22:15:07.715401+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:05.403372+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.403372+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:06.652225+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"93'13","reported_seq":279,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.723186+0000","last_change":"2024-03-11T22:14:26.127827+0000","last_active":"2024-03-11T22:15:07.723186+0000","last_peered":"2024-03-11T22:15:07.723186+0000","last_clean":"2024-03-11T22:15:07.723186+0000","last_became_active":"2024-03-11T22:08:35.085787+0000","last_became_peered":"2024-03-11T22:08:35.085787+0000","last_unstale":"2024-03-11T22:15:07.723186+0000","last_undegraded":"2024-03-11T22:15:07.723186+0000","last_fullsized":"2024-03-11T22:15:07.723186+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:14:26.127737+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.127737+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:40.826957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":26,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717863+0000","last_change":"2024-03-11T22:14:52.247626+0000","last_active":"2024-03-11T22:15:07.717863+0000","last_peered":"2024-03-11T22:15:07.717863+0000","last_clean":"2024-03-11T22:15:07.717863+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:15:07.717863+0000","last_undegraded":"2024-03-11T22:15:07.717863+0000","last_fullsized":"2024-03-11T22:15:07.717863+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:11.092328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:11.092328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:20.856434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.723927+0000","last_change":"2024-03-11T22:14:14.850354+0000","last_active":"2024-03-11T22:15:07.723927+0000","last_peered":"2024-03-11T22:15:07.723927+0000","last_clean":"2024-03-11T22:15:07.723927+0000","last_became_active":"2024-03-11T22:08:37.069611+0000","last_became_peered":"2024-03-11T22:08:37.069611+0000","last_unstale":"2024-03-11T22:15:07.723927+0000","last_undegraded":"2024-03-11T22:15:07.723927+0000","last_fullsized":"2024-03-11T22:15:07.723927+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.850128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.850128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:39.356529+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":259,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.711345+0000","last_change":"2024-03-11T22:14:19.987755+0000","last_active":"2024-03-11T22:15:07.711345+0000","last_peered":"2024-03-11T22:15:07.711345+0000","last_clean":"2024-03-11T22:15:07.711345+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:15:07.711345+0000","last_undegraded":"2024-03-11T22:15:07.711345+0000","last_fullsized":"2024-03-11T22:15:07.711345+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:19.987672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:19.987672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:44.507245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":275,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717451+0000","last_change":"2024-03-11T22:13:30.515940+0000","last_active":"2024-03-11T22:15:07.717451+0000","last_peered":"2024-03-11T22:15:07.717451+0000","last_clean":"2024-03-11T22:15:07.717451+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:15:07.717451+0000","last_undegraded":"2024-03-11T22:15:07.717451+0000","last_fullsized":"2024-03-11T22:15:07.717451+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:13:30.515836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:30.515836+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716228+0000","last_change":"2024-03-11T22:14:33.012119+0000","last_active":"2024-03-11T22:15:07.716228+0000","last_peered":"2024-03-11T22:15:07.716228+0000","last_clean":"2024-03-11T22:15:07.716228+0000","last_became_active":"2024-03-11T22:08:33.032850+0000","last_became_peered":"2024-03-11T22:08:33.032850+0000","last_unstale":"2024-03-11T22:15:07.716228+0000","last_undegraded":"2024-03-11T22:15:07.716228+0000","last_fullsized":"2024-03-11T22:15:07.716228+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:33.012017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:33.012017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:02.262664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":346,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.719408+0000","last_change":"2024-03-11T22:14:26.011406+0000","last_active":"2024-03-11T22:15:07.719408+0000","last_peered":"2024-03-11T22:15:07.719408+0000","last_clean":"2024-03-11T22:15:07.719408+0000","last_became_active":"2024-03-11T22:08:37.071202+0000","last_became_peered":"2024-03-11T22:08:37.071202+0000","last_unstale":"2024-03-11T22:15:07.719408+0000","last_undegraded":"2024-03-11T22:15:07.719408+0000","last_fullsized":"2024-03-11T22:15:07.719408+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:14:26.011319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.011319+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:45.370951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":22,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.711170+0000","last_change":"2024-03-11T22:14:52.244222+0000","last_active":"2024-03-11T22:15:07.711170+0000","last_peered":"2024-03-11T22:15:07.711170+0000","last_clean":"2024-03-11T22:15:07.711170+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:15:07.711170+0000","last_undegraded":"2024-03-11T22:15:07.711170+0000","last_fullsized":"2024-03-11T22:15:07.711170+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:31.308103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:31.308103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"93'130","reported_seq":538,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716338+0000","last_change":"2024-03-11T22:15:07.646561+0000","last_active":"2024-03-11T22:15:07.716338+0000","last_peered":"2024-03-11T22:15:07.716338+0000","last_clean":"2024-03-11T22:15:07.716338+0000","last_became_active":"2024-03-11T22:14:43.194350+0000","last_became_peered":"2024-03-11T22:14:43.194350+0000","last_unstale":"2024-03-11T22:15:07.716338+0000","last_undegraded":"2024-03-11T22:15:07.716338+0000","last_fullsized":"2024-03-11T22:15:07.716338+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:07.646448+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:15:07.646448+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:30.541086+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":266,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.719167+0000","last_change":"2024-03-11T22:12:20.295562+0000","last_active":"2024-03-11T22:15:07.719167+0000","last_peered":"2024-03-11T22:15:07.719167+0000","last_clean":"2024-03-11T22:15:07.719167+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:15:07.719167+0000","last_undegraded":"2024-03-11T22:15:07.719167+0000","last_fullsized":"2024-03-11T22:15:07.719167+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:12:20.295427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:20.295427+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.967794+0000","last_change":"2024-03-11T22:15:07.967880+0000","last_active":"2024-03-11T22:15:07.967794+0000","last_peered":"2024-03-11T22:15:07.967794+0000","last_clean":"2024-03-11T22:15:07.967794+0000","last_became_active":"2024-03-11T22:14:52.242397+0000","last_became_peered":"2024-03-11T22:14:52.242397+0000","last_unstale":"2024-03-11T22:15:07.967794+0000","last_undegraded":"2024-03-11T22:15:07.967794+0000","last_fullsized":"2024-03-11T22:15:07.967794+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:07.967713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:07.967713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:28.435655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":343,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717029+0000","last_change":"2024-03-11T22:13:28.520411+0000","last_active":"2024-03-11T22:15:07.717029+0000","last_peered":"2024-03-11T22:15:07.717029+0000","last_clean":"2024-03-11T22:15:07.717029+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:15:07.717029+0000","last_undegraded":"2024-03-11T22:15:07.717029+0000","last_fullsized":"2024-03-11T22:15:07.717029+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:13:28.520283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:13:28.520283+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":243,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718415+0000","last_change":"2024-03-11T22:14:14.969800+0000","last_active":"2024-03-11T22:15:07.718415+0000","last_peered":"2024-03-11T22:15:07.718415+0000","last_clean":"2024-03-11T22:15:07.718415+0000","last_became_active":"2024-03-11T22:08:39.082938+0000","last_became_peered":"2024-03-11T22:08:39.082938+0000","last_unstale":"2024-03-11T22:15:07.718415+0000","last_undegraded":"2024-03-11T22:15:07.718415+0000","last_fullsized":"2024-03-11T22:15:07.718415+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.969572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.969572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:44.759439+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"93'130","reported_seq":494,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718194+0000","last_change":"2024-03-11T22:15:06.779041+0000","last_active":"2024-03-11T22:15:07.718194+0000","last_peered":"2024-03-11T22:15:07.718194+0000","last_clean":"2024-03-11T22:15:07.718194+0000","last_became_active":"2024-03-11T22:14:43.191297+0000","last_became_peered":"2024-03-11T22:14:43.191297+0000","last_unstale":"2024-03-11T22:15:07.718194+0000","last_undegraded":"2024-03-11T22:15:07.718194+0000","last_fullsized":"2024-03-11T22:15:07.718194+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.778786+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.778786+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.184868+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":60,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718009+0000","last_change":"2024-03-11T22:14:52.240268+0000","last_active":"2024-03-11T22:15:07.718009+0000","last_peered":"2024-03-11T22:15:07.718009+0000","last_clean":"2024-03-11T22:15:07.718009+0000","last_became_active":"2024-03-11T22:14:52.239810+0000","last_became_peered":"2024-03-11T22:14:52.239810+0000","last_unstale":"2024-03-11T22:15:07.718009+0000","last_undegraded":"2024-03-11T22:15:07.718009+0000","last_fullsized":"2024-03-11T22:15:07.718009+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:14:32.853664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:32.853664+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:34.208129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":31,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718129+0000","last_change":"2024-03-11T22:14:52.240206+0000","last_active":"2024-03-11T22:15:07.718129+0000","last_peered":"2024-03-11T22:15:07.718129+0000","last_clean":"2024-03-11T22:15:07.718129+0000","last_became_active":"2024-03-11T22:14:52.239817+0000","last_became_peered":"2024-03-11T22:14:52.239817+0000","last_unstale":"2024-03-11T22:15:07.718129+0000","last_undegraded":"2024-03-11T22:15:07.718129+0000","last_fullsized":"2024-03-11T22:15:07.718129+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:12:18.154515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:18.154515+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.713562+0000","last_change":"2024-03-11T22:14:07.844362+0000","last_active":"2024-03-11T22:15:07.713562+0000","last_peered":"2024-03-11T22:15:07.713562+0000","last_clean":"2024-03-11T22:15:07.713562+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:15:07.713562+0000","last_undegraded":"2024-03-11T22:15:07.713562+0000","last_fullsized":"2024-03-11T22:15:07.713562+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:07.844240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:07.844240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:27.085453+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":243,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.721665+0000","last_change":"2024-03-11T22:12:46.492608+0000","last_active":"2024-03-11T22:15:07.721665+0000","last_peered":"2024-03-11T22:15:07.721665+0000","last_clean":"2024-03-11T22:15:07.721665+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:15:07.721665+0000","last_undegraded":"2024-03-11T22:15:07.721665+0000","last_fullsized":"2024-03-11T22:15:07.721665+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.492494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.492494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":275,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.713864+0000","last_change":"2024-03-11T22:13:31.796744+0000","last_active":"2024-03-11T22:15:07.713864+0000","last_peered":"2024-03-11T22:15:07.713864+0000","last_clean":"2024-03-11T22:15:07.713864+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:15:07.713864+0000","last_undegraded":"2024-03-11T22:15:07.713864+0000","last_fullsized":"2024-03-11T22:15:07.713864+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:13:31.796611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:31.796611+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":194,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.722623+0000","last_change":"2024-03-11T22:14:21.857065+0000","last_active":"2024-03-11T22:15:07.722623+0000","last_peered":"2024-03-11T22:15:07.722623+0000","last_clean":"2024-03-11T22:15:07.722623+0000","last_became_active":"2024-03-11T22:10:35.414121+0000","last_became_peered":"2024-03-11T22:10:35.414121+0000","last_unstale":"2024-03-11T22:15:07.722623+0000","last_undegraded":"2024-03-11T22:15:07.722623+0000","last_fullsized":"2024-03-11T22:15:07.722623+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:21.856963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.856963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:24.600199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.720956+0000","last_change":"2024-03-11T22:14:15.858384+0000","last_active":"2024-03-11T22:15:07.720956+0000","last_peered":"2024-03-11T22:15:07.720956+0000","last_clean":"2024-03-11T22:15:07.720956+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:15:07.720956+0000","last_undegraded":"2024-03-11T22:15:07.720956+0000","last_fullsized":"2024-03-11T22:15:07.720956+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:15.857082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:14:15.857082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:29.241026+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":243,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.715920+0000","last_change":"2024-03-11T22:12:46.829082+0000","last_active":"2024-03-11T22:15:07.715920+0000","last_peered":"2024-03-11T22:15:07.715920+0000","last_clean":"2024-03-11T22:15:07.715920+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:15:07.715920+0000","last_undegraded":"2024-03-11T22:15:07.715920+0000","last_fullsized":"2024-03-11T22:15:07.715920+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.828942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.828942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"93'130","reported_seq":539,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.710737+0000","last_change":"2024-03-11T22:15:06.951256+0000","last_active":"2024-03-11T22:15:07.710737+0000","last_peered":"2024-03-11T22:15:07.710737+0000","last_clean":"2024-03-11T22:15:07.710737+0000","last_became_active":"2024-03-11T22:14:43.194955+0000","last_became_peered":"2024-03-11T22:14:43.194955+0000","last_unstale":"2024-03-11T22:15:07.710737+0000","last_undegraded":"2024-03-11T22:15:07.710737+0000","last_fullsized":"2024-03-11T22:15:07.710737+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.951073+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.951073+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.482728+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":270,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.719278+0000","last_change":"2024-03-11T22:13:18.486183+0000","last_active":"2024-03-11T22:15:07.719278+0000","last_peered":"2024-03-11T22:15:07.719278+0000","last_clean":"2024-03-11T22:15:07.719278+0000","last_became_active":"2024-03-11T22:08:35.085928+0000","last_became_peered":"2024-03-11T22:08:35.085928+0000","last_unstale":"2024-03-11T22:15:07.719278+0000","last_undegraded":"2024-03-11T22:15:07.719278+0000","last_fullsized":"2024-03-11T22:15:07.719278+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:13:18.486039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:18.486039+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716968+0000","last_change":"2024-03-11T22:13:29.550985+0000","last_active":"2024-03-11T22:15:07.716968+0000","last_peered":"2024-03-11T22:15:07.716968+0000","last_clean":"2024-03-11T22:15:07.716968+0000","last_became_active":"2024-03-11T22:08:33.039349+0000","last_became_peered":"2024-03-11T22:08:33.039349+0000","last_unstale":"2024-03-11T22:15:07.716968+0000","last_undegraded":"2024-03-11T22:15:07.716968+0000","last_fullsized":"2024-03-11T22:15:07.716968+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:29.550889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:29.550889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717479+0000","last_change":"2024-03-11T22:14:52.248368+0000","last_active":"2024-03-11T22:15:07.717479+0000","last_peered":"2024-03-11T22:15:07.717479+0000","last_clean":"2024-03-11T22:15:07.717479+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:15:07.717479+0000","last_undegraded":"2024-03-11T22:15:07.717479+0000","last_fullsized":"2024-03-11T22:15:07.717479+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:11:42.175852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":251,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.719163+0000","last_change":"2024-03-11T22:14:23.930193+0000","last_active":"2024-03-11T22:15:07.719163+0000","last_peered":"2024-03-11T22:15:07.719163+0000","last_clean":"2024-03-11T22:15:07.719163+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:15:07.719163+0000","last_undegraded":"2024-03-11T22:15:07.719163+0000","last_fullsized":"2024-03-11T22:15:07.719163+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.930091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.930091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:52.770218+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":277,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.719326+0000","last_change":"2024-03-11T22:13:31.491151+0000","last_active":"2024-03-11T22:15:07.719326+0000","last_peered":"2024-03-11T22:15:07.719326+0000","last_clean":"2024-03-11T22:15:07.719326+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:15:07.719326+0000","last_undegraded":"2024-03-11T22:15:07.719326+0000","last_fullsized":"2024-03-11T22:15:07.719326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-11T22:13:31.490878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:31.490878+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717385+0000","last_change":"2024-03-11T22:13:32.033557+0000","last_active":"2024-03-11T22:15:07.717385+0000","last_peered":"2024-03-11T22:15:07.717385+0000","last_clean":"2024-03-11T22:15:07.717385+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:15:07.717385+0000","last_undegraded":"2024-03-11T22:15:07.717385+0000","last_fullsized":"2024-03-11T22:15:07.717385+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:32.033444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:32.033444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":67,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.712914+0000","last_change":"2024-03-11T22:14:25.982151+0000","last_active":"2024-03-11T22:15:07.712914+0000","last_peered":"2024-03-11T22:15:07.712914+0000","last_clean":"2024-03-11T22:15:07.712914+0000","last_became_active":"2024-03-11T22:14:08.442596+0000","last_became_peered":"2024-03-11T22:14:08.442596+0000","last_unstale":"2024-03-11T22:15:07.712914+0000","last_undegraded":"2024-03-11T22:15:07.712914+0000","last_fullsized":"2024-03-11T22:15:07.712914+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:25.982072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:25.982072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:40.493713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":193,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.710165+0000","last_change":"2024-03-11T22:13:27.763590+0000","last_active":"2024-03-11T22:15:07.710165+0000","last_peered":"2024-03-11T22:15:07.710165+0000","last_clean":"2024-03-11T22:15:07.710165+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:15:07.710165+0000","last_undegraded":"2024-03-11T22:15:07.710165+0000","last_fullsized":"2024-03-11T22:15:07.710165+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:27.763466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:27.763466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":279,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718705+0000","last_change":"2024-03-11T22:13:17.923943+0000","last_active":"2024-03-11T22:15:07.718705+0000","last_peered":"2024-03-11T22:15:07.718705+0000","last_clean":"2024-03-11T22:15:07.718705+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:15:07.718705+0000","last_undegraded":"2024-03-11T22:15:07.718705+0000","last_fullsized":"2024-03-11T22:15:07.718705+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:13:17.923797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:17.923797+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.719234+0000","last_change":"2024-03-11T22:14:07.822714+0000","last_active":"2024-03-11T22:15:07.719234+0000","last_peered":"2024-03-11T22:15:07.719234+0000","last_clean":"2024-03-11T22:15:07.719234+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:15:07.719234+0000","last_undegraded":"2024-03-11T22:15:07.719234+0000","last_fullsized":"2024-03-11T22:15:07.719234+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:07.822036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:07.822036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:12.568652+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":256,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.723363+0000","last_change":"2024-03-11T22:14:23.034943+0000","last_active":"2024-03-11T22:15:07.723363+0000","last_peered":"2024-03-11T22:15:07.723363+0000","last_clean":"2024-03-11T22:15:07.723363+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:15:07.723363+0000","last_undegraded":"2024-03-11T22:15:07.723363+0000","last_fullsized":"2024-03-11T22:15:07.723363+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.034850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.034850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:37.593852+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":251,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.713404+0000","last_change":"2024-03-11T22:13:30.794816+0000","last_active":"2024-03-11T22:15:07.713404+0000","last_peered":"2024-03-11T22:15:07.713404+0000","last_clean":"2024-03-11T22:15:07.713404+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:15:07.713404+0000","last_undegraded":"2024-03-11T22:15:07.713404+0000","last_fullsized":"2024-03-11T22:15:07.713404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:30.794739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:30.794739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":267,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.723858+0000","last_change":"2024-03-11T22:13:16.935454+0000","last_active":"2024-03-11T22:15:07.723858+0000","last_peered":"2024-03-11T22:15:07.723858+0000","last_clean":"2024-03-11T22:15:07.723858+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:15:07.723858+0000","last_undegraded":"2024-03-11T22:15:07.723858+0000","last_fullsized":"2024-03-11T22:15:07.723858+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:13:16.935159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:16.935159+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.713294+0000","last_change":"2024-03-11T22:14:23.995912+0000","last_active":"2024-03-11T22:15:07.713294+0000","last_peered":"2024-03-11T22:15:07.713294+0000","last_clean":"2024-03-11T22:15:07.713294+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:15:07.713294+0000","last_undegraded":"2024-03-11T22:15:07.713294+0000","last_fullsized":"2024-03-11T22:15:07.713294+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:14:23.995781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.995781+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:33.646778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":355,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718696+0000","last_change":"2024-03-11T22:14:21.037744+0000","last_active":"2024-03-11T22:15:07.718696+0000","last_peered":"2024-03-11T22:15:07.718696+0000","last_clean":"2024-03-11T22:15:07.718696+0000","last_became_active":"2024-03-11T22:08:37.078588+0000","last_became_peered":"2024-03-11T22:08:37.078588+0000","last_unstale":"2024-03-11T22:15:07.718696+0000","last_undegraded":"2024-03-11T22:15:07.718696+0000","last_fullsized":"2024-03-11T22:15:07.718696+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:14:21.037625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.037625+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:38.139047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":204,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716096+0000","last_change":"2024-03-11T22:13:21.532369+0000","last_active":"2024-03-11T22:15:07.716096+0000","last_peered":"2024-03-11T22:15:07.716096+0000","last_clean":"2024-03-11T22:15:07.716096+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:15:07.716096+0000","last_undegraded":"2024-03-11T22:15:07.716096+0000","last_fullsized":"2024-03-11T22:15:07.716096+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:13:21.532280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:21.532280+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":282,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718243+0000","last_change":"2024-03-11T22:14:27.998984+0000","last_active":"2024-03-11T22:15:07.718243+0000","last_peered":"2024-03-11T22:15:07.718243+0000","last_clean":"2024-03-11T22:15:07.718243+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:15:07.718243+0000","last_undegraded":"2024-03-11T22:15:07.718243+0000","last_fullsized":"2024-03-11T22:15:07.718243+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:14:27.998819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:27.998819+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:50.184055+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.713409+0000","last_change":"2024-03-11T22:14:52.245696+0000","last_active":"2024-03-11T22:15:07.713409+0000","last_peered":"2024-03-11T22:15:07.713409+0000","last_clean":"2024-03-11T22:15:07.713409+0000","last_became_active":"2024-03-11T22:14:52.245199+0000","last_became_peered":"2024-03-11T22:14:52.245199+0000","last_unstale":"2024-03-11T22:15:07.713409+0000","last_undegraded":"2024-03-11T22:15:07.713409+0000","last_fullsized":"2024-03-11T22:15:07.713409+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:22.058795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:22.058795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":22,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.719925+0000","last_change":"2024-03-11T22:14:52.233517+0000","last_active":"2024-03-11T22:15:07.719925+0000","last_peered":"2024-03-11T22:15:07.719925+0000","last_clean":"2024-03-11T22:15:07.719925+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:15:07.719925+0000","last_undegraded":"2024-03-11T22:15:07.719925+0000","last_fullsized":"2024-03-11T22:15:07.719925+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.849679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.849679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:46.294927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":244,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.723566+0000","last_change":"2024-03-11T22:12:44.464428+0000","last_active":"2024-03-11T22:15:07.723566+0000","last_peered":"2024-03-11T22:15:07.723566+0000","last_clean":"2024-03-11T22:15:07.723566+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:15:07.723566+0000","last_undegraded":"2024-03-11T22:15:07.723566+0000","last_fullsized":"2024-03-11T22:15:07.723566+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.464308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.464308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":292,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.710921+0000","last_change":"2024-03-11T22:14:12.989218+0000","last_active":"2024-03-11T22:15:07.710921+0000","last_peered":"2024-03-11T22:15:07.710921+0000","last_clean":"2024-03-11T22:15:07.710921+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:15:07.710921+0000","last_undegraded":"2024-03-11T22:15:07.710921+0000","last_fullsized":"2024-03-11T22:15:07.710921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:14:12.989078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:12.989078+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:19.385123+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718702+0000","last_change":"2024-03-11T22:14:32.050907+0000","last_active":"2024-03-11T22:15:07.718702+0000","last_peered":"2024-03-11T22:15:07.718702+0000","last_clean":"2024-03-11T22:15:07.718702+0000","last_became_active":"2024-03-11T22:08:33.035624+0000","last_became_peered":"2024-03-11T22:08:33.035624+0000","last_unstale":"2024-03-11T22:15:07.718702+0000","last_undegraded":"2024-03-11T22:15:07.718702+0000","last_fullsized":"2024-03-11T22:15:07.718702+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:32.050761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:32.050761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:45.298909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.713264+0000","last_change":"2024-03-11T22:14:26.935697+0000","last_active":"2024-03-11T22:15:07.713264+0000","last_peered":"2024-03-11T22:15:07.713264+0000","last_clean":"2024-03-11T22:15:07.713264+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:15:07.713264+0000","last_undegraded":"2024-03-11T22:15:07.713264+0000","last_fullsized":"2024-03-11T22:15:07.713264+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:26.935615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.935615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:39.182791+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":22,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718410+0000","last_change":"2024-03-11T22:14:52.240697+0000","last_active":"2024-03-11T22:15:07.718410+0000","last_peered":"2024-03-11T22:15:07.718410+0000","last_clean":"2024-03-11T22:15:07.718410+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:15:07.718410+0000","last_undegraded":"2024-03-11T22:15:07.718410+0000","last_fullsized":"2024-03-11T22:15:07.718410+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:32.044320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:32.044320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"95'7","reported_seq":96,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718496+0000","last_change":"2024-03-11T22:14:52.242050+0000","last_active":"2024-03-11T22:15:07.718496+0000","last_peered":"2024-03-11T22:15:07.718496+0000","last_clean":"2024-03-11T22:15:07.718496+0000","last_became_active":"2024-03-11T22:14:52.238411+0000","last_became_peered":"2024-03-11T22:14:52.238411+0000","last_unstale":"2024-03-11T22:15:07.718496+0000","last_undegraded":"2024-03-11T22:15:07.718496+0000","last_fullsized":"2024-03-11T22:15:07.718496+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'7","last_scrub_stamp":"2024-03-11T22:13:32.041219+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:13:32.041219+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716320+0000","last_change":"2024-03-11T22:14:30.716953+0000","last_active":"2024-03-11T22:15:07.716320+0000","last_peered":"2024-03-11T22:15:07.716320+0000","last_clean":"2024-03-11T22:15:07.716320+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:15:07.716320+0000","last_undegraded":"2024-03-11T22:15:07.716320+0000","last_fullsized":"2024-03-11T22:15:07.716320+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:14:30.716844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:39.360065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.713235+0000","last_change":"2024-03-11T22:14:52.239698+0000","last_active":"2024-03-11T22:15:07.713235+0000","last_peered":"2024-03-11T22:15:07.713235+0000","last_clean":"2024-03-11T22:15:07.713235+0000","last_became_active":"2024-03-11T22:14:52.239391+0000","last_became_peered":"2024-03-11T22:14:52.239391+0000","last_unstale":"2024-03-11T22:15:07.713235+0000","last_undegraded":"2024-03-11T22:15:07.713235+0000","last_fullsized":"2024-03-11T22:15:07.713235+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:36.039591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.270166+0000","last_clean_scrub_stamp":"2024-03-11T22:13:36.039591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":251,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716064+0000","last_change":"2024-03-11T22:14:22.631560+0000","last_active":"2024-03-11T22:15:07.716064+0000","last_peered":"2024-03-11T22:15:07.716064+0000","last_clean":"2024-03-11T22:15:07.716064+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:15:07.716064+0000","last_undegraded":"2024-03-11T22:15:07.716064+0000","last_fullsized":"2024-03-11T22:15:07.716064+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:22.631456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:22.631456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:48.639261+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":57,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718378+0000","last_change":"2024-03-11T22:14:52.237478+0000","last_active":"2024-03-11T22:15:07.718378+0000","last_peered":"2024-03-11T22:15:07.718378+0000","last_clean":"2024-03-11T22:15:07.718378+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:15:07.718378+0000","last_undegraded":"2024-03-11T22:15:07.718378+0000","last_fullsized":"2024-03-11T22:15:07.718378+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:11:45.169706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:11:45.169706+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":263,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717252+0000","last_change":"2024-03-11T22:12:47.432839+0000","last_active":"2024-03-11T22:15:07.717252+0000","last_peered":"2024-03-11T22:15:07.717252+0000","last_clean":"2024-03-11T22:15:07.717252+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:15:07.717252+0000","last_undegraded":"2024-03-11T22:15:07.717252+0000","last_fullsized":"2024-03-11T22:15:07.717252+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:47.432769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:12:47.432769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716459+0000","last_change":"2024-03-11T22:14:28.675081+0000","last_active":"2024-03-11T22:15:07.716459+0000","last_peered":"2024-03-11T22:15:07.716459+0000","last_clean":"2024-03-11T22:15:07.716459+0000","last_became_active":"2024-03-11T22:08:37.081996+0000","last_became_peered":"2024-03-11T22:08:37.081996+0000","last_unstale":"2024-03-11T22:15:07.716459+0000","last_undegraded":"2024-03-11T22:15:07.716459+0000","last_fullsized":"2024-03-11T22:15:07.716459+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:28.674962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:14:28.674962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:55.434068+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":243,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716272+0000","last_change":"2024-03-11T22:13:25.042445+0000","last_active":"2024-03-11T22:15:07.716272+0000","last_peered":"2024-03-11T22:15:07.716272+0000","last_clean":"2024-03-11T22:15:07.716272+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:15:07.716272+0000","last_undegraded":"2024-03-11T22:15:07.716272+0000","last_fullsized":"2024-03-11T22:15:07.716272+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:25.042307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:25.042307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":276,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.719380+0000","last_change":"2024-03-11T22:12:49.510144+0000","last_active":"2024-03-11T22:15:07.719380+0000","last_peered":"2024-03-11T22:15:07.719380+0000","last_clean":"2024-03-11T22:15:07.719380+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:15:07.719380+0000","last_undegraded":"2024-03-11T22:15:07.719380+0000","last_fullsized":"2024-03-11T22:15:07.719380+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:12:49.509860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.509860+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":270,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.722762+0000","last_change":"2024-03-11T22:13:19.463087+0000","last_active":"2024-03-11T22:15:07.722762+0000","last_peered":"2024-03-11T22:15:07.722762+0000","last_clean":"2024-03-11T22:15:07.722762+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:15:07.722762+0000","last_undegraded":"2024-03-11T22:15:07.722762+0000","last_fullsized":"2024-03-11T22:15:07.722762+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:13:19.462955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.462955+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":22,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717559+0000","last_change":"2024-03-11T22:14:52.248197+0000","last_active":"2024-03-11T22:15:07.717559+0000","last_peered":"2024-03-11T22:15:07.717559+0000","last_clean":"2024-03-11T22:15:07.717559+0000","last_became_active":"2024-03-11T22:14:52.247689+0000","last_became_peered":"2024-03-11T22:14:52.247689+0000","last_unstale":"2024-03-11T22:15:07.717559+0000","last_undegraded":"2024-03-11T22:15:07.717559+0000","last_fullsized":"2024-03-11T22:15:07.717559+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:43.150872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:11:43.150872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":243,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.718745+0000","last_change":"2024-03-11T22:12:50.533192+0000","last_active":"2024-03-11T22:15:07.718745+0000","last_peered":"2024-03-11T22:15:07.718745+0000","last_clean":"2024-03-11T22:15:07.718745+0000","last_became_active":"2024-03-11T22:08:39.079045+0000","last_became_peered":"2024-03-11T22:08:39.079045+0000","last_unstale":"2024-03-11T22:15:07.718745+0000","last_undegraded":"2024-03-11T22:15:07.718745+0000","last_fullsized":"2024-03-11T22:15:07.718745+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.533067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.533067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":57,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.717755+0000","last_change":"2024-03-11T22:14:52.248071+0000","last_active":"2024-03-11T22:15:07.717755+0000","last_peered":"2024-03-11T22:15:07.717755+0000","last_clean":"2024-03-11T22:15:07.717755+0000","last_became_active":"2024-03-11T22:14:52.247411+0000","last_became_peered":"2024-03-11T22:14:52.247411+0000","last_unstale":"2024-03-11T22:15:07.717755+0000","last_undegraded":"2024-03-11T22:15:07.717755+0000","last_fullsized":"2024-03-11T22:15:07.717755+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:14:15.059599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:15.059599+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:37.879905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":264,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716477+0000","last_change":"2024-03-11T22:14:28.062521+0000","last_active":"2024-03-11T22:15:07.716477+0000","last_peered":"2024-03-11T22:15:07.716477+0000","last_clean":"2024-03-11T22:15:07.716477+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:15:07.716477+0000","last_undegraded":"2024-03-11T22:15:07.716477+0000","last_fullsized":"2024-03-11T22:15:07.716477+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:28.062439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:28.062439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:46.792592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":255,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:09.519013+0000","last_change":"2024-03-11T22:15:09.519052+0000","last_active":"2024-03-11T22:15:09.519013+0000","last_peered":"2024-03-11T22:15:09.519013+0000","last_clean":"2024-03-11T22:15:09.519013+0000","last_became_active":"2024-03-11T22:08:37.073775+0000","last_became_peered":"2024-03-11T22:08:37.073775+0000","last_unstale":"2024-03-11T22:15:09.519013+0000","last_undegraded":"2024-03-11T22:15:09.519013+0000","last_fullsized":"2024-03-11T22:15:09.519013+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:09.518937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:09.518937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:10.825295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":22,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-11T22:15:07.716883+0000","last_change":"2024-03-11T22:14:52.247635+0000","last_active":"2024-03-11T22:15:07.716883+0000","last_peered":"2024-03-11T22:15:07.716883+0000","last_clean":"2024-03-11T22:15:07.716883+0000","last_became_active":"2024-03-11T22:14:52.247273+0000","last_became_peered":"2024-03-11T22:14:52.247273+0000","last_unstale":"2024-03-11T22:15:07.716883+0000","last_undegraded":"2024-03-11T22:15:07.716883+0000","last_fullsized":"2024-03-11T22:15:07.716883+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:11:30.350540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:11:30.350540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1439694848,"num_objects":21968,"num_object_clones":0,"num_object_copies":65904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21968,"num_write_kb":1405952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4295753728,"data_stored":4295753728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21968,"ondisk_log_size":21968,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":364,"num_read_kb":359,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2730,"ondisk_log_size":2730,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":688,"num_read_kb":445,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005140,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":521956,"kb_used_data":386652,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93223196,"statfs":{"total":95995035648,"available":95460552704,"internally_reserved":0,"allocated":395931648,"data_stored":395588579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":138541515},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332185,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":514960,"kb_used_data":379656,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93230192,"statfs":{"total":95995035648,"available":95467716608,"internally_reserved":0,"allocated":388767744,"data_stored":388444965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":138541510},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574963,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":664452,"kb_used_data":547192,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93080700,"statfs":{"total":95995035648,"available":95314636800,"internally_reserved":0,"allocated":560324608,"data_stored":559952873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738489,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":655264,"kb_used_data":493592,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93089888,"statfs":{"total":95995035648,"available":95324045312,"internally_reserved":0,"allocated":505438208,"data_stored":505098567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411438,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":649104,"kb_used_data":501320,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93096048,"statfs":{"total":95995035648,"available":95330353152,"internally_reserved":0,"allocated":513351680,"data_stored":513029271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":683644,"kb_used_data":512240,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93061508,"statfs":{"total":95995035648,"available":95294984192,"internally_reserved":0,"allocated":524533760,"data_stored":524171071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":175503815},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920868,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537000,"kb_used_data":407264,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93208152,"statfs":{"total":95995035648,"available":95445147648,"internally_reserved":0,"allocated":417038336,"data_stored":416690712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659229,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":753240,"kb_used_data":641552,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92991912,"statfs":{"total":95995035648,"available":95223717888,"internally_reserved":0,"allocated":656949248,"data_stored":656618094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":556924928,"data_stored":556924928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":623902720,"data_stored":623902720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":558366720,"data_stored":558366720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":543883264,"data_stored":543883264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":432537600,"data_stored":432537600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":694747136,"data_stored":694747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":440795136,"data_stored":440795136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":444596224,"data_stored":444596224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:15:18.904 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-11T22:15:18.937 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:18.974 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:18.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:19.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:18 smithi045 ceph-mon[30178]: 5.10 scrub starts 2024-03-11T22:15:19.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:18 smithi045 ceph-mon[30178]: 5.10 scrub ok 2024-03-11T22:15:19.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:18 smithi045 ceph-mon[30178]: pgmap v283: 168 pgs: 168 active+clean; 1.9 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 688 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-03-11T22:15:19.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:18 smithi045 ceph-mon[30178]: 3.0 scrub starts 2024-03-11T22:15:19.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:18 smithi045 ceph-mon[30178]: 3.0 scrub ok 2024-03-11T22:15:19.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:18 smithi045 ceph-mon[39715]: 5.10 scrub starts 2024-03-11T22:15:19.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:18 smithi045 ceph-mon[39715]: 5.10 scrub ok 2024-03-11T22:15:19.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:18 smithi045 ceph-mon[39715]: pgmap v283: 168 pgs: 168 active+clean; 1.9 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 688 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-03-11T22:15:19.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:18 smithi045 ceph-mon[39715]: 3.0 scrub starts 2024-03-11T22:15:19.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:18 smithi045 ceph-mon[39715]: 3.0 scrub ok 2024-03-11T22:15:19.051 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:19.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:19.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:18 smithi144 ceph-mon[33984]: 5.10 scrub starts 2024-03-11T22:15:19.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:18 smithi144 ceph-mon[33984]: 5.10 scrub ok 2024-03-11T22:15:19.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:18 smithi144 ceph-mon[33984]: pgmap v283: 168 pgs: 168 active+clean; 1.9 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 688 B/s rd, 75 MiB/s wr, 1.19k op/s 2024-03-11T22:15:19.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:18 smithi144 ceph-mon[33984]: 3.0 scrub starts 2024-03-11T22:15:19.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:18 smithi144 ceph-mon[33984]: 3.0 scrub ok 2024-03-11T22:15:19.120 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:19.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:19.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:19.222 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:19.251 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:19.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:19.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:19.312 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 17 16 36881 36865 135.503 118.188 0.00598291 0.00737357 2024-03-11T22:15:19.353 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:19.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:19.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:19.495 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:19.533 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:19.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:19.634 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:19.666 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:19.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:19.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:19.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:19.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:19.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:19.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:19.913 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:19.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:19.948 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:19.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:20.022 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[30178]: 6.f scrub starts 2024-03-11T22:15:20.022 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[30178]: 6.f scrub ok 2024-03-11T22:15:20.022 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[30178]: 4.1a scrub starts 2024-03-11T22:15:20.022 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[30178]: 4.1a scrub ok 2024-03-11T22:15:20.022 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[30178]: 5.b scrub starts 2024-03-11T22:15:20.022 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[30178]: 5.b scrub ok 2024-03-11T22:15:20.022 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:15:20.022 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:15:20.023 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:20.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[39715]: 6.f scrub starts 2024-03-11T22:15:20.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[39715]: 6.f scrub ok 2024-03-11T22:15:20.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[39715]: 4.1a scrub starts 2024-03-11T22:15:20.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[39715]: 4.1a scrub ok 2024-03-11T22:15:20.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[39715]: 5.b scrub starts 2024-03-11T22:15:20.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[39715]: 5.b scrub ok 2024-03-11T22:15:20.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:15:20.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:15:20.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:20.049 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:20.080 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:20.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:20.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:19 smithi144 ceph-mon[33984]: 6.f scrub starts 2024-03-11T22:15:20.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:19 smithi144 ceph-mon[33984]: 6.f scrub ok 2024-03-11T22:15:20.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:19 smithi144 ceph-mon[33984]: 4.1a scrub starts 2024-03-11T22:15:20.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:19 smithi144 ceph-mon[33984]: 4.1a scrub ok 2024-03-11T22:15:20.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:19 smithi144 ceph-mon[33984]: 5.b scrub starts 2024-03-11T22:15:20.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:19 smithi144 ceph-mon[33984]: 5.b scrub ok 2024-03-11T22:15:20.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:15:20.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:15:20.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:20.181 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:20.215 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:20.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:20.313 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 18 16 38855 38839 134.828 123.375 0.00740271 0.0074109 2024-03-11T22:15:20.315 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:20.346 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:20.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:20.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:20.448 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:20.481 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:20.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:20.583 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:20.613 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:20.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:20.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:20.715 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:20.757 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:20.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:20.859 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:20.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:20.893 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:20.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:20.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:21.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:20 smithi045 ceph-mon[30178]: 5.a scrub starts 2024-03-11T22:15:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:20 smithi045 ceph-mon[30178]: 5.a scrub ok 2024-03-11T22:15:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:20 smithi045 ceph-mon[30178]: 4.1e scrub starts 2024-03-11T22:15:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:20 smithi045 ceph-mon[30178]: 4.1e scrub ok 2024-03-11T22:15:21.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:20 smithi045 ceph-mon[30178]: pgmap v284: 168 pgs: 168 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 568 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-11T22:15:21.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:20 smithi045 ceph-mon[39715]: 5.a scrub starts 2024-03-11T22:15:21.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:20 smithi045 ceph-mon[39715]: 5.a scrub ok 2024-03-11T22:15:21.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:20 smithi045 ceph-mon[39715]: 4.1e scrub starts 2024-03-11T22:15:21.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:20 smithi045 ceph-mon[39715]: 4.1e scrub ok 2024-03-11T22:15:21.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:20 smithi045 ceph-mon[39715]: pgmap v284: 168 pgs: 168 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 568 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-11T22:15:21.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:21.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:21.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:20 smithi144 ceph-mon[33984]: 5.a scrub starts 2024-03-11T22:15:21.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:20 smithi144 ceph-mon[33984]: 5.a scrub ok 2024-03-11T22:15:21.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:20 smithi144 ceph-mon[33984]: 4.1e scrub starts 2024-03-11T22:15:21.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:20 smithi144 ceph-mon[33984]: 4.1e scrub ok 2024-03-11T22:15:21.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:20 smithi144 ceph-mon[33984]: pgmap v284: 168 pgs: 168 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 568 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-03-11T22:15:21.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:21.190 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:21.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:21.291 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:21.313 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 19 16 40693 40677 133.774 114.875 0.011875 0.00746875 2024-03-11T22:15:21.335 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:21.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:21.436 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:21.469 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:21.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:21.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:21.570 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:21.604 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:21.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:21.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:21.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:21.750 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:21.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:21.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:21.886 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:15:21.886 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:15:21.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:21.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:21.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:22.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[39715]: 6.12 scrub starts 2024-03-11T22:15:22.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[39715]: 6.12 scrub ok 2024-03-11T22:15:22.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[39715]: 6.18 scrub starts 2024-03-11T22:15:22.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[39715]: 6.18 scrub ok 2024-03-11T22:15:22.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[39715]: osdmap e185: 8 total, 8 up, 8 in 2024-03-11T22:15:22.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[39715]: 6.1 scrub starts 2024-03-11T22:15:22.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[39715]: 6.1 scrub ok 2024-03-11T22:15:22.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:22.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[30178]: 6.12 scrub starts 2024-03-11T22:15:22.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[30178]: 6.12 scrub ok 2024-03-11T22:15:22.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[30178]: 6.18 scrub starts 2024-03-11T22:15:22.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[30178]: 6.18 scrub ok 2024-03-11T22:15:22.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[30178]: osdmap e185: 8 total, 8 up, 8 in 2024-03-11T22:15:22.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[30178]: 6.1 scrub starts 2024-03-11T22:15:22.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:21 smithi045 ceph-mon[30178]: 6.1 scrub ok 2024-03-11T22:15:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:22.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:22.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:21 smithi144 ceph-mon[33984]: 6.12 scrub starts 2024-03-11T22:15:22.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:21 smithi144 ceph-mon[33984]: 6.12 scrub ok 2024-03-11T22:15:22.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:21 smithi144 ceph-mon[33984]: 6.18 scrub starts 2024-03-11T22:15:22.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:21 smithi144 ceph-mon[33984]: 6.18 scrub ok 2024-03-11T22:15:22.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:21 smithi144 ceph-mon[33984]: osdmap e185: 8 total, 8 up, 8 in 2024-03-11T22:15:22.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:21 smithi144 ceph-mon[33984]: 6.1 scrub starts 2024-03-11T22:15:22.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:21 smithi144 ceph-mon[33984]: 6.1 scrub ok 2024-03-11T22:15:22.134 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:22.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:22.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:22.313 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:15:22.315785+0000 min lat: 0.00125214 max lat: 0.224138 avg lat: 0.00752455 2024-03-11T22:15:22.314 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:15:22.314 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 20 16 42518 42502 132.787 114.062 0.00548403 0.00752455 2024-03-11T22:15:22.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:22.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:22.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:22.525 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:22.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:22.627 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:22.658 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:22.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:22.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T22:15:22.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:22.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:22.808 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:22.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:22.909 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:22.939 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:22.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:22.964 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[30178]: 6.a scrub starts 2024-03-11T22:15:22.965 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[30178]: 6.a scrub ok 2024-03-11T22:15:22.965 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[30178]: 2.1 scrub starts 2024-03-11T22:15:22.965 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[30178]: 2.1 scrub ok 2024-03-11T22:15:22.965 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[30178]: pgmap v286: 168 pgs: 168 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-03-11T22:15:22.965 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[30178]: 6.1d scrub starts 2024-03-11T22:15:22.965 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[30178]: 6.1d scrub ok 2024-03-11T22:15:22.965 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:15:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:15:22.965 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[39715]: 6.a scrub starts 2024-03-11T22:15:22.966 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[39715]: 6.a scrub ok 2024-03-11T22:15:22.966 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[39715]: 2.1 scrub starts 2024-03-11T22:15:22.966 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[39715]: 2.1 scrub ok 2024-03-11T22:15:22.966 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[39715]: pgmap v286: 168 pgs: 168 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-03-11T22:15:22.966 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[39715]: 6.1d scrub starts 2024-03-11T22:15:22.966 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:22 smithi045 ceph-mon[39715]: 6.1d scrub ok 2024-03-11T22:15:23.040 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:23.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:23.078 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:23.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:23.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:22 smithi144 ceph-mon[33984]: 6.a scrub starts 2024-03-11T22:15:23.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:22 smithi144 ceph-mon[33984]: 6.a scrub ok 2024-03-11T22:15:23.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:22 smithi144 ceph-mon[33984]: 2.1 scrub starts 2024-03-11T22:15:23.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:22 smithi144 ceph-mon[33984]: 2.1 scrub ok 2024-03-11T22:15:23.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:22 smithi144 ceph-mon[33984]: pgmap v286: 168 pgs: 168 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-03-11T22:15:23.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:22 smithi144 ceph-mon[33984]: 6.1d scrub starts 2024-03-11T22:15:23.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:22 smithi144 ceph-mon[33984]: 6.1d scrub ok 2024-03-11T22:15:23.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:23.223 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:23.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:23.313 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 21 16 44300 44284 131.767 111.375 0.00541315 0.00758369 2024-03-11T22:15:23.326 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:23.360 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:23.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:23.461 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:23.492 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:23.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:23.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:23.594 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:23.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:23.627 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:23.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:23.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:23.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:23.775 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:23.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:23.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:23.905 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:15:23.905 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:15:23.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:15:23.924 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:23.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:23.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:23.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[39715]: 4.e scrub starts 2024-03-11T22:15:23.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[39715]: 4.e scrub ok 2024-03-11T22:15:23.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[39715]: 4.3 scrub starts 2024-03-11T22:15:23.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[39715]: 4.3 scrub ok 2024-03-11T22:15:23.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[39715]: 4.1c scrub starts 2024-03-11T22:15:23.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[39715]: 4.1c scrub ok 2024-03-11T22:15:23.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[39715]: 3.1a scrub starts 2024-03-11T22:15:23.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[39715]: 3.1a scrub ok 2024-03-11T22:15:23.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[30178]: 4.e scrub starts 2024-03-11T22:15:23.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[30178]: 4.e scrub ok 2024-03-11T22:15:23.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[30178]: 4.3 scrub starts 2024-03-11T22:15:23.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[30178]: 4.3 scrub ok 2024-03-11T22:15:23.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[30178]: 4.1c scrub starts 2024-03-11T22:15:23.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[30178]: 4.1c scrub ok 2024-03-11T22:15:23.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[30178]: 3.1a scrub starts 2024-03-11T22:15:23.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:23 smithi045 ceph-mon[30178]: 3.1a scrub ok 2024-03-11T22:15:24.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:24.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:23 smithi144 ceph-mon[33984]: 4.e scrub starts 2024-03-11T22:15:24.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:23 smithi144 ceph-mon[33984]: 4.e scrub ok 2024-03-11T22:15:24.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:23 smithi144 ceph-mon[33984]: 4.3 scrub starts 2024-03-11T22:15:24.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:23 smithi144 ceph-mon[33984]: 4.3 scrub ok 2024-03-11T22:15:24.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:23 smithi144 ceph-mon[33984]: 4.1c scrub starts 2024-03-11T22:15:24.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:23 smithi144 ceph-mon[33984]: 4.1c scrub ok 2024-03-11T22:15:24.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:23 smithi144 ceph-mon[33984]: 3.1a scrub starts 2024-03-11T22:15:24.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:23 smithi144 ceph-mon[33984]: 3.1a scrub ok 2024-03-11T22:15:24.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:24.109 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:24.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:24.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:24.265 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:24.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:24.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:24.314 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 22 16 45812 45796 130.07 94.5 0.00316746 0.00768221 2024-03-11T22:15:24.367 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:24.397 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:24.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:24.406 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:24.453 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:15:24.499 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:24.500 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:15:24.546 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:15:24.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:24.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:24.593 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:15:24.634 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:15:24.657 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:24.678 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:15:24.690 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:24.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:24.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:24.724 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:15:24.732 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:24.770 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:15:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:15:24.780 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:15:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:15:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:15:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:15:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:15:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:15:24.781 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:15:24.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:24.827 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:15:24.843 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:24.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:24.945 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:24.976 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:24.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:25.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:24 smithi045 ceph-mon[39715]: pgmap v287: 168 pgs: 168 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 638 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-11T22:15:25.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:24 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/429234893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:15:25.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:24 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:15:25.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:24 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1910455790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:15:25.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:24 smithi045 ceph-mon[39715]: 4.12 scrub starts 2024-03-11T22:15:25.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:24 smithi045 ceph-mon[39715]: 4.12 scrub ok 2024-03-11T22:15:25.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:24 smithi045 ceph-mon[30178]: pgmap v287: 168 pgs: 168 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 638 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-11T22:15:25.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:24 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/429234893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:15:25.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:24 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:15:25.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:24 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1910455790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:15:25.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:24 smithi045 ceph-mon[30178]: 4.12 scrub starts 2024-03-11T22:15:25.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:24 smithi045 ceph-mon[30178]: 4.12 scrub ok 2024-03-11T22:15:25.077 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:25.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:24 smithi144 ceph-mon[33984]: pgmap v287: 168 pgs: 168 active+clean; 2.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 638 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-11T22:15:25.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:24 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/429234893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:15:25.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:24 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:15:25.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:24 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1910455790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:15:25.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:24 smithi144 ceph-mon[33984]: 4.12 scrub starts 2024-03-11T22:15:25.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:24 smithi144 ceph-mon[33984]: 4.12 scrub ok 2024-03-11T22:15:25.112 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:25.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:25.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:25.215 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:25.249 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:25.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:25.314 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 23 16 47379 47363 128.672 97.9375 0.0113874 0.00776585 2024-03-11T22:15:25.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:25.392 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:25.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:25.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:25.538 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:25.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:25.640 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:25.672 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:25.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:25.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:25.774 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:25.807 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:25.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:25.910 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:25.943 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:25.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:26.045 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:26.079 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:26.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:26.094 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:15:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:15:26.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:25 smithi144 ceph-mon[33984]: 3.5 scrub starts 2024-03-11T22:15:26.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:25 smithi144 ceph-mon[33984]: 3.5 scrub ok 2024-03-11T22:15:26.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:25 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:15:26.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:25 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:15:26.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:25 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:15:26.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:25 smithi144 ceph-mon[33984]: osdmap e186: 8 total, 8 up, 8 in 2024-03-11T22:15:26.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:25 smithi144 ceph-mon[33984]: 3.3 scrub starts 2024-03-11T22:15:26.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:25 smithi144 ceph-mon[33984]: 3.3 scrub ok 2024-03-11T22:15:26.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[39715]: 3.5 scrub starts 2024-03-11T22:15:26.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[39715]: 3.5 scrub ok 2024-03-11T22:15:26.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:15:26.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:15:26.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:15:26.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[39715]: osdmap e186: 8 total, 8 up, 8 in 2024-03-11T22:15:26.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[39715]: 3.3 scrub starts 2024-03-11T22:15:26.132 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[39715]: 3.3 scrub ok 2024-03-11T22:15:26.133 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[30178]: 3.5 scrub starts 2024-03-11T22:15:26.133 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[30178]: 3.5 scrub ok 2024-03-11T22:15:26.133 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:15:26.133 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:15:26.133 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:15:26.133 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[30178]: osdmap e186: 8 total, 8 up, 8 in 2024-03-11T22:15:26.133 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[30178]: 3.3 scrub starts 2024-03-11T22:15:26.133 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:25 smithi045 ceph-mon[30178]: 3.3 scrub ok 2024-03-11T22:15:26.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:26.220 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:26.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:26.314 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 24 16 49116 49100 127.834 108.562 0.00433935 0.00781767 2024-03-11T22:15:26.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:26.362 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:26.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:26.463 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:26.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:26.500 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:26.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:26.601 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:26.632 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:26.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:26.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:26.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:26.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:26.776 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:26.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:26.877 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:26.907 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:26.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:26.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:27.008 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:26 smithi045 ceph-mon[39715]: 4.8 scrub starts 2024-03-11T22:15:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:26 smithi045 ceph-mon[39715]: 4.8 scrub ok 2024-03-11T22:15:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:26 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:26 smithi045 ceph-mon[39715]: osdmap e187: 8 total, 8 up, 8 in 2024-03-11T22:15:27.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:26 smithi045 ceph-mon[39715]: pgmap v290: 168 pgs: 168 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 158 MiB/s wr, 2.52k op/s 2024-03-11T22:15:27.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:26 smithi045 ceph-mon[30178]: 4.8 scrub starts 2024-03-11T22:15:27.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:26 smithi045 ceph-mon[30178]: 4.8 scrub ok 2024-03-11T22:15:27.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:26 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:27.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:26 smithi045 ceph-mon[30178]: osdmap e187: 8 total, 8 up, 8 in 2024-03-11T22:15:27.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:26 smithi045 ceph-mon[30178]: pgmap v290: 168 pgs: 168 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 158 MiB/s wr, 2.52k op/s 2024-03-11T22:15:27.041 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:27.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:27.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:27.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:26 smithi144 ceph-mon[33984]: 4.8 scrub starts 2024-03-11T22:15:27.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:26 smithi144 ceph-mon[33984]: 4.8 scrub ok 2024-03-11T22:15:27.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:26 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:27.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:26 smithi144 ceph-mon[33984]: osdmap e187: 8 total, 8 up, 8 in 2024-03-11T22:15:27.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:26 smithi144 ceph-mon[33984]: pgmap v290: 168 pgs: 168 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 158 MiB/s wr, 2.52k op/s 2024-03-11T22:15:27.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:27.189 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:27.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:27.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:27.291 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:27.315 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 25 16 50736 50720 126.77 101.25 0.0159238 0.00788299 2024-03-11T22:15:27.333 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:27.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:27.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:27.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:27.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:27.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:27.576 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:27.609 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:27.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:27.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:27.711 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:27.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:27.734 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:27.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:27.835 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:27.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:27.868 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:27.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:27.969 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:28.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:28.005 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:28.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:28.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:27 smithi144 ceph-mon[33984]: 3.14 scrub starts 2024-03-11T22:15:28.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:27 smithi144 ceph-mon[33984]: 3.14 scrub ok 2024-03-11T22:15:28.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:27 smithi144 ceph-mon[33984]: 6.0 scrub starts 2024-03-11T22:15:28.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:27 smithi144 ceph-mon[33984]: 6.0 scrub ok 2024-03-11T22:15:28.106 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:28.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:28.140 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:28.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:28.224 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:27 smithi045 ceph-mon[30178]: 3.14 scrub starts 2024-03-11T22:15:28.224 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:27 smithi045 ceph-mon[30178]: 3.14 scrub ok 2024-03-11T22:15:28.224 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:27 smithi045 ceph-mon[30178]: 6.0 scrub starts 2024-03-11T22:15:28.224 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:27 smithi045 ceph-mon[30178]: 6.0 scrub ok 2024-03-11T22:15:28.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:27 smithi045 ceph-mon[39715]: 3.14 scrub starts 2024-03-11T22:15:28.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:27 smithi045 ceph-mon[39715]: 3.14 scrub ok 2024-03-11T22:15:28.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:27 smithi045 ceph-mon[39715]: 6.0 scrub starts 2024-03-11T22:15:28.224 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:27 smithi045 ceph-mon[39715]: 6.0 scrub ok 2024-03-11T22:15:28.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:28.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:28.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:28.315 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 26 16 52219 52203 125.459 92.6875 0.0121237 0.00796433 2024-03-11T22:15:28.369 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:28.402 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:28.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:28.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:28.503 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:28.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:28.528 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:28.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:28.629 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:28.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:28.664 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:28.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:28.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:28.804 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:28.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:28.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:28.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:28.947 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:28.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:29.049 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:29.082 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:29.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:29.183 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:29.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:29.220 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:29.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:28 smithi045 ceph-mon[39715]: 4.17 scrub starts 2024-03-11T22:15:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:28 smithi045 ceph-mon[39715]: 4.17 scrub ok 2024-03-11T22:15:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:28 smithi045 ceph-mon[39715]: pgmap v291: 168 pgs: 168 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 730 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-11T22:15:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:28 smithi045 ceph-mon[39715]: 5.1a deep-scrub starts 2024-03-11T22:15:29.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:28 smithi045 ceph-mon[39715]: 5.1a deep-scrub ok 2024-03-11T22:15:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:28 smithi045 ceph-mon[30178]: 4.17 scrub starts 2024-03-11T22:15:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:28 smithi045 ceph-mon[30178]: 4.17 scrub ok 2024-03-11T22:15:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:28 smithi045 ceph-mon[30178]: pgmap v291: 168 pgs: 168 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 730 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-11T22:15:29.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:28 smithi045 ceph-mon[30178]: 5.1a deep-scrub starts 2024-03-11T22:15:29.282 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:28 smithi045 ceph-mon[30178]: 5.1a deep-scrub ok 2024-03-11T22:15:29.315 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 27 16 53802 53786 124.477 98.9375 0.00358469 0.00802786 2024-03-11T22:15:29.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:29.343 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:28 smithi144 ceph-mon[33984]: 4.17 scrub starts 2024-03-11T22:15:29.343 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:28 smithi144 ceph-mon[33984]: 4.17 scrub ok 2024-03-11T22:15:29.343 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:28 smithi144 ceph-mon[33984]: pgmap v291: 168 pgs: 168 active+clean; 2.9 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 730 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-11T22:15:29.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:28 smithi144 ceph-mon[33984]: 5.1a deep-scrub starts 2024-03-11T22:15:29.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:28 smithi144 ceph-mon[33984]: 5.1a deep-scrub ok 2024-03-11T22:15:29.358 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:29.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:29.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:15:29.461 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:29.493 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:29.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:29.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:29.597 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:29.629 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:29.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:29.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:29.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:29.785 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:29.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:29.886 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:29.938 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:29.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:29.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:30.040 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:30.092 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:30.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:30.174 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[30178]: 6.1e scrub starts 2024-03-11T22:15:30.174 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[30178]: 6.1e scrub ok 2024-03-11T22:15:30.174 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:15:30.174 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:15:30.174 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[30178]: osdmap e188: 8 total, 8 up, 8 in 2024-03-11T22:15:30.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:15:30.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:15:30.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:30.175 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:15:30.175 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[39715]: 6.1e scrub starts 2024-03-11T22:15:30.175 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[39715]: 6.1e scrub ok 2024-03-11T22:15:30.175 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:15:30.175 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:15:30.175 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[39715]: osdmap e188: 8 total, 8 up, 8 in 2024-03-11T22:15:30.175 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:15:30.175 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:15:30.175 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:30.176 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:29 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:15:30.194 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:30.227 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:30.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:30.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:30.315 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 28 16 55323 55307 123.426 95.0625 0.0111993 0.0080967 2024-03-11T22:15:30.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:30.343 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:29 smithi144 ceph-mon[33984]: 6.1e scrub starts 2024-03-11T22:15:30.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:29 smithi144 ceph-mon[33984]: 6.1e scrub ok 2024-03-11T22:15:30.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:29 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:15:30.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:29 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:15:30.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:29 smithi144 ceph-mon[33984]: osdmap e188: 8 total, 8 up, 8 in 2024-03-11T22:15:30.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:29 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:15:30.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:29 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:15:30.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:29 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:15:30.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:29 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:15:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:30.371 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:30.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:30.472 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:30.504 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:30.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:30.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:30.606 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:30.635 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:30.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:30.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:30.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:30.782 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:30.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:30.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:30.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:30.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:30.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:31.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:31.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:31.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:31.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:31.183 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[39715]: 4.1f scrub starts 2024-03-11T22:15:31.183 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[39715]: 4.1f scrub ok 2024-03-11T22:15:31.183 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[39715]: 4.14 scrub starts 2024-03-11T22:15:31.183 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[39715]: 4.14 scrub ok 2024-03-11T22:15:31.184 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[39715]: 4.15 deep-scrub starts 2024-03-11T22:15:31.184 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[39715]: 4.15 deep-scrub ok 2024-03-11T22:15:31.184 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[39715]: osdmap e189: 8 total, 8 up, 8 in 2024-03-11T22:15:31.184 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[39715]: pgmap v294: 168 pgs: 168 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 956 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-11T22:15:31.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[30178]: 4.1f scrub starts 2024-03-11T22:15:31.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[30178]: 4.1f scrub ok 2024-03-11T22:15:31.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[30178]: 4.14 scrub starts 2024-03-11T22:15:31.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[30178]: 4.14 scrub ok 2024-03-11T22:15:31.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[30178]: 4.15 deep-scrub starts 2024-03-11T22:15:31.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[30178]: 4.15 deep-scrub ok 2024-03-11T22:15:31.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[30178]: osdmap e189: 8 total, 8 up, 8 in 2024-03-11T22:15:31.184 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:30 smithi045 ceph-mon[30178]: pgmap v294: 168 pgs: 168 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 956 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-11T22:15:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:31.224 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:31.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:31.315 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 29 16 56759 56743 122.263 89.75 0.0120851 0.00817362 2024-03-11T22:15:31.325 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:31.326 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:15:31.326 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:15:31.326 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:56762 8.f 8:f7181eae:::benchmark_data_smithi045_115206_object56761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-11T22:15:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:15:31.321747+0000", 2024-03-11T22:15:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.002155087, 2024-03-11T22:15:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0022003270000000002, 2024-03-11T22:15:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:15:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:15:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:15:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:15:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:15:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 56762 2024-03-11T22:15:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:15:31.327 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.328 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:15:31.328 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.321747+0000", 2024-03-11T22:15:31.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:15:31.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.328 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:15:31.328 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.321747+0000", 2024-03-11T22:15:31.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:15:31.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.328 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:15:31.328 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.321748+0000", 2024-03-11T22:15:31.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.542e-06 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.321771+0000", 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2289e-05 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.321786+0000", 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5733999999999999e-05 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:15:31.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.321795+0000", 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2300000000000008e-06 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.321883+0000", 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8747999999999998e-05 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.322028+0000", 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014410000000000001 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:15:31.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.322180+0000", 2024-03-11T22:15:31.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000152039 2024-03-11T22:15:31.331 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:15:31.331 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:15:31.331 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:15:31.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.331 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:56753 8.b 8:dfe4eb13:::benchmark_data_smithi045_115206_object56752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-11T22:15:31.331 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:15:31.312550+0000", 2024-03-11T22:15:31.331 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.011352018, 2024-03-11T22:15:31.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011512747, 2024-03-11T22:15:31.331 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:15:31.331 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:15:31.331 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 56753 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.312550+0000", 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.312550+0000", 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:15:31.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.312551+0000", 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.37e-06 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.312573+0000", 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1851000000000001e-05 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.312585+0000", 2024-03-11T22:15:31.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1810999999999999e-05 2024-03-11T22:15:31.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:15:31.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.312593+0000", 2024-03-11T22:15:31.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4909999999999999e-06 2024-03-11T22:15:31.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:15:31.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.312750+0000", 2024-03-11T22:15:31.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015785300000000001 2024-03-11T22:15:31.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.312899+0000", 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000148999 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.313130+0000", 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023094700000000001 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.315884+0000", 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027538789999999999 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:15:31.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.317259+0000", 2024-03-11T22:15:31.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013752370000000001 2024-03-11T22:15:31.336 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:15:31.336 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:15:31.336 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:15:31.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.336 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:56761 8.b 8:d2b1da27:::benchmark_data_smithi045_115206_object56760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-11T22:15:31.336 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:15:31.318824+0000", 2024-03-11T22:15:31.336 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0050776650000000003, 2024-03-11T22:15:31.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0053445710000000002, 2024-03-11T22:15:31.336 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 56761 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.318824+0000", 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.337 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.318824+0000", 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.318826+0000", 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5409999999999999e-06 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.318855+0000", 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8935999999999999e-05 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.318870+0000", 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5248999999999999e-05 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.318878+0000", 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0539999999999995e-06 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.318937+0000", 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9299999999999998e-05 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:15:31.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.319085+0000", 2024-03-11T22:15:31.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000147987 2024-03-11T22:15:31.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-03-11T22:15:31.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.319266+0000", 2024-03-11T22:15:31.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018060400000000001 2024-03-11T22:15:31.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:15:31.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.322506+0000", 2024-03-11T22:15:31.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032397329999999998 2024-03-11T22:15:31.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:15:31.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:15:31.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:15:31.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:15:31.323019+0000", 2024-03-11T22:15:31.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00051303199999999996 2024-03-11T22:15:31.341 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:15:31.341 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:15:31.341 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:15:31.341 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:15:31.341 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:15:31.341 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 3 2024-03-11T22:15:31.341 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:15:31.343 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:30 smithi144 ceph-mon[33984]: 4.1f scrub starts 2024-03-11T22:15:31.343 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:30 smithi144 ceph-mon[33984]: 4.1f scrub ok 2024-03-11T22:15:31.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:30 smithi144 ceph-mon[33984]: 4.14 scrub starts 2024-03-11T22:15:31.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:30 smithi144 ceph-mon[33984]: 4.14 scrub ok 2024-03-11T22:15:31.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:30 smithi144 ceph-mon[33984]: 4.15 deep-scrub starts 2024-03-11T22:15:31.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:30 smithi144 ceph-mon[33984]: 4.15 deep-scrub ok 2024-03-11T22:15:31.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:30 smithi144 ceph-mon[33984]: osdmap e189: 8 total, 8 up, 8 in 2024-03-11T22:15:31.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:30 smithi144 ceph-mon[33984]: pgmap v294: 168 pgs: 168 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 956 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-11T22:15:31.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:31.398 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:31.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:31.499 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:31.527 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:31.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:31.629 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:31.662 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:31.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:31.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:31.803 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:31.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:31.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:31.905 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:31.931 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:31.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:31.940 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:15:31.944 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:15:32.033 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:32.065 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:32.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:32.166 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:32.200 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:32.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:32.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[39715]: 5.4 scrub starts 2024-03-11T22:15:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[39715]: 5.4 scrub ok 2024-03-11T22:15:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[39715]: 5.5 scrub starts 2024-03-11T22:15:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[39715]: 5.5 scrub ok 2024-03-11T22:15:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[39715]: osdmap e190: 8 total, 8 up, 8 in 2024-03-11T22:15:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[39715]: 4.f scrub starts 2024-03-11T22:15:32.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[39715]: 4.f scrub ok 2024-03-11T22:15:32.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[30178]: 5.4 scrub starts 2024-03-11T22:15:32.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[30178]: 5.4 scrub ok 2024-03-11T22:15:32.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[30178]: 5.5 scrub starts 2024-03-11T22:15:32.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[30178]: 5.5 scrub ok 2024-03-11T22:15:32.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[30178]: osdmap e190: 8 total, 8 up, 8 in 2024-03-11T22:15:32.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[30178]: 4.f scrub starts 2024-03-11T22:15:32.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:31 smithi045 ceph-mon[30178]: 4.f scrub ok 2024-03-11T22:15:32.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:32.316 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 30 16 58178 58162 121.14 88.6875 0.0121531 0.00824925 2024-03-11T22:15:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:32.342 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:32.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:32.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:31 smithi144 ceph-mon[33984]: 5.4 scrub starts 2024-03-11T22:15:32.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:31 smithi144 ceph-mon[33984]: 5.4 scrub ok 2024-03-11T22:15:32.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:31 smithi144 ceph-mon[33984]: 5.5 scrub starts 2024-03-11T22:15:32.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:31 smithi144 ceph-mon[33984]: 5.5 scrub ok 2024-03-11T22:15:32.347 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:31 smithi144 ceph-mon[33984]: osdmap e190: 8 total, 8 up, 8 in 2024-03-11T22:15:32.347 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:31 smithi144 ceph-mon[33984]: 4.f scrub starts 2024-03-11T22:15:32.347 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:31 smithi144 ceph-mon[33984]: 4.f scrub ok 2024-03-11T22:15:32.443 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:32.477 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:32.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:32.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:32.579 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:32.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:15:32.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:32.613 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:32.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:32.714 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:32.737 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:32.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:32.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:32.839 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:32.872 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:32.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:32.884 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:15:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:15:32.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[30178]: 5.0 scrub starts 2024-03-11T22:15:32.974 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:33.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:33.018 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:33.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:33.118 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:33.152 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:33.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:33.243 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[30178]: 5.0 scrub ok 2024-03-11T22:15:33.244 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[30178]: 5.15 scrub starts 2024-03-11T22:15:33.244 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[30178]: 5.15 scrub ok 2024-03-11T22:15:33.244 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[30178]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:15:33.244 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[30178]: pgmap v296: 168 pgs: 168 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:15:33.244 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[30178]: 5.12 scrub starts 2024-03-11T22:15:33.244 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[30178]: 5.12 scrub ok 2024-03-11T22:15:33.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[39715]: 5.0 scrub starts 2024-03-11T22:15:33.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[39715]: 5.0 scrub ok 2024-03-11T22:15:33.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[39715]: 5.15 scrub starts 2024-03-11T22:15:33.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[39715]: 5.15 scrub ok 2024-03-11T22:15:33.245 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[39715]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:15:33.245 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[39715]: pgmap v296: 168 pgs: 168 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:15:33.245 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[39715]: 5.12 scrub starts 2024-03-11T22:15:33.245 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:32 smithi045 ceph-mon[39715]: 5.12 scrub ok 2024-03-11T22:15:33.254 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:33.289 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:33.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:33.316 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 31 16 59527 59511 119.952 84.3125 0.0122972 0.00833175 2024-03-11T22:15:33.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:32 smithi144 ceph-mon[33984]: 5.0 scrub starts 2024-03-11T22:15:33.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:32 smithi144 ceph-mon[33984]: 5.0 scrub ok 2024-03-11T22:15:33.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:32 smithi144 ceph-mon[33984]: 5.15 scrub starts 2024-03-11T22:15:33.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:32 smithi144 ceph-mon[33984]: 5.15 scrub ok 2024-03-11T22:15:33.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:32 smithi144 ceph-mon[33984]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:15:33.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:32 smithi144 ceph-mon[33984]: pgmap v296: 168 pgs: 168 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:15:33.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:32 smithi144 ceph-mon[33984]: 5.12 scrub starts 2024-03-11T22:15:33.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:32 smithi144 ceph-mon[33984]: 5.12 scrub ok 2024-03-11T22:15:33.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:33.431 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:33.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:33.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:33.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:33.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:33.670 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:33.707 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:33.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:33.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T22:15:33.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:33.867 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:33.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:33.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:33.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:34.014 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:34.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:34.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:33 smithi045 ceph-mon[30178]: 6.6 scrub starts 2024-03-11T22:15:34.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:33 smithi045 ceph-mon[30178]: 6.6 scrub ok 2024-03-11T22:15:34.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:33 smithi045 ceph-mon[30178]: 4.2 scrub starts 2024-03-11T22:15:34.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:33 smithi045 ceph-mon[30178]: 4.2 scrub ok 2024-03-11T22:15:34.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:33 smithi045 ceph-mon[39715]: 6.6 scrub starts 2024-03-11T22:15:34.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:33 smithi045 ceph-mon[39715]: 6.6 scrub ok 2024-03-11T22:15:34.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:33 smithi045 ceph-mon[39715]: 4.2 scrub starts 2024-03-11T22:15:34.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:33 smithi045 ceph-mon[39715]: 4.2 scrub ok 2024-03-11T22:15:34.115 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:34.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:34.160 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:34.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:34.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:34.300 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:34.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:34.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:34.318 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 32 16 60845 60829 118.778 82.375 0.0130813 0.00841428 2024-03-11T22:15:34.343 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:33 smithi144 ceph-mon[33984]: 6.6 scrub starts 2024-03-11T22:15:34.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:33 smithi144 ceph-mon[33984]: 6.6 scrub ok 2024-03-11T22:15:34.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:33 smithi144 ceph-mon[33984]: 4.2 scrub starts 2024-03-11T22:15:34.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:33 smithi144 ceph-mon[33984]: 4.2 scrub ok 2024-03-11T22:15:34.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:34.443 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:34.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:34.545 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:34.580 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:34.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:34.681 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:34.713 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:34.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:34.815 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:34.872 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:34.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:34.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:34.974 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:35.015 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:35.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:35.117 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:35.156 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:35.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:35.244 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:34 smithi144 ceph-mon[33984]: 4.0 scrub starts 2024-03-11T22:15:35.245 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:34 smithi144 ceph-mon[33984]: 4.0 scrub ok 2024-03-11T22:15:35.245 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:34 smithi144 ceph-mon[33984]: 6.1c scrub starts 2024-03-11T22:15:35.245 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:34 smithi144 ceph-mon[33984]: 6.1c scrub ok 2024-03-11T22:15:35.245 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:34 smithi144 ceph-mon[33984]: pgmap v297: 168 pgs: 168 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:15:35.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:35.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:34 smithi045 ceph-mon[30178]: 4.0 scrub starts 2024-03-11T22:15:35.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:34 smithi045 ceph-mon[30178]: 4.0 scrub ok 2024-03-11T22:15:35.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:34 smithi045 ceph-mon[30178]: 6.1c scrub starts 2024-03-11T22:15:35.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:34 smithi045 ceph-mon[30178]: 6.1c scrub ok 2024-03-11T22:15:35.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:34 smithi045 ceph-mon[30178]: pgmap v297: 168 pgs: 168 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:15:35.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:34 smithi045 ceph-mon[39715]: 4.0 scrub starts 2024-03-11T22:15:35.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:34 smithi045 ceph-mon[39715]: 4.0 scrub ok 2024-03-11T22:15:35.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:34 smithi045 ceph-mon[39715]: 6.1c scrub starts 2024-03-11T22:15:35.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:34 smithi045 ceph-mon[39715]: 6.1c scrub ok 2024-03-11T22:15:35.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:34 smithi045 ceph-mon[39715]: pgmap v297: 168 pgs: 168 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:15:35.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:35.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:35.317 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 33 16 62340 62324 118.01 93.4375 0.00804317 0.00846884 2024-03-11T22:15:35.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:35.416 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-03-11T22:15:31.163464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3636992348,"num_objects":55685,"num_object_clones":0,"num_object_copies":167055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55685,"num_whiteouts":0,"num_read":1079,"num_read_kb":831,"num_write":55923,"num_write_kb":3551213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58712,"ondisk_log_size":58712,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11888784,"kb_used_data":10491244,"kb_used_omap":12,"kb_used_meta":1397427,"kb_avail":738072432,"statfs":{"total":767960285184,"available":755786170368,"internally_reserved":0,"allocated":10743033856,"data_stored":10740045156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":1430965811},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,4,17,8,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":518455296,"num_objects":7911,"num_object_clones":0,"num_object_copies":23733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7911,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7911,"num_write_kb":506304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.003191"},"pg_stats":[{"pgid":"8.0","version":"189'3403","reported_seq":3440,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.533432+0000","last_change":"2024-03-11T22:14:52.240741+0000","last_active":"2024-03-11T22:15:30.533432+0000","last_peered":"2024-03-11T22:15:30.533432+0000","last_clean":"2024-03-11T22:15:30.533432+0000","last_became_active":"2024-03-11T22:14:52.240283+0000","last_became_peered":"2024-03-11T22:14:52.240283+0000","last_unstale":"2024-03-11T22:15:30.533432+0000","last_undegraded":"2024-03-11T22:15:30.533432+0000","last_fullsized":"2024-03-11T22:15:30.533432+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3403,"log_dups_size":0,"ondisk_log_size":3403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:01.057390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223019008,"num_objects":3403,"num_object_clones":0,"num_object_copies":10209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3403,"num_write_kb":217792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":72,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.874091+0000","last_change":"2024-03-11T22:14:52.237967+0000","last_active":"2024-03-11T22:15:29.874091+0000","last_peered":"2024-03-11T22:15:29.874091+0000","last_clean":"2024-03-11T22:15:29.874091+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:15:29.874091+0000","last_undegraded":"2024-03-11T22:15:29.874091+0000","last_fullsized":"2024-03-11T22:15:29.874091+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:14:24.847623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:24.847623+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:30.799272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873596+0000","last_change":"2024-03-11T22:14:12.801583+0000","last_active":"2024-03-11T22:15:29.873596+0000","last_peered":"2024-03-11T22:15:29.873596+0000","last_clean":"2024-03-11T22:15:29.873596+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:15:29.873596+0000","last_undegraded":"2024-03-11T22:15:29.873596+0000","last_fullsized":"2024-03-11T22:15:29.873596+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:12.801470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:12.801470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":373,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.874408+0000","last_change":"2024-03-11T22:14:29.061747+0000","last_active":"2024-03-11T22:15:29.874408+0000","last_peered":"2024-03-11T22:15:29.874408+0000","last_clean":"2024-03-11T22:15:29.874408+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:15:29.874408+0000","last_undegraded":"2024-03-11T22:15:29.874408+0000","last_fullsized":"2024-03-11T22:15:29.874408+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:14:29.061624+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_clean_scrub_stamp":"2024-03-11T22:14:29.061624+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:46.082482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":38,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867588+0000","last_change":"2024-03-11T22:14:52.244107+0000","last_active":"2024-03-11T22:15:29.867588+0000","last_peered":"2024-03-11T22:15:29.867588+0000","last_clean":"2024-03-11T22:15:29.867588+0000","last_became_active":"2024-03-11T22:14:52.243690+0000","last_became_peered":"2024-03-11T22:14:52.243690+0000","last_unstale":"2024-03-11T22:15:29.867588+0000","last_undegraded":"2024-03-11T22:15:29.867588+0000","last_fullsized":"2024-03-11T22:15:29.867588+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:09.099443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:09.099443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:38.958957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":539,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.871637+0000","last_change":"2024-03-11T22:14:10.906814+0000","last_active":"2024-03-11T22:15:29.871637+0000","last_peered":"2024-03-11T22:15:29.871637+0000","last_clean":"2024-03-11T22:15:29.871637+0000","last_became_active":"2024-03-11T22:13:38.571540+0000","last_became_peered":"2024-03-11T22:13:38.571540+0000","last_unstale":"2024-03-11T22:15:29.871637+0000","last_undegraded":"2024-03-11T22:15:29.871637+0000","last_fullsized":"2024-03-11T22:15:29.871637+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:10.906610+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:10.906610+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"189'3578","reported_seq":3616,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.450170+0000","last_change":"2024-03-11T22:14:52.245149+0000","last_active":"2024-03-11T22:15:30.450170+0000","last_peered":"2024-03-11T22:15:30.450170+0000","last_clean":"2024-03-11T22:15:30.450170+0000","last_became_active":"2024-03-11T22:14:52.244799+0000","last_became_peered":"2024-03-11T22:14:52.244799+0000","last_unstale":"2024-03-11T22:15:30.450170+0000","last_undegraded":"2024-03-11T22:15:30.450170+0000","last_fullsized":"2024-03-11T22:15:30.450170+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3578,"log_dups_size":0,"ondisk_log_size":3578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:09.738242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234487808,"num_objects":3578,"num_object_clones":0,"num_object_copies":10734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3578,"num_write_kb":228992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":93,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.874261+0000","last_change":"2024-03-11T22:15:05.001672+0000","last_active":"2024-03-11T22:15:29.874261+0000","last_peered":"2024-03-11T22:15:29.874261+0000","last_clean":"2024-03-11T22:15:29.874261+0000","last_became_active":"2024-03-11T22:14:52.239649+0000","last_became_peered":"2024-03-11T22:14:52.239649+0000","last_unstale":"2024-03-11T22:15:29.874261+0000","last_undegraded":"2024-03-11T22:15:29.874261+0000","last_fullsized":"2024-03-11T22:15:29.874261+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-11T22:15:05.001522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.001522+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.788107+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":288,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868483+0000","last_change":"2024-03-11T22:14:14.990482+0000","last_active":"2024-03-11T22:15:29.868483+0000","last_peered":"2024-03-11T22:15:29.868483+0000","last_clean":"2024-03-11T22:15:29.868483+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:15:29.868483+0000","last_undegraded":"2024-03-11T22:15:29.868483+0000","last_fullsized":"2024-03-11T22:15:29.868483+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.990368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.990368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":272,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868331+0000","last_change":"2024-03-11T22:14:18.859904+0000","last_active":"2024-03-11T22:15:29.868331+0000","last_peered":"2024-03-11T22:15:29.868331+0000","last_clean":"2024-03-11T22:15:29.868331+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:15:29.868331+0000","last_undegraded":"2024-03-11T22:15:29.868331+0000","last_fullsized":"2024-03-11T22:15:29.868331+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:18.859546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:14:18.859546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:41.507002+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":276,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.878008+0000","last_change":"2024-03-11T22:15:18.067803+0000","last_active":"2024-03-11T22:15:29.878008+0000","last_peered":"2024-03-11T22:15:29.878008+0000","last_clean":"2024-03-11T22:15:29.878008+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:15:29.878008+0000","last_undegraded":"2024-03-11T22:15:29.878008+0000","last_fullsized":"2024-03-11T22:15:29.878008+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:18.067284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:18.067284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:46.629798+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":539,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.865988+0000","last_change":"2024-03-11T22:14:11.673622+0000","last_active":"2024-03-11T22:15:29.865988+0000","last_peered":"2024-03-11T22:15:29.865988+0000","last_clean":"2024-03-11T22:15:29.865988+0000","last_became_active":"2024-03-11T22:13:38.569606+0000","last_became_peered":"2024-03-11T22:13:38.569606+0000","last_unstale":"2024-03-11T22:15:29.865988+0000","last_undegraded":"2024-03-11T22:15:29.865988+0000","last_fullsized":"2024-03-11T22:15:29.865988+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:11.673454+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:11.673454+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:37.168404+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"189'3478","reported_seq":3516,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.666662+0000","last_change":"2024-03-11T22:14:52.243433+0000","last_active":"2024-03-11T22:15:30.666662+0000","last_peered":"2024-03-11T22:15:30.666662+0000","last_clean":"2024-03-11T22:15:30.666662+0000","last_became_active":"2024-03-11T22:14:52.243038+0000","last_became_peered":"2024-03-11T22:14:52.243038+0000","last_unstale":"2024-03-11T22:15:30.666662+0000","last_undegraded":"2024-03-11T22:15:30.666662+0000","last_fullsized":"2024-03-11T22:15:30.666662+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3478,"log_dups_size":0,"ondisk_log_size":3478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:52.893231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227934208,"num_objects":3478,"num_object_clones":0,"num_object_copies":10434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3478,"num_write_kb":222592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":84,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.869247+0000","last_change":"2024-03-11T22:15:22.254990+0000","last_active":"2024-03-11T22:15:29.869247+0000","last_peered":"2024-03-11T22:15:29.869247+0000","last_clean":"2024-03-11T22:15:29.869247+0000","last_became_active":"2024-03-11T22:14:52.240437+0000","last_became_peered":"2024-03-11T22:14:52.240437+0000","last_unstale":"2024-03-11T22:15:29.869247+0000","last_undegraded":"2024-03-11T22:15:29.869247+0000","last_fullsized":"2024-03-11T22:15:29.869247+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:15:22.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:22.254869+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:26.708596+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":39,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.869290+0000","last_change":"2024-03-11T22:14:52.242036+0000","last_active":"2024-03-11T22:15:29.869290+0000","last_peered":"2024-03-11T22:15:29.869290+0000","last_clean":"2024-03-11T22:15:29.869290+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:15:29.869290+0000","last_undegraded":"2024-03-11T22:15:29.869290+0000","last_fullsized":"2024-03-11T22:15:29.869290+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:25.892396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:25.892396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:35.013158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":272,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866947+0000","last_change":"2024-03-11T22:14:26.672389+0000","last_active":"2024-03-11T22:15:29.866947+0000","last_peered":"2024-03-11T22:15:29.866947+0000","last_clean":"2024-03-11T22:15:29.866947+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:15:29.866947+0000","last_undegraded":"2024-03-11T22:15:29.866947+0000","last_fullsized":"2024-03-11T22:15:29.866947+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:26.672153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.672153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:47.573533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":260,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873553+0000","last_change":"2024-03-11T22:13:17.533805+0000","last_active":"2024-03-11T22:15:29.873553+0000","last_peered":"2024-03-11T22:15:29.873553+0000","last_clean":"2024-03-11T22:15:29.873553+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:15:29.873553+0000","last_undegraded":"2024-03-11T22:15:29.873553+0000","last_fullsized":"2024-03-11T22:15:29.873553+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:17.533607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:17.533607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"93'130","reported_seq":560,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.874934+0000","last_change":"2024-03-11T22:15:04.424215+0000","last_active":"2024-03-11T22:15:29.874934+0000","last_peered":"2024-03-11T22:15:29.874934+0000","last_clean":"2024-03-11T22:15:29.874934+0000","last_became_active":"2024-03-11T22:14:43.194793+0000","last_became_peered":"2024-03-11T22:14:43.194793+0000","last_unstale":"2024-03-11T22:15:29.874934+0000","last_undegraded":"2024-03-11T22:15:29.874934+0000","last_fullsized":"2024-03-11T22:15:29.874934+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:04.424114+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:04.424114+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:32.665794+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3","version":"189'3335","reported_seq":3373,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.448311+0000","last_change":"2024-03-11T22:14:52.245584+0000","last_active":"2024-03-11T22:15:30.448311+0000","last_peered":"2024-03-11T22:15:30.448311+0000","last_clean":"2024-03-11T22:15:30.448311+0000","last_became_active":"2024-03-11T22:14:52.245192+0000","last_became_peered":"2024-03-11T22:14:52.245192+0000","last_unstale":"2024-03-11T22:15:30.448311+0000","last_undegraded":"2024-03-11T22:15:30.448311+0000","last_fullsized":"2024-03-11T22:15:30.448311+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3335,"log_dups_size":0,"ondisk_log_size":3335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:11.735324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218562560,"num_objects":3335,"num_object_clones":0,"num_object_copies":10005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3335,"num_write_kb":213440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":301,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872531+0000","last_change":"2024-03-11T22:14:14.861720+0000","last_active":"2024-03-11T22:15:29.872531+0000","last_peered":"2024-03-11T22:15:29.872531+0000","last_clean":"2024-03-11T22:15:29.872531+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:15:29.872531+0000","last_undegraded":"2024-03-11T22:15:29.872531+0000","last_fullsized":"2024-03-11T22:15:29.872531+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:14:14.861573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.861573+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873168+0000","last_change":"2024-03-11T22:14:38.013139+0000","last_active":"2024-03-11T22:15:29.873168+0000","last_peered":"2024-03-11T22:15:29.873168+0000","last_clean":"2024-03-11T22:15:29.873168+0000","last_became_active":"2024-03-11T22:08:33.037537+0000","last_became_peered":"2024-03-11T22:08:33.037537+0000","last_unstale":"2024-03-11T22:15:29.873168+0000","last_undegraded":"2024-03-11T22:15:29.873168+0000","last_fullsized":"2024-03-11T22:15:29.873168+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:38.013058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:38.013058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:44.722852+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":72,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866259+0000","last_change":"2024-03-11T22:14:52.248753+0000","last_active":"2024-03-11T22:15:29.866259+0000","last_peered":"2024-03-11T22:15:29.866259+0000","last_clean":"2024-03-11T22:15:29.866259+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:15:29.866259+0000","last_undegraded":"2024-03-11T22:15:29.866259+0000","last_fullsized":"2024-03-11T22:15:29.866259+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:13:35.126665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:13:35.126665+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":268,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867340+0000","last_change":"2024-03-11T22:15:14.620431+0000","last_active":"2024-03-11T22:15:29.867340+0000","last_peered":"2024-03-11T22:15:29.867340+0000","last_clean":"2024-03-11T22:15:29.867340+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:15:29.867340+0000","last_undegraded":"2024-03-11T22:15:29.867340+0000","last_fullsized":"2024-03-11T22:15:29.867340+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.620337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.620337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:14.661179+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":539,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866184+0000","last_change":"2024-03-11T22:14:12.712999+0000","last_active":"2024-03-11T22:15:29.866184+0000","last_peered":"2024-03-11T22:15:29.866184+0000","last_clean":"2024-03-11T22:15:29.866184+0000","last_became_active":"2024-03-11T22:13:38.573794+0000","last_became_peered":"2024-03-11T22:13:38.573794+0000","last_unstale":"2024-03-11T22:15:29.866184+0000","last_undegraded":"2024-03-11T22:15:29.866184+0000","last_fullsized":"2024-03-11T22:15:29.866184+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:12.712874+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:12.712874+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:40.396091+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"189'3455","reported_seq":3493,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.658109+0000","last_change":"2024-03-11T22:14:52.236231+0000","last_active":"2024-03-11T22:15:30.658109+0000","last_peered":"2024-03-11T22:15:30.658109+0000","last_clean":"2024-03-11T22:15:30.658109+0000","last_became_active":"2024-03-11T22:14:52.235927+0000","last_became_peered":"2024-03-11T22:14:52.235927+0000","last_unstale":"2024-03-11T22:15:30.658109+0000","last_undegraded":"2024-03-11T22:15:30.658109+0000","last_fullsized":"2024-03-11T22:15:30.658109+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3455,"log_dups_size":0,"ondisk_log_size":3455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:56.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226426880,"num_objects":3455,"num_object_clones":0,"num_object_copies":10365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3455,"num_write_kb":221120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":46,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872085+0000","last_change":"2024-03-11T22:15:14.294473+0000","last_active":"2024-03-11T22:15:29.872085+0000","last_peered":"2024-03-11T22:15:29.872085+0000","last_clean":"2024-03-11T22:15:29.872085+0000","last_became_active":"2024-03-11T22:14:52.233157+0000","last_became_peered":"2024-03-11T22:14:52.233157+0000","last_unstale":"2024-03-11T22:15:29.872085+0000","last_undegraded":"2024-03-11T22:15:29.872085+0000","last_fullsized":"2024-03-11T22:15:29.872085+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.293977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.293977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:32.648842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":303,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873256+0000","last_change":"2024-03-11T22:12:39.447777+0000","last_active":"2024-03-11T22:15:29.873256+0000","last_peered":"2024-03-11T22:15:29.873256+0000","last_clean":"2024-03-11T22:15:29.873256+0000","last_became_active":"2024-03-11T22:08:35.085979+0000","last_became_peered":"2024-03-11T22:08:35.085979+0000","last_unstale":"2024-03-11T22:15:29.873256+0000","last_undegraded":"2024-03-11T22:15:29.873256+0000","last_fullsized":"2024-03-11T22:15:29.873256+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:12:39.447312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:39.447312+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.877263+0000","last_change":"2024-03-11T22:13:26.481234+0000","last_active":"2024-03-11T22:15:29.877263+0000","last_peered":"2024-03-11T22:15:29.877263+0000","last_clean":"2024-03-11T22:15:29.877263+0000","last_became_active":"2024-03-11T22:08:33.036469+0000","last_became_peered":"2024-03-11T22:08:33.036469+0000","last_unstale":"2024-03-11T22:15:29.877263+0000","last_undegraded":"2024-03-11T22:15:29.877263+0000","last_fullsized":"2024-03-11T22:15:29.877263+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:26.481122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:26.481122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":46,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.871866+0000","last_change":"2024-03-11T22:15:12.339167+0000","last_active":"2024-03-11T22:15:29.871866+0000","last_peered":"2024-03-11T22:15:29.871866+0000","last_clean":"2024-03-11T22:15:29.871866+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:15:29.871866+0000","last_undegraded":"2024-03-11T22:15:29.871866+0000","last_fullsized":"2024-03-11T22:15:29.871866+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:12.339050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:12.339050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:15.388005+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":484,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.874497+0000","last_change":"2024-03-11T22:15:04.043741+0000","last_active":"2024-03-11T22:15:29.874497+0000","last_peered":"2024-03-11T22:15:29.874497+0000","last_clean":"2024-03-11T22:15:29.874497+0000","last_became_active":"2024-03-11T22:14:52.240971+0000","last_became_peered":"2024-03-11T22:14:52.240971+0000","last_unstale":"2024-03-11T22:15:29.874497+0000","last_undegraded":"2024-03-11T22:15:29.874497+0000","last_fullsized":"2024-03-11T22:15:29.874497+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:04.043590+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:04.043590+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:13.992528+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"189'3516","reported_seq":3553,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.163888+0000","last_change":"2024-03-11T22:14:52.243501+0000","last_active":"2024-03-11T22:15:30.163888+0000","last_peered":"2024-03-11T22:15:30.163888+0000","last_clean":"2024-03-11T22:15:30.163888+0000","last_became_active":"2024-03-11T22:14:52.243195+0000","last_became_peered":"2024-03-11T22:14:52.243195+0000","last_unstale":"2024-03-11T22:15:30.163888+0000","last_undegraded":"2024-03-11T22:15:30.163888+0000","last_fullsized":"2024-03-11T22:15:30.163888+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:55.466333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230424576,"num_objects":3516,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3516,"num_write_kb":225024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":356,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873362+0000","last_change":"2024-03-11T22:12:31.264333+0000","last_active":"2024-03-11T22:15:29.873362+0000","last_peered":"2024-03-11T22:15:29.873362+0000","last_clean":"2024-03-11T22:15:29.873362+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:15:29.873362+0000","last_undegraded":"2024-03-11T22:15:29.873362+0000","last_fullsized":"2024-03-11T22:15:29.873362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:12:31.264213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:12:31.264213+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":78,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866077+0000","last_change":"2024-03-11T22:14:52.248658+0000","last_active":"2024-03-11T22:15:29.866077+0000","last_peered":"2024-03-11T22:15:29.866077+0000","last_clean":"2024-03-11T22:15:29.866077+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:15:29.866077+0000","last_undegraded":"2024-03-11T22:15:29.866077+0000","last_fullsized":"2024-03-11T22:15:29.866077+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:13:27.028171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:27.028171+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":281,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.878520+0000","last_change":"2024-03-11T22:15:08.820614+0000","last_active":"2024-03-11T22:15:29.878520+0000","last_peered":"2024-03-11T22:15:29.878520+0000","last_clean":"2024-03-11T22:15:29.878520+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:15:29.878520+0000","last_undegraded":"2024-03-11T22:15:29.878520+0000","last_fullsized":"2024-03-11T22:15:29.878520+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:15:08.820296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:08.820296+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:20.880401+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":218,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.869658+0000","last_change":"2024-03-11T22:14:33.134910+0000","last_active":"2024-03-11T22:15:29.869658+0000","last_peered":"2024-03-11T22:15:29.869658+0000","last_clean":"2024-03-11T22:15:29.869658+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:15:29.869658+0000","last_undegraded":"2024-03-11T22:15:29.869658+0000","last_fullsized":"2024-03-11T22:15:29.869658+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:33.134806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:14:33.134806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:53.275391+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":552,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873408+0000","last_change":"2024-03-11T22:15:06.243411+0000","last_active":"2024-03-11T22:15:29.873408+0000","last_peered":"2024-03-11T22:15:29.873408+0000","last_clean":"2024-03-11T22:15:29.873408+0000","last_became_active":"2024-03-11T22:14:38.164924+0000","last_became_peered":"2024-03-11T22:14:38.164924+0000","last_unstale":"2024-03-11T22:15:29.873408+0000","last_undegraded":"2024-03-11T22:15:29.873408+0000","last_fullsized":"2024-03-11T22:15:29.873408+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.243267+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.243267+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:13.005193+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"189'3430","reported_seq":3468,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.672515+0000","last_change":"2024-03-11T22:14:52.242543+0000","last_active":"2024-03-11T22:15:30.672515+0000","last_peered":"2024-03-11T22:15:30.672515+0000","last_clean":"2024-03-11T22:15:30.672515+0000","last_became_active":"2024-03-11T22:14:52.240224+0000","last_became_peered":"2024-03-11T22:14:52.240224+0000","last_unstale":"2024-03-11T22:15:30.672515+0000","last_undegraded":"2024-03-11T22:15:30.672515+0000","last_fullsized":"2024-03-11T22:15:30.672515+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:53.025545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224788480,"num_objects":3430,"num_object_clones":0,"num_object_copies":10290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3430,"num_write_kb":219520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":364,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.876470+0000","last_change":"2024-03-11T22:14:27.105480+0000","last_active":"2024-03-11T22:15:29.876470+0000","last_peered":"2024-03-11T22:15:29.876470+0000","last_clean":"2024-03-11T22:15:29.876470+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:15:29.876470+0000","last_undegraded":"2024-03-11T22:15:29.876470+0000","last_fullsized":"2024-03-11T22:15:29.876470+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:14:27.105394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:27.105394+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:56.745563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":291,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.871920+0000","last_change":"2024-03-11T22:14:28.936592+0000","last_active":"2024-03-11T22:15:29.871920+0000","last_peered":"2024-03-11T22:15:29.871920+0000","last_clean":"2024-03-11T22:15:29.871920+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:15:29.871920+0000","last_undegraded":"2024-03-11T22:15:29.871920+0000","last_fullsized":"2024-03-11T22:15:29.871920+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:14:28.936501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:28.936501+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:32.315105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":288,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.869175+0000","last_change":"2024-03-11T22:15:24.618272+0000","last_active":"2024-03-11T22:15:29.869175+0000","last_peered":"2024-03-11T22:15:29.869175+0000","last_clean":"2024-03-11T22:15:29.869175+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:15:29.869175+0000","last_undegraded":"2024-03-11T22:15:29.869175+0000","last_fullsized":"2024-03-11T22:15:29.869175+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:24.618157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:24.618157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:38.099831+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":268,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.875180+0000","last_change":"2024-03-11T22:15:26.900176+0000","last_active":"2024-03-11T22:15:29.875180+0000","last_peered":"2024-03-11T22:15:29.875180+0000","last_clean":"2024-03-11T22:15:29.875180+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:15:29.875180+0000","last_undegraded":"2024-03-11T22:15:29.875180+0000","last_fullsized":"2024-03-11T22:15:29.875180+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:26.899954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:26.899954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:31.796881+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":440,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.869047+0000","last_change":"2024-03-11T22:15:08.189185+0000","last_active":"2024-03-11T22:15:29.869047+0000","last_peered":"2024-03-11T22:15:29.869047+0000","last_clean":"2024-03-11T22:15:29.869047+0000","last_became_active":"2024-03-11T22:14:38.164708+0000","last_became_peered":"2024-03-11T22:14:38.164708+0000","last_unstale":"2024-03-11T22:15:29.869047+0000","last_undegraded":"2024-03-11T22:15:29.869047+0000","last_fullsized":"2024-03-11T22:15:29.869047+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:08.189043+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:08.189043+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:10.087716+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.f","version":"189'3579","reported_seq":3617,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.524907+0000","last_change":"2024-03-11T22:14:52.238556+0000","last_active":"2024-03-11T22:15:30.524907+0000","last_peered":"2024-03-11T22:15:30.524907+0000","last_clean":"2024-03-11T22:15:30.524907+0000","last_became_active":"2024-03-11T22:14:52.238241+0000","last_became_peered":"2024-03-11T22:14:52.238241+0000","last_unstale":"2024-03-11T22:15:30.524907+0000","last_undegraded":"2024-03-11T22:15:30.524907+0000","last_fullsized":"2024-03-11T22:15:30.524907+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3579,"log_dups_size":0,"ondisk_log_size":3579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:08.580556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234553344,"num_objects":3579,"num_object_clones":0,"num_object_copies":10737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3579,"num_write_kb":229056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868407+0000","last_change":"2024-03-11T22:14:18.961752+0000","last_active":"2024-03-11T22:15:29.868407+0000","last_peered":"2024-03-11T22:15:29.868407+0000","last_clean":"2024-03-11T22:15:29.868407+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:15:29.868407+0000","last_undegraded":"2024-03-11T22:15:29.868407+0000","last_fullsized":"2024-03-11T22:15:29.868407+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:18.961673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:14:18.961673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:45.145191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":312,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.876075+0000","last_change":"2024-03-11T22:15:22.846514+0000","last_active":"2024-03-11T22:15:29.876075+0000","last_peered":"2024-03-11T22:15:29.876075+0000","last_clean":"2024-03-11T22:15:29.876075+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:15:29.876075+0000","last_undegraded":"2024-03-11T22:15:29.876075+0000","last_fullsized":"2024-03-11T22:15:29.876075+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:15:22.846352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:22.846352+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:37.818474+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868083+0000","last_change":"2024-03-11T22:14:36.079416+0000","last_active":"2024-03-11T22:15:29.868083+0000","last_peered":"2024-03-11T22:15:29.868083+0000","last_clean":"2024-03-11T22:15:29.868083+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:15:29.868083+0000","last_undegraded":"2024-03-11T22:15:29.868083+0000","last_fullsized":"2024-03-11T22:15:29.868083+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:36.079334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:36.079334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:41.253472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":276,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872592+0000","last_change":"2024-03-11T22:15:21.454660+0000","last_active":"2024-03-11T22:15:29.872592+0000","last_peered":"2024-03-11T22:15:29.872592+0000","last_clean":"2024-03-11T22:15:29.872592+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:15:29.872592+0000","last_undegraded":"2024-03-11T22:15:29.872592+0000","last_fullsized":"2024-03-11T22:15:29.872592+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:21.454272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:21.454272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:50.718809+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":440,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868954+0000","last_change":"2024-03-11T22:15:07.207510+0000","last_active":"2024-03-11T22:15:29.868954+0000","last_peered":"2024-03-11T22:15:29.868954+0000","last_clean":"2024-03-11T22:15:29.868954+0000","last_became_active":"2024-03-11T22:14:38.166870+0000","last_became_peered":"2024-03-11T22:14:38.166870+0000","last_unstale":"2024-03-11T22:15:29.868954+0000","last_undegraded":"2024-03-11T22:15:29.868954+0000","last_fullsized":"2024-03-11T22:15:29.868954+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:07.207365+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:07.207365+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:28.797642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"189'3537","reported_seq":3575,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.426994+0000","last_change":"2024-03-11T22:14:52.247032+0000","last_active":"2024-03-11T22:15:30.426994+0000","last_peered":"2024-03-11T22:15:30.426994+0000","last_clean":"2024-03-11T22:15:30.426994+0000","last_became_active":"2024-03-11T22:14:52.246614+0000","last_became_peered":"2024-03-11T22:14:52.246614+0000","last_unstale":"2024-03-11T22:15:30.426994+0000","last_undegraded":"2024-03-11T22:15:30.426994+0000","last_fullsized":"2024-03-11T22:15:30.426994+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3537,"log_dups_size":0,"ondisk_log_size":3537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:54.213666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231800832,"num_objects":3537,"num_object_clones":0,"num_object_copies":10611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":226368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":272,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867057+0000","last_change":"2024-03-11T22:14:29.697272+0000","last_active":"2024-03-11T22:15:29.867057+0000","last_peered":"2024-03-11T22:15:29.867057+0000","last_clean":"2024-03-11T22:15:29.867057+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:15:29.867057+0000","last_undegraded":"2024-03-11T22:15:29.867057+0000","last_fullsized":"2024-03-11T22:15:29.867057+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:29.697144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:29.697144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:37.479317+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":294,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.878429+0000","last_change":"2024-03-11T22:14:21.065394+0000","last_active":"2024-03-11T22:15:29.878429+0000","last_peered":"2024-03-11T22:15:29.878429+0000","last_clean":"2024-03-11T22:15:29.878429+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:15:29.878429+0000","last_undegraded":"2024-03-11T22:15:29.878429+0000","last_fullsized":"2024-03-11T22:15:29.878429+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:14:21.065306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.065306+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:37.928690+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":679,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867234+0000","last_change":"2024-03-11T22:14:13.848646+0000","last_active":"2024-03-11T22:15:29.867234+0000","last_peered":"2024-03-11T22:15:29.867234+0000","last_clean":"2024-03-11T22:15:29.867234+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:15:29.867234+0000","last_undegraded":"2024-03-11T22:15:29.867234+0000","last_fullsized":"2024-03-11T22:15:29.867234+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:14:13.848518+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:14:13.848518+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:36.660519+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":384,"num_read_kb":379,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.875970+0000","last_change":"2024-03-11T22:14:35.025497+0000","last_active":"2024-03-11T22:15:29.875970+0000","last_peered":"2024-03-11T22:15:29.875970+0000","last_clean":"2024-03-11T22:15:29.875970+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:15:29.875970+0000","last_undegraded":"2024-03-11T22:15:29.875970+0000","last_fullsized":"2024-03-11T22:15:29.875970+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:35.025383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:35.025383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:52.119736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":276,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867362+0000","last_change":"2024-03-11T22:14:38.073584+0000","last_active":"2024-03-11T22:15:29.867362+0000","last_peered":"2024-03-11T22:15:29.867362+0000","last_clean":"2024-03-11T22:15:29.867362+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:15:29.867362+0000","last_undegraded":"2024-03-11T22:15:29.867362+0000","last_fullsized":"2024-03-11T22:15:29.867362+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:38.073415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:38.073415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:40.723809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":552,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.875374+0000","last_change":"2024-03-11T22:15:06.523755+0000","last_active":"2024-03-11T22:15:29.875374+0000","last_peered":"2024-03-11T22:15:29.875374+0000","last_clean":"2024-03-11T22:15:29.875374+0000","last_became_active":"2024-03-11T22:14:38.164101+0000","last_became_peered":"2024-03-11T22:14:38.164101+0000","last_unstale":"2024-03-11T22:15:29.875374+0000","last_undegraded":"2024-03-11T22:15:29.875374+0000","last_fullsized":"2024-03-11T22:15:29.875374+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.523513+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.523513+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:09.939331+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"189'3391","reported_seq":3429,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.184756+0000","last_change":"2024-03-11T22:14:52.240542+0000","last_active":"2024-03-11T22:15:30.184756+0000","last_peered":"2024-03-11T22:15:30.184756+0000","last_clean":"2024-03-11T22:15:30.184756+0000","last_became_active":"2024-03-11T22:14:52.240210+0000","last_became_peered":"2024-03-11T22:14:52.240210+0000","last_unstale":"2024-03-11T22:15:30.184756+0000","last_undegraded":"2024-03-11T22:15:30.184756+0000","last_fullsized":"2024-03-11T22:15:30.184756+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:06.873414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222232576,"num_objects":3391,"num_object_clones":0,"num_object_copies":10173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":217024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":218,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.874122+0000","last_change":"2024-03-11T22:14:23.041388+0000","last_active":"2024-03-11T22:15:29.874122+0000","last_peered":"2024-03-11T22:15:29.874122+0000","last_clean":"2024-03-11T22:15:29.874122+0000","last_became_active":"2024-03-11T22:10:35.412547+0000","last_became_peered":"2024-03-11T22:10:35.412547+0000","last_unstale":"2024-03-11T22:15:29.874122+0000","last_undegraded":"2024-03-11T22:15:29.874122+0000","last_fullsized":"2024-03-11T22:15:29.874122+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.041267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.041267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":390,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868817+0000","last_change":"2024-03-11T22:14:37.137570+0000","last_active":"2024-03-11T22:15:29.868817+0000","last_peered":"2024-03-11T22:15:29.868817+0000","last_clean":"2024-03-11T22:15:29.868817+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:15:29.868817+0000","last_undegraded":"2024-03-11T22:15:29.868817+0000","last_fullsized":"2024-03-11T22:15:29.868817+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-11T22:14:37.137464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:37.137464+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:49.689233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.871620+0000","last_change":"2024-03-11T22:13:33.460800+0000","last_active":"2024-03-11T22:15:29.871620+0000","last_peered":"2024-03-11T22:15:29.871620+0000","last_clean":"2024-03-11T22:15:29.871620+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:15:29.871620+0000","last_undegraded":"2024-03-11T22:15:29.871620+0000","last_fullsized":"2024-03-11T22:15:29.871620+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:33.460308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:13:33.460308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":552,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.875291+0000","last_change":"2024-03-11T22:15:03.739785+0000","last_active":"2024-03-11T22:15:29.875291+0000","last_peered":"2024-03-11T22:15:29.875291+0000","last_clean":"2024-03-11T22:15:29.875291+0000","last_became_active":"2024-03-11T22:14:38.165091+0000","last_became_peered":"2024-03-11T22:14:38.165091+0000","last_unstale":"2024-03-11T22:15:29.875291+0000","last_undegraded":"2024-03-11T22:15:29.875291+0000","last_fullsized":"2024-03-11T22:15:29.875291+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:03.739634+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:15:03.739634+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:26.259765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":268,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866617+0000","last_change":"2024-03-11T22:13:34.478753+0000","last_active":"2024-03-11T22:15:29.866617+0000","last_peered":"2024-03-11T22:15:29.866617+0000","last_clean":"2024-03-11T22:15:29.866617+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:15:29.866617+0000","last_undegraded":"2024-03-11T22:15:29.866617+0000","last_fullsized":"2024-03-11T22:15:29.866617+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:34.478634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:34.478634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"189'3354","reported_seq":3392,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.543829+0000","last_change":"2024-03-11T22:14:52.240355+0000","last_active":"2024-03-11T22:15:30.543829+0000","last_peered":"2024-03-11T22:15:30.543829+0000","last_clean":"2024-03-11T22:15:30.543829+0000","last_became_active":"2024-03-11T22:14:52.238824+0000","last_became_peered":"2024-03-11T22:14:52.238824+0000","last_unstale":"2024-03-11T22:15:30.543829+0000","last_undegraded":"2024-03-11T22:15:30.543829+0000","last_fullsized":"2024-03-11T22:15:30.543829+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3354,"log_dups_size":0,"ondisk_log_size":3354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:07.988297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219807744,"num_objects":3354,"num_object_clones":0,"num_object_copies":10062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3354,"num_write_kb":214656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.877670+0000","last_change":"2024-03-11T22:14:21.033755+0000","last_active":"2024-03-11T22:15:29.877670+0000","last_peered":"2024-03-11T22:15:29.877670+0000","last_clean":"2024-03-11T22:15:29.877670+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:15:29.877670+0000","last_undegraded":"2024-03-11T22:15:29.877670+0000","last_fullsized":"2024-03-11T22:15:29.877670+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:21.033664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.033664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:38.113619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":311,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872672+0000","last_change":"2024-03-11T22:15:15.581703+0000","last_active":"2024-03-11T22:15:29.872672+0000","last_peered":"2024-03-11T22:15:29.872672+0000","last_clean":"2024-03-11T22:15:29.872672+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:15:29.872672+0000","last_undegraded":"2024-03-11T22:15:29.872672+0000","last_fullsized":"2024-03-11T22:15:29.872672+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-03-11T22:15:15.581545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:15.581545+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:15.895749+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":305,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.869435+0000","last_change":"2024-03-11T22:15:21.847867+0000","last_active":"2024-03-11T22:15:29.869435+0000","last_peered":"2024-03-11T22:15:29.869435+0000","last_clean":"2024-03-11T22:15:29.869435+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:15:29.869435+0000","last_undegraded":"2024-03-11T22:15:29.869435+0000","last_fullsized":"2024-03-11T22:15:29.869435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:21.846681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:15:21.846681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.774603+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":288,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867888+0000","last_change":"2024-03-11T22:15:18.479060+0000","last_active":"2024-03-11T22:15:29.867888+0000","last_peered":"2024-03-11T22:15:29.867888+0000","last_clean":"2024-03-11T22:15:29.867888+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:15:29.867888+0000","last_undegraded":"2024-03-11T22:15:29.867888+0000","last_fullsized":"2024-03-11T22:15:29.867888+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:18.478850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:18.478850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:25.236419+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":218,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868902+0000","last_change":"2024-03-11T22:14:34.131682+0000","last_active":"2024-03-11T22:15:29.868902+0000","last_peered":"2024-03-11T22:15:29.868902+0000","last_clean":"2024-03-11T22:15:29.868902+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:15:29.868902+0000","last_undegraded":"2024-03-11T22:15:29.868902+0000","last_fullsized":"2024-03-11T22:15:29.868902+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.131530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:02.487163+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":559,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.876109+0000","last_change":"2024-03-11T22:15:04.771002+0000","last_active":"2024-03-11T22:15:29.876109+0000","last_peered":"2024-03-11T22:15:29.876109+0000","last_clean":"2024-03-11T22:15:29.876109+0000","last_became_active":"2024-03-11T22:14:38.166129+0000","last_became_peered":"2024-03-11T22:14:38.166129+0000","last_unstale":"2024-03-11T22:15:29.876109+0000","last_undegraded":"2024-03-11T22:15:29.876109+0000","last_fullsized":"2024-03-11T22:15:29.876109+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:04.770858+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:04.770858+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:30.991906+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"189'3509","reported_seq":3547,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.146840+0000","last_change":"2024-03-11T22:14:52.243442+0000","last_active":"2024-03-11T22:15:30.146840+0000","last_peered":"2024-03-11T22:15:30.146840+0000","last_clean":"2024-03-11T22:15:30.146840+0000","last_became_active":"2024-03-11T22:14:52.243066+0000","last_became_peered":"2024-03-11T22:14:52.243066+0000","last_unstale":"2024-03-11T22:15:30.146840+0000","last_undegraded":"2024-03-11T22:15:30.146840+0000","last_fullsized":"2024-03-11T22:15:30.146840+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3509,"log_dups_size":0,"ondisk_log_size":3509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:11.959240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229965824,"num_objects":3509,"num_object_clones":0,"num_object_copies":10527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3509,"num_write_kb":224576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":272,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.877984+0000","last_change":"2024-03-11T22:14:16.998358+0000","last_active":"2024-03-11T22:15:29.877984+0000","last_peered":"2024-03-11T22:15:29.877984+0000","last_clean":"2024-03-11T22:15:29.877984+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:15:29.877984+0000","last_undegraded":"2024-03-11T22:15:29.877984+0000","last_fullsized":"2024-03-11T22:15:29.877984+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:16.998238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:16.998238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":416,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868176+0000","last_change":"2024-03-11T22:15:11.547339+0000","last_active":"2024-03-11T22:15:29.868176+0000","last_peered":"2024-03-11T22:15:29.868176+0000","last_clean":"2024-03-11T22:15:29.868176+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:15:29.868176+0000","last_undegraded":"2024-03-11T22:15:29.868176+0000","last_fullsized":"2024-03-11T22:15:29.868176+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-11T22:15:11.547182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:11.547182+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:33.121742+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":47,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.875983+0000","last_change":"2024-03-11T22:15:24.899285+0000","last_active":"2024-03-11T22:15:29.875983+0000","last_peered":"2024-03-11T22:15:29.875983+0000","last_clean":"2024-03-11T22:15:29.875983+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:15:29.875983+0000","last_undegraded":"2024-03-11T22:15:29.875983+0000","last_fullsized":"2024-03-11T22:15:29.875983+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:24.899168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:24.899168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:50.013607+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":292,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866370+0000","last_change":"2024-03-11T22:14:16.662735+0000","last_active":"2024-03-11T22:15:29.866370+0000","last_peered":"2024-03-11T22:15:29.866370+0000","last_clean":"2024-03-11T22:15:29.866370+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:15:29.866370+0000","last_undegraded":"2024-03-11T22:15:29.866370+0000","last_fullsized":"2024-03-11T22:15:29.866370+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:14:16.662596+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:14:16.662596+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:45.012386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":260,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872769+0000","last_change":"2024-03-11T22:12:45.439711+0000","last_active":"2024-03-11T22:15:29.872769+0000","last_peered":"2024-03-11T22:15:29.872769+0000","last_clean":"2024-03-11T22:15:29.872769+0000","last_became_active":"2024-03-11T22:08:39.083738+0000","last_became_peered":"2024-03-11T22:08:39.083738+0000","last_unstale":"2024-03-11T22:15:29.872769+0000","last_undegraded":"2024-03-11T22:15:29.872769+0000","last_fullsized":"2024-03-11T22:15:29.872769+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:12:45.439584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":551,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.865967+0000","last_change":"2024-03-11T22:15:06.673117+0000","last_active":"2024-03-11T22:15:29.865967+0000","last_peered":"2024-03-11T22:15:29.865967+0000","last_clean":"2024-03-11T22:15:29.865967+0000","last_became_active":"2024-03-11T22:14:38.164859+0000","last_became_peered":"2024-03-11T22:14:38.164859+0000","last_unstale":"2024-03-11T22:15:29.865967+0000","last_undegraded":"2024-03-11T22:15:29.865967+0000","last_fullsized":"2024-03-11T22:15:29.865967+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.672938+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.672938+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:25.674338+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"189'3436","reported_seq":3472,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.447703+0000","last_change":"2024-03-11T22:14:52.242768+0000","last_active":"2024-03-11T22:15:30.447703+0000","last_peered":"2024-03-11T22:15:30.447703+0000","last_clean":"2024-03-11T22:15:30.447703+0000","last_became_active":"2024-03-11T22:14:52.242455+0000","last_became_peered":"2024-03-11T22:14:52.242455+0000","last_unstale":"2024-03-11T22:15:30.447703+0000","last_undegraded":"2024-03-11T22:15:30.447703+0000","last_fullsized":"2024-03-11T22:15:30.447703+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3436,"log_dups_size":0,"ondisk_log_size":3436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:15.907365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225181696,"num_objects":3436,"num_object_clones":0,"num_object_copies":10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3436,"num_write_kb":219904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":308,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.876366+0000","last_change":"2024-03-11T22:15:11.902221+0000","last_active":"2024-03-11T22:15:29.876366+0000","last_peered":"2024-03-11T22:15:29.876366+0000","last_clean":"2024-03-11T22:15:29.876366+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:15:29.876366+0000","last_undegraded":"2024-03-11T22:15:29.876366+0000","last_fullsized":"2024-03-11T22:15:29.876366+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:15:11.902080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:11.902080+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:27.107447+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867144+0000","last_change":"2024-03-11T22:14:37.615454+0000","last_active":"2024-03-11T22:15:29.867144+0000","last_peered":"2024-03-11T22:15:29.867144+0000","last_clean":"2024-03-11T22:15:29.867144+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:15:29.867144+0000","last_undegraded":"2024-03-11T22:15:29.867144+0000","last_fullsized":"2024-03-11T22:15:29.867144+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:37.615375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:37.615375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:04.760537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.870999+0000","last_change":"2024-03-11T22:15:19.936668+0000","last_active":"2024-03-11T22:15:29.870999+0000","last_peered":"2024-03-11T22:15:29.870999+0000","last_clean":"2024-03-11T22:15:29.870999+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:15:29.870999+0000","last_undegraded":"2024-03-11T22:15:29.870999+0000","last_fullsized":"2024-03-11T22:15:29.870999+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:19.936555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:15:19.936555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:35.701409+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":268,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.876184+0000","last_change":"2024-03-11T22:14:36.008065+0000","last_active":"2024-03-11T22:15:29.876184+0000","last_peered":"2024-03-11T22:15:29.876184+0000","last_clean":"2024-03-11T22:15:29.876184+0000","last_became_active":"2024-03-11T22:08:39.085496+0000","last_became_peered":"2024-03-11T22:08:39.085496+0000","last_unstale":"2024-03-11T22:15:29.876184+0000","last_undegraded":"2024-03-11T22:15:29.876184+0000","last_fullsized":"2024-03-11T22:15:29.876184+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:36.007981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:36.007981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:55.604651+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"93'130","reported_seq":367,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.871972+0000","last_change":"2024-03-11T22:15:09.205252+0000","last_active":"2024-03-11T22:15:29.871972+0000","last_peered":"2024-03-11T22:15:29.871972+0000","last_clean":"2024-03-11T22:15:29.871972+0000","last_became_active":"2024-03-11T22:14:52.236711+0000","last_became_peered":"2024-03-11T22:14:52.236711+0000","last_unstale":"2024-03-11T22:15:29.871972+0000","last_undegraded":"2024-03-11T22:15:29.871972+0000","last_fullsized":"2024-03-11T22:15:29.871972+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:09.205116+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:09.205116+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:10.406518+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"189'3614","reported_seq":3652,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.669918+0000","last_change":"2024-03-11T22:14:52.243118+0000","last_active":"2024-03-11T22:15:30.669918+0000","last_peered":"2024-03-11T22:15:30.669918+0000","last_clean":"2024-03-11T22:15:30.669918+0000","last_became_active":"2024-03-11T22:14:52.242807+0000","last_became_peered":"2024-03-11T22:14:52.242807+0000","last_unstale":"2024-03-11T22:15:30.669918+0000","last_undegraded":"2024-03-11T22:15:30.669918+0000","last_fullsized":"2024-03-11T22:15:30.669918+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:19.483709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236847104,"num_objects":3614,"num_object_clones":0,"num_object_copies":10842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3614,"num_write_kb":231296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":315,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868320+0000","last_change":"2024-03-11T22:14:26.007003+0000","last_active":"2024-03-11T22:15:29.868320+0000","last_peered":"2024-03-11T22:15:29.868320+0000","last_clean":"2024-03-11T22:15:29.868320+0000","last_became_active":"2024-03-11T22:08:35.087046+0000","last_became_peered":"2024-03-11T22:08:35.087046+0000","last_unstale":"2024-03-11T22:15:29.868320+0000","last_undegraded":"2024-03-11T22:15:29.868320+0000","last_fullsized":"2024-03-11T22:15:29.868320+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:14:26.006872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.006872+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:52.717574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":219,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.869313+0000","last_change":"2024-03-11T22:14:34.649271+0000","last_active":"2024-03-11T22:15:29.869313+0000","last_peered":"2024-03-11T22:15:29.869313+0000","last_clean":"2024-03-11T22:15:29.869313+0000","last_became_active":"2024-03-11T22:10:35.418715+0000","last_became_peered":"2024-03-11T22:10:35.418715+0000","last_unstale":"2024-03-11T22:15:29.869313+0000","last_undegraded":"2024-03-11T22:15:29.869313+0000","last_fullsized":"2024-03-11T22:15:29.869313+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.649172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.649172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:51.298499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.871465+0000","last_change":"2024-03-11T22:15:18.917780+0000","last_active":"2024-03-11T22:15:29.871465+0000","last_peered":"2024-03-11T22:15:29.871465+0000","last_clean":"2024-03-11T22:15:29.871465+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:15:29.871465+0000","last_undegraded":"2024-03-11T22:15:29.871465+0000","last_fullsized":"2024-03-11T22:15:29.871465+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:18.917625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.710319+0000","last_clean_scrub_stamp":"2024-03-11T22:15:18.917625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:31.494091+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":218,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866560+0000","last_change":"2024-03-11T22:14:14.005962+0000","last_active":"2024-03-11T22:15:29.866560+0000","last_peered":"2024-03-11T22:15:29.866560+0000","last_clean":"2024-03-11T22:15:29.866560+0000","last_became_active":"2024-03-11T22:10:35.410893+0000","last_became_peered":"2024-03-11T22:10:35.410893+0000","last_unstale":"2024-03-11T22:15:29.866560+0000","last_undegraded":"2024-03-11T22:15:29.866560+0000","last_fullsized":"2024-03-11T22:15:29.866560+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.005795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.005795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:32.664197+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"93'130","reported_seq":448,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868562+0000","last_change":"2024-03-11T22:15:06.157130+0000","last_active":"2024-03-11T22:15:29.868562+0000","last_peered":"2024-03-11T22:15:29.868562+0000","last_clean":"2024-03-11T22:15:29.868562+0000","last_became_active":"2024-03-11T22:14:43.195433+0000","last_became_peered":"2024-03-11T22:14:43.195433+0000","last_unstale":"2024-03-11T22:15:29.868562+0000","last_undegraded":"2024-03-11T22:15:29.868562+0000","last_fullsized":"2024-03-11T22:15:29.868562+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.157005+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.157005+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:06.486166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"189'3419","reported_seq":3457,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.156205+0000","last_change":"2024-03-11T22:14:52.240970+0000","last_active":"2024-03-11T22:15:30.156205+0000","last_peered":"2024-03-11T22:15:30.156205+0000","last_clean":"2024-03-11T22:15:30.156205+0000","last_became_active":"2024-03-11T22:14:52.240614+0000","last_became_peered":"2024-03-11T22:14:52.240614+0000","last_unstale":"2024-03-11T22:15:30.156205+0000","last_undegraded":"2024-03-11T22:15:30.156205+0000","last_fullsized":"2024-03-11T22:15:30.156205+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3419,"log_dups_size":0,"ondisk_log_size":3419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:17.716033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224067584,"num_objects":3419,"num_object_clones":0,"num_object_copies":10257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3419,"num_write_kb":218816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866259+0000","last_change":"2024-03-11T22:14:24.051145+0000","last_active":"2024-03-11T22:15:29.866259+0000","last_peered":"2024-03-11T22:15:29.866259+0000","last_clean":"2024-03-11T22:15:29.866259+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:15:29.866259+0000","last_undegraded":"2024-03-11T22:15:29.866259+0000","last_fullsized":"2024-03-11T22:15:29.866259+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:24.051020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:14:24.051020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:48.994239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":95,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866324+0000","last_change":"2024-03-11T22:15:11.678247+0000","last_active":"2024-03-11T22:15:29.866324+0000","last_peered":"2024-03-11T22:15:29.866324+0000","last_clean":"2024-03-11T22:15:29.866324+0000","last_became_active":"2024-03-11T22:14:52.247400+0000","last_became_peered":"2024-03-11T22:14:52.247400+0000","last_unstale":"2024-03-11T22:15:29.866324+0000","last_undegraded":"2024-03-11T22:15:29.866324+0000","last_fullsized":"2024-03-11T22:15:29.866324+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:15:11.678114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:11.678114+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:38.522520+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":50,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867984+0000","last_change":"2024-03-11T22:15:08.478721+0000","last_active":"2024-03-11T22:15:29.867984+0000","last_peered":"2024-03-11T22:15:29.867984+0000","last_clean":"2024-03-11T22:15:29.867984+0000","last_became_active":"2024-03-11T22:14:52.239329+0000","last_became_peered":"2024-03-11T22:14:52.239329+0000","last_unstale":"2024-03-11T22:15:29.867984+0000","last_undegraded":"2024-03-11T22:15:29.867984+0000","last_fullsized":"2024-03-11T22:15:29.867984+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:08.477912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:08.477912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:13.992163+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":262,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873854+0000","last_change":"2024-03-11T22:12:36.449430+0000","last_active":"2024-03-11T22:15:29.873854+0000","last_peered":"2024-03-11T22:15:29.873854+0000","last_clean":"2024-03-11T22:15:29.873854+0000","last_became_active":"2024-03-11T22:08:39.083141+0000","last_became_peered":"2024-03-11T22:08:39.083141+0000","last_unstale":"2024-03-11T22:15:29.873854+0000","last_undegraded":"2024-03-11T22:15:29.873854+0000","last_fullsized":"2024-03-11T22:15:29.873854+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:12:36.449307+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:12:36.449307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"93'130","reported_seq":566,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873294+0000","last_change":"2024-03-11T22:15:05.249082+0000","last_active":"2024-03-11T22:15:29.873294+0000","last_peered":"2024-03-11T22:15:29.873294+0000","last_clean":"2024-03-11T22:15:29.873294+0000","last_became_active":"2024-03-11T22:14:43.196117+0000","last_became_peered":"2024-03-11T22:14:43.196117+0000","last_unstale":"2024-03-11T22:15:29.873294+0000","last_undegraded":"2024-03-11T22:15:29.873294+0000","last_fullsized":"2024-03-11T22:15:29.873294+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:05.248940+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.248940+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:24.043494+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"189'3453","reported_seq":3491,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:30.144142+0000","last_change":"2024-03-11T22:14:52.249598+0000","last_active":"2024-03-11T22:15:30.144142+0000","last_peered":"2024-03-11T22:15:30.144142+0000","last_clean":"2024-03-11T22:15:30.144142+0000","last_became_active":"2024-03-11T22:14:52.248793+0000","last_became_peered":"2024-03-11T22:14:52.248793+0000","last_unstale":"2024-03-11T22:15:30.144142+0000","last_undegraded":"2024-03-11T22:15:30.144142+0000","last_fullsized":"2024-03-11T22:15:30.144142+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3453,"log_dups_size":0,"ondisk_log_size":3453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:16.661255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226295808,"num_objects":3453,"num_object_clones":0,"num_object_copies":10359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3453,"num_write_kb":220992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"99'9","reported_seq":303,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868870+0000","last_change":"2024-03-11T22:14:36.152155+0000","last_active":"2024-03-11T22:15:29.868870+0000","last_peered":"2024-03-11T22:15:29.868870+0000","last_clean":"2024-03-11T22:15:29.868870+0000","last_became_active":"2024-03-11T22:10:35.413089+0000","last_became_peered":"2024-03-11T22:10:35.413089+0000","last_unstale":"2024-03-11T22:15:29.868870+0000","last_undegraded":"2024-03-11T22:15:29.868870+0000","last_fullsized":"2024-03-11T22:15:29.868870+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'9","last_scrub_stamp":"2024-03-11T22:14:36.152067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:36.152067+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:59.182225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"91'22","reported_seq":316,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.865803+0000","last_change":"2024-03-11T22:15:25.619081+0000","last_active":"2024-03-11T22:15:29.865803+0000","last_peered":"2024-03-11T22:15:29.865803+0000","last_clean":"2024-03-11T22:15:29.865803+0000","last_became_active":"2024-03-11T22:08:35.085916+0000","last_became_peered":"2024-03-11T22:08:35.085916+0000","last_unstale":"2024-03-11T22:15:29.865803+0000","last_undegraded":"2024-03-11T22:15:29.865803+0000","last_fullsized":"2024-03-11T22:15:29.865803+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'22","last_scrub_stamp":"2024-03-11T22:15:25.618938+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:15:25.618938+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:36.248853+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":56,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.878320+0000","last_change":"2024-03-11T22:15:15.946889+0000","last_active":"2024-03-11T22:15:29.878320+0000","last_peered":"2024-03-11T22:15:29.878320+0000","last_clean":"2024-03-11T22:15:29.878320+0000","last_became_active":"2024-03-11T22:14:52.239442+0000","last_became_peered":"2024-03-11T22:14:52.239442+0000","last_unstale":"2024-03-11T22:15:29.878320+0000","last_undegraded":"2024-03-11T22:15:29.878320+0000","last_fullsized":"2024-03-11T22:15:29.878320+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:15:15.946740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:15.946740+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:39.769968+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":268,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867759+0000","last_change":"2024-03-11T22:15:21.641164+0000","last_active":"2024-03-11T22:15:29.867759+0000","last_peered":"2024-03-11T22:15:29.867759+0000","last_clean":"2024-03-11T22:15:29.867759+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:15:29.867759+0000","last_undegraded":"2024-03-11T22:15:29.867759+0000","last_fullsized":"2024-03-11T22:15:29.867759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:21.641053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:21.641053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:37.378649+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"93'130","reported_seq":551,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867608+0000","last_change":"2024-03-11T22:15:07.482011+0000","last_active":"2024-03-11T22:15:29.867608+0000","last_peered":"2024-03-11T22:15:29.867608+0000","last_clean":"2024-03-11T22:15:29.867608+0000","last_became_active":"2024-03-11T22:14:52.240083+0000","last_became_peered":"2024-03-11T22:14:52.240083+0000","last_unstale":"2024-03-11T22:15:29.867608+0000","last_undegraded":"2024-03-11T22:15:29.867608+0000","last_fullsized":"2024-03-11T22:15:29.867608+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:07.481870+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:07.481870+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:26.692915+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":284,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.877526+0000","last_change":"2024-03-11T22:15:27.448259+0000","last_active":"2024-03-11T22:15:29.877526+0000","last_peered":"2024-03-11T22:15:29.877526+0000","last_clean":"2024-03-11T22:15:29.877526+0000","last_became_active":"2024-03-11T22:08:35.085914+0000","last_became_peered":"2024-03-11T22:08:35.085914+0000","last_unstale":"2024-03-11T22:15:29.877526+0000","last_undegraded":"2024-03-11T22:15:29.877526+0000","last_fullsized":"2024-03-11T22:15:29.877526+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:15:27.448002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:27.448002+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:54.310384+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867709+0000","last_change":"2024-03-11T22:13:28.086496+0000","last_active":"2024-03-11T22:15:29.867709+0000","last_peered":"2024-03-11T22:15:29.867709+0000","last_clean":"2024-03-11T22:15:29.867709+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:15:29.867709+0000","last_undegraded":"2024-03-11T22:15:29.867709+0000","last_fullsized":"2024-03-11T22:15:29.867709+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:28.086387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:13:28.086387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866550+0000","last_change":"2024-03-11T22:13:25.550514+0000","last_active":"2024-03-11T22:15:29.866550+0000","last_peered":"2024-03-11T22:15:29.866550+0000","last_clean":"2024-03-11T22:15:29.866550+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:15:29.866550+0000","last_undegraded":"2024-03-11T22:15:29.866550+0000","last_fullsized":"2024-03-11T22:15:29.866550+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:25.550429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:07.195289+0000","last_clean_scrub_stamp":"2024-03-11T22:13:25.550429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":210,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868987+0000","last_change":"2024-03-11T22:13:37.172811+0000","last_active":"2024-03-11T22:15:29.868987+0000","last_peered":"2024-03-11T22:15:29.868987+0000","last_clean":"2024-03-11T22:15:29.868987+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:15:29.868987+0000","last_undegraded":"2024-03-11T22:15:29.868987+0000","last_fullsized":"2024-03-11T22:15:29.868987+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:13:37.172700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"93'130","reported_seq":561,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866065+0000","last_change":"2024-03-11T22:15:05.687130+0000","last_active":"2024-03-11T22:15:29.866065+0000","last_peered":"2024-03-11T22:15:29.866065+0000","last_clean":"2024-03-11T22:15:29.866065+0000","last_became_active":"2024-03-11T22:14:43.194413+0000","last_became_peered":"2024-03-11T22:14:43.194413+0000","last_unstale":"2024-03-11T22:15:29.866065+0000","last_undegraded":"2024-03-11T22:15:29.866065+0000","last_fullsized":"2024-03-11T22:15:29.866065+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:05.687010+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.687010+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:20.099917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"93'130","reported_seq":560,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.874797+0000","last_change":"2024-03-11T22:15:05.403493+0000","last_active":"2024-03-11T22:15:29.874797+0000","last_peered":"2024-03-11T22:15:29.874797+0000","last_clean":"2024-03-11T22:15:29.874797+0000","last_became_active":"2024-03-11T22:14:43.198216+0000","last_became_peered":"2024-03-11T22:14:43.198216+0000","last_unstale":"2024-03-11T22:15:29.874797+0000","last_undegraded":"2024-03-11T22:15:29.874797+0000","last_fullsized":"2024-03-11T22:15:29.874797+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:05.403372+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.403372+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:06.652225+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"93'13","reported_seq":295,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.878693+0000","last_change":"2024-03-11T22:14:26.127827+0000","last_active":"2024-03-11T22:15:29.878693+0000","last_peered":"2024-03-11T22:15:29.878693+0000","last_clean":"2024-03-11T22:15:29.878693+0000","last_became_active":"2024-03-11T22:08:35.085787+0000","last_became_peered":"2024-03-11T22:08:35.085787+0000","last_unstale":"2024-03-11T22:15:29.878693+0000","last_undegraded":"2024-03-11T22:15:29.878693+0000","last_fullsized":"2024-03-11T22:15:29.878693+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:14:26.127737+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.127737+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:40.826957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":42,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868736+0000","last_change":"2024-03-11T22:14:52.247626+0000","last_active":"2024-03-11T22:15:29.868736+0000","last_peered":"2024-03-11T22:15:29.868736+0000","last_clean":"2024-03-11T22:15:29.868736+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:15:29.868736+0000","last_undegraded":"2024-03-11T22:15:29.868736+0000","last_fullsized":"2024-03-11T22:15:29.868736+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:11.092328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:11.092328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.877444+0000","last_change":"2024-03-11T22:14:14.850354+0000","last_active":"2024-03-11T22:15:29.877444+0000","last_peered":"2024-03-11T22:15:29.877444+0000","last_clean":"2024-03-11T22:15:29.877444+0000","last_became_active":"2024-03-11T22:08:37.069611+0000","last_became_peered":"2024-03-11T22:08:37.069611+0000","last_unstale":"2024-03-11T22:15:29.877444+0000","last_undegraded":"2024-03-11T22:15:29.877444+0000","last_fullsized":"2024-03-11T22:15:29.877444+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.850128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.850128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:39.356529+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":275,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.874038+0000","last_change":"2024-03-11T22:14:19.987755+0000","last_active":"2024-03-11T22:15:29.874038+0000","last_peered":"2024-03-11T22:15:29.874038+0000","last_clean":"2024-03-11T22:15:29.874038+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:15:29.874038+0000","last_undegraded":"2024-03-11T22:15:29.874038+0000","last_fullsized":"2024-03-11T22:15:29.874038+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:19.987672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:19.987672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:44.507245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"188'13","reported_seq":303,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.869285+0000","last_change":"2024-03-11T22:15:29.636731+0000","last_active":"2024-03-11T22:15:29.869285+0000","last_peered":"2024-03-11T22:15:29.869285+0000","last_clean":"2024-03-11T22:15:29.869285+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:15:29.869285+0000","last_undegraded":"2024-03-11T22:15:29.869285+0000","last_fullsized":"2024-03-11T22:15:29.869285+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'13","last_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.636639+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:36.914657+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.876014+0000","last_change":"2024-03-11T22:14:33.012119+0000","last_active":"2024-03-11T22:15:29.876014+0000","last_peered":"2024-03-11T22:15:29.876014+0000","last_clean":"2024-03-11T22:15:29.876014+0000","last_became_active":"2024-03-11T22:08:33.032850+0000","last_became_peered":"2024-03-11T22:08:33.032850+0000","last_unstale":"2024-03-11T22:15:29.876014+0000","last_undegraded":"2024-03-11T22:15:29.876014+0000","last_fullsized":"2024-03-11T22:15:29.876014+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:33.012017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:33.012017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:02.262664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":366,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873847+0000","last_change":"2024-03-11T22:14:26.011406+0000","last_active":"2024-03-11T22:15:29.873847+0000","last_peered":"2024-03-11T22:15:29.873847+0000","last_clean":"2024-03-11T22:15:29.873847+0000","last_became_active":"2024-03-11T22:08:37.071202+0000","last_became_peered":"2024-03-11T22:08:37.071202+0000","last_unstale":"2024-03-11T22:15:29.873847+0000","last_undegraded":"2024-03-11T22:15:29.873847+0000","last_fullsized":"2024-03-11T22:15:29.873847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:14:26.011319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.011319+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:45.370951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":46,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.876625+0000","last_change":"2024-03-11T22:15:12.007160+0000","last_active":"2024-03-11T22:15:29.876625+0000","last_peered":"2024-03-11T22:15:29.876625+0000","last_clean":"2024-03-11T22:15:29.876625+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:15:29.876625+0000","last_undegraded":"2024-03-11T22:15:29.876625+0000","last_fullsized":"2024-03-11T22:15:29.876625+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:12.007023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:12.007023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.489583+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"93'130","reported_seq":554,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866620+0000","last_change":"2024-03-11T22:15:07.646561+0000","last_active":"2024-03-11T22:15:29.866620+0000","last_peered":"2024-03-11T22:15:29.866620+0000","last_clean":"2024-03-11T22:15:29.866620+0000","last_became_active":"2024-03-11T22:14:43.194350+0000","last_became_peered":"2024-03-11T22:14:43.194350+0000","last_unstale":"2024-03-11T22:15:29.866620+0000","last_undegraded":"2024-03-11T22:15:29.866620+0000","last_fullsized":"2024-03-11T22:15:29.866620+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:07.646448+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:15:07.646448+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:30.541086+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":290,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873749+0000","last_change":"2024-03-11T22:15:29.460432+0000","last_active":"2024-03-11T22:15:29.873749+0000","last_peered":"2024-03-11T22:15:29.873749+0000","last_clean":"2024-03-11T22:15:29.873749+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:15:29.873749+0000","last_undegraded":"2024-03-11T22:15:29.873749+0000","last_fullsized":"2024-03-11T22:15:29.873749+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:15:29.460297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.460297+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:45.658272+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":50,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.871371+0000","last_change":"2024-03-11T22:15:07.967880+0000","last_active":"2024-03-11T22:15:29.871371+0000","last_peered":"2024-03-11T22:15:29.871371+0000","last_clean":"2024-03-11T22:15:29.871371+0000","last_became_active":"2024-03-11T22:14:52.242397+0000","last_became_peered":"2024-03-11T22:14:52.242397+0000","last_unstale":"2024-03-11T22:15:29.871371+0000","last_undegraded":"2024-03-11T22:15:29.871371+0000","last_fullsized":"2024-03-11T22:15:29.871371+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:07.967713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:07.967713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:28.435655+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":363,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.869079+0000","last_change":"2024-03-11T22:13:28.520411+0000","last_active":"2024-03-11T22:15:29.869079+0000","last_peered":"2024-03-11T22:15:29.869079+0000","last_clean":"2024-03-11T22:15:29.869079+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:15:29.869079+0000","last_undegraded":"2024-03-11T22:15:29.869079+0000","last_fullsized":"2024-03-11T22:15:29.869079+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:13:28.520283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:13:28.520283+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":259,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.878334+0000","last_change":"2024-03-11T22:14:14.969800+0000","last_active":"2024-03-11T22:15:29.878334+0000","last_peered":"2024-03-11T22:15:29.878334+0000","last_clean":"2024-03-11T22:15:29.878334+0000","last_became_active":"2024-03-11T22:08:39.082938+0000","last_became_peered":"2024-03-11T22:08:39.082938+0000","last_unstale":"2024-03-11T22:15:29.878334+0000","last_undegraded":"2024-03-11T22:15:29.878334+0000","last_fullsized":"2024-03-11T22:15:29.878334+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.969572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.969572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:44.759439+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"93'130","reported_seq":510,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.875377+0000","last_change":"2024-03-11T22:15:06.779041+0000","last_active":"2024-03-11T22:15:29.875377+0000","last_peered":"2024-03-11T22:15:29.875377+0000","last_clean":"2024-03-11T22:15:29.875377+0000","last_became_active":"2024-03-11T22:14:43.191297+0000","last_became_peered":"2024-03-11T22:14:43.191297+0000","last_unstale":"2024-03-11T22:15:29.875377+0000","last_undegraded":"2024-03-11T22:15:29.875377+0000","last_fullsized":"2024-03-11T22:15:29.875377+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.778786+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.778786+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.184868+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":76,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.869501+0000","last_change":"2024-03-11T22:14:52.240268+0000","last_active":"2024-03-11T22:15:29.869501+0000","last_peered":"2024-03-11T22:15:29.869501+0000","last_clean":"2024-03-11T22:15:29.869501+0000","last_became_active":"2024-03-11T22:14:52.239810+0000","last_became_peered":"2024-03-11T22:14:52.239810+0000","last_unstale":"2024-03-11T22:15:29.869501+0000","last_undegraded":"2024-03-11T22:15:29.869501+0000","last_fullsized":"2024-03-11T22:15:29.869501+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:14:32.853664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:32.853664+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:34.208129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":55,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868340+0000","last_change":"2024-03-11T22:15:26.154367+0000","last_active":"2024-03-11T22:15:29.868340+0000","last_peered":"2024-03-11T22:15:29.868340+0000","last_clean":"2024-03-11T22:15:29.868340+0000","last_became_active":"2024-03-11T22:14:52.239817+0000","last_became_peered":"2024-03-11T22:14:52.239817+0000","last_unstale":"2024-03-11T22:15:29.868340+0000","last_undegraded":"2024-03-11T22:15:29.868340+0000","last_fullsized":"2024-03-11T22:15:29.868340+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:15:26.154167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:26.154167+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:26.166133+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.871809+0000","last_change":"2024-03-11T22:14:07.844362+0000","last_active":"2024-03-11T22:15:29.871809+0000","last_peered":"2024-03-11T22:15:29.871809+0000","last_clean":"2024-03-11T22:15:29.871809+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:15:29.871809+0000","last_undegraded":"2024-03-11T22:15:29.871809+0000","last_fullsized":"2024-03-11T22:15:29.871809+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:07.844240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:07.844240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":259,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.871811+0000","last_change":"2024-03-11T22:12:46.492608+0000","last_active":"2024-03-11T22:15:29.871811+0000","last_peered":"2024-03-11T22:15:29.871811+0000","last_clean":"2024-03-11T22:15:29.871811+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:15:29.871811+0000","last_undegraded":"2024-03-11T22:15:29.871811+0000","last_fullsized":"2024-03-11T22:15:29.871811+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.492494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.492494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":299,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872006+0000","last_change":"2024-03-11T22:15:24.432542+0000","last_active":"2024-03-11T22:15:29.872006+0000","last_peered":"2024-03-11T22:15:29.872006+0000","last_clean":"2024-03-11T22:15:29.872006+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:15:29.872006+0000","last_undegraded":"2024-03-11T22:15:29.872006+0000","last_fullsized":"2024-03-11T22:15:29.872006+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:15:24.432264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:24.432264+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:45.938006+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":211,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872120+0000","last_change":"2024-03-11T22:14:21.857065+0000","last_active":"2024-03-11T22:15:29.872120+0000","last_peered":"2024-03-11T22:15:29.872120+0000","last_clean":"2024-03-11T22:15:29.872120+0000","last_became_active":"2024-03-11T22:10:35.414121+0000","last_became_peered":"2024-03-11T22:10:35.414121+0000","last_unstale":"2024-03-11T22:15:29.872120+0000","last_undegraded":"2024-03-11T22:15:29.872120+0000","last_fullsized":"2024-03-11T22:15:29.872120+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:21.856963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.856963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873675+0000","last_change":"2024-03-11T22:14:15.858384+0000","last_active":"2024-03-11T22:15:29.873675+0000","last_peered":"2024-03-11T22:15:29.873675+0000","last_clean":"2024-03-11T22:15:29.873675+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:15:29.873675+0000","last_undegraded":"2024-03-11T22:15:29.873675+0000","last_fullsized":"2024-03-11T22:15:29.873675+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:15.857082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:14:15.857082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":267,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.875067+0000","last_change":"2024-03-11T22:15:10.896807+0000","last_active":"2024-03-11T22:15:29.875067+0000","last_peered":"2024-03-11T22:15:29.875067+0000","last_clean":"2024-03-11T22:15:29.875067+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:15:29.875067+0000","last_undegraded":"2024-03-11T22:15:29.875067+0000","last_fullsized":"2024-03-11T22:15:29.875067+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:10.896676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:10.896676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:16.899428+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"93'130","reported_seq":555,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873515+0000","last_change":"2024-03-11T22:15:06.951256+0000","last_active":"2024-03-11T22:15:29.873515+0000","last_peered":"2024-03-11T22:15:29.873515+0000","last_clean":"2024-03-11T22:15:29.873515+0000","last_became_active":"2024-03-11T22:14:43.194955+0000","last_became_peered":"2024-03-11T22:14:43.194955+0000","last_unstale":"2024-03-11T22:15:29.873515+0000","last_undegraded":"2024-03-11T22:15:29.873515+0000","last_fullsized":"2024-03-11T22:15:29.873515+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.951073+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.951073+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.482728+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":286,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872533+0000","last_change":"2024-03-11T22:13:18.486183+0000","last_active":"2024-03-11T22:15:29.872533+0000","last_peered":"2024-03-11T22:15:29.872533+0000","last_clean":"2024-03-11T22:15:29.872533+0000","last_became_active":"2024-03-11T22:08:35.085928+0000","last_became_peered":"2024-03-11T22:08:35.085928+0000","last_unstale":"2024-03-11T22:15:29.872533+0000","last_undegraded":"2024-03-11T22:15:29.872533+0000","last_fullsized":"2024-03-11T22:15:29.872533+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:13:18.486039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:18.486039+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866469+0000","last_change":"2024-03-11T22:13:29.550985+0000","last_active":"2024-03-11T22:15:29.866469+0000","last_peered":"2024-03-11T22:15:29.866469+0000","last_clean":"2024-03-11T22:15:29.866469+0000","last_became_active":"2024-03-11T22:08:33.039349+0000","last_became_peered":"2024-03-11T22:08:33.039349+0000","last_unstale":"2024-03-11T22:15:29.866469+0000","last_undegraded":"2024-03-11T22:15:29.866469+0000","last_fullsized":"2024-03-11T22:15:29.866469+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:29.550889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:29.550889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":49,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867525+0000","last_change":"2024-03-11T22:15:17.246483+0000","last_active":"2024-03-11T22:15:29.867525+0000","last_peered":"2024-03-11T22:15:29.867525+0000","last_clean":"2024-03-11T22:15:29.867525+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:15:29.867525+0000","last_undegraded":"2024-03-11T22:15:29.867525+0000","last_fullsized":"2024-03-11T22:15:29.867525+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:17.246407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:15:17.246407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:22.492312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":267,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873951+0000","last_change":"2024-03-11T22:14:23.930193+0000","last_active":"2024-03-11T22:15:29.873951+0000","last_peered":"2024-03-11T22:15:29.873951+0000","last_clean":"2024-03-11T22:15:29.873951+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:15:29.873951+0000","last_undegraded":"2024-03-11T22:15:29.873951+0000","last_fullsized":"2024-03-11T22:15:29.873951+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.930091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.930091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:52.770218+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":293,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873071+0000","last_change":"2024-03-11T22:13:31.491151+0000","last_active":"2024-03-11T22:15:29.873071+0000","last_peered":"2024-03-11T22:15:29.873071+0000","last_clean":"2024-03-11T22:15:29.873071+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:15:29.873071+0000","last_undegraded":"2024-03-11T22:15:29.873071+0000","last_fullsized":"2024-03-11T22:15:29.873071+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-11T22:13:31.490878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:31.490878+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.878414+0000","last_change":"2024-03-11T22:15:14.878681+0000","last_active":"2024-03-11T22:15:29.878414+0000","last_peered":"2024-03-11T22:15:29.878414+0000","last_clean":"2024-03-11T22:15:29.878414+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:15:29.878414+0000","last_undegraded":"2024-03-11T22:15:29.878414+0000","last_fullsized":"2024-03-11T22:15:29.878414+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.878563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.878563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:43.850059+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":83,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.869986+0000","last_change":"2024-03-11T22:14:25.982151+0000","last_active":"2024-03-11T22:15:29.869986+0000","last_peered":"2024-03-11T22:15:29.869986+0000","last_clean":"2024-03-11T22:15:29.869986+0000","last_became_active":"2024-03-11T22:14:08.442596+0000","last_became_peered":"2024-03-11T22:14:08.442596+0000","last_unstale":"2024-03-11T22:15:29.869986+0000","last_undegraded":"2024-03-11T22:15:29.869986+0000","last_fullsized":"2024-03-11T22:15:29.869986+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:25.982072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:25.982072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:40.493713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":218,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.871192+0000","last_change":"2024-03-11T22:15:20.894393+0000","last_active":"2024-03-11T22:15:29.871192+0000","last_peered":"2024-03-11T22:15:29.871192+0000","last_clean":"2024-03-11T22:15:29.871192+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:15:29.871192+0000","last_undegraded":"2024-03-11T22:15:29.871192+0000","last_fullsized":"2024-03-11T22:15:29.871192+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:20.894271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:20.894271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:29.495255+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":303,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.869128+0000","last_change":"2024-03-11T22:15:29.232861+0000","last_active":"2024-03-11T22:15:29.869128+0000","last_peered":"2024-03-11T22:15:29.869128+0000","last_clean":"2024-03-11T22:15:29.869128+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:15:29.869128+0000","last_undegraded":"2024-03-11T22:15:29.869128+0000","last_fullsized":"2024-03-11T22:15:29.869128+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:15:29.232725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.232725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:53.191290+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872076+0000","last_change":"2024-03-11T22:14:07.822714+0000","last_active":"2024-03-11T22:15:29.872076+0000","last_peered":"2024-03-11T22:15:29.872076+0000","last_clean":"2024-03-11T22:15:29.872076+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:15:29.872076+0000","last_undegraded":"2024-03-11T22:15:29.872076+0000","last_fullsized":"2024-03-11T22:15:29.872076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:07.822036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:07.822036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":272,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.878839+0000","last_change":"2024-03-11T22:14:23.034943+0000","last_active":"2024-03-11T22:15:29.878839+0000","last_peered":"2024-03-11T22:15:29.878839+0000","last_clean":"2024-03-11T22:15:29.878839+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:15:29.878839+0000","last_undegraded":"2024-03-11T22:15:29.878839+0000","last_fullsized":"2024-03-11T22:15:29.878839+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.034850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.034850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:37.593852+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":275,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868145+0000","last_change":"2024-03-11T22:15:22.441543+0000","last_active":"2024-03-11T22:15:29.868145+0000","last_peered":"2024-03-11T22:15:29.868145+0000","last_clean":"2024-03-11T22:15:29.868145+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:15:29.868145+0000","last_undegraded":"2024-03-11T22:15:29.868145+0000","last_fullsized":"2024-03-11T22:15:29.868145+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:22.441417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:22.441417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:28.609256+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":291,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.878607+0000","last_change":"2024-03-11T22:15:19.959754+0000","last_active":"2024-03-11T22:15:29.878607+0000","last_peered":"2024-03-11T22:15:29.878607+0000","last_clean":"2024-03-11T22:15:29.878607+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:15:29.878607+0000","last_undegraded":"2024-03-11T22:15:29.878607+0000","last_fullsized":"2024-03-11T22:15:29.878607+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:15:19.958690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:19.958690+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:46.912976+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872787+0000","last_change":"2024-03-11T22:14:23.995912+0000","last_active":"2024-03-11T22:15:29.872787+0000","last_peered":"2024-03-11T22:15:29.872787+0000","last_clean":"2024-03-11T22:15:29.872787+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:15:29.872787+0000","last_undegraded":"2024-03-11T22:15:29.872787+0000","last_fullsized":"2024-03-11T22:15:29.872787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:14:23.995781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.995781+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:33.646778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":375,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868799+0000","last_change":"2024-03-11T22:14:21.037744+0000","last_active":"2024-03-11T22:15:29.868799+0000","last_peered":"2024-03-11T22:15:29.868799+0000","last_clean":"2024-03-11T22:15:29.868799+0000","last_became_active":"2024-03-11T22:08:37.078588+0000","last_became_peered":"2024-03-11T22:08:37.078588+0000","last_unstale":"2024-03-11T22:15:29.868799+0000","last_undegraded":"2024-03-11T22:15:29.868799+0000","last_fullsized":"2024-03-11T22:15:29.868799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:14:21.037625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.037625+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:38.139047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":220,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866840+0000","last_change":"2024-03-11T22:13:21.532369+0000","last_active":"2024-03-11T22:15:29.866840+0000","last_peered":"2024-03-11T22:15:29.866840+0000","last_clean":"2024-03-11T22:15:29.866840+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:15:29.866840+0000","last_undegraded":"2024-03-11T22:15:29.866840+0000","last_fullsized":"2024-03-11T22:15:29.866840+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:13:21.532280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:21.532280+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":298,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.871854+0000","last_change":"2024-03-11T22:14:27.998984+0000","last_active":"2024-03-11T22:15:29.871854+0000","last_peered":"2024-03-11T22:15:29.871854+0000","last_clean":"2024-03-11T22:15:29.871854+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:15:29.871854+0000","last_undegraded":"2024-03-11T22:15:29.871854+0000","last_fullsized":"2024-03-11T22:15:29.871854+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:14:27.998819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:27.998819+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:50.184055+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":47,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868659+0000","last_change":"2024-03-11T22:15:23.422984+0000","last_active":"2024-03-11T22:15:29.868659+0000","last_peered":"2024-03-11T22:15:29.868659+0000","last_clean":"2024-03-11T22:15:29.868659+0000","last_became_active":"2024-03-11T22:14:52.245199+0000","last_became_peered":"2024-03-11T22:14:52.245199+0000","last_unstale":"2024-03-11T22:15:29.868659+0000","last_undegraded":"2024-03-11T22:15:29.868659+0000","last_fullsized":"2024-03-11T22:15:29.868659+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:23.418102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:23.418102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:41.174600+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873994+0000","last_change":"2024-03-11T22:14:52.233517+0000","last_active":"2024-03-11T22:15:29.873994+0000","last_peered":"2024-03-11T22:15:29.873994+0000","last_clean":"2024-03-11T22:15:29.873994+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:15:29.873994+0000","last_undegraded":"2024-03-11T22:15:29.873994+0000","last_fullsized":"2024-03-11T22:15:29.873994+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.849679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.849679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:46.294927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":260,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.877114+0000","last_change":"2024-03-11T22:12:44.464428+0000","last_active":"2024-03-11T22:15:29.877114+0000","last_peered":"2024-03-11T22:15:29.877114+0000","last_clean":"2024-03-11T22:15:29.877114+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:15:29.877114+0000","last_undegraded":"2024-03-11T22:15:29.877114+0000","last_fullsized":"2024-03-11T22:15:29.877114+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.464308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.464308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":316,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873679+0000","last_change":"2024-03-11T22:15:22.863847+0000","last_active":"2024-03-11T22:15:29.873679+0000","last_peered":"2024-03-11T22:15:29.873679+0000","last_clean":"2024-03-11T22:15:29.873679+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:15:29.873679+0000","last_undegraded":"2024-03-11T22:15:29.873679+0000","last_fullsized":"2024-03-11T22:15:29.873679+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:15:22.863711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:22.863711+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:48.485553+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.876172+0000","last_change":"2024-03-11T22:14:32.050907+0000","last_active":"2024-03-11T22:15:29.876172+0000","last_peered":"2024-03-11T22:15:29.876172+0000","last_clean":"2024-03-11T22:15:29.876172+0000","last_became_active":"2024-03-11T22:08:33.035624+0000","last_became_peered":"2024-03-11T22:08:33.035624+0000","last_unstale":"2024-03-11T22:15:29.876172+0000","last_undegraded":"2024-03-11T22:15:29.876172+0000","last_fullsized":"2024-03-11T22:15:29.876172+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:32.050761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:32.050761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:45.298909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867991+0000","last_change":"2024-03-11T22:14:26.935697+0000","last_active":"2024-03-11T22:15:29.867991+0000","last_peered":"2024-03-11T22:15:29.867991+0000","last_clean":"2024-03-11T22:15:29.867991+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:15:29.867991+0000","last_undegraded":"2024-03-11T22:15:29.867991+0000","last_fullsized":"2024-03-11T22:15:29.867991+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:26.935615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.935615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:39.182791+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":46,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.869145+0000","last_change":"2024-03-11T22:15:28.217629+0000","last_active":"2024-03-11T22:15:29.869145+0000","last_peered":"2024-03-11T22:15:29.869145+0000","last_clean":"2024-03-11T22:15:29.869145+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:15:29.869145+0000","last_undegraded":"2024-03-11T22:15:29.869145+0000","last_fullsized":"2024-03-11T22:15:29.869145+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:28.217499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:28.217499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:54.886059+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"95'7","reported_seq":112,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873506+0000","last_change":"2024-03-11T22:14:52.242050+0000","last_active":"2024-03-11T22:15:29.873506+0000","last_peered":"2024-03-11T22:15:29.873506+0000","last_clean":"2024-03-11T22:15:29.873506+0000","last_became_active":"2024-03-11T22:14:52.238411+0000","last_became_peered":"2024-03-11T22:14:52.238411+0000","last_unstale":"2024-03-11T22:15:29.873506+0000","last_undegraded":"2024-03-11T22:15:29.873506+0000","last_fullsized":"2024-03-11T22:15:29.873506+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'7","last_scrub_stamp":"2024-03-11T22:13:32.041219+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:13:32.041219+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867155+0000","last_change":"2024-03-11T22:14:30.716953+0000","last_active":"2024-03-11T22:15:29.867155+0000","last_peered":"2024-03-11T22:15:29.867155+0000","last_clean":"2024-03-11T22:15:29.867155+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:15:29.867155+0000","last_undegraded":"2024-03-11T22:15:29.867155+0000","last_fullsized":"2024-03-11T22:15:29.867155+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:14:30.716844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:39.360065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":49,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.868529+0000","last_change":"2024-03-11T22:15:28.434583+0000","last_active":"2024-03-11T22:15:29.868529+0000","last_peered":"2024-03-11T22:15:29.868529+0000","last_clean":"2024-03-11T22:15:29.868529+0000","last_became_active":"2024-03-11T22:14:52.239391+0000","last_became_peered":"2024-03-11T22:14:52.239391+0000","last_unstale":"2024-03-11T22:15:29.868529+0000","last_undegraded":"2024-03-11T22:15:29.868529+0000","last_fullsized":"2024-03-11T22:15:29.868529+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_clean_scrub_stamp":"2024-03-11T22:15:28.434334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:45.678839+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":267,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866178+0000","last_change":"2024-03-11T22:14:22.631560+0000","last_active":"2024-03-11T22:15:29.866178+0000","last_peered":"2024-03-11T22:15:29.866178+0000","last_clean":"2024-03-11T22:15:29.866178+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:15:29.866178+0000","last_undegraded":"2024-03-11T22:15:29.866178+0000","last_fullsized":"2024-03-11T22:15:29.866178+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:22.631456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:22.631456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:48.639261+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":81,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872992+0000","last_change":"2024-03-11T22:15:18.401675+0000","last_active":"2024-03-11T22:15:29.872992+0000","last_peered":"2024-03-11T22:15:29.872992+0000","last_clean":"2024-03-11T22:15:29.872992+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:15:29.872992+0000","last_undegraded":"2024-03-11T22:15:29.872992+0000","last_fullsized":"2024-03-11T22:15:29.872992+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:15:18.401549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:18.401549+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.261566+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":287,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867389+0000","last_change":"2024-03-11T22:15:15.590670+0000","last_active":"2024-03-11T22:15:29.867389+0000","last_peered":"2024-03-11T22:15:29.867389+0000","last_clean":"2024-03-11T22:15:29.867389+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:15:29.867389+0000","last_undegraded":"2024-03-11T22:15:29.867389+0000","last_fullsized":"2024-03-11T22:15:29.867389+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:15.590551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:15.590551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:42.253111+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866156+0000","last_change":"2024-03-11T22:14:28.675081+0000","last_active":"2024-03-11T22:15:29.866156+0000","last_peered":"2024-03-11T22:15:29.866156+0000","last_clean":"2024-03-11T22:15:29.866156+0000","last_became_active":"2024-03-11T22:08:37.081996+0000","last_became_peered":"2024-03-11T22:08:37.081996+0000","last_unstale":"2024-03-11T22:15:29.866156+0000","last_undegraded":"2024-03-11T22:15:29.866156+0000","last_fullsized":"2024-03-11T22:15:29.866156+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:28.674962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:14:28.674962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:55.434068+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":267,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.875454+0000","last_change":"2024-03-11T22:15:20.947535+0000","last_active":"2024-03-11T22:15:29.875454+0000","last_peered":"2024-03-11T22:15:29.875454+0000","last_clean":"2024-03-11T22:15:29.875454+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:15:29.875454+0000","last_undegraded":"2024-03-11T22:15:29.875454+0000","last_fullsized":"2024-03-11T22:15:29.875454+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:20.947405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:20.947405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:48.995693+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":292,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872637+0000","last_change":"2024-03-11T22:12:49.510144+0000","last_active":"2024-03-11T22:15:29.872637+0000","last_peered":"2024-03-11T22:15:29.872637+0000","last_clean":"2024-03-11T22:15:29.872637+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:15:29.872637+0000","last_undegraded":"2024-03-11T22:15:29.872637+0000","last_fullsized":"2024-03-11T22:15:29.872637+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:12:49.509860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.509860+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":286,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872145+0000","last_change":"2024-03-11T22:13:19.463087+0000","last_active":"2024-03-11T22:15:29.872145+0000","last_peered":"2024-03-11T22:15:29.872145+0000","last_clean":"2024-03-11T22:15:29.872145+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:15:29.872145+0000","last_undegraded":"2024-03-11T22:15:29.872145+0000","last_fullsized":"2024-03-11T22:15:29.872145+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:13:19.462955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.462955+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":46,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867723+0000","last_change":"2024-03-11T22:15:16.288357+0000","last_active":"2024-03-11T22:15:29.867723+0000","last_peered":"2024-03-11T22:15:29.867723+0000","last_clean":"2024-03-11T22:15:29.867723+0000","last_became_active":"2024-03-11T22:14:52.247689+0000","last_became_peered":"2024-03-11T22:14:52.247689+0000","last_unstale":"2024-03-11T22:15:29.867723+0000","last_undegraded":"2024-03-11T22:15:29.867723+0000","last_fullsized":"2024-03-11T22:15:29.867723+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:16.288232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:16.288232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:28.360920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":259,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.873759+0000","last_change":"2024-03-11T22:12:50.533192+0000","last_active":"2024-03-11T22:15:29.873759+0000","last_peered":"2024-03-11T22:15:29.873759+0000","last_clean":"2024-03-11T22:15:29.873759+0000","last_became_active":"2024-03-11T22:08:39.079045+0000","last_became_peered":"2024-03-11T22:08:39.079045+0000","last_unstale":"2024-03-11T22:15:29.873759+0000","last_undegraded":"2024-03-11T22:15:29.873759+0000","last_fullsized":"2024-03-11T22:15:29.873759+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.533067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.533067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":73,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.867831+0000","last_change":"2024-03-11T22:14:52.248071+0000","last_active":"2024-03-11T22:15:29.867831+0000","last_peered":"2024-03-11T22:15:29.867831+0000","last_clean":"2024-03-11T22:15:29.867831+0000","last_became_active":"2024-03-11T22:14:52.247411+0000","last_became_peered":"2024-03-11T22:14:52.247411+0000","last_unstale":"2024-03-11T22:15:29.867831+0000","last_undegraded":"2024-03-11T22:15:29.867831+0000","last_fullsized":"2024-03-11T22:15:29.867831+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:14:15.059599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:15.059599+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:37.879905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":280,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.877830+0000","last_change":"2024-03-11T22:14:28.062521+0000","last_active":"2024-03-11T22:15:29.877830+0000","last_peered":"2024-03-11T22:15:29.877830+0000","last_clean":"2024-03-11T22:15:29.877830+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:15:29.877830+0000","last_undegraded":"2024-03-11T22:15:29.877830+0000","last_fullsized":"2024-03-11T22:15:29.877830+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:28.062439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:28.062439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:46.792592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.872692+0000","last_change":"2024-03-11T22:15:09.519052+0000","last_active":"2024-03-11T22:15:29.872692+0000","last_peered":"2024-03-11T22:15:29.872692+0000","last_clean":"2024-03-11T22:15:29.872692+0000","last_became_active":"2024-03-11T22:08:37.073775+0000","last_became_peered":"2024-03-11T22:08:37.073775+0000","last_unstale":"2024-03-11T22:15:29.872692+0000","last_undegraded":"2024-03-11T22:15:29.872692+0000","last_fullsized":"2024-03-11T22:15:29.872692+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:09.518937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:09.518937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:10.825295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":46,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-11T22:15:29.866287+0000","last_change":"2024-03-11T22:15:12.647958+0000","last_active":"2024-03-11T22:15:29.866287+0000","last_peered":"2024-03-11T22:15:29.866287+0000","last_clean":"2024-03-11T22:15:29.866287+0000","last_became_active":"2024-03-11T22:14:52.247273+0000","last_became_peered":"2024-03-11T22:14:52.247273+0000","last_unstale":"2024-03-11T22:15:29.866287+0000","last_undegraded":"2024-03-11T22:15:29.866287+0000","last_fullsized":"2024-03-11T22:15:29.866287+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:12.647884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:12.647884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:27.129974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3636396032,"num_objects":55487,"num_object_clones":0,"num_object_copies":166461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55487,"num_write_kb":3551168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10894245888,"data_stored":10894245888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55487,"ondisk_log_size":55487,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":384,"num_read_kb":379,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2730,"ondisk_log_size":2730,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":688,"num_read_kb":445,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":410,"ondisk_log_size":410,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005144,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1228856,"kb_used_data":1085232,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92516296,"statfs":{"total":95995035648,"available":94736687104,"internally_reserved":0,"allocated":1111277568,"data_stored":1110905599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":147061195},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332189,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1211368,"kb_used_data":1067740,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92533784,"statfs":{"total":95995035648,"available":94754594816,"internally_reserved":0,"allocated":1093365760,"data_stored":1093014081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1703464,"kb_used_data":1545944,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92041688,"statfs":{"total":95995035648,"available":94250688512,"internally_reserved":0,"allocated":1583046656,"data_stored":1582641921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,6,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738493,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1567944,"kb_used_data":1320184,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92177208,"statfs":{"total":95995035648,"available":94389460992,"internally_reserved":0,"allocated":1351868416,"data_stored":1351495775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411442,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1510184,"kb_used_data":1330460,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92234968,"statfs":{"total":95995035648,"available":94448607232,"internally_reserved":0,"allocated":1362391040,"data_stored":1362039731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1566304,"kb_used_data":1328272,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92178848,"statfs":{"total":95995035648,"available":94391140352,"internally_reserved":0,"allocated":1360150528,"data_stored":1359754839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":243726791},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920872,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1220492,"kb_used_data":1081024,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92524660,"statfs":{"total":95995035648,"available":94745251840,"internally_reserved":0,"allocated":1106968576,"data_stored":1106587952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142801350},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659233,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1880172,"kb_used_data":1732388,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":91864980,"statfs":{"total":95995035648,"available":94069739520,"internally_reserved":0,"allocated":1773965312,"data_stored":1773605258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1376649216,"data_stored":1376649216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1585971200,"data_stored":1585971200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1393229824,"data_stored":1393229824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1362427904,"data_stored":1362427904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1114570752,"data_stored":1114570752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1800470528,"data_stored":1800470528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1120075776,"data_stored":1120075776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1140850688,"data_stored":1140850688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:15:35.438 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:35.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:35.465 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:35.534 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:35.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:35.549 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:35.606 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:35.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:35.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:35.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:35.754 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:35.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:35.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:35.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:35.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:35.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:36.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:36.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:36.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:36.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:36.146 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:36.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:36 smithi045 ceph-mon[39715]: 3.10 scrub starts 2024-03-11T22:15:36.157 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:36 smithi045 ceph-mon[39715]: 3.10 scrub ok 2024-03-11T22:15:36.157 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:36 smithi045 ceph-mon[30178]: 3.10 scrub starts 2024-03-11T22:15:36.157 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:36 smithi045 ceph-mon[30178]: 3.10 scrub ok 2024-03-11T22:15:36.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:36.181 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:36.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:36.282 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:36.318 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 34 16 63724 63708 117.081 86.5 0.0139021 0.00853611 2024-03-11T22:15:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:36.319 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:36.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:36.343 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:36 smithi144 ceph-mon[33984]: 3.10 scrub starts 2024-03-11T22:15:36.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:36 smithi144 ceph-mon[33984]: 3.10 scrub ok 2024-03-11T22:15:36.344 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:15:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:15:36.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:36.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:36.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:36.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:36.610 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:36.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:36.711 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:36.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:36.744 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:36.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:36.845 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:36.879 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:36.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:36.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:36.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:37.024 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:37.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:37.125 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:37.164 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:37.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:37.265 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:37.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:37.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:37 smithi045 ceph-mon[39715]: pgmap v298: 168 pgs: 168 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-03-11T22:15:37.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:37 smithi045 ceph-mon[39715]: 4.1 scrub starts 2024-03-11T22:15:37.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:37 smithi045 ceph-mon[39715]: 4.1 scrub ok 2024-03-11T22:15:37.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:37.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:37 smithi045 ceph-mon[30178]: pgmap v298: 168 pgs: 168 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-03-11T22:15:37.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:37 smithi045 ceph-mon[30178]: 4.1 scrub starts 2024-03-11T22:15:37.280 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:37 smithi045 ceph-mon[30178]: 4.1 scrub ok 2024-03-11T22:15:37.301 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:37.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:37.318 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 35 16 65199 65183 116.369 92.1875 0.019242 0.00858825 2024-03-11T22:15:37.403 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:37.445 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:37.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:37.547 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:37.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:37.581 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:37.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:37.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:37.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:37 smithi144 ceph-mon[33984]: pgmap v298: 168 pgs: 168 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-03-11T22:15:37.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:37 smithi144 ceph-mon[33984]: 4.1 scrub starts 2024-03-11T22:15:37.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:37 smithi144 ceph-mon[33984]: 4.1 scrub ok 2024-03-11T22:15:37.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:37.729 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:37.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:37.830 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:37.885 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:37.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:37.986 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:38.020 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:38.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:38.122 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:38.156 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:38.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:38.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:38.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:38.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:38.318 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 36 16 66638 66622 115.635 89.9375 0.00622756 0.00864201 2024-03-11T22:15:38.398 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:38.434 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:38.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:38 smithi045 ceph-mon[30178]: 3.19 scrub starts 2024-03-11T22:15:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:38 smithi045 ceph-mon[30178]: 3.19 scrub ok 2024-03-11T22:15:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:38 smithi045 ceph-mon[30178]: 6.8 scrub starts 2024-03-11T22:15:38.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:38 smithi045 ceph-mon[30178]: 6.8 scrub ok 2024-03-11T22:15:38.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:38 smithi045 ceph-mon[39715]: 3.19 scrub starts 2024-03-11T22:15:38.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:38 smithi045 ceph-mon[39715]: 3.19 scrub ok 2024-03-11T22:15:38.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:38 smithi045 ceph-mon[39715]: 6.8 scrub starts 2024-03-11T22:15:38.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:38 smithi045 ceph-mon[39715]: 6.8 scrub ok 2024-03-11T22:15:38.535 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:38.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:38.570 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:38.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:38.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:38 smithi144 ceph-mon[33984]: 3.19 scrub starts 2024-03-11T22:15:38.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:38 smithi144 ceph-mon[33984]: 3.19 scrub ok 2024-03-11T22:15:38.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:38 smithi144 ceph-mon[33984]: 6.8 scrub starts 2024-03-11T22:15:38.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:38 smithi144 ceph-mon[33984]: 6.8 scrub ok 2024-03-11T22:15:38.671 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:38.700 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:38.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:38.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:38.847 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:38.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:38.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:38.991 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:38.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:39.093 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:39.130 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:39.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:39.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:39.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:39.279 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:39.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:39.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:39.318 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 37 16 68018 68002 114.841 86.25 0.00798123 0.00870162 2024-03-11T22:15:39.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:39.424 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:39.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:39.481 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:39 smithi045 ceph-mon[30178]: pgmap v299: 168 pgs: 168 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-03-11T22:15:39.481 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:39 smithi045 ceph-mon[30178]: 2.0 scrub starts 2024-03-11T22:15:39.481 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:39 smithi045 ceph-mon[30178]: 2.0 scrub ok 2024-03-11T22:15:39.482 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:39 smithi045 ceph-mon[39715]: pgmap v299: 168 pgs: 168 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-03-11T22:15:39.482 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:39 smithi045 ceph-mon[39715]: 2.0 scrub starts 2024-03-11T22:15:39.482 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:39 smithi045 ceph-mon[39715]: 2.0 scrub ok 2024-03-11T22:15:39.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:39.562 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:39.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:39.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:39 smithi144 ceph-mon[33984]: pgmap v299: 168 pgs: 168 active+clean; 3.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-03-11T22:15:39.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:39 smithi144 ceph-mon[33984]: 2.0 scrub starts 2024-03-11T22:15:39.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:39 smithi144 ceph-mon[33984]: 2.0 scrub ok 2024-03-11T22:15:39.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:39.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:39.706 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:39.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:39.803 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:15:39.803 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:15:39.803 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-03-11T22:15:39.803 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-03-11T22:15:39.803 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:15:39.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:39.849 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:39.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:39.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:39.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:39.976 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:39.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:40.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:40.117 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:40.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:40.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:40.246 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:40.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:40.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:40.318 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 38 15 69065 69050 113.542 65.5 0.0208361 0.00880197 2024-03-11T22:15:40.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:40.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:40.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:40.424 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:15:40.424 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:15:40.424 INFO:tasks.thrashosds.thrasher:Reweighting osd 7 to 0.3585811336606596 2024-03-11T22:15:40.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 7 0.3585811336606596 2024-03-11T22:15:40.488 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:40.521 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:40.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:40.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:40.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:40 smithi045 ceph-mon[30178]: 6.15 scrub starts 2024-03-11T22:15:40.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:40 smithi045 ceph-mon[30178]: 6.15 scrub ok 2024-03-11T22:15:40.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:40 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3953662923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:15:40.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:40 smithi045 ceph-mon[30178]: 5.e scrub starts 2024-03-11T22:15:40.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:40 smithi045 ceph-mon[30178]: 5.e scrub ok 2024-03-11T22:15:40.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:40 smithi045 ceph-mon[39715]: 6.15 scrub starts 2024-03-11T22:15:40.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:40 smithi045 ceph-mon[39715]: 6.15 scrub ok 2024-03-11T22:15:40.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:40 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3953662923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:15:40.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:40 smithi045 ceph-mon[39715]: 5.e scrub starts 2024-03-11T22:15:40.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:40 smithi045 ceph-mon[39715]: 5.e scrub ok 2024-03-11T22:15:40.593 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:40 smithi144 ceph-mon[33984]: 6.15 scrub starts 2024-03-11T22:15:40.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:40 smithi144 ceph-mon[33984]: 6.15 scrub ok 2024-03-11T22:15:40.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:40 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3953662923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:15:40.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:40 smithi144 ceph-mon[33984]: 5.e scrub starts 2024-03-11T22:15:40.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:40 smithi144 ceph-mon[33984]: 5.e scrub ok 2024-03-11T22:15:40.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:40.668 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:40.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:40.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:40.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:40.798 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:40.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:40.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:40.924 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:40.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:41.025 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:41.060 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:41.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:41.161 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:41.192 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:41.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:41.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:41.318 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 39 16 70044 70028 112.198 61.125 0.00922576 0.00890689 2024-03-11T22:15:41.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:41.332 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:41.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:41.434 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:41.466 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:41.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:41.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:41 smithi045 ceph-mon[30178]: pgmap v300: 168 pgs: 168 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-11T22:15:41.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:41 smithi045 ceph-mon[30178]: 6.e scrub starts 2024-03-11T22:15:41.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:41 smithi045 ceph-mon[30178]: 6.e scrub ok 2024-03-11T22:15:41.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:41 smithi045 ceph-mon[39715]: pgmap v300: 168 pgs: 168 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-11T22:15:41.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:41 smithi045 ceph-mon[39715]: 6.e scrub starts 2024-03-11T22:15:41.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:41 smithi045 ceph-mon[39715]: 6.e scrub ok 2024-03-11T22:15:41.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:41.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:41 smithi144 ceph-mon[33984]: pgmap v300: 168 pgs: 168 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-11T22:15:41.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:41 smithi144 ceph-mon[33984]: 6.e scrub starts 2024-03-11T22:15:41.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:41 smithi144 ceph-mon[33984]: 6.e scrub ok 2024-03-11T22:15:41.605 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:41.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:41.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:41.749 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:41.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:41.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:41.892 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:41.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:41.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:41.994 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:42.026 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:42.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:42.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:42.128 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:42.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:42.165 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:42.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:42.266 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:42.301 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:42.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:42.318 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:15:42.320184+0000 min lat: 0.00125214 max lat: 0.224138 avg lat: 0.00898188 2024-03-11T22:15:42.318 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:15:42.319 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 40 16 71247 71231 111.273 75.1875 0.0187327 0.00898188 2024-03-11T22:15:42.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:42.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:42.431 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:42.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:42.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[39715]: 6.b scrub starts 2024-03-11T22:15:42.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[39715]: 6.b scrub ok 2024-03-11T22:15:42.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:15:42.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:15:42.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[39715]: osdmap e191: 8 total, 8 up, 8 in 2024-03-11T22:15:42.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[39715]: 5.1d scrub starts 2024-03-11T22:15:42.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[39715]: 5.1d scrub ok 2024-03-11T22:15:42.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[39715]: 5.16 scrub starts 2024-03-11T22:15:42.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[39715]: 5.16 scrub ok 2024-03-11T22:15:42.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[39715]: 3.9 scrub starts 2024-03-11T22:15:42.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[39715]: 3.9 scrub ok 2024-03-11T22:15:42.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[30178]: 6.b scrub starts 2024-03-11T22:15:42.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[30178]: 6.b scrub ok 2024-03-11T22:15:42.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:15:42.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:15:42.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[30178]: osdmap e191: 8 total, 8 up, 8 in 2024-03-11T22:15:42.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[30178]: 5.1d scrub starts 2024-03-11T22:15:42.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[30178]: 5.1d scrub ok 2024-03-11T22:15:42.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[30178]: 5.16 scrub starts 2024-03-11T22:15:42.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[30178]: 5.16 scrub ok 2024-03-11T22:15:42.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[30178]: 3.9 scrub starts 2024-03-11T22:15:42.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-mon[30178]: 3.9 scrub ok 2024-03-11T22:15:42.532 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:42.562 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:42.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:42.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:42 smithi144 ceph-mon[33984]: 6.b scrub starts 2024-03-11T22:15:42.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:42 smithi144 ceph-mon[33984]: 6.b scrub ok 2024-03-11T22:15:42.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:42 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:15:42.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:42 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:15:42.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:42 smithi144 ceph-mon[33984]: osdmap e191: 8 total, 8 up, 8 in 2024-03-11T22:15:42.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:42 smithi144 ceph-mon[33984]: 5.1d scrub starts 2024-03-11T22:15:42.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:42 smithi144 ceph-mon[33984]: 5.1d scrub ok 2024-03-11T22:15:42.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:42 smithi144 ceph-mon[33984]: 5.16 scrub starts 2024-03-11T22:15:42.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:42 smithi144 ceph-mon[33984]: 5.16 scrub ok 2024-03-11T22:15:42.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:42 smithi144 ceph-mon[33984]: 3.9 scrub starts 2024-03-11T22:15:42.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:42 smithi144 ceph-mon[33984]: 3.9 scrub ok 2024-03-11T22:15:42.664 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:42.698 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:42.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:42.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:42.840 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:42.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:42.862 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:15:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:15:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:15:42.941 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:42.978 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:42.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:42.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:43.080 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:43.112 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:43.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:43.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:43.214 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:43.257 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:43.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:43.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:43.318 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 41 16 72511 72495 110.486 79 0.0238103 0.00904588 2024-03-11T22:15:43.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:43.413 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:43.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:43.473 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:15:43.473 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:15:43.473 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:15:43.473 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:15:43.473 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:15:43.473 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:15:43.473 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:15:43.473 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:15:43.474 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:15:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:15:43.514 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:43.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:43 smithi045 ceph-mon[30178]: 3.a deep-scrub starts 2024-03-11T22:15:43.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:43 smithi045 ceph-mon[30178]: 3.a deep-scrub ok 2024-03-11T22:15:43.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:43 smithi045 ceph-mon[30178]: pgmap v302: 168 pgs: 168 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-11T22:15:43.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:43 smithi045 ceph-mon[39715]: 3.a deep-scrub starts 2024-03-11T22:15:43.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:43 smithi045 ceph-mon[39715]: 3.a deep-scrub ok 2024-03-11T22:15:43.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:43 smithi045 ceph-mon[39715]: pgmap v302: 168 pgs: 168 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-11T22:15:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:43.547 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:43.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:43.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:43 smithi144 ceph-mon[33984]: 3.a deep-scrub starts 2024-03-11T22:15:43.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:43 smithi144 ceph-mon[33984]: 3.a deep-scrub ok 2024-03-11T22:15:43.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:43 smithi144 ceph-mon[33984]: pgmap v302: 168 pgs: 168 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-11T22:15:43.649 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:43.681 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:43.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:43.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:43.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:43.825 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:43.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:43.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:43.927 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:43.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:43.964 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:43.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:44.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:44.109 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:44.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:44.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:44.269 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:44.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:44.319 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 42 16 73851 73835 109.848 83.75 0.0119667 0.0090987 2024-03-11T22:15:44.370 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:44.403 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:44.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:44.505 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:44.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:44 smithi045 ceph-mon[39715]: 3.11 scrub starts 2024-03-11T22:15:44.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:44 smithi045 ceph-mon[39715]: 3.11 scrub ok 2024-03-11T22:15:44.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:44 smithi045 ceph-mon[39715]: 5.11 scrub starts 2024-03-11T22:15:44.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:44 smithi045 ceph-mon[39715]: 5.11 scrub ok 2024-03-11T22:15:44.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:44 smithi045 ceph-mon[30178]: 3.11 scrub starts 2024-03-11T22:15:44.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:44 smithi045 ceph-mon[30178]: 3.11 scrub ok 2024-03-11T22:15:44.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:44 smithi045 ceph-mon[30178]: 5.11 scrub starts 2024-03-11T22:15:44.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:44 smithi045 ceph-mon[30178]: 5.11 scrub ok 2024-03-11T22:15:44.540 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:44.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:44.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:44.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:44 smithi144 ceph-mon[33984]: 3.11 scrub starts 2024-03-11T22:15:44.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:44 smithi144 ceph-mon[33984]: 3.11 scrub ok 2024-03-11T22:15:44.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:44 smithi144 ceph-mon[33984]: 5.11 scrub starts 2024-03-11T22:15:44.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:44 smithi144 ceph-mon[33984]: 5.11 scrub ok 2024-03-11T22:15:44.643 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:44.670 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:44.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:44.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:44.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T22:15:44.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:44.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:44.836 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:44.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:44.938 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:44.968 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:44.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:45.070 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:45.105 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:45.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:45.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:45.207 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:45.238 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:45.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:45.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:45.282 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:45.319 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 43 16 75101 75085 109.111 78.125 0.0295952 0.00915827 2024-03-11T22:15:45.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:45.385 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:45.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:45.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:45.487 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:45.521 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:45.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:45.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[30178]: 5.1e scrub starts 2024-03-11T22:15:45.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[30178]: 5.1e scrub ok 2024-03-11T22:15:45.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[30178]: 4.13 scrub starts 2024-03-11T22:15:45.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[30178]: 4.13 scrub ok 2024-03-11T22:15:45.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[30178]: pgmap v303: 168 pgs: 168 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-11T22:15:45.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[30178]: 6.4 scrub starts 2024-03-11T22:15:45.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[30178]: 6.4 scrub ok 2024-03-11T22:15:45.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:15:45.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[30178]: 5.6 scrub starts 2024-03-11T22:15:45.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[30178]: 5.6 scrub ok 2024-03-11T22:15:45.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[39715]: 5.1e scrub starts 2024-03-11T22:15:45.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[39715]: 5.1e scrub ok 2024-03-11T22:15:45.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[39715]: 4.13 scrub starts 2024-03-11T22:15:45.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[39715]: 4.13 scrub ok 2024-03-11T22:15:45.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[39715]: pgmap v303: 168 pgs: 168 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-11T22:15:45.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[39715]: 6.4 scrub starts 2024-03-11T22:15:45.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[39715]: 6.4 scrub ok 2024-03-11T22:15:45.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:15:45.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[39715]: 5.6 scrub starts 2024-03-11T22:15:45.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:45 smithi045 ceph-mon[39715]: 5.6 scrub ok 2024-03-11T22:15:45.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:45 smithi144 ceph-mon[33984]: 5.1e scrub starts 2024-03-11T22:15:45.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:45 smithi144 ceph-mon[33984]: 5.1e scrub ok 2024-03-11T22:15:45.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:45 smithi144 ceph-mon[33984]: 4.13 scrub starts 2024-03-11T22:15:45.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:45 smithi144 ceph-mon[33984]: 4.13 scrub ok 2024-03-11T22:15:45.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:45 smithi144 ceph-mon[33984]: pgmap v303: 168 pgs: 168 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-11T22:15:45.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:45 smithi144 ceph-mon[33984]: 6.4 scrub starts 2024-03-11T22:15:45.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:45 smithi144 ceph-mon[33984]: 6.4 scrub ok 2024-03-11T22:15:45.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:15:45.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:45 smithi144 ceph-mon[33984]: 5.6 scrub starts 2024-03-11T22:15:45.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:45 smithi144 ceph-mon[33984]: 5.6 scrub ok 2024-03-11T22:15:45.624 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:45.652 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:45.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:45.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:45.754 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:45.780 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:45.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:45.881 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:45.914 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:45.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:46.015 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:46.050 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:46.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:46.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:46.152 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:46.165 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:15:46.177 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:46.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:46.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:46.279 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:46.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:15:46.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:15:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:15:46.309 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:46.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:46.319 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 44 15 76284 76269 108.313 74 0.0288564 0.00922769 2024-03-11T22:15:46.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:46.472 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:46.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:46.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:46.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[30178]: 4.6 scrub starts 2024-03-11T22:15:46.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[30178]: 4.6 scrub ok 2024-03-11T22:15:46.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2425784753' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:15:46.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[30178]: 3.16 scrub starts 2024-03-11T22:15:46.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[30178]: 3.16 scrub ok 2024-03-11T22:15:46.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:15:46.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2425784753' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:15:46.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[30178]: osdmap e192: 8 total, 8 up, 8 in 2024-03-11T22:15:46.512 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[39715]: 4.6 scrub starts 2024-03-11T22:15:46.513 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[39715]: 4.6 scrub ok 2024-03-11T22:15:46.513 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2425784753' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:15:46.513 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[39715]: 3.16 scrub starts 2024-03-11T22:15:46.513 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[39715]: 3.16 scrub ok 2024-03-11T22:15:46.513 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:15:46.513 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2425784753' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:15:46.513 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:46 smithi045 ceph-mon[39715]: osdmap e192: 8 total, 8 up, 8 in 2024-03-11T22:15:46.574 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:46.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:46 smithi144 ceph-mon[33984]: 4.6 scrub starts 2024-03-11T22:15:46.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:46 smithi144 ceph-mon[33984]: 4.6 scrub ok 2024-03-11T22:15:46.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:46 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2425784753' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:15:46.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:46 smithi144 ceph-mon[33984]: 3.16 scrub starts 2024-03-11T22:15:46.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:46 smithi144 ceph-mon[33984]: 3.16 scrub ok 2024-03-11T22:15:46.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:46 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:15:46.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:46 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2425784753' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:15:46.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:46 smithi144 ceph-mon[33984]: osdmap e192: 8 total, 8 up, 8 in 2024-03-11T22:15:46.606 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:46.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:46.708 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:46.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:46.748 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:46.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:46.761 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:46.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:46.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:46.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:46.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:47.037 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:47.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:47.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:47.139 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:47.171 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:47.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:47.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:47.273 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:47.313 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:47.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:47.319 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 45 16 77585 77569 107.711 81.25 0.0110301 0.00927939 2024-03-11T22:15:47.414 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:47.439 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:47.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:47.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:47.541 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:47.576 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:47.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:47.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:47.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:47 smithi144 ceph-mon[33984]: pgmap v305: 168 pgs: 168 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T22:15:47.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:47.713 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:47.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:47.720 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:47.721 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:47 smithi045 ceph-mon[39715]: pgmap v305: 168 pgs: 168 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T22:15:47.723 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:47.723 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:47 smithi045 ceph-mon[30178]: pgmap v305: 168 pgs: 168 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T22:15:47.816 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:47.847 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:47.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:47.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:47.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:47.994 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:47.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:47.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:48.096 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:48.129 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:48.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:48.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:48.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:48.275 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:48.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:48.321 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 46 16 78849 78833 107.081 79 0.00814333 0.00933269 2024-03-11T22:15:48.376 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:48.407 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:48.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:48.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:48.550 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:48.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:48.652 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:48.683 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:48.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:48.785 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:48.818 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:48.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:48.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:48.920 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:48.963 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:48.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:48.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:49.065 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:49.093 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:49.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:49.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:49.195 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:49.225 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:49.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:49.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:49.313 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:49.322 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 47 16 79967 79951 106.289 69.875 0.0241941 0.00940104 2024-03-11T22:15:49.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:49.387 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:49.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:49.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:49.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:49 smithi045 ceph-mon[39715]: pgmap v306: 168 pgs: 168 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T22:15:49.445 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:49 smithi045 ceph-mon[30178]: pgmap v306: 168 pgs: 168 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T22:15:49.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:49.544 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:49.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:49.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:49 smithi144 ceph-mon[33984]: pgmap v306: 168 pgs: 168 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T22:15:49.646 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:49.675 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:49.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:49.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:49.777 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:49.821 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:49.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:49.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:49.923 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:49.962 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:49.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:50.064 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:50.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:50.093 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:50.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:50.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:50.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:50.235 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:50.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:50.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:15:50.322 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 48 16 81034 81018 105.464 66.6875 0.00906703 0.00947638 2024-03-11T22:15:50.336 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:50.356 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.7 to 0.358581 (5bcb) 2024-03-11T22:15:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:50.396 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:50.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:50.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:50.521 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:50 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:15:50.521 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:50 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:15:50.521 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:50 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1976599932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.3585811336606596}]: dispatch 2024-03-11T22:15:50.521 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:50 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.3585811336606596}]: dispatch 2024-03-11T22:15:50.521 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:50 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:15:50.521 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:50 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:15:50.521 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:50 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1976599932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.3585811336606596}]: dispatch 2024-03-11T22:15:50.522 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:50 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.3585811336606596}]: dispatch 2024-03-11T22:15:50.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:50.522 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:50.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:50.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:50 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:15:50.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:50 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:15:50.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:50 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1976599932' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.3585811336606596}]: dispatch 2024-03-11T22:15:50.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:50 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.3585811336606596}]: dispatch 2024-03-11T22:15:50.624 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:50.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:50.648 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:50.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:50.749 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:50.774 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:50.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:50.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:50.876 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:50.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:50.900 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:50.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:51.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:51.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:51.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:51.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:51.155 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:51.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:51.256 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:51.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:51.280 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:51.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:51.322 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 49 16 81174 81158 103.49 8.75 0.0032825 0.00948375 2024-03-11T22:15:51.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:51.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:51.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:51.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:51.416 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:51 smithi045 ceph-mon[39715]: pgmap v307: 168 pgs: 168 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 573 B/s rd, 42 MiB/s wr, 671 op/s 2024-03-11T22:15:51.416 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:51 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.3585811336606596}]': finished 2024-03-11T22:15:51.416 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:51 smithi045 ceph-mon[39715]: osdmap e193: 8 total, 8 up, 8 in 2024-03-11T22:15:51.416 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:51 smithi045 ceph-mon[39715]: osdmap e194: 8 total, 8 up, 8 in 2024-03-11T22:15:51.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:51 smithi045 ceph-mon[30178]: pgmap v307: 168 pgs: 168 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 573 B/s rd, 42 MiB/s wr, 671 op/s 2024-03-11T22:15:51.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:51 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.3585811336606596}]': finished 2024-03-11T22:15:51.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:51 smithi045 ceph-mon[30178]: osdmap e193: 8 total, 8 up, 8 in 2024-03-11T22:15:51.417 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:51 smithi045 ceph-mon[30178]: osdmap e194: 8 total, 8 up, 8 in 2024-03-11T22:15:51.508 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:51.532 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:51.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:51.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:51.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:51 smithi144 ceph-mon[33984]: pgmap v307: 168 pgs: 168 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 573 B/s rd, 42 MiB/s wr, 671 op/s 2024-03-11T22:15:51.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:51 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.3585811336606596}]': finished 2024-03-11T22:15:51.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:51 smithi144 ceph-mon[33984]: osdmap e193: 8 total, 8 up, 8 in 2024-03-11T22:15:51.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:51 smithi144 ceph-mon[33984]: osdmap e194: 8 total, 8 up, 8 in 2024-03-11T22:15:51.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:51.658 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:51.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:51.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:51.761 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:51.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:51.786 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:51.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:51.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:51.913 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:51.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:51.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:52.015 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:52.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:52.040 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:52.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:52.141 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:52.168 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:52.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:52.269 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:52.299 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:52.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:52.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:52.322 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 50 16 81208 81192 101.463 2.125 0.00349077 0.00963837 2024-03-11T22:15:52.401 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:52.434 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:52.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:52.536 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:52.571 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:52.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:52.654 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:15:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:15:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:15:52.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:52.726 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:52.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:52.769 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:15:52.769 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:15:52.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:52.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:52.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:52.975 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:53.012 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:53.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:53.113 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:53.145 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:53.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:53.254 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:53.289 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:53.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:53.322 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 51 16 82324 82308 100.842 69.75 0.0104515 0.00991135 2024-03-11T22:15:53.391 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:53.426 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:53.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:53.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:53.528 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:53.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:53 smithi045 ceph-mon[39715]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 192'5069 2024-03-11T22:15:53.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:53 smithi045 ceph-mon[39715]: osdmap e195: 8 total, 8 up, 8 in 2024-03-11T22:15:53.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:53 smithi045 ceph-mon[39715]: pgmap v311: 168 pgs: 31 peering, 137 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 61 MiB/s wr, 969 op/s 2024-03-11T22:15:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:53 smithi045 ceph-mon[30178]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 192'5069 2024-03-11T22:15:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:53 smithi045 ceph-mon[30178]: osdmap e195: 8 total, 8 up, 8 in 2024-03-11T22:15:53.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:53 smithi045 ceph-mon[30178]: pgmap v311: 168 pgs: 31 peering, 137 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 61 MiB/s wr, 969 op/s 2024-03-11T22:15:53.564 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:53.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:53.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:53 smithi144 ceph-mon[33984]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 192'5069 2024-03-11T22:15:53.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:53 smithi144 ceph-mon[33984]: osdmap e195: 8 total, 8 up, 8 in 2024-03-11T22:15:53.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:53 smithi144 ceph-mon[33984]: pgmap v311: 168 pgs: 31 peering, 137 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 61 MiB/s wr, 969 op/s 2024-03-11T22:15:53.666 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:53.693 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:53.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:53.795 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:53.831 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:53.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:53.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:53.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:53.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:53.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:54.072 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:54.098 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:54.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:54.200 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:54.237 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:54.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:54.258 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:15:54.300 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:15:54.301 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:54 smithi045 ceph-mon[30178]: osdmap e196: 8 total, 8 up, 8 in 2024-03-11T22:15:54.301 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:54 smithi045 ceph-mon[39715]: osdmap e196: 8 total, 8 up, 8 in 2024-03-11T22:15:54.322 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 52 16 83411 83395 100.209 67.9375 0.00850972 0.0099714 2024-03-11T22:15:54.335 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:15:54.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:54.377 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:15:54.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:54.379 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:54.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:54.422 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:15:54.460 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:15:54.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:54.498 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:15:54.520 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:54.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:54.540 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:15:54.543 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:15:54.543 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:15:54.543 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:15:54.543 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:15:54.543 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:15:54.543 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:15:54.543 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:15:54.543 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:15:54.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:54 smithi144 ceph-mon[33984]: osdmap e196: 8 total, 8 up, 8 in 2024-03-11T22:15:54.622 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:54.651 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:54.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:54.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:54.753 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:54.789 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:54.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:54.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:54.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:54.925 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:54.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:55.027 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:55.060 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:55.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:55.161 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:55.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:55.201 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:55.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:55.302 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:55.322 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 53 16 84518 84502 99.6235 69.1875 0.00460114 0.0100321 2024-03-11T22:15:55.340 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:55.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:55.442 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:55.477 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:55.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:55.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:55.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:55 smithi045 ceph-mon[39715]: pgmap v314: 168 pgs: 31 peering, 137 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 65 MiB/s wr, 1.03k op/s 2024-03-11T22:15:55.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:55 smithi045 ceph-mon[39715]: osdmap e197: 8 total, 8 up, 8 in 2024-03-11T22:15:55.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:55 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2628258763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:15:55.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:55 smithi045 ceph-mon[30178]: pgmap v314: 168 pgs: 31 peering, 137 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 65 MiB/s wr, 1.03k op/s 2024-03-11T22:15:55.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:55 smithi045 ceph-mon[30178]: osdmap e197: 8 total, 8 up, 8 in 2024-03-11T22:15:55.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:55 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2628258763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:15:55.579 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:55.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:55 smithi144 ceph-mon[33984]: pgmap v314: 168 pgs: 31 peering, 137 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 65 MiB/s wr, 1.03k op/s 2024-03-11T22:15:55.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:55 smithi144 ceph-mon[33984]: osdmap e197: 8 total, 8 up, 8 in 2024-03-11T22:15:55.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:55 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2628258763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:15:55.617 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:55.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:55.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:55.760 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:55.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:55.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:55.788 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T22:15:55.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:55.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:55.901 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:55.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:56.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:56.042 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:56.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:56.046 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:56.094 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:15:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:15:56.144 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:56.162 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:15:56.162 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:15:56.163 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-11T22:15:56.163 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:15:56.199 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:56.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:56.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:56.204 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:15:56.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:56.323 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 54 16 85695 85679 99.1405 73.5625 0.00843003 0.0100804 2024-03-11T22:15:56.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:56.348 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:56.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:56.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:56.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:56.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:56.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:56.600 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:56.638 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:56.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:56.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:56.781 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:56.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:56.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:56.883 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:56.936 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:56.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:57.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:15:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:57.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:57.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:15:57.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:15:57.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:57.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:57.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:15:57.323 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:57.323 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 55 16 86836 86820 98.6345 71.3125 0.0417521 0.0101323 2024-03-11T22:15:57.358 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:57.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:57.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:57.458 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:15:57.460 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:57.492 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:57.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:57.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:57.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:57.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:57 smithi045 ceph-mon[39715]: osdmap e198: 8 total, 8 up, 8 in 2024-03-11T22:15:57.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:57 smithi045 ceph-mon[39715]: pgmap v316: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 907 op/s; 9530/254733 objects misplaced (3.741%); 13 MiB/s, 213 objects/s recovering 2024-03-11T22:15:57.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:57 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:15:57.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:57 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:15:57.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:57.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:57 smithi045 ceph-mon[30178]: osdmap e198: 8 total, 8 up, 8 in 2024-03-11T22:15:57.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:57 smithi045 ceph-mon[30178]: pgmap v316: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 907 op/s; 9530/254733 objects misplaced (3.741%); 13 MiB/s, 213 objects/s recovering 2024-03-11T22:15:57.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:57 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:15:57.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:57 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:15:57.532 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:15:57.532 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:57 smithi144 ceph-mon[33984]: osdmap e198: 8 total, 8 up, 8 in 2024-03-11T22:15:57.532 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:57 smithi144 ceph-mon[33984]: pgmap v316: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 907 op/s; 9530/254733 objects misplaced (3.741%); 13 MiB/s, 213 objects/s recovering 2024-03-11T22:15:57.532 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:57 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:15:57.532 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:57 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:15:57.594 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:57.628 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:57.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:57.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:57.730 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:57.767 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:57.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:57.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:57.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:57.907 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:57.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:58.008 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:58.046 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:58.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:58.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:58.186 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:58.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:58.288 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:58.323 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 56 16 87989 87973 98.1596 72.0625 0.0169552 0.0101825 2024-03-11T22:15:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:58.325 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:58.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:58.426 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:58.472 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:58.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:58.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:58.491 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:58 smithi045 ceph-mon[39715]: osdmap e199: 8 total, 8 up, 8 in 2024-03-11T22:15:58.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:58 smithi045 ceph-mon[30178]: osdmap e199: 8 total, 8 up, 8 in 2024-03-11T22:15:58.509 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:58 smithi144 ceph-mon[33984]: osdmap e199: 8 total, 8 up, 8 in 2024-03-11T22:15:58.545 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:15:58.574 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:15:58.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:58.612 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:58.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:15:58.713 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:58.751 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:58.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:58.852 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:15:58.859 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:15:58.859 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:15:58.859 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.859 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:88600 8.b e199/175 8:d9eababa:::benchmark_data_smithi045_115206_object88599:head v 199'5350, mlcod=199'5348)", 2024-03-11T22:15:58.859 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:15:58.853671+0000", 2024-03-11T22:15:58.859 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.0025788370000000001, 2024-03-11T22:15:58.859 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026114760000000002, 2024-03-11T22:15:58.859 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:15:58.859 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:15:58.859 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:15:58.859 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.859 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:15:58.859 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.853671+0000", 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.853671+0000", 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.853672+0000", 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.266e-06 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.860 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.853709+0000", 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7610000000000001e-05 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.853727+0000", 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7807000000000001e-05 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.853737+0000", 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.9359999999999994e-06 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.861 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.854141+0000", 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000403442 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.854173+0000", 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2076999999999998e-05 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.862 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:88601 8.e e199/175 8:7a2d415e:::benchmark_data_smithi045_115206_object88600:head v 199'5467, mlcod=199'5465)", 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:15:58.854890+0000", 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.00135969, 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001459239, 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.854890+0000", 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.854890+0000", 2024-03-11T22:15:58.863 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.854890+0000", 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.44e-07 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.854943+0000", 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2598e-05 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:15:58.864 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.854960+0000", 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6535999999999999e-05 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.854968+0000", 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.354e-06 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.855048+0000", 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0242999999999997e-05 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:15:58.865 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.855072+0000", 2024-03-11T22:15:58.866 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4043999999999999e-05 2024-03-11T22:15:58.866 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:15:58.866 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:15:58.866 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:15:58.866 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:15:58.866 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-03-11T22:15:58.866 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 2 2024-03-11T22:15:58.866 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:15:58.890 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:58.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:15:58.992 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:59.022 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:59.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:59.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:59.124 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:59.163 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:59.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:59.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:59.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:59.304 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:59.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:15:59.323 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 57 16 89162 89146 97.7237 73.3125 0.033349 0.0102279 2024-03-11T22:15:59.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:15:59.454 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:59.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:59.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:15:59.456 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:15:59 smithi045 ceph-mon[39715]: pgmap v318: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 727 op/s; 9530/254733 objects misplaced (3.741%); 10 MiB/s, 171 objects/s recovering 2024-03-11T22:15:59.456 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:15:59 smithi045 ceph-mon[30178]: pgmap v318: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 727 op/s; 9530/254733 objects misplaced (3.741%); 10 MiB/s, 171 objects/s recovering 2024-03-11T22:15:59.491 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:15:59 smithi144 ceph-mon[33984]: pgmap v318: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 727 op/s; 9530/254733 objects misplaced (3.741%); 10 MiB/s, 171 objects/s recovering 2024-03-11T22:15:59.539 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T22:15:59.556 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:15:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:59.626 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:59.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:15:59.729 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:15:59.768 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:15:59.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:59.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:15:59.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:15:59.927 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:15:59.928 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:15:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:15:59.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:00.030 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:00.069 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:00.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:00.170 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:00.207 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:00.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:00.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:00.309 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:00.324 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 58 16 90248 90232 97.209 67.875 0.0147596 0.0102772 2024-03-11T22:16:00.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:00.343 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:00.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:00.402 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:00.444 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:00.474 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:00.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:00.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:00.575 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:00.608 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:00.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:00.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:00.710 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:00.713 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:16:00.751 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:00.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:00.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:00.853 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:16:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:00.892 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:00.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:00.893 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:16:00.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:01.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:16:01.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:16:01.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:00 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1126356646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:01.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:16:01.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:16:01.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:00 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1126356646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:01.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:01.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:01.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:01.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:16:01.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:16:01.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:00 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1126356646' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:01.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:01.169 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:01.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:01.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:01.271 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:01.311 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:01.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:01.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:01.323 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 59 16 91295 91279 96.6705 65.4375 0.0117039 0.0103395 2024-03-11T22:16:01.413 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:01.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:01.451 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:01.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:01.552 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:01.593 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:01.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:01.695 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:01.731 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:01.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:01.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:01.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:01.871 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:01.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:01.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:01.969 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:16:01.969 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-03-11T22:16:01.970 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-03-11T22:16:01.970 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-03-11T22:16:01.970 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:16:01.970 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:01 smithi144 ceph-mon[33984]: pgmap v319: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 38 MiB/s wr, 605 op/s; 9530/254733 objects misplaced (3.741%); 8.4 MiB/s, 142 objects/s recovering 2024-03-11T22:16:01.970 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:01 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:01.970 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:01 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1126356646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:16:01.970 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:01 smithi144 ceph-mon[33984]: osdmap e200: 8 total, 8 up, 8 in 2024-03-11T22:16:01.973 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:02.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:02.010 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:02.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:02.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:01 smithi045 ceph-mon[39715]: pgmap v319: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 38 MiB/s wr, 605 op/s; 9530/254733 objects misplaced (3.741%); 8.4 MiB/s, 142 objects/s recovering 2024-03-11T22:16:02.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:01 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:02.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:01 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1126356646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:16:02.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:01 smithi045 ceph-mon[39715]: osdmap e200: 8 total, 8 up, 8 in 2024-03-11T22:16:02.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:01 smithi045 ceph-mon[30178]: pgmap v319: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 38 MiB/s wr, 605 op/s; 9530/254733 objects misplaced (3.741%); 8.4 MiB/s, 142 objects/s recovering 2024-03-11T22:16:02.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:01 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:02.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:01 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1126356646' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:16:02.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:01 smithi045 ceph-mon[30178]: osdmap e200: 8 total, 8 up, 8 in 2024-03-11T22:16:02.111 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:02.150 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:02.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:02.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:02.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:02.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:02.324 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:16:02.325371+0000 min lat: 0.00125214 max lat: 2.01083 avg lat: 0.0103783 2024-03-11T22:16:02.324 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:16:02.324 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 60 16 92489 92473 96.303 74.625 0.00766488 0.0103783 2024-03-11T22:16:02.391 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:02.425 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:02.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:02.526 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:02.561 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:02.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:02.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:02.672 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T22:16:02.701 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:02.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:02.803 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:02.834 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:02.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:02.936 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:02.978 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:02.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:03.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:02 smithi045 ceph-mon[30178]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:16:03.030 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:16:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:16:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:16:03.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:02 smithi045 ceph-mon[39715]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:16:03.058 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:16:03.081 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:03.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:02 smithi144 ceph-mon[33984]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:16:03.108 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:03.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:03.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:03.250 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:03.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:03.324 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 61 16 93499 93483 95.7591 63.125 0.00836448 0.0104376 2024-03-11T22:16:03.352 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:03.388 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:03.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:03.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:03.536 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:03.573 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:03.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:03.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:03.675 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:03.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:03.711 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:03.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:03.812 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:03.850 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:03.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:03.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:03.992 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:03.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:04.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:03 smithi045 ceph-mon[30178]: pgmap v321: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 166 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 851 B/s rd, 60 MiB/s wr, 958 op/s; 8671/272004 objects misplaced (3.188%); 12 MiB/s, 189 objects/s recovering 2024-03-11T22:16:04.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:03 smithi045 ceph-mon[39715]: pgmap v321: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 166 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 851 B/s rd, 60 MiB/s wr, 958 op/s; 8671/272004 objects misplaced (3.188%); 12 MiB/s, 189 objects/s recovering 2024-03-11T22:16:04.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:03 smithi144 ceph-mon[33984]: pgmap v321: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 166 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 851 B/s rd, 60 MiB/s wr, 958 op/s; 8671/272004 objects misplaced (3.188%); 12 MiB/s, 189 objects/s recovering 2024-03-11T22:16:04.094 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:04.130 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:04.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:04.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:04.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:16:04.232 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:04.268 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:04.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:04.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:04.324 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 62 16 94474 94458 95.1975 60.9375 0.00397679 0.0104976 2024-03-11T22:16:04.370 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:04.403 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:04.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:04.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:04.422 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-03-11T22:16:00.689726+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5764421980,"num_objects":88147,"num_object_clones":0,"num_object_copies":264441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8750,"num_objects_unfound":0,"num_objects_dirty":88147,"num_whiteouts":0,"num_read":1109,"num_read_kb":861,"num_write":88385,"num_write_kb":5628781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1924,"num_bytes_recovered":122553593,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73068,"ondisk_log_size":73068,"up":504,"acting":503,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":506,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18555548,"kb_used_data":17002516,"kb_used_omap":12,"kb_used_meta":1552947,"kb_avail":731405668,"statfs":{"total":767960285184,"available":748959404032,"internally_reserved":0,"allocated":17410576384,"data_stored":17407282550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":1590218291},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,7,4,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":450166784,"num_objects":6869,"num_object_clones":0,"num_object_copies":20607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8750,"num_objects_unfound":0,"num_objects_dirty":6869,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6869,"num_write_kb":439616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1918,"num_bytes_recovered":122553483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.515562"},"pg_stats":[{"pgid":"8.0","version":"197'5268","reported_seq":5322,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:55.685078+0000","last_change":"2024-03-11T22:14:52.240741+0000","last_active":"2024-03-11T22:15:55.685078+0000","last_peered":"2024-03-11T22:15:55.685078+0000","last_clean":"2024-03-11T22:15:55.685078+0000","last_became_active":"2024-03-11T22:14:52.240283+0000","last_became_peered":"2024-03-11T22:14:52.240283+0000","last_unstale":"2024-03-11T22:15:55.685078+0000","last_undegraded":"2024-03-11T22:15:55.685078+0000","last_fullsized":"2024-03-11T22:15:55.685078+0000","mapping_epoch":175,"log_start":"181'1188","ondisk_log_start":"181'1188","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4080,"log_dups_size":0,"ondisk_log_size":4080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:01.057390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345243648,"num_objects":5268,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5268,"num_write_kb":337152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":88,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.188241+0000","last_change":"2024-03-11T22:14:52.237967+0000","last_active":"2024-03-11T22:15:54.188241+0000","last_peered":"2024-03-11T22:15:54.188241+0000","last_clean":"2024-03-11T22:15:54.188241+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:15:54.188241+0000","last_undegraded":"2024-03-11T22:15:54.188241+0000","last_fullsized":"2024-03-11T22:15:54.188241+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:14:24.847623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:24.847623+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":296,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.180864+0000","last_change":"2024-03-11T22:14:12.801583+0000","last_active":"2024-03-11T22:15:54.180864+0000","last_peered":"2024-03-11T22:15:54.180864+0000","last_clean":"2024-03-11T22:15:54.180864+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:15:54.180864+0000","last_undegraded":"2024-03-11T22:15:54.180864+0000","last_fullsized":"2024-03-11T22:15:54.180864+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:12.801470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:12.801470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":394,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.186850+0000","last_change":"2024-03-11T22:14:29.061747+0000","last_active":"2024-03-11T22:15:54.186850+0000","last_peered":"2024-03-11T22:15:54.186850+0000","last_clean":"2024-03-11T22:15:54.186850+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:15:54.186850+0000","last_undegraded":"2024-03-11T22:15:54.186850+0000","last_fullsized":"2024-03-11T22:15:54.186850+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:14:29.061624+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_clean_scrub_stamp":"2024-03-11T22:14:29.061624+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":66,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.196441+0000","last_change":"2024-03-11T22:15:40.342976+0000","last_active":"2024-03-11T22:15:57.196441+0000","last_peered":"2024-03-11T22:15:57.196441+0000","last_clean":"2024-03-11T22:15:57.196441+0000","last_became_active":"2024-03-11T22:14:52.243690+0000","last_became_peered":"2024-03-11T22:14:52.243690+0000","last_unstale":"2024-03-11T22:15:57.196441+0000","last_undegraded":"2024-03-11T22:15:57.196441+0000","last_fullsized":"2024-03-11T22:15:57.196441+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:40.342855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:40.342855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:07.685609+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":565,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.201107+0000","last_change":"2024-03-11T22:15:54.201107+0000","last_active":"2024-03-11T22:15:54.201107+0000","last_peered":"2024-03-11T22:15:54.201107+0000","last_clean":"2024-03-11T22:15:54.201107+0000","last_became_active":"2024-03-11T22:15:54.200742+0000","last_became_peered":"2024-03-11T22:15:54.200742+0000","last_unstale":"2024-03-11T22:15:54.201107+0000","last_undegraded":"2024-03-11T22:15:54.201107+0000","last_fullsized":"2024-03-11T22:15:54.201107+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:10.906610+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:10.906610+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"199'5675","reported_seq":5733,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:16:00.476984+0000","last_change":"2024-03-11T22:14:52.245149+0000","last_active":"2024-03-11T22:16:00.476984+0000","last_peered":"2024-03-11T22:16:00.476984+0000","last_clean":"2024-03-11T22:16:00.476984+0000","last_became_active":"2024-03-11T22:14:52.244799+0000","last_became_peered":"2024-03-11T22:14:52.244799+0000","last_unstale":"2024-03-11T22:16:00.476984+0000","last_undegraded":"2024-03-11T22:16:00.476984+0000","last_fullsized":"2024-03-11T22:16:00.476984+0000","mapping_epoch":175,"log_start":"183'1725","ondisk_log_start":"183'1725","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3950,"log_dups_size":0,"ondisk_log_size":3950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:09.738242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371916800,"num_objects":5675,"num_object_clones":0,"num_object_copies":17025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5675,"num_write_kb":363200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":109,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.186744+0000","last_change":"2024-03-11T22:15:05.001672+0000","last_active":"2024-03-11T22:15:54.186744+0000","last_peered":"2024-03-11T22:15:54.186744+0000","last_clean":"2024-03-11T22:15:54.186744+0000","last_became_active":"2024-03-11T22:14:52.239649+0000","last_became_peered":"2024-03-11T22:14:52.239649+0000","last_unstale":"2024-03-11T22:15:54.186744+0000","last_undegraded":"2024-03-11T22:15:54.186744+0000","last_fullsized":"2024-03-11T22:15:54.186744+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-11T22:15:05.001522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.001522+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.788107+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":316,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.200146+0000","last_change":"2024-03-11T22:15:42.136659+0000","last_active":"2024-03-11T22:15:57.200146+0000","last_peered":"2024-03-11T22:15:57.200146+0000","last_clean":"2024-03-11T22:15:57.200146+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:15:57.200146+0000","last_undegraded":"2024-03-11T22:15:57.200146+0000","last_fullsized":"2024-03-11T22:15:57.200146+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:15:42.136540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:56.439912+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":292,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.197169+0000","last_change":"2024-03-11T22:14:18.859904+0000","last_active":"2024-03-11T22:15:57.197169+0000","last_peered":"2024-03-11T22:15:57.197169+0000","last_clean":"2024-03-11T22:15:57.197169+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:15:57.197169+0000","last_undegraded":"2024-03-11T22:15:57.197169+0000","last_fullsized":"2024-03-11T22:15:57.197169+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:18.859546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:14:18.859546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:01.240812+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":292,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.186295+0000","last_change":"2024-03-11T22:15:18.067803+0000","last_active":"2024-03-11T22:15:54.186295+0000","last_peered":"2024-03-11T22:15:54.186295+0000","last_clean":"2024-03-11T22:15:54.186295+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:15:54.186295+0000","last_undegraded":"2024-03-11T22:15:54.186295+0000","last_fullsized":"2024-03-11T22:15:54.186295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:18.067284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:18.067284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:46.629798+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":560,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.198526+0000","last_change":"2024-03-11T22:14:11.673622+0000","last_active":"2024-03-11T22:15:57.198526+0000","last_peered":"2024-03-11T22:15:57.198526+0000","last_clean":"2024-03-11T22:15:57.198526+0000","last_became_active":"2024-03-11T22:13:38.569606+0000","last_became_peered":"2024-03-11T22:13:38.569606+0000","last_unstale":"2024-03-11T22:15:57.198526+0000","last_undegraded":"2024-03-11T22:15:57.198526+0000","last_fullsized":"2024-03-11T22:15:57.198526+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:11.673454+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:11.673454+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"197'5288","reported_seq":5342,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:55.615220+0000","last_change":"2024-03-11T22:14:52.243433+0000","last_active":"2024-03-11T22:15:55.615220+0000","last_peered":"2024-03-11T22:15:55.615220+0000","last_clean":"2024-03-11T22:15:55.615220+0000","last_became_active":"2024-03-11T22:14:52.243038+0000","last_became_peered":"2024-03-11T22:14:52.243038+0000","last_unstale":"2024-03-11T22:15:55.615220+0000","last_undegraded":"2024-03-11T22:15:55.615220+0000","last_fullsized":"2024-03-11T22:15:55.615220+0000","mapping_epoch":175,"log_start":"180'589","ondisk_log_start":"180'589","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4699,"log_dups_size":0,"ondisk_log_size":4699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346554368,"num_objects":5288,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5288,"num_write_kb":338432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":104,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201284+0000","last_change":"2024-03-11T22:15:22.254990+0000","last_active":"2024-03-11T22:15:57.201284+0000","last_peered":"2024-03-11T22:15:57.201284+0000","last_clean":"2024-03-11T22:15:57.201284+0000","last_became_active":"2024-03-11T22:14:52.240437+0000","last_became_peered":"2024-03-11T22:14:52.240437+0000","last_unstale":"2024-03-11T22:15:57.201284+0000","last_undegraded":"2024-03-11T22:15:57.201284+0000","last_fullsized":"2024-03-11T22:15:57.201284+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:15:22.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:22.254869+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:26.708596+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":63,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.185518+0000","last_change":"2024-03-11T22:15:41.871554+0000","last_active":"2024-03-11T22:15:54.185518+0000","last_peered":"2024-03-11T22:15:54.185518+0000","last_clean":"2024-03-11T22:15:54.185518+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:15:54.185518+0000","last_undegraded":"2024-03-11T22:15:54.185518+0000","last_fullsized":"2024-03-11T22:15:54.185518+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:41.870596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:41.870596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:56.404780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":292,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199505+0000","last_change":"2024-03-11T22:14:26.672389+0000","last_active":"2024-03-11T22:15:57.199505+0000","last_peered":"2024-03-11T22:15:57.199505+0000","last_clean":"2024-03-11T22:15:57.199505+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:15:57.199505+0000","last_undegraded":"2024-03-11T22:15:57.199505+0000","last_fullsized":"2024-03-11T22:15:57.199505+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:26.672153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.672153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":276,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.187683+0000","last_change":"2024-03-11T22:13:17.533805+0000","last_active":"2024-03-11T22:15:54.187683+0000","last_peered":"2024-03-11T22:15:54.187683+0000","last_clean":"2024-03-11T22:15:54.187683+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:15:54.187683+0000","last_undegraded":"2024-03-11T22:15:54.187683+0000","last_fullsized":"2024-03-11T22:15:54.187683+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:17.533607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:17.533607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"93'130","reported_seq":580,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.197229+0000","last_change":"2024-03-11T22:15:04.424215+0000","last_active":"2024-03-11T22:15:57.197229+0000","last_peered":"2024-03-11T22:15:57.197229+0000","last_clean":"2024-03-11T22:15:57.197229+0000","last_became_active":"2024-03-11T22:14:43.194793+0000","last_became_peered":"2024-03-11T22:14:43.194793+0000","last_unstale":"2024-03-11T22:15:57.197229+0000","last_undegraded":"2024-03-11T22:15:57.197229+0000","last_fullsized":"2024-03-11T22:15:57.197229+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:04.424114+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:04.424114+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:32.665794+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3","version":"199'5503","reported_seq":5561,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:16:00.489163+0000","last_change":"2024-03-11T22:14:52.245584+0000","last_active":"2024-03-11T22:16:00.489163+0000","last_peered":"2024-03-11T22:16:00.489163+0000","last_clean":"2024-03-11T22:16:00.489163+0000","last_became_active":"2024-03-11T22:14:52.245192+0000","last_became_peered":"2024-03-11T22:14:52.245192+0000","last_unstale":"2024-03-11T22:16:00.489163+0000","last_undegraded":"2024-03-11T22:16:00.489163+0000","last_fullsized":"2024-03-11T22:16:00.489163+0000","mapping_epoch":175,"log_start":"183'1488","ondisk_log_start":"183'1488","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:11.735324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360644608,"num_objects":5503,"num_object_clones":0,"num_object_copies":16509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5503,"num_write_kb":352192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":329,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.202286+0000","last_change":"2024-03-11T22:15:31.357450+0000","last_active":"2024-03-11T22:15:57.202286+0000","last_peered":"2024-03-11T22:15:57.202286+0000","last_clean":"2024-03-11T22:15:57.202286+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:15:57.202286+0000","last_undegraded":"2024-03-11T22:15:57.202286+0000","last_fullsized":"2024-03-11T22:15:57.202286+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:15:31.357146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:31.357146+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:39.125944+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":302,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.190830+0000","last_change":"2024-03-11T22:15:51.183637+0000","last_active":"2024-03-11T22:15:54.190830+0000","last_peered":"2024-03-11T22:15:54.190830+0000","last_clean":"2024-03-11T22:15:54.190830+0000","last_became_active":"2024-03-11T22:15:51.182962+0000","last_became_peered":"2024-03-11T22:15:51.182962+0000","last_unstale":"2024-03-11T22:15:54.190830+0000","last_undegraded":"2024-03-11T22:15:54.190830+0000","last_fullsized":"2024-03-11T22:15:54.190830+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:38.013058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:38.013058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:03.265410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":106,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.198850+0000","last_change":"2024-03-11T22:15:39.663447+0000","last_active":"2024-03-11T22:15:57.198850+0000","last_peered":"2024-03-11T22:15:57.198850+0000","last_clean":"2024-03-11T22:15:57.198850+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:15:57.198850+0000","last_undegraded":"2024-03-11T22:15:57.198850+0000","last_fullsized":"2024-03-11T22:15:57.198850+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:15:39.663324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:39.663324+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:57.759753+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":288,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.198940+0000","last_change":"2024-03-11T22:15:14.620431+0000","last_active":"2024-03-11T22:15:57.198940+0000","last_peered":"2024-03-11T22:15:57.198940+0000","last_clean":"2024-03-11T22:15:57.198940+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:15:57.198940+0000","last_undegraded":"2024-03-11T22:15:57.198940+0000","last_fullsized":"2024-03-11T22:15:57.198940+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.620337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.620337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:14.661179+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":560,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201534+0000","last_change":"2024-03-11T22:14:12.712999+0000","last_active":"2024-03-11T22:15:57.201534+0000","last_peered":"2024-03-11T22:15:57.201534+0000","last_clean":"2024-03-11T22:15:57.201534+0000","last_became_active":"2024-03-11T22:13:38.573794+0000","last_became_peered":"2024-03-11T22:13:38.573794+0000","last_unstale":"2024-03-11T22:15:57.201534+0000","last_undegraded":"2024-03-11T22:15:57.201534+0000","last_fullsized":"2024-03-11T22:15:57.201534+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:12.712874+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:12.712874+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"197'5346","reported_seq":5490,"reported_epoch":197,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-11T22:15:55.771174+0000","last_change":"2024-03-11T22:15:52.376011+0000","last_active":"2024-03-11T22:15:55.771174+0000","last_peered":"2024-03-11T22:15:55.771174+0000","last_clean":"2024-03-11T22:15:50.311709+0000","last_became_active":"2024-03-11T22:15:52.324400+0000","last_became_peered":"2024-03-11T22:15:52.324400+0000","last_unstale":"2024-03-11T22:15:55.771174+0000","last_undegraded":"2024-03-11T22:15:55.771174+0000","last_fullsized":"2024-03-11T22:15:52.170552+0000","mapping_epoch":194,"log_start":"180'578","ondisk_log_start":"180'578","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350355456,"num_objects":5346,"num_object_clones":0,"num_object_copies":16038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5022,"num_objects_unfound":0,"num_objects_dirty":5346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5346,"num_write_kb":342144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":2818048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3","objects":324},{"shards":"0,3,7","objects":5022}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":68,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.179728+0000","last_change":"2024-03-11T22:15:51.192288+0000","last_active":"2024-03-11T22:15:54.179728+0000","last_peered":"2024-03-11T22:15:54.179728+0000","last_clean":"2024-03-11T22:15:54.179728+0000","last_became_active":"2024-03-11T22:15:51.191837+0000","last_became_peered":"2024-03-11T22:15:51.191837+0000","last_unstale":"2024-03-11T22:15:54.179728+0000","last_undegraded":"2024-03-11T22:15:54.179728+0000","last_fullsized":"2024-03-11T22:15:54.179728+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.293977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.293977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:22.606686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":351,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.190932+0000","last_change":"2024-03-11T22:15:51.206019+0000","last_active":"2024-03-11T22:15:54.190932+0000","last_peered":"2024-03-11T22:15:54.190932+0000","last_clean":"2024-03-11T22:15:54.190932+0000","last_became_active":"2024-03-11T22:15:51.183718+0000","last_became_peered":"2024-03-11T22:15:51.183718+0000","last_unstale":"2024-03-11T22:15:54.190932+0000","last_undegraded":"2024-03-11T22:15:54.190932+0000","last_fullsized":"2024-03-11T22:15:54.190932+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:15:33.497732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:33.497732+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:54.637072+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":302,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.181891+0000","last_change":"2024-03-11T22:15:51.187210+0000","last_active":"2024-03-11T22:15:54.181891+0000","last_peered":"2024-03-11T22:15:54.181891+0000","last_clean":"2024-03-11T22:15:54.181891+0000","last_became_active":"2024-03-11T22:15:51.186074+0000","last_became_peered":"2024-03-11T22:15:51.186074+0000","last_unstale":"2024-03-11T22:15:54.181891+0000","last_undegraded":"2024-03-11T22:15:54.181891+0000","last_fullsized":"2024-03-11T22:15:54.181891+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:26.481122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:26.481122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:56.633401+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":62,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.180632+0000","last_change":"2024-03-11T22:15:12.339167+0000","last_active":"2024-03-11T22:15:54.180632+0000","last_peered":"2024-03-11T22:15:54.180632+0000","last_clean":"2024-03-11T22:15:54.180632+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:15:54.180632+0000","last_undegraded":"2024-03-11T22:15:54.180632+0000","last_fullsized":"2024-03-11T22:15:54.180632+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:12.339050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:12.339050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:15.388005+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":500,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.186931+0000","last_change":"2024-03-11T22:15:04.043741+0000","last_active":"2024-03-11T22:15:54.186931+0000","last_peered":"2024-03-11T22:15:54.186931+0000","last_clean":"2024-03-11T22:15:54.186931+0000","last_became_active":"2024-03-11T22:14:52.240971+0000","last_became_peered":"2024-03-11T22:14:52.240971+0000","last_unstale":"2024-03-11T22:15:54.186931+0000","last_undegraded":"2024-03-11T22:15:54.186931+0000","last_fullsized":"2024-03-11T22:15:54.186931+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:04.043590+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:04.043590+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:13.992528+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"199'5665","reported_seq":9383,"reported_epoch":199,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T22:16:00.202954+0000","last_change":"2024-03-11T22:15:52.238076+0000","last_active":"2024-03-11T22:16:00.202954+0000","last_peered":"2024-03-11T22:16:00.202954+0000","last_clean":"2024-03-11T22:15:50.323325+0000","last_became_active":"2024-03-11T22:15:52.223886+0000","last_became_peered":"2024-03-11T22:15:52.223886+0000","last_unstale":"2024-03-11T22:16:00.202954+0000","last_undegraded":"2024-03-11T22:16:00.202954+0000","last_fullsized":"2024-03-11T22:16:00.202954+0000","mapping_epoch":194,"log_start":"181'1344","ondisk_log_start":"181'1344","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371261440,"num_objects":5665,"num_object_clones":0,"num_object_copies":16995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3728,"num_objects_unfound":0,"num_objects_dirty":5665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5665,"num_write_kb":362560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1827,"num_bytes_recovered":119734272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,7],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":5665}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":385,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.191100+0000","last_change":"2024-03-11T22:15:31.519229+0000","last_active":"2024-03-11T22:15:54.191100+0000","last_peered":"2024-03-11T22:15:54.191100+0000","last_clean":"2024-03-11T22:15:54.191100+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:15:54.191100+0000","last_undegraded":"2024-03-11T22:15:54.191100+0000","last_fullsized":"2024-03-11T22:15:54.191100+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:15:31.513336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:31.513336+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:55.692490+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":106,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.198930+0000","last_change":"2024-03-11T22:15:36.626072+0000","last_active":"2024-03-11T22:15:57.198930+0000","last_peered":"2024-03-11T22:15:57.198930+0000","last_clean":"2024-03-11T22:15:57.198930+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:15:57.198930+0000","last_undegraded":"2024-03-11T22:15:57.198930+0000","last_fullsized":"2024-03-11T22:15:57.198930+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:15:36.625916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:36.625916+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:58.734572+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":297,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.189068+0000","last_change":"2024-03-11T22:15:08.820614+0000","last_active":"2024-03-11T22:15:54.189068+0000","last_peered":"2024-03-11T22:15:54.189068+0000","last_clean":"2024-03-11T22:15:54.189068+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:15:54.189068+0000","last_undegraded":"2024-03-11T22:15:54.189068+0000","last_fullsized":"2024-03-11T22:15:54.189068+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:15:08.820296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:08.820296+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:20.880401+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":238,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.200516+0000","last_change":"2024-03-11T22:14:33.134910+0000","last_active":"2024-03-11T22:15:57.200516+0000","last_peered":"2024-03-11T22:15:57.200516+0000","last_clean":"2024-03-11T22:15:57.200516+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:15:57.200516+0000","last_undegraded":"2024-03-11T22:15:57.200516+0000","last_fullsized":"2024-03-11T22:15:57.200516+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:33.134806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:14:33.134806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":578,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.193467+0000","last_change":"2024-03-11T22:15:54.193467+0000","last_active":"2024-03-11T22:15:54.193467+0000","last_peered":"2024-03-11T22:15:54.193467+0000","last_clean":"2024-03-11T22:15:54.193467+0000","last_became_active":"2024-03-11T22:15:54.193112+0000","last_became_peered":"2024-03-11T22:15:54.193112+0000","last_unstale":"2024-03-11T22:15:54.193467+0000","last_undegraded":"2024-03-11T22:15:54.193467+0000","last_fullsized":"2024-03-11T22:15:54.193467+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.243267+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.243267+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:22.005211+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"197'5240","reported_seq":5294,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:55.711724+0000","last_change":"2024-03-11T22:14:52.242543+0000","last_active":"2024-03-11T22:15:55.711724+0000","last_peered":"2024-03-11T22:15:55.711724+0000","last_clean":"2024-03-11T22:15:55.711724+0000","last_became_active":"2024-03-11T22:14:52.240224+0000","last_became_peered":"2024-03-11T22:14:52.240224+0000","last_unstale":"2024-03-11T22:15:55.711724+0000","last_undegraded":"2024-03-11T22:15:55.711724+0000","last_fullsized":"2024-03-11T22:15:55.711724+0000","mapping_epoch":175,"log_start":"180'529","ondisk_log_start":"180'529","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343408640,"num_objects":5240,"num_object_clones":0,"num_object_copies":15720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5240,"num_write_kb":335360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":385,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.190801+0000","last_change":"2024-03-11T22:14:27.105480+0000","last_active":"2024-03-11T22:15:54.190801+0000","last_peered":"2024-03-11T22:15:54.190801+0000","last_clean":"2024-03-11T22:15:54.190801+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:15:54.190801+0000","last_undegraded":"2024-03-11T22:15:54.190801+0000","last_fullsized":"2024-03-11T22:15:54.190801+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:14:27.105394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:27.105394+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:56.745563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":319,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.197981+0000","last_change":"2024-03-11T22:15:33.327216+0000","last_active":"2024-03-11T22:15:57.197981+0000","last_peered":"2024-03-11T22:15:57.197981+0000","last_clean":"2024-03-11T22:15:57.197981+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:15:57.197981+0000","last_undegraded":"2024-03-11T22:15:57.197981+0000","last_fullsized":"2024-03-11T22:15:57.197981+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:15:33.327074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:33.327074+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:35.844295+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":308,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199251+0000","last_change":"2024-03-11T22:15:24.618272+0000","last_active":"2024-03-11T22:15:57.199251+0000","last_peered":"2024-03-11T22:15:57.199251+0000","last_clean":"2024-03-11T22:15:57.199251+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:15:57.199251+0000","last_undegraded":"2024-03-11T22:15:57.199251+0000","last_fullsized":"2024-03-11T22:15:57.199251+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:24.618157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:24.618157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:38.099831+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":284,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.184351+0000","last_change":"2024-03-11T22:15:26.900176+0000","last_active":"2024-03-11T22:15:54.184351+0000","last_peered":"2024-03-11T22:15:54.184351+0000","last_clean":"2024-03-11T22:15:54.184351+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:15:54.184351+0000","last_undegraded":"2024-03-11T22:15:54.184351+0000","last_fullsized":"2024-03-11T22:15:54.184351+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:26.899954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:26.899954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:31.796881+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":470,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201146+0000","last_change":"2024-03-11T22:15:54.202839+0000","last_active":"2024-03-11T22:15:57.201146+0000","last_peered":"2024-03-11T22:15:57.201146+0000","last_clean":"2024-03-11T22:15:57.201146+0000","last_became_active":"2024-03-11T22:15:54.202497+0000","last_became_peered":"2024-03-11T22:15:54.202497+0000","last_unstale":"2024-03-11T22:15:57.201146+0000","last_undegraded":"2024-03-11T22:15:57.201146+0000","last_fullsized":"2024-03-11T22:15:57.201146+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:08.189043+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:08.189043+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:37.414091+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.f","version":"197'5358","reported_seq":5412,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:55.680780+0000","last_change":"2024-03-11T22:14:52.238556+0000","last_active":"2024-03-11T22:15:55.680780+0000","last_peered":"2024-03-11T22:15:55.680780+0000","last_clean":"2024-03-11T22:15:55.680780+0000","last_became_active":"2024-03-11T22:14:52.238241+0000","last_became_peered":"2024-03-11T22:14:52.238241+0000","last_unstale":"2024-03-11T22:15:55.680780+0000","last_undegraded":"2024-03-11T22:15:55.680780+0000","last_fullsized":"2024-03-11T22:15:55.680780+0000","mapping_epoch":175,"log_start":"181'1263","ondisk_log_start":"181'1263","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4095,"log_dups_size":0,"ondisk_log_size":4095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:08.580556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351141888,"num_objects":5358,"num_object_clones":0,"num_object_copies":16074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":342912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":300,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.200119+0000","last_change":"2024-03-11T22:14:18.961752+0000","last_active":"2024-03-11T22:15:57.200119+0000","last_peered":"2024-03-11T22:15:57.200119+0000","last_clean":"2024-03-11T22:15:57.200119+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:15:57.200119+0000","last_undegraded":"2024-03-11T22:15:57.200119+0000","last_fullsized":"2024-03-11T22:15:57.200119+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:18.961673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:14:18.961673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":328,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.190446+0000","last_change":"2024-03-11T22:15:22.846514+0000","last_active":"2024-03-11T22:15:54.190446+0000","last_peered":"2024-03-11T22:15:54.190446+0000","last_clean":"2024-03-11T22:15:54.190446+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:15:54.190446+0000","last_undegraded":"2024-03-11T22:15:54.190446+0000","last_fullsized":"2024-03-11T22:15:54.190446+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:15:22.846352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:22.846352+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:37.818474+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":300,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.196874+0000","last_change":"2024-03-11T22:14:36.079416+0000","last_active":"2024-03-11T22:15:57.196874+0000","last_peered":"2024-03-11T22:15:57.196874+0000","last_clean":"2024-03-11T22:15:57.196874+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:15:57.196874+0000","last_undegraded":"2024-03-11T22:15:57.196874+0000","last_fullsized":"2024-03-11T22:15:57.196874+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:36.079334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:36.079334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:01.240783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":296,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.196078+0000","last_change":"2024-03-11T22:15:21.454660+0000","last_active":"2024-03-11T22:15:57.196078+0000","last_peered":"2024-03-11T22:15:57.196078+0000","last_clean":"2024-03-11T22:15:57.196078+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:15:57.196078+0000","last_undegraded":"2024-03-11T22:15:57.196078+0000","last_fullsized":"2024-03-11T22:15:57.196078+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:21.454272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:21.454272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:50.718809+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":461,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201029+0000","last_change":"2024-03-11T22:15:07.207510+0000","last_active":"2024-03-11T22:15:57.201029+0000","last_peered":"2024-03-11T22:15:57.201029+0000","last_clean":"2024-03-11T22:15:57.201029+0000","last_became_active":"2024-03-11T22:14:38.166870+0000","last_became_peered":"2024-03-11T22:14:38.166870+0000","last_unstale":"2024-03-11T22:15:57.201029+0000","last_undegraded":"2024-03-11T22:15:57.201029+0000","last_fullsized":"2024-03-11T22:15:57.201029+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:07.207365+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:07.207365+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:28.797642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"199'5818","reported_seq":5876,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:16:00.487123+0000","last_change":"2024-03-11T22:14:52.247032+0000","last_active":"2024-03-11T22:16:00.487123+0000","last_peered":"2024-03-11T22:16:00.487123+0000","last_clean":"2024-03-11T22:16:00.487123+0000","last_became_active":"2024-03-11T22:14:52.246614+0000","last_became_peered":"2024-03-11T22:14:52.246614+0000","last_unstale":"2024-03-11T22:16:00.487123+0000","last_undegraded":"2024-03-11T22:16:00.487123+0000","last_fullsized":"2024-03-11T22:16:00.487123+0000","mapping_epoch":175,"log_start":"183'1861","ondisk_log_start":"183'1861","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3957,"log_dups_size":0,"ondisk_log_size":3957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381288448,"num_objects":5818,"num_object_clones":0,"num_object_copies":17454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5818,"num_write_kb":372352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":292,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.200485+0000","last_change":"2024-03-11T22:14:29.697272+0000","last_active":"2024-03-11T22:15:57.200485+0000","last_peered":"2024-03-11T22:15:57.200485+0000","last_clean":"2024-03-11T22:15:57.200485+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:15:57.200485+0000","last_undegraded":"2024-03-11T22:15:57.200485+0000","last_fullsized":"2024-03-11T22:15:57.200485+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:29.697144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:29.697144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":318,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.188549+0000","last_change":"2024-03-11T22:15:45.150431+0000","last_active":"2024-03-11T22:15:54.188549+0000","last_peered":"2024-03-11T22:15:54.188549+0000","last_clean":"2024-03-11T22:15:54.188549+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:15:54.188549+0000","last_undegraded":"2024-03-11T22:15:54.188549+0000","last_fullsized":"2024-03-11T22:15:54.188549+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:15:45.150297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:45.150297+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:46.686298+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":737,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:59.413495+0000","last_change":"2024-03-11T22:15:38.304410+0000","last_active":"2024-03-11T22:15:59.413495+0000","last_peered":"2024-03-11T22:15:59.413495+0000","last_clean":"2024-03-11T22:15:59.413495+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:15:59.413495+0000","last_undegraded":"2024-03-11T22:15:59.413495+0000","last_fullsized":"2024-03-11T22:15:59.413495+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:15:38.304286+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:15:38.304286+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:51.893991+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":414,"num_read_kb":409,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":296,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.197263+0000","last_change":"2024-03-11T22:14:35.025497+0000","last_active":"2024-03-11T22:15:54.197263+0000","last_peered":"2024-03-11T22:15:54.197263+0000","last_clean":"2024-03-11T22:15:54.197263+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:15:54.197263+0000","last_undegraded":"2024-03-11T22:15:54.197263+0000","last_fullsized":"2024-03-11T22:15:54.197263+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:35.025383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:35.025383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":304,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.196643+0000","last_change":"2024-03-11T22:15:44.401163+0000","last_active":"2024-03-11T22:15:57.196643+0000","last_peered":"2024-03-11T22:15:57.196643+0000","last_clean":"2024-03-11T22:15:57.196643+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:15:57.196643+0000","last_undegraded":"2024-03-11T22:15:57.196643+0000","last_fullsized":"2024-03-11T22:15:57.196643+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:44.401045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:44.401045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:00.692967+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":573,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.198309+0000","last_change":"2024-03-11T22:15:06.523755+0000","last_active":"2024-03-11T22:15:57.198309+0000","last_peered":"2024-03-11T22:15:57.198309+0000","last_clean":"2024-03-11T22:15:57.198309+0000","last_became_active":"2024-03-11T22:14:38.164101+0000","last_became_peered":"2024-03-11T22:14:38.164101+0000","last_unstale":"2024-03-11T22:15:57.198309+0000","last_undegraded":"2024-03-11T22:15:57.198309+0000","last_fullsized":"2024-03-11T22:15:57.198309+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.523513+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.523513+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:09.939331+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"199'5567","reported_seq":5625,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:16:00.112034+0000","last_change":"2024-03-11T22:14:52.240542+0000","last_active":"2024-03-11T22:16:00.112034+0000","last_peered":"2024-03-11T22:16:00.112034+0000","last_clean":"2024-03-11T22:16:00.112034+0000","last_became_active":"2024-03-11T22:14:52.240210+0000","last_became_peered":"2024-03-11T22:14:52.240210+0000","last_unstale":"2024-03-11T22:16:00.112034+0000","last_undegraded":"2024-03-11T22:16:00.112034+0000","last_fullsized":"2024-03-11T22:16:00.112034+0000","mapping_epoch":175,"log_start":"180'673","ondisk_log_start":"180'673","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:06.873414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364838912,"num_objects":5567,"num_object_clones":0,"num_object_copies":16701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5567,"num_write_kb":356288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":248,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.186708+0000","last_change":"2024-03-11T22:15:51.185575+0000","last_active":"2024-03-11T22:15:54.186708+0000","last_peered":"2024-03-11T22:15:54.186708+0000","last_clean":"2024-03-11T22:15:54.186708+0000","last_became_active":"2024-03-11T22:15:51.185218+0000","last_became_peered":"2024-03-11T22:15:51.185218+0000","last_unstale":"2024-03-11T22:15:54.186708+0000","last_undegraded":"2024-03-11T22:15:54.186708+0000","last_fullsized":"2024-03-11T22:15:54.186708+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:30.829461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:15:30.829461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:59.862897+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":416,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.200860+0000","last_change":"2024-03-11T22:14:37.137570+0000","last_active":"2024-03-11T22:15:57.200860+0000","last_peered":"2024-03-11T22:15:57.200860+0000","last_clean":"2024-03-11T22:15:57.200860+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:15:57.200860+0000","last_undegraded":"2024-03-11T22:15:57.200860+0000","last_fullsized":"2024-03-11T22:15:57.200860+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-11T22:14:37.137464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:37.137464+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":296,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.178889+0000","last_change":"2024-03-11T22:13:33.460800+0000","last_active":"2024-03-11T22:15:54.178889+0000","last_peered":"2024-03-11T22:15:54.178889+0000","last_clean":"2024-03-11T22:15:54.178889+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:15:54.178889+0000","last_undegraded":"2024-03-11T22:15:54.178889+0000","last_fullsized":"2024-03-11T22:15:54.178889+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:33.460308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:13:33.460308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":577,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.203009+0000","last_change":"2024-03-11T22:15:54.203009+0000","last_active":"2024-03-11T22:15:54.203009+0000","last_peered":"2024-03-11T22:15:54.203009+0000","last_clean":"2024-03-11T22:15:54.203009+0000","last_became_active":"2024-03-11T22:15:54.199718+0000","last_became_peered":"2024-03-11T22:15:54.199718+0000","last_unstale":"2024-03-11T22:15:54.203009+0000","last_undegraded":"2024-03-11T22:15:54.203009+0000","last_fullsized":"2024-03-11T22:15:54.203009+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:03.739634+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:15:03.739634+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:17.461979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":288,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199198+0000","last_change":"2024-03-11T22:13:34.478753+0000","last_active":"2024-03-11T22:15:57.199198+0000","last_peered":"2024-03-11T22:15:57.199198+0000","last_clean":"2024-03-11T22:15:57.199198+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:15:57.199198+0000","last_undegraded":"2024-03-11T22:15:57.199198+0000","last_fullsized":"2024-03-11T22:15:57.199198+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:34.478634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:34.478634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"197'5139","reported_seq":5193,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:55.661726+0000","last_change":"2024-03-11T22:14:52.240355+0000","last_active":"2024-03-11T22:15:55.661726+0000","last_peered":"2024-03-11T22:15:55.661726+0000","last_clean":"2024-03-11T22:15:55.661726+0000","last_became_active":"2024-03-11T22:14:52.238824+0000","last_became_peered":"2024-03-11T22:14:52.238824+0000","last_unstale":"2024-03-11T22:15:55.661726+0000","last_undegraded":"2024-03-11T22:15:55.661726+0000","last_fullsized":"2024-03-11T22:15:55.661726+0000","mapping_epoch":175,"log_start":"181'1041","ondisk_log_start":"181'1041","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4098,"log_dups_size":0,"ondisk_log_size":4098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:07.988297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336789504,"num_objects":5139,"num_object_clones":0,"num_object_copies":15417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5139,"num_write_kb":328896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":304,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.186311+0000","last_change":"2024-03-11T22:15:44.873768+0000","last_active":"2024-03-11T22:15:54.186311+0000","last_peered":"2024-03-11T22:15:54.186311+0000","last_clean":"2024-03-11T22:15:54.186311+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:15:54.186311+0000","last_undegraded":"2024-03-11T22:15:54.186311+0000","last_fullsized":"2024-03-11T22:15:54.186311+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:44.873386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:44.873386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:12.700596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":331,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.202476+0000","last_change":"2024-03-11T22:15:15.581703+0000","last_active":"2024-03-11T22:15:57.202476+0000","last_peered":"2024-03-11T22:15:57.202476+0000","last_clean":"2024-03-11T22:15:57.202476+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:15:57.202476+0000","last_undegraded":"2024-03-11T22:15:57.202476+0000","last_fullsized":"2024-03-11T22:15:57.202476+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-03-11T22:15:15.581545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:15.581545+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:15.895749+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":321,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.185669+0000","last_change":"2024-03-11T22:15:21.847867+0000","last_active":"2024-03-11T22:15:54.185669+0000","last_peered":"2024-03-11T22:15:54.185669+0000","last_clean":"2024-03-11T22:15:54.185669+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:15:54.185669+0000","last_undegraded":"2024-03-11T22:15:54.185669+0000","last_fullsized":"2024-03-11T22:15:54.185669+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:21.846681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:15:21.846681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.774603+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":308,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.198504+0000","last_change":"2024-03-11T22:15:18.479060+0000","last_active":"2024-03-11T22:15:57.198504+0000","last_peered":"2024-03-11T22:15:57.198504+0000","last_clean":"2024-03-11T22:15:57.198504+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:15:57.198504+0000","last_undegraded":"2024-03-11T22:15:57.198504+0000","last_fullsized":"2024-03-11T22:15:57.198504+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:18.478850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:18.478850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:25.236419+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":238,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.200819+0000","last_change":"2024-03-11T22:14:34.131682+0000","last_active":"2024-03-11T22:15:57.200819+0000","last_peered":"2024-03-11T22:15:57.200819+0000","last_clean":"2024-03-11T22:15:57.200819+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:15:57.200819+0000","last_undegraded":"2024-03-11T22:15:57.200819+0000","last_fullsized":"2024-03-11T22:15:57.200819+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.131530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:02.487163+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":585,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.187363+0000","last_change":"2024-03-11T22:15:54.187363+0000","last_active":"2024-03-11T22:15:54.187363+0000","last_peered":"2024-03-11T22:15:54.187363+0000","last_clean":"2024-03-11T22:15:54.187363+0000","last_became_active":"2024-03-11T22:15:54.186991+0000","last_became_peered":"2024-03-11T22:15:54.186991+0000","last_unstale":"2024-03-11T22:15:54.187363+0000","last_undegraded":"2024-03-11T22:15:54.187363+0000","last_fullsized":"2024-03-11T22:15:54.187363+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:04.770858+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:04.770858+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:09.014981+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"199'5708","reported_seq":5766,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:16:00.204481+0000","last_change":"2024-03-11T22:14:52.243442+0000","last_active":"2024-03-11T22:16:00.204481+0000","last_peered":"2024-03-11T22:16:00.204481+0000","last_clean":"2024-03-11T22:16:00.204481+0000","last_became_active":"2024-03-11T22:14:52.243066+0000","last_became_peered":"2024-03-11T22:14:52.243066+0000","last_unstale":"2024-03-11T22:16:00.204481+0000","last_undegraded":"2024-03-11T22:16:00.204481+0000","last_fullsized":"2024-03-11T22:16:00.204481+0000","mapping_epoch":175,"log_start":"181'1439","ondisk_log_start":"181'1439","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4269,"log_dups_size":0,"ondisk_log_size":4269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:11.959240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374079488,"num_objects":5708,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5708,"num_write_kb":365312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":296,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.188548+0000","last_change":"2024-03-11T22:15:30.922162+0000","last_active":"2024-03-11T22:15:54.188548+0000","last_peered":"2024-03-11T22:15:54.188548+0000","last_clean":"2024-03-11T22:15:54.188548+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:15:54.188548+0000","last_undegraded":"2024-03-11T22:15:54.188548+0000","last_fullsized":"2024-03-11T22:15:54.188548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:30.922001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:30.922001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:59.152783+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":442,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.197062+0000","last_change":"2024-03-11T22:15:11.547339+0000","last_active":"2024-03-11T22:15:57.197062+0000","last_peered":"2024-03-11T22:15:57.197062+0000","last_clean":"2024-03-11T22:15:57.197062+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:15:57.197062+0000","last_undegraded":"2024-03-11T22:15:57.197062+0000","last_fullsized":"2024-03-11T22:15:57.197062+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-11T22:15:11.547182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:11.547182+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:33.121742+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":63,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.190347+0000","last_change":"2024-03-11T22:15:24.899285+0000","last_active":"2024-03-11T22:15:54.190347+0000","last_peered":"2024-03-11T22:15:54.190347+0000","last_clean":"2024-03-11T22:15:54.190347+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:15:54.190347+0000","last_undegraded":"2024-03-11T22:15:54.190347+0000","last_fullsized":"2024-03-11T22:15:54.190347+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:24.899168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:24.899168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:50.013607+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":312,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201650+0000","last_change":"2024-03-11T22:14:16.662735+0000","last_active":"2024-03-11T22:15:57.201650+0000","last_peered":"2024-03-11T22:15:57.201650+0000","last_clean":"2024-03-11T22:15:57.201650+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:15:57.201650+0000","last_undegraded":"2024-03-11T22:15:57.201650+0000","last_fullsized":"2024-03-11T22:15:57.201650+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:14:16.662596+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:14:16.662596+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":290,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.186795+0000","last_change":"2024-03-11T22:15:51.192142+0000","last_active":"2024-03-11T22:15:54.186795+0000","last_peered":"2024-03-11T22:15:54.186795+0000","last_clean":"2024-03-11T22:15:54.186795+0000","last_became_active":"2024-03-11T22:15:51.191481+0000","last_became_peered":"2024-03-11T22:15:51.191481+0000","last_unstale":"2024-03-11T22:15:54.186795+0000","last_undegraded":"2024-03-11T22:15:54.186795+0000","last_fullsized":"2024-03-11T22:15:54.186795+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:32.518054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:15:32.518054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:58.828297+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":572,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.198873+0000","last_change":"2024-03-11T22:15:06.673117+0000","last_active":"2024-03-11T22:15:57.198873+0000","last_peered":"2024-03-11T22:15:57.198873+0000","last_clean":"2024-03-11T22:15:57.198873+0000","last_became_active":"2024-03-11T22:14:38.164859+0000","last_became_peered":"2024-03-11T22:14:38.164859+0000","last_unstale":"2024-03-11T22:15:57.198873+0000","last_undegraded":"2024-03-11T22:15:57.198873+0000","last_fullsized":"2024-03-11T22:15:57.198873+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.672938+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.672938+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:25.674338+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"199'5643","reported_seq":5701,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:16:00.470717+0000","last_change":"2024-03-11T22:14:52.242768+0000","last_active":"2024-03-11T22:16:00.470717+0000","last_peered":"2024-03-11T22:16:00.470717+0000","last_clean":"2024-03-11T22:16:00.470717+0000","last_became_active":"2024-03-11T22:14:52.242455+0000","last_became_peered":"2024-03-11T22:14:52.242455+0000","last_unstale":"2024-03-11T22:16:00.470717+0000","last_undegraded":"2024-03-11T22:16:00.470717+0000","last_fullsized":"2024-03-11T22:16:00.470717+0000","mapping_epoch":175,"log_start":"183'1639","ondisk_log_start":"183'1639","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4004,"log_dups_size":0,"ondisk_log_size":4004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:15.907365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369819648,"num_objects":5643,"num_object_clones":0,"num_object_copies":16929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5643,"num_write_kb":361152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":324,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.190698+0000","last_change":"2024-03-11T22:15:11.902221+0000","last_active":"2024-03-11T22:15:54.190698+0000","last_peered":"2024-03-11T22:15:54.190698+0000","last_clean":"2024-03-11T22:15:54.190698+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:15:54.190698+0000","last_undegraded":"2024-03-11T22:15:54.190698+0000","last_fullsized":"2024-03-11T22:15:54.190698+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:15:11.902080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:11.902080+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:27.107447+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":300,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201310+0000","last_change":"2024-03-11T22:14:37.615454+0000","last_active":"2024-03-11T22:15:57.201310+0000","last_peered":"2024-03-11T22:15:57.201310+0000","last_clean":"2024-03-11T22:15:57.201310+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:15:57.201310+0000","last_undegraded":"2024-03-11T22:15:57.201310+0000","last_fullsized":"2024-03-11T22:15:57.201310+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:37.615375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:37.615375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:04.760537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":296,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.185458+0000","last_change":"2024-03-11T22:15:19.936668+0000","last_active":"2024-03-11T22:15:54.185458+0000","last_peered":"2024-03-11T22:15:54.185458+0000","last_clean":"2024-03-11T22:15:54.185458+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:15:54.185458+0000","last_undegraded":"2024-03-11T22:15:54.185458+0000","last_fullsized":"2024-03-11T22:15:54.185458+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:19.936555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:15:19.936555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:35.701409+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":290,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.197454+0000","last_change":"2024-03-11T22:15:51.185584+0000","last_active":"2024-03-11T22:15:54.197454+0000","last_peered":"2024-03-11T22:15:54.197454+0000","last_clean":"2024-03-11T22:15:54.197454+0000","last_became_active":"2024-03-11T22:15:51.184473+0000","last_became_peered":"2024-03-11T22:15:51.184473+0000","last_unstale":"2024-03-11T22:15:54.197454+0000","last_undegraded":"2024-03-11T22:15:54.197454+0000","last_fullsized":"2024-03-11T22:15:54.197454+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:36.007981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:36.007981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"93'130","reported_seq":392,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.206755+0000","last_change":"2024-03-11T22:15:54.206755+0000","last_active":"2024-03-11T22:15:54.206755+0000","last_peered":"2024-03-11T22:15:54.206755+0000","last_clean":"2024-03-11T22:15:54.206755+0000","last_became_active":"2024-03-11T22:15:54.206146+0000","last_became_peered":"2024-03-11T22:15:54.206146+0000","last_unstale":"2024-03-11T22:15:54.206755+0000","last_undegraded":"2024-03-11T22:15:54.206755+0000","last_fullsized":"2024-03-11T22:15:54.206755+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:09.205116+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:09.205116+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.999298+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"197'5408","reported_seq":5462,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:55.723315+0000","last_change":"2024-03-11T22:14:52.243118+0000","last_active":"2024-03-11T22:15:55.723315+0000","last_peered":"2024-03-11T22:15:55.723315+0000","last_clean":"2024-03-11T22:15:55.723315+0000","last_became_active":"2024-03-11T22:14:52.242807+0000","last_became_peered":"2024-03-11T22:14:52.242807+0000","last_unstale":"2024-03-11T22:15:55.723315+0000","last_undegraded":"2024-03-11T22:15:55.723315+0000","last_fullsized":"2024-03-11T22:15:55.723315+0000","mapping_epoch":175,"log_start":"180'652","ondisk_log_start":"180'652","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4756,"log_dups_size":0,"ondisk_log_size":4756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:19.483709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354418688,"num_objects":5408,"num_object_clones":0,"num_object_copies":16224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5408,"num_write_kb":346112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":361,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.200084+0000","last_change":"2024-03-11T22:15:51.208759+0000","last_active":"2024-03-11T22:15:57.200084+0000","last_peered":"2024-03-11T22:15:57.200084+0000","last_clean":"2024-03-11T22:15:57.200084+0000","last_became_active":"2024-03-11T22:15:51.190082+0000","last_became_peered":"2024-03-11T22:15:51.190082+0000","last_unstale":"2024-03-11T22:15:57.200084+0000","last_undegraded":"2024-03-11T22:15:57.200084+0000","last_fullsized":"2024-03-11T22:15:57.200084+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:14:26.006872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.006872+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":245,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199334+0000","last_change":"2024-03-11T22:15:51.186287+0000","last_active":"2024-03-11T22:15:57.199334+0000","last_peered":"2024-03-11T22:15:57.199334+0000","last_clean":"2024-03-11T22:15:57.199334+0000","last_became_active":"2024-03-11T22:15:51.185946+0000","last_became_peered":"2024-03-11T22:15:51.185946+0000","last_unstale":"2024-03-11T22:15:57.199334+0000","last_undegraded":"2024-03-11T22:15:57.199334+0000","last_fullsized":"2024-03-11T22:15:57.199334+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.649172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.649172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":296,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.185865+0000","last_change":"2024-03-11T22:15:18.917780+0000","last_active":"2024-03-11T22:15:54.185865+0000","last_peered":"2024-03-11T22:15:54.185865+0000","last_clean":"2024-03-11T22:15:54.185865+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:15:54.185865+0000","last_undegraded":"2024-03-11T22:15:54.185865+0000","last_fullsized":"2024-03-11T22:15:54.185865+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:18.917625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.710319+0000","last_clean_scrub_stamp":"2024-03-11T22:15:18.917625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:31.494091+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":248,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.185178+0000","last_change":"2024-03-11T22:15:51.186904+0000","last_active":"2024-03-11T22:15:54.185178+0000","last_peered":"2024-03-11T22:15:54.185178+0000","last_clean":"2024-03-11T22:15:54.185178+0000","last_became_active":"2024-03-11T22:15:51.186315+0000","last_became_peered":"2024-03-11T22:15:51.186315+0000","last_unstale":"2024-03-11T22:15:54.185178+0000","last_undegraded":"2024-03-11T22:15:54.185178+0000","last_fullsized":"2024-03-11T22:15:54.185178+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:37.928924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:37.928924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:55.758297+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"93'130","reported_seq":468,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.200935+0000","last_change":"2024-03-11T22:15:06.157130+0000","last_active":"2024-03-11T22:15:57.200935+0000","last_peered":"2024-03-11T22:15:57.200935+0000","last_clean":"2024-03-11T22:15:57.200935+0000","last_became_active":"2024-03-11T22:14:43.195433+0000","last_became_peered":"2024-03-11T22:14:43.195433+0000","last_unstale":"2024-03-11T22:15:57.200935+0000","last_undegraded":"2024-03-11T22:15:57.200935+0000","last_fullsized":"2024-03-11T22:15:57.200935+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.157005+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.157005+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:06.486166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"199'5610","reported_seq":5668,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:16:00.163660+0000","last_change":"2024-03-11T22:14:52.240970+0000","last_active":"2024-03-11T22:16:00.163660+0000","last_peered":"2024-03-11T22:16:00.163660+0000","last_clean":"2024-03-11T22:16:00.163660+0000","last_became_active":"2024-03-11T22:14:52.240614+0000","last_became_peered":"2024-03-11T22:14:52.240614+0000","last_unstale":"2024-03-11T22:16:00.163660+0000","last_undegraded":"2024-03-11T22:16:00.163660+0000","last_fullsized":"2024-03-11T22:16:00.163660+0000","mapping_epoch":175,"log_start":"180'712","ondisk_log_start":"180'712","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4898,"log_dups_size":0,"ondisk_log_size":4898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:17.716033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367656960,"num_objects":5610,"num_object_clones":0,"num_object_copies":16830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5610,"num_write_kb":359040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":296,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.185266+0000","last_change":"2024-03-11T22:14:24.051145+0000","last_active":"2024-03-11T22:15:54.185266+0000","last_peered":"2024-03-11T22:15:54.185266+0000","last_clean":"2024-03-11T22:15:54.185266+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:15:54.185266+0000","last_undegraded":"2024-03-11T22:15:54.185266+0000","last_fullsized":"2024-03-11T22:15:54.185266+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:24.051020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:14:24.051020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:56.893738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":115,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199217+0000","last_change":"2024-03-11T22:15:11.678247+0000","last_active":"2024-03-11T22:15:57.199217+0000","last_peered":"2024-03-11T22:15:57.199217+0000","last_clean":"2024-03-11T22:15:57.199217+0000","last_became_active":"2024-03-11T22:14:52.247400+0000","last_became_peered":"2024-03-11T22:14:52.247400+0000","last_unstale":"2024-03-11T22:15:57.199217+0000","last_undegraded":"2024-03-11T22:15:57.199217+0000","last_fullsized":"2024-03-11T22:15:57.199217+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:15:11.678114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:11.678114+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:38.522520+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":76,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.196581+0000","last_change":"2024-03-11T22:15:51.187301+0000","last_active":"2024-03-11T22:15:57.196581+0000","last_peered":"2024-03-11T22:15:57.196581+0000","last_clean":"2024-03-11T22:15:57.196581+0000","last_became_active":"2024-03-11T22:15:51.186974+0000","last_became_peered":"2024-03-11T22:15:51.186974+0000","last_unstale":"2024-03-11T22:15:57.196581+0000","last_undegraded":"2024-03-11T22:15:57.196581+0000","last_fullsized":"2024-03-11T22:15:57.196581+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:08.477912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:08.477912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:29.966511+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":296,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.181145+0000","last_change":"2024-03-11T22:15:51.203126+0000","last_active":"2024-03-11T22:15:54.181145+0000","last_peered":"2024-03-11T22:15:54.181145+0000","last_clean":"2024-03-11T22:15:54.181145+0000","last_became_active":"2024-03-11T22:15:51.191693+0000","last_became_peered":"2024-03-11T22:15:51.191693+0000","last_unstale":"2024-03-11T22:15:54.181145+0000","last_undegraded":"2024-03-11T22:15:54.181145+0000","last_fullsized":"2024-03-11T22:15:54.181145+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:15:40.678629+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:15:40.678629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:57.268143+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"93'130","reported_seq":582,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.187130+0000","last_change":"2024-03-11T22:15:05.249082+0000","last_active":"2024-03-11T22:15:54.187130+0000","last_peered":"2024-03-11T22:15:54.187130+0000","last_clean":"2024-03-11T22:15:54.187130+0000","last_became_active":"2024-03-11T22:14:43.196117+0000","last_became_peered":"2024-03-11T22:14:43.196117+0000","last_unstale":"2024-03-11T22:15:54.187130+0000","last_undegraded":"2024-03-11T22:15:54.187130+0000","last_fullsized":"2024-03-11T22:15:54.187130+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:05.248940+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.248940+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:24.043494+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"199'5713","reported_seq":5771,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:16:00.188499+0000","last_change":"2024-03-11T22:14:52.249598+0000","last_active":"2024-03-11T22:16:00.188499+0000","last_peered":"2024-03-11T22:16:00.188499+0000","last_clean":"2024-03-11T22:16:00.188499+0000","last_became_active":"2024-03-11T22:14:52.248793+0000","last_became_peered":"2024-03-11T22:14:52.248793+0000","last_unstale":"2024-03-11T22:16:00.188499+0000","last_undegraded":"2024-03-11T22:16:00.188499+0000","last_fullsized":"2024-03-11T22:16:00.188499+0000","mapping_epoch":175,"log_start":"181'1388","ondisk_log_start":"181'1388","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4325,"log_dups_size":0,"ondisk_log_size":4325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:16.661255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374407168,"num_objects":5713,"num_object_clones":0,"num_object_copies":17139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5713,"num_write_kb":365632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"194'10","reported_seq":340,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.200060+0000","last_change":"2024-03-11T22:15:51.199307+0000","last_active":"2024-03-11T22:15:57.200060+0000","last_peered":"2024-03-11T22:15:57.200060+0000","last_clean":"2024-03-11T22:15:57.200060+0000","last_became_active":"2024-03-11T22:15:51.190539+0000","last_became_peered":"2024-03-11T22:15:51.190539+0000","last_unstale":"2024-03-11T22:15:57.200060+0000","last_undegraded":"2024-03-11T22:15:57.200060+0000","last_fullsized":"2024-03-11T22:15:57.200060+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'9","last_scrub_stamp":"2024-03-11T22:14:36.152067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:36.152067+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"91'22","reported_seq":358,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.198294+0000","last_change":"2024-03-11T22:15:51.200837+0000","last_active":"2024-03-11T22:15:57.198294+0000","last_peered":"2024-03-11T22:15:57.198294+0000","last_clean":"2024-03-11T22:15:57.198294+0000","last_became_active":"2024-03-11T22:15:51.187016+0000","last_became_peered":"2024-03-11T22:15:51.187016+0000","last_unstale":"2024-03-11T22:15:57.198294+0000","last_undegraded":"2024-03-11T22:15:57.198294+0000","last_fullsized":"2024-03-11T22:15:57.198294+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'22","last_scrub_stamp":"2024-03-11T22:15:25.618938+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:15:25.618938+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:53.118409+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":84,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.189143+0000","last_change":"2024-03-11T22:15:51.196449+0000","last_active":"2024-03-11T22:15:54.189143+0000","last_peered":"2024-03-11T22:15:54.189143+0000","last_clean":"2024-03-11T22:15:54.189143+0000","last_became_active":"2024-03-11T22:15:51.184840+0000","last_became_peered":"2024-03-11T22:15:51.184840+0000","last_unstale":"2024-03-11T22:15:54.189143+0000","last_undegraded":"2024-03-11T22:15:54.189143+0000","last_fullsized":"2024-03-11T22:15:54.189143+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:15:15.946740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:15.946740+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:44.748212+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":288,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201714+0000","last_change":"2024-03-11T22:15:21.641164+0000","last_active":"2024-03-11T22:15:57.201714+0000","last_peered":"2024-03-11T22:15:57.201714+0000","last_clean":"2024-03-11T22:15:57.201714+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:15:57.201714+0000","last_undegraded":"2024-03-11T22:15:57.201714+0000","last_fullsized":"2024-03-11T22:15:57.201714+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:21.641053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:21.641053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:37.378649+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"93'130","reported_seq":580,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.195821+0000","last_change":"2024-03-11T22:15:54.205126+0000","last_active":"2024-03-11T22:15:57.195821+0000","last_peered":"2024-03-11T22:15:57.195821+0000","last_clean":"2024-03-11T22:15:57.195821+0000","last_became_active":"2024-03-11T22:15:54.204174+0000","last_became_peered":"2024-03-11T22:15:54.204174+0000","last_unstale":"2024-03-11T22:15:57.195821+0000","last_undegraded":"2024-03-11T22:15:57.195821+0000","last_fullsized":"2024-03-11T22:15:57.195821+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:07.481870+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:07.481870+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:19.825589+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":315,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.186022+0000","last_change":"2024-03-11T22:15:51.197712+0000","last_active":"2024-03-11T22:15:54.186022+0000","last_peered":"2024-03-11T22:15:54.186022+0000","last_clean":"2024-03-11T22:15:54.186022+0000","last_became_active":"2024-03-11T22:15:51.186719+0000","last_became_peered":"2024-03-11T22:15:51.186719+0000","last_unstale":"2024-03-11T22:15:54.186022+0000","last_undegraded":"2024-03-11T22:15:54.186022+0000","last_fullsized":"2024-03-11T22:15:54.186022+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:15:27.448002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:27.448002+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:54.939322+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":308,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199601+0000","last_change":"2024-03-11T22:15:35.101348+0000","last_active":"2024-03-11T22:15:57.199601+0000","last_peered":"2024-03-11T22:15:57.199601+0000","last_clean":"2024-03-11T22:15:57.199601+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:15:57.199601+0000","last_undegraded":"2024-03-11T22:15:57.199601+0000","last_fullsized":"2024-03-11T22:15:57.199601+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:35.101214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:15:35.101214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:35.754390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":300,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.198924+0000","last_change":"2024-03-11T22:15:41.760176+0000","last_active":"2024-03-11T22:15:57.198924+0000","last_peered":"2024-03-11T22:15:57.198924+0000","last_clean":"2024-03-11T22:15:57.198924+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:15:57.198924+0000","last_undegraded":"2024-03-11T22:15:57.198924+0000","last_fullsized":"2024-03-11T22:15:57.198924+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:41.760044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:07.195289+0000","last_clean_scrub_stamp":"2024-03-11T22:15:41.760044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:54.196637+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":238,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.200916+0000","last_change":"2024-03-11T22:15:39.142832+0000","last_active":"2024-03-11T22:15:57.200916+0000","last_peered":"2024-03-11T22:15:57.200916+0000","last_clean":"2024-03-11T22:15:57.200916+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:15:57.200916+0000","last_undegraded":"2024-03-11T22:15:57.200916+0000","last_fullsized":"2024-03-11T22:15:57.200916+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:39.142690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:15:39.142690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:46.639107+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"93'130","reported_seq":581,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201439+0000","last_change":"2024-03-11T22:15:05.687130+0000","last_active":"2024-03-11T22:15:57.201439+0000","last_peered":"2024-03-11T22:15:57.201439+0000","last_clean":"2024-03-11T22:15:57.201439+0000","last_became_active":"2024-03-11T22:14:43.194413+0000","last_became_peered":"2024-03-11T22:14:43.194413+0000","last_unstale":"2024-03-11T22:15:57.201439+0000","last_undegraded":"2024-03-11T22:15:57.201439+0000","last_fullsized":"2024-03-11T22:15:57.201439+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:05.687010+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.687010+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:20.099917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"93'130","reported_seq":580,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.197128+0000","last_change":"2024-03-11T22:15:05.403493+0000","last_active":"2024-03-11T22:15:57.197128+0000","last_peered":"2024-03-11T22:15:57.197128+0000","last_clean":"2024-03-11T22:15:57.197128+0000","last_became_active":"2024-03-11T22:14:43.198216+0000","last_became_peered":"2024-03-11T22:14:43.198216+0000","last_unstale":"2024-03-11T22:15:57.197128+0000","last_undegraded":"2024-03-11T22:15:57.197128+0000","last_fullsized":"2024-03-11T22:15:57.197128+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:05.403372+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.403372+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:06.652225+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"93'13","reported_seq":328,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.189253+0000","last_change":"2024-03-11T22:15:51.198003+0000","last_active":"2024-03-11T22:15:54.189253+0000","last_peered":"2024-03-11T22:15:54.189253+0000","last_clean":"2024-03-11T22:15:54.189253+0000","last_became_active":"2024-03-11T22:15:51.183890+0000","last_became_peered":"2024-03-11T22:15:51.183890+0000","last_unstale":"2024-03-11T22:15:54.189253+0000","last_undegraded":"2024-03-11T22:15:54.189253+0000","last_fullsized":"2024-03-11T22:15:54.189253+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:14:26.127737+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.127737+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":70,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199907+0000","last_change":"2024-03-11T22:15:43.115122+0000","last_active":"2024-03-11T22:15:57.199907+0000","last_peered":"2024-03-11T22:15:57.199907+0000","last_clean":"2024-03-11T22:15:57.199907+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:15:57.199907+0000","last_undegraded":"2024-03-11T22:15:57.199907+0000","last_fullsized":"2024-03-11T22:15:57.199907+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:43.114976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:43.114976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:59.330233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":294,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.195459+0000","last_change":"2024-03-11T22:15:51.187935+0000","last_active":"2024-03-11T22:15:54.195459+0000","last_peered":"2024-03-11T22:15:54.195459+0000","last_clean":"2024-03-11T22:15:54.195459+0000","last_became_active":"2024-03-11T22:15:51.187605+0000","last_became_peered":"2024-03-11T22:15:51.187605+0000","last_unstale":"2024-03-11T22:15:54.195459+0000","last_undegraded":"2024-03-11T22:15:54.195459+0000","last_fullsized":"2024-03-11T22:15:54.195459+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.850128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.850128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:56.633437+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":292,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.186564+0000","last_change":"2024-03-11T22:14:19.987755+0000","last_active":"2024-03-11T22:15:54.186564+0000","last_peered":"2024-03-11T22:15:54.186564+0000","last_clean":"2024-03-11T22:15:54.186564+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:15:54.186564+0000","last_undegraded":"2024-03-11T22:15:54.186564+0000","last_fullsized":"2024-03-11T22:15:54.186564+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:19.987672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:19.987672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"188'13","reported_seq":324,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199390+0000","last_change":"2024-03-11T22:15:29.636731+0000","last_active":"2024-03-11T22:15:57.199390+0000","last_peered":"2024-03-11T22:15:57.199390+0000","last_clean":"2024-03-11T22:15:57.199390+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:15:57.199390+0000","last_undegraded":"2024-03-11T22:15:57.199390+0000","last_fullsized":"2024-03-11T22:15:57.199390+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'13","last_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.636639+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:36.914657+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":302,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.184417+0000","last_change":"2024-03-11T22:15:51.182980+0000","last_active":"2024-03-11T22:15:54.184417+0000","last_peered":"2024-03-11T22:15:54.184417+0000","last_clean":"2024-03-11T22:15:54.184417+0000","last_became_active":"2024-03-11T22:15:51.182535+0000","last_became_peered":"2024-03-11T22:15:51.182535+0000","last_unstale":"2024-03-11T22:15:54.184417+0000","last_undegraded":"2024-03-11T22:15:54.184417+0000","last_fullsized":"2024-03-11T22:15:54.184417+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:33.012017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:33.012017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"194'9","reported_seq":399,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.187995+0000","last_change":"2024-03-11T22:15:51.197565+0000","last_active":"2024-03-11T22:15:54.187995+0000","last_peered":"2024-03-11T22:15:54.187995+0000","last_clean":"2024-03-11T22:15:54.187995+0000","last_became_active":"2024-03-11T22:15:51.189032+0000","last_became_peered":"2024-03-11T22:15:51.189032+0000","last_unstale":"2024-03-11T22:15:54.187995+0000","last_undegraded":"2024-03-11T22:15:54.187995+0000","last_fullsized":"2024-03-11T22:15:54.187995+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:14:26.011319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.011319+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:56.633456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":62,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.186086+0000","last_change":"2024-03-11T22:15:12.007160+0000","last_active":"2024-03-11T22:15:54.186086+0000","last_peered":"2024-03-11T22:15:54.186086+0000","last_clean":"2024-03-11T22:15:54.186086+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:15:54.186086+0000","last_undegraded":"2024-03-11T22:15:54.186086+0000","last_fullsized":"2024-03-11T22:15:54.186086+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:12.007023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:12.007023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.489583+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"93'130","reported_seq":574,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.198648+0000","last_change":"2024-03-11T22:15:07.646561+0000","last_active":"2024-03-11T22:15:57.198648+0000","last_peered":"2024-03-11T22:15:57.198648+0000","last_clean":"2024-03-11T22:15:57.198648+0000","last_became_active":"2024-03-11T22:14:43.194350+0000","last_became_peered":"2024-03-11T22:14:43.194350+0000","last_unstale":"2024-03-11T22:15:57.198648+0000","last_undegraded":"2024-03-11T22:15:57.198648+0000","last_fullsized":"2024-03-11T22:15:57.198648+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:07.646448+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:15:07.646448+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:30.541086+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":307,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.187905+0000","last_change":"2024-03-11T22:15:29.460432+0000","last_active":"2024-03-11T22:15:54.187905+0000","last_peered":"2024-03-11T22:15:54.187905+0000","last_clean":"2024-03-11T22:15:54.187905+0000","last_became_active":"2024-03-11T22:08:35.085797+0000","last_became_peered":"2024-03-11T22:08:35.085797+0000","last_unstale":"2024-03-11T22:15:54.187905+0000","last_undegraded":"2024-03-11T22:15:54.187905+0000","last_fullsized":"2024-03-11T22:15:54.187905+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:15:29.460297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.460297+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:45.658272+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":72,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.185768+0000","last_change":"2024-03-11T22:15:51.186696+0000","last_active":"2024-03-11T22:15:54.185768+0000","last_peered":"2024-03-11T22:15:54.185768+0000","last_clean":"2024-03-11T22:15:54.185768+0000","last_became_active":"2024-03-11T22:15:51.186355+0000","last_became_peered":"2024-03-11T22:15:51.186355+0000","last_unstale":"2024-03-11T22:15:54.185768+0000","last_undegraded":"2024-03-11T22:15:54.185768+0000","last_fullsized":"2024-03-11T22:15:54.185768+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:07.967713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:07.967713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.395034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":398,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199202+0000","last_change":"2024-03-11T22:15:31.661867+0000","last_active":"2024-03-11T22:15:57.199202+0000","last_peered":"2024-03-11T22:15:57.199202+0000","last_clean":"2024-03-11T22:15:57.199202+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:15:57.199202+0000","last_undegraded":"2024-03-11T22:15:57.199202+0000","last_fullsized":"2024-03-11T22:15:57.199202+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:15:31.661660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:31.661660+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:43.545860+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":282,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.188487+0000","last_change":"2024-03-11T22:15:51.189902+0000","last_active":"2024-03-11T22:15:54.188487+0000","last_peered":"2024-03-11T22:15:54.188487+0000","last_clean":"2024-03-11T22:15:54.188487+0000","last_became_active":"2024-03-11T22:15:51.188000+0000","last_became_peered":"2024-03-11T22:15:51.188000+0000","last_unstale":"2024-03-11T22:15:54.188487+0000","last_undegraded":"2024-03-11T22:15:54.188487+0000","last_fullsized":"2024-03-11T22:15:54.188487+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.969572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.969572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"93'130","reported_seq":526,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.197012+0000","last_change":"2024-03-11T22:15:06.779041+0000","last_active":"2024-03-11T22:15:54.197012+0000","last_peered":"2024-03-11T22:15:54.197012+0000","last_clean":"2024-03-11T22:15:54.197012+0000","last_became_active":"2024-03-11T22:14:43.191297+0000","last_became_peered":"2024-03-11T22:14:43.191297+0000","last_unstale":"2024-03-11T22:15:54.197012+0000","last_undegraded":"2024-03-11T22:15:54.197012+0000","last_fullsized":"2024-03-11T22:15:54.197012+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.778786+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.778786+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.184868+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":104,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.200382+0000","last_change":"2024-03-11T22:15:44.126540+0000","last_active":"2024-03-11T22:15:57.200382+0000","last_peered":"2024-03-11T22:15:57.200382+0000","last_clean":"2024-03-11T22:15:57.200382+0000","last_became_active":"2024-03-11T22:14:52.239810+0000","last_became_peered":"2024-03-11T22:14:52.239810+0000","last_unstale":"2024-03-11T22:15:57.200382+0000","last_undegraded":"2024-03-11T22:15:57.200382+0000","last_fullsized":"2024-03-11T22:15:57.200382+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:15:44.126413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:44.126413+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:01.968842+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":85,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.200488+0000","last_change":"2024-03-11T22:15:51.191241+0000","last_active":"2024-03-11T22:15:57.200488+0000","last_peered":"2024-03-11T22:15:57.200488+0000","last_clean":"2024-03-11T22:15:57.200488+0000","last_became_active":"2024-03-11T22:15:51.185312+0000","last_became_peered":"2024-03-11T22:15:51.185312+0000","last_unstale":"2024-03-11T22:15:57.200488+0000","last_undegraded":"2024-03-11T22:15:57.200488+0000","last_fullsized":"2024-03-11T22:15:57.200488+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:15:26.154167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:26.154167+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:56.069284+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":300,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.197892+0000","last_change":"2024-03-11T22:15:32.310641+0000","last_active":"2024-03-11T22:15:57.197892+0000","last_peered":"2024-03-11T22:15:57.197892+0000","last_clean":"2024-03-11T22:15:57.197892+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:15:57.197892+0000","last_undegraded":"2024-03-11T22:15:57.197892+0000","last_fullsized":"2024-03-11T22:15:57.197892+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:32.310504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:32.310504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:37.531286+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":276,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.179478+0000","last_change":"2024-03-11T22:12:46.492608+0000","last_active":"2024-03-11T22:15:54.179478+0000","last_peered":"2024-03-11T22:15:54.179478+0000","last_clean":"2024-03-11T22:15:54.179478+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:15:54.179478+0000","last_undegraded":"2024-03-11T22:15:54.179478+0000","last_fullsized":"2024-03-11T22:15:54.179478+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.492494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.492494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":320,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.198075+0000","last_change":"2024-03-11T22:15:24.432542+0000","last_active":"2024-03-11T22:15:57.198075+0000","last_peered":"2024-03-11T22:15:57.198075+0000","last_clean":"2024-03-11T22:15:57.198075+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:15:57.198075+0000","last_undegraded":"2024-03-11T22:15:57.198075+0000","last_fullsized":"2024-03-11T22:15:57.198075+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:15:24.432264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:24.432264+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:45.938006+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":233,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.181691+0000","last_change":"2024-03-11T22:15:51.190683+0000","last_active":"2024-03-11T22:15:54.181691+0000","last_peered":"2024-03-11T22:15:54.181691+0000","last_clean":"2024-03-11T22:15:54.181691+0000","last_became_active":"2024-03-11T22:15:51.189292+0000","last_became_peered":"2024-03-11T22:15:51.189292+0000","last_unstale":"2024-03-11T22:15:54.181691+0000","last_undegraded":"2024-03-11T22:15:54.181691+0000","last_fullsized":"2024-03-11T22:15:54.181691+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:21.856963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.856963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:56.633473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":288,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.180948+0000","last_change":"2024-03-11T22:14:15.858384+0000","last_active":"2024-03-11T22:15:54.180948+0000","last_peered":"2024-03-11T22:15:54.180948+0000","last_clean":"2024-03-11T22:15:54.180948+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:15:54.180948+0000","last_undegraded":"2024-03-11T22:15:54.180948+0000","last_fullsized":"2024-03-11T22:15:54.180948+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:15.857082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:14:15.857082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":284,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.184234+0000","last_change":"2024-03-11T22:15:10.896807+0000","last_active":"2024-03-11T22:15:54.184234+0000","last_peered":"2024-03-11T22:15:54.184234+0000","last_clean":"2024-03-11T22:15:54.184234+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:15:54.184234+0000","last_undegraded":"2024-03-11T22:15:54.184234+0000","last_fullsized":"2024-03-11T22:15:54.184234+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:10.896676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:10.896676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:16.899428+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"93'130","reported_seq":571,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.186121+0000","last_change":"2024-03-11T22:15:06.951256+0000","last_active":"2024-03-11T22:15:54.186121+0000","last_peered":"2024-03-11T22:15:54.186121+0000","last_clean":"2024-03-11T22:15:54.186121+0000","last_became_active":"2024-03-11T22:14:43.194955+0000","last_became_peered":"2024-03-11T22:14:43.194955+0000","last_unstale":"2024-03-11T22:15:54.186121+0000","last_undegraded":"2024-03-11T22:15:54.186121+0000","last_fullsized":"2024-03-11T22:15:54.186121+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.951073+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.951073+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.482728+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":321,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.179812+0000","last_change":"2024-03-11T22:15:51.204960+0000","last_active":"2024-03-11T22:15:54.179812+0000","last_peered":"2024-03-11T22:15:54.179812+0000","last_clean":"2024-03-11T22:15:54.179812+0000","last_became_active":"2024-03-11T22:15:51.191282+0000","last_became_peered":"2024-03-11T22:15:51.191282+0000","last_unstale":"2024-03-11T22:15:54.179812+0000","last_undegraded":"2024-03-11T22:15:54.179812+0000","last_fullsized":"2024-03-11T22:15:54.179812+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:13:18.486039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:18.486039+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:56.633421+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":314,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.198934+0000","last_change":"2024-03-11T22:15:51.184445+0000","last_active":"2024-03-11T22:15:57.198934+0000","last_peered":"2024-03-11T22:15:57.198934+0000","last_clean":"2024-03-11T22:15:57.198934+0000","last_became_active":"2024-03-11T22:15:51.184115+0000","last_became_peered":"2024-03-11T22:15:51.184115+0000","last_unstale":"2024-03-11T22:15:57.198934+0000","last_undegraded":"2024-03-11T22:15:57.198934+0000","last_fullsized":"2024-03-11T22:15:57.198934+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:45.801375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:45.801375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:57.764800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":69,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199585+0000","last_change":"2024-03-11T22:15:17.246483+0000","last_active":"2024-03-11T22:15:57.199585+0000","last_peered":"2024-03-11T22:15:57.199585+0000","last_clean":"2024-03-11T22:15:57.199585+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:15:57.199585+0000","last_undegraded":"2024-03-11T22:15:57.199585+0000","last_fullsized":"2024-03-11T22:15:57.199585+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:17.246407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:15:17.246407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:22.492312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":284,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.181254+0000","last_change":"2024-03-11T22:14:23.930193+0000","last_active":"2024-03-11T22:15:54.181254+0000","last_peered":"2024-03-11T22:15:54.181254+0000","last_clean":"2024-03-11T22:15:54.181254+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:15:54.181254+0000","last_undegraded":"2024-03-11T22:15:54.181254+0000","last_fullsized":"2024-03-11T22:15:54.181254+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.930091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.930091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":310,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.190716+0000","last_change":"2024-03-11T22:13:31.491151+0000","last_active":"2024-03-11T22:15:54.190716+0000","last_peered":"2024-03-11T22:15:54.190716+0000","last_clean":"2024-03-11T22:15:54.190716+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:15:54.190716+0000","last_undegraded":"2024-03-11T22:15:54.190716+0000","last_fullsized":"2024-03-11T22:15:54.190716+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-11T22:13:31.490878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:31.490878+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":296,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.189246+0000","last_change":"2024-03-11T22:15:14.878681+0000","last_active":"2024-03-11T22:15:54.189246+0000","last_peered":"2024-03-11T22:15:54.189246+0000","last_clean":"2024-03-11T22:15:54.189246+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:15:54.189246+0000","last_undegraded":"2024-03-11T22:15:54.189246+0000","last_fullsized":"2024-03-11T22:15:54.189246+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.878563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.878563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:43.850059+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":111,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.196234+0000","last_change":"2024-03-11T22:15:43.405176+0000","last_active":"2024-03-11T22:15:57.196234+0000","last_peered":"2024-03-11T22:15:57.196234+0000","last_clean":"2024-03-11T22:15:57.196234+0000","last_became_active":"2024-03-11T22:14:08.442596+0000","last_became_peered":"2024-03-11T22:14:08.442596+0000","last_unstale":"2024-03-11T22:15:57.196234+0000","last_undegraded":"2024-03-11T22:15:57.196234+0000","last_fullsized":"2024-03-11T22:15:57.196234+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:43.405052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:43.405052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:45.864784+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":234,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.185696+0000","last_change":"2024-03-11T22:15:20.894393+0000","last_active":"2024-03-11T22:15:54.185696+0000","last_peered":"2024-03-11T22:15:54.185696+0000","last_clean":"2024-03-11T22:15:54.185696+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:15:54.185696+0000","last_undegraded":"2024-03-11T22:15:54.185696+0000","last_fullsized":"2024-03-11T22:15:54.185696+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:20.894271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:20.894271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:29.495255+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":323,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201243+0000","last_change":"2024-03-11T22:15:29.232861+0000","last_active":"2024-03-11T22:15:57.201243+0000","last_peered":"2024-03-11T22:15:57.201243+0000","last_clean":"2024-03-11T22:15:57.201243+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:15:57.201243+0000","last_undegraded":"2024-03-11T22:15:57.201243+0000","last_fullsized":"2024-03-11T22:15:57.201243+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:15:29.232725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.232725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:53.191290+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":296,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.181998+0000","last_change":"2024-03-11T22:14:07.822714+0000","last_active":"2024-03-11T22:15:54.181998+0000","last_peered":"2024-03-11T22:15:54.181998+0000","last_clean":"2024-03-11T22:15:54.181998+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:15:54.181998+0000","last_undegraded":"2024-03-11T22:15:54.181998+0000","last_fullsized":"2024-03-11T22:15:54.181998+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:07.822036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:07.822036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":296,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.189400+0000","last_change":"2024-03-11T22:15:44.071290+0000","last_active":"2024-03-11T22:15:54.189400+0000","last_peered":"2024-03-11T22:15:54.189400+0000","last_clean":"2024-03-11T22:15:54.189400+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:15:54.189400+0000","last_undegraded":"2024-03-11T22:15:54.189400+0000","last_fullsized":"2024-03-11T22:15:54.189400+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:44.070947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:44.070947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:45.798443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":295,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.196286+0000","last_change":"2024-03-11T22:15:22.441543+0000","last_active":"2024-03-11T22:15:57.196286+0000","last_peered":"2024-03-11T22:15:57.196286+0000","last_clean":"2024-03-11T22:15:57.196286+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:15:57.196286+0000","last_undegraded":"2024-03-11T22:15:57.196286+0000","last_fullsized":"2024-03-11T22:15:57.196286+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:22.441417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:22.441417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:28.609256+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":325,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.189160+0000","last_change":"2024-03-11T22:15:51.200779+0000","last_active":"2024-03-11T22:15:54.189160+0000","last_peered":"2024-03-11T22:15:54.189160+0000","last_clean":"2024-03-11T22:15:54.189160+0000","last_became_active":"2024-03-11T22:15:51.188441+0000","last_became_peered":"2024-03-11T22:15:51.188441+0000","last_unstale":"2024-03-11T22:15:54.189160+0000","last_undegraded":"2024-03-11T22:15:54.189160+0000","last_fullsized":"2024-03-11T22:15:54.189160+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:15:19.958690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:19.958690+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:36.726748+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":309,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.196328+0000","last_change":"2024-03-11T22:15:37.329154+0000","last_active":"2024-03-11T22:15:57.196328+0000","last_peered":"2024-03-11T22:15:57.196328+0000","last_clean":"2024-03-11T22:15:57.196328+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:15:57.196328+0000","last_undegraded":"2024-03-11T22:15:57.196328+0000","last_fullsized":"2024-03-11T22:15:57.196328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:15:37.329026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:37.329026+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:49.312912+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"194'9","reported_seq":412,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201092+0000","last_change":"2024-03-11T22:15:51.193247+0000","last_active":"2024-03-11T22:15:57.201092+0000","last_peered":"2024-03-11T22:15:57.201092+0000","last_clean":"2024-03-11T22:15:57.201092+0000","last_became_active":"2024-03-11T22:15:51.188803+0000","last_became_peered":"2024-03-11T22:15:51.188803+0000","last_unstale":"2024-03-11T22:15:57.201092+0000","last_undegraded":"2024-03-11T22:15:57.201092+0000","last_fullsized":"2024-03-11T22:15:57.201092+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:14:21.037625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.037625+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":249,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201158+0000","last_change":"2024-03-11T22:15:33.601133+0000","last_active":"2024-03-11T22:15:57.201158+0000","last_peered":"2024-03-11T22:15:57.201158+0000","last_clean":"2024-03-11T22:15:57.201158+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:15:57.201158+0000","last_undegraded":"2024-03-11T22:15:57.201158+0000","last_fullsized":"2024-03-11T22:15:57.201158+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:15:33.601006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:33.601006+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:34.316683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":314,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.186485+0000","last_change":"2024-03-11T22:14:27.998984+0000","last_active":"2024-03-11T22:15:54.186485+0000","last_peered":"2024-03-11T22:15:54.186485+0000","last_clean":"2024-03-11T22:15:54.186485+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:15:54.186485+0000","last_undegraded":"2024-03-11T22:15:54.186485+0000","last_fullsized":"2024-03-11T22:15:54.186485+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:14:27.998819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:27.998819+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:56.633492+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":67,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.197751+0000","last_change":"2024-03-11T22:15:23.422984+0000","last_active":"2024-03-11T22:15:57.197751+0000","last_peered":"2024-03-11T22:15:57.197751+0000","last_clean":"2024-03-11T22:15:57.197751+0000","last_became_active":"2024-03-11T22:14:52.245199+0000","last_became_peered":"2024-03-11T22:14:52.245199+0000","last_unstale":"2024-03-11T22:15:57.197751+0000","last_undegraded":"2024-03-11T22:15:57.197751+0000","last_fullsized":"2024-03-11T22:15:57.197751+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:23.418102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:23.418102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:41.174600+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":54,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.188150+0000","last_change":"2024-03-11T22:14:52.233517+0000","last_active":"2024-03-11T22:15:54.188150+0000","last_peered":"2024-03-11T22:15:54.188150+0000","last_clean":"2024-03-11T22:15:54.188150+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:15:54.188150+0000","last_undegraded":"2024-03-11T22:15:54.188150+0000","last_fullsized":"2024-03-11T22:15:54.188150+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.849679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.849679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":276,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.181738+0000","last_change":"2024-03-11T22:12:44.464428+0000","last_active":"2024-03-11T22:15:54.181738+0000","last_peered":"2024-03-11T22:15:54.181738+0000","last_clean":"2024-03-11T22:15:54.181738+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:15:54.181738+0000","last_undegraded":"2024-03-11T22:15:54.181738+0000","last_fullsized":"2024-03-11T22:15:54.181738+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.464308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.464308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":332,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.186302+0000","last_change":"2024-03-11T22:15:22.863847+0000","last_active":"2024-03-11T22:15:54.186302+0000","last_peered":"2024-03-11T22:15:54.186302+0000","last_clean":"2024-03-11T22:15:54.186302+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:15:54.186302+0000","last_undegraded":"2024-03-11T22:15:54.186302+0000","last_fullsized":"2024-03-11T22:15:54.186302+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:15:22.863711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:22.863711+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:48.485553+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":301,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.190546+0000","last_change":"2024-03-11T22:15:51.186643+0000","last_active":"2024-03-11T22:15:54.190546+0000","last_peered":"2024-03-11T22:15:54.190546+0000","last_clean":"2024-03-11T22:15:54.190546+0000","last_became_active":"2024-03-11T22:15:51.186230+0000","last_became_peered":"2024-03-11T22:15:51.186230+0000","last_unstale":"2024-03-11T22:15:54.190546+0000","last_undegraded":"2024-03-11T22:15:54.190546+0000","last_fullsized":"2024-03-11T22:15:54.190546+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:32.050761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:32.050761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:57.955282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":299,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.196159+0000","last_change":"2024-03-11T22:15:41.407553+0000","last_active":"2024-03-11T22:15:57.196159+0000","last_peered":"2024-03-11T22:15:57.196159+0000","last_clean":"2024-03-11T22:15:57.196159+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:15:57.196159+0000","last_undegraded":"2024-03-11T22:15:57.196159+0000","last_fullsized":"2024-03-11T22:15:57.196159+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:41.407438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:41.407438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:58.057536+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":66,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201190+0000","last_change":"2024-03-11T22:15:28.217629+0000","last_active":"2024-03-11T22:15:57.201190+0000","last_peered":"2024-03-11T22:15:57.201190+0000","last_clean":"2024-03-11T22:15:57.201190+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:15:57.201190+0000","last_undegraded":"2024-03-11T22:15:57.201190+0000","last_fullsized":"2024-03-11T22:15:57.201190+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:28.217499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:28.217499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:54.886059+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"95'7","reported_seq":128,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.180754+0000","last_change":"2024-03-11T22:14:52.242050+0000","last_active":"2024-03-11T22:15:54.180754+0000","last_peered":"2024-03-11T22:15:54.180754+0000","last_clean":"2024-03-11T22:15:54.180754+0000","last_became_active":"2024-03-11T22:14:52.238411+0000","last_became_peered":"2024-03-11T22:14:52.238411+0000","last_unstale":"2024-03-11T22:15:54.180754+0000","last_undegraded":"2024-03-11T22:15:54.180754+0000","last_fullsized":"2024-03-11T22:15:54.180754+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'7","last_scrub_stamp":"2024-03-11T22:13:32.041219+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:13:32.041219+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":299,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201267+0000","last_change":"2024-03-11T22:14:30.716953+0000","last_active":"2024-03-11T22:15:57.201267+0000","last_peered":"2024-03-11T22:15:57.201267+0000","last_clean":"2024-03-11T22:15:57.201267+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:15:57.201267+0000","last_undegraded":"2024-03-11T22:15:57.201267+0000","last_fullsized":"2024-03-11T22:15:57.201267+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:14:30.716844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":69,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.197606+0000","last_change":"2024-03-11T22:15:28.434583+0000","last_active":"2024-03-11T22:15:57.197606+0000","last_peered":"2024-03-11T22:15:57.197606+0000","last_clean":"2024-03-11T22:15:57.197606+0000","last_became_active":"2024-03-11T22:14:52.239391+0000","last_became_peered":"2024-03-11T22:14:52.239391+0000","last_unstale":"2024-03-11T22:15:57.197606+0000","last_undegraded":"2024-03-11T22:15:57.197606+0000","last_fullsized":"2024-03-11T22:15:57.197606+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_clean_scrub_stamp":"2024-03-11T22:15:28.434334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:45.678839+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":288,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199315+0000","last_change":"2024-03-11T22:14:22.631560+0000","last_active":"2024-03-11T22:15:57.199315+0000","last_peered":"2024-03-11T22:15:57.199315+0000","last_clean":"2024-03-11T22:15:57.199315+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:15:57.199315+0000","last_undegraded":"2024-03-11T22:15:57.199315+0000","last_fullsized":"2024-03-11T22:15:57.199315+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:22.631456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:22.631456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":97,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.179052+0000","last_change":"2024-03-11T22:15:18.401675+0000","last_active":"2024-03-11T22:15:54.179052+0000","last_peered":"2024-03-11T22:15:54.179052+0000","last_clean":"2024-03-11T22:15:54.179052+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:15:54.179052+0000","last_undegraded":"2024-03-11T22:15:54.179052+0000","last_fullsized":"2024-03-11T22:15:54.179052+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:15:18.401549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:18.401549+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:23.261566+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":307,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199074+0000","last_change":"2024-03-11T22:15:15.590670+0000","last_active":"2024-03-11T22:15:57.199074+0000","last_peered":"2024-03-11T22:15:57.199074+0000","last_clean":"2024-03-11T22:15:57.199074+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:15:57.199074+0000","last_undegraded":"2024-03-11T22:15:57.199074+0000","last_fullsized":"2024-03-11T22:15:57.199074+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:15.590551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:15.590551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:42.253111+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":297,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199495+0000","last_change":"2024-03-11T22:15:51.185431+0000","last_active":"2024-03-11T22:15:57.199495+0000","last_peered":"2024-03-11T22:15:57.199495+0000","last_clean":"2024-03-11T22:15:57.199495+0000","last_became_active":"2024-03-11T22:15:51.184944+0000","last_became_peered":"2024-03-11T22:15:51.184944+0000","last_unstale":"2024-03-11T22:15:57.199495+0000","last_undegraded":"2024-03-11T22:15:57.199495+0000","last_fullsized":"2024-03-11T22:15:57.199495+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:28.674962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:14:28.674962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":284,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.193063+0000","last_change":"2024-03-11T22:15:20.947535+0000","last_active":"2024-03-11T22:15:54.193063+0000","last_peered":"2024-03-11T22:15:54.193063+0000","last_clean":"2024-03-11T22:15:54.193063+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:15:54.193063+0000","last_undegraded":"2024-03-11T22:15:54.193063+0000","last_fullsized":"2024-03-11T22:15:54.193063+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:20.947405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:20.947405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:48.995693+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":309,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.179956+0000","last_change":"2024-03-11T22:12:49.510144+0000","last_active":"2024-03-11T22:15:54.179956+0000","last_peered":"2024-03-11T22:15:54.179956+0000","last_clean":"2024-03-11T22:15:54.179956+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:15:54.179956+0000","last_undegraded":"2024-03-11T22:15:54.179956+0000","last_fullsized":"2024-03-11T22:15:54.179956+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:12:49.509860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.509860+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":302,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.181965+0000","last_change":"2024-03-11T22:13:19.463087+0000","last_active":"2024-03-11T22:15:54.181965+0000","last_peered":"2024-03-11T22:15:54.181965+0000","last_clean":"2024-03-11T22:15:54.181965+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:15:54.181965+0000","last_undegraded":"2024-03-11T22:15:54.181965+0000","last_fullsized":"2024-03-11T22:15:54.181965+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:13:19.462955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.462955+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":66,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199681+0000","last_change":"2024-03-11T22:15:16.288357+0000","last_active":"2024-03-11T22:15:57.199681+0000","last_peered":"2024-03-11T22:15:57.199681+0000","last_clean":"2024-03-11T22:15:57.199681+0000","last_became_active":"2024-03-11T22:14:52.247689+0000","last_became_peered":"2024-03-11T22:14:52.247689+0000","last_unstale":"2024-03-11T22:15:57.199681+0000","last_undegraded":"2024-03-11T22:15:57.199681+0000","last_fullsized":"2024-03-11T22:15:57.199681+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:16.288232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:16.288232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:28.360920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":281,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.181048+0000","last_change":"2024-03-11T22:15:51.191909+0000","last_active":"2024-03-11T22:15:54.181048+0000","last_peered":"2024-03-11T22:15:54.181048+0000","last_clean":"2024-03-11T22:15:54.181048+0000","last_became_active":"2024-03-11T22:15:51.191529+0000","last_became_peered":"2024-03-11T22:15:51.191529+0000","last_unstale":"2024-03-11T22:15:54.181048+0000","last_undegraded":"2024-03-11T22:15:54.181048+0000","last_fullsized":"2024-03-11T22:15:54.181048+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.533067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.533067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:15:56.633382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":93,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.199761+0000","last_change":"2024-03-11T22:14:52.248071+0000","last_active":"2024-03-11T22:15:57.199761+0000","last_peered":"2024-03-11T22:15:57.199761+0000","last_clean":"2024-03-11T22:15:57.199761+0000","last_became_active":"2024-03-11T22:14:52.247411+0000","last_became_peered":"2024-03-11T22:14:52.247411+0000","last_unstale":"2024-03-11T22:15:57.199761+0000","last_undegraded":"2024-03-11T22:15:57.199761+0000","last_fullsized":"2024-03-11T22:15:57.199761+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:14:15.059599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:15.059599+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":296,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-11T22:15:54.188988+0000","last_change":"2024-03-11T22:14:28.062521+0000","last_active":"2024-03-11T22:15:54.188988+0000","last_peered":"2024-03-11T22:15:54.188988+0000","last_clean":"2024-03-11T22:15:54.188988+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:15:54.188988+0000","last_undegraded":"2024-03-11T22:15:54.188988+0000","last_fullsized":"2024-03-11T22:15:54.188988+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:28.062439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:28.062439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":298,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.196027+0000","last_change":"2024-03-11T22:15:51.183838+0000","last_active":"2024-03-11T22:15:57.196027+0000","last_peered":"2024-03-11T22:15:57.196027+0000","last_clean":"2024-03-11T22:15:57.196027+0000","last_became_active":"2024-03-11T22:15:51.183195+0000","last_became_peered":"2024-03-11T22:15:51.183195+0000","last_unstale":"2024-03-11T22:15:57.196027+0000","last_undegraded":"2024-03-11T22:15:57.196027+0000","last_fullsized":"2024-03-11T22:15:57.196027+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:09.518937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:09.518937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:33.810676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":66,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-03-11T22:15:57.201551+0000","last_change":"2024-03-11T22:15:12.647958+0000","last_active":"2024-03-11T22:15:57.201551+0000","last_peered":"2024-03-11T22:15:57.201551+0000","last_clean":"2024-03-11T22:15:57.201551+0000","last_became_active":"2024-03-11T22:14:52.247273+0000","last_became_peered":"2024-03-11T22:14:52.247273+0000","last_unstale":"2024-03-11T22:15:57.201551+0000","last_undegraded":"2024-03-11T22:15:57.201551+0000","last_fullsized":"2024-03-11T22:15:57.201551+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:12.647884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:12.647884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:16:27.129974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5763825664,"num_objects":87949,"num_object_clones":0,"num_object_copies":263847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8750,"num_objects_unfound":0,"num_objects_dirty":87949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87949,"num_write_kb":5628736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1870,"num_bytes_recovered":122552320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17487953920,"data_stored":17487953920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69840,"ondisk_log_size":69840,"up":48,"acting":47,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":414,"num_read_kb":409,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2730,"ondisk_log_size":2730,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":688,"num_read_kb":445,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":410,"ondisk_log_size":410,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1921748,"kb_used_data":1764236,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":91823404,"statfs":{"total":95995035648,"available":94027165696,"internally_reserved":0,"allocated":1806577664,"data_stored":1806202253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":161282507},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332195,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2080556,"kb_used_data":1909156,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91664596,"statfs":{"total":95995035648,"available":93864546304,"internally_reserved":0,"allocated":1954975744,"data_stored":1954576743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175503814},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574973,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2730856,"kb_used_data":2519200,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":91014296,"statfs":{"total":95995035648,"available":93198639104,"internally_reserved":0,"allocated":2579660800,"data_stored":2579200854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738498,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2284512,"kb_used_data":2028440,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":91460640,"statfs":{"total":95995035648,"available":93655695360,"internally_reserved":0,"allocated":2077122560,"data_stored":2076718906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411447,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2208472,"kb_used_data":2037072,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91536680,"statfs":{"total":95995035648,"available":93733560320,"internally_reserved":0,"allocated":2085961728,"data_stored":2085567622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":175503813},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2283200,"kb_used_data":2035440,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91461952,"statfs":{"total":95995035648,"available":93657038848,"internally_reserved":0,"allocated":2084290560,"data_stored":2083863858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":253688263},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920878,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1948356,"kb_used_data":1782524,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91796796,"statfs":{"total":95995035648,"available":93999919104,"internally_reserved":0,"allocated":1825304576,"data_stored":1824880827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169802182},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659239,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3097848,"kb_used_data":2926448,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":90647304,"statfs":{"total":95995035648,"available":92822839296,"internally_reserved":0,"allocated":2996682752,"data_stored":2996271487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175503814},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2094530560,"data_stored":2094530560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2585591808,"data_stored":2585591808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2107703296,"data_stored":2107703296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2083454976,"data_stored":2083454976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1829371904,"data_stored":1829371904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3005284352,"data_stored":3005284352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1962672128,"data_stored":1962672128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1819344896,"data_stored":1819344896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:16:04.432 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-11T22:16:04.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pg_num 31 2024-03-11T22:16:04.504 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:04.539 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:04.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:04.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:04.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:04.666 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:04.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:04.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:04.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:04.792 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:04.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:04.893 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:04.926 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:04.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:05.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:05.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:05.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:05.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:05.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:05.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:05.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:05.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:05.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:05.328 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 63 16 95152 95136 94.3562 42.375 0.0120836 0.0105935 2024-03-11T22:16:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:05.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:05.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:05.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:05.466 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:05.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:05.567 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:05.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:05.602 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:05.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:05.704 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:05.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:05.739 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:05.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:05.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:05.845 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:05.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:05.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:05.913 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:05 smithi045 ceph-mon[39715]: pgmap v322: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 166 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 719 op/s; 8671/272004 objects misplaced (3.188%); 8.9 MiB/s, 142 objects/s recovering 2024-03-11T22:16:05.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:05 smithi045 ceph-mon[30178]: pgmap v322: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 166 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 719 op/s; 8671/272004 objects misplaced (3.188%); 8.9 MiB/s, 142 objects/s recovering 2024-03-11T22:16:06.004 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:06.040 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:05 smithi144 ceph-mon[33984]: pgmap v322: 168 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 166 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 719 op/s; 8671/272004 objects misplaced (3.188%); 8.9 MiB/s, 142 objects/s recovering 2024-03-11T22:16:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:06.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:06.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:06.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:06.183 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:06.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:06.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:06.322 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:06.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:06.326 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 64 16 96285 96269 93.9883 70.8125 0.0144733 0.0106348 2024-03-11T22:16:06.344 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:16:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:16:06.425 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:06.453 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:06.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:06.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:06.604 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:06.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:06.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:06.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:06.745 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:06.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:06.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:06.756 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:06 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3179360208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:06.756 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:06 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3179360208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:06.847 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:06.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:06.887 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:06.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:06.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:07.007 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:16:07.007 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:16:07.007 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:16:07.007 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:16:07.007 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:16:07.007 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:16:07.007 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:16:07.007 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:16:07.007 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:16:07.007 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:16:07.008 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:16:07.008 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:16:07.008 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:16:07.008 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:16:07.008 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:16:07.008 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:16:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:07.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:07.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:07.050 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:06 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3179360208' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:07.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:07.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:07.166 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:07.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:07.246 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:07.267 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:07.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:07.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:07.326 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 65 16 97237 97221 93.4577 59.5 0.0105502 0.0106945 2024-03-11T22:16:07.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:07.457 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:07.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:07.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:07.559 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:07.595 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:07.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:07.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:07.696 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:07.733 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:07.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:07.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:07.745 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:07.745 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:07 smithi144 ceph-mon[33984]: pgmap v323: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 7622/286803 objects misplaced (2.658%); 17 MiB/s, 264 objects/s recovering 2024-03-11T22:16:07.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:07.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:07.892 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:07.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:07.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:08.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:08.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:08.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:08.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:07 smithi045 ceph-mon[39715]: pgmap v323: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 7622/286803 objects misplaced (2.658%); 17 MiB/s, 264 objects/s recovering 2024-03-11T22:16:08.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:08.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:07 smithi045 ceph-mon[30178]: pgmap v323: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 7622/286803 objects misplaced (2.658%); 17 MiB/s, 264 objects/s recovering 2024-03-11T22:16:08.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:08.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:08.169 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:08.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:08.270 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:08.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:08.309 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:08.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:08.326 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 66 16 98380 98364 93.124 71.4375 0.0306902 0.0107329 2024-03-11T22:16:08.410 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:08.451 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:08.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:08.554 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:08.593 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:08.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:08.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:08.690 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:16:08.690 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:16:08.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:08.737 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:08.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:08.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:08.839 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:08.877 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:08.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:08.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:09.020 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:09.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:09.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:09.157 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:09.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:09.259 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:09.295 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:09.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:09.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:09.326 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 67 16 99578 99562 92.8516 74.875 0.0129221 0.0107642 2024-03-11T22:16:09.388 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T22:16:09.396 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:09.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:09.476 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:09.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:09.578 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:09.622 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:09.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:09.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:09.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:09.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:09 smithi045 ceph-mon[30178]: pgmap v324: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s; 7622/286803 objects misplaced (2.658%); 15 MiB/s, 237 objects/s recovering 2024-03-11T22:16:09.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:09 smithi045 ceph-mon[39715]: pgmap v324: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s; 7622/286803 objects misplaced (2.658%); 15 MiB/s, 237 objects/s recovering 2024-03-11T22:16:09.784 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:16:09.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:09.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:09.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:10.000 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:10.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:10.038 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:10.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:10.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:09 smithi144 ceph-mon[33984]: pgmap v324: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s; 7622/286803 objects misplaced (2.658%); 15 MiB/s, 237 objects/s recovering 2024-03-11T22:16:10.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:10.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:10.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:10.286 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:10.320 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:10.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:10.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:10.326 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 68 16 100718 100702 92.5339 71.25 0.00684364 0.0108019 2024-03-11T22:16:10.423 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:10.458 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:10.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:10.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:10.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:16:10.561 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:10.593 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:10.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:10.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:10.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:10.718 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:10.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:10.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:10.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:10.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:10.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:10.975 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:11.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:11.010 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:11.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:11.111 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:11.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:11.149 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:11.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:11.250 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:11.275 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:11.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:11.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:11.326 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 69 16 101717 101701 92.0977 62.4375 0.0901013 0.010846 2024-03-11T22:16:11.377 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:11.410 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:11.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:11.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:11.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:11.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:11.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:11.656 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:11.689 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:11.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:11.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:11.768 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:11 smithi045 ceph-mon[30178]: pgmap v325: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s; 7622/286803 objects misplaced (2.658%); 15 MiB/s, 237 objects/s recovering 2024-03-11T22:16:11.769 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:11 smithi045 ceph-mon[39715]: pgmap v325: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s; 7622/286803 objects misplaced (2.658%); 15 MiB/s, 237 objects/s recovering 2024-03-11T22:16:11.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:11.832 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:11.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:11.934 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:11.945 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:11.950 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:11 smithi144 ceph-mon[33984]: pgmap v325: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s; 7622/286803 objects misplaced (2.658%); 15 MiB/s, 237 objects/s recovering 2024-03-11T22:16:11.967 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:11.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:11.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:12.069 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:12.107 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:12.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:12.167 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:16:12.167 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:16:12.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:12.254 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:12.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:12.326 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 70 16 102788 102772 91.7382 66.9375 0.00735423 0.0108956 2024-03-11T22:16:12.355 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:12.388 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:12.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:12.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:12.490 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:12.525 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:12.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:12.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:12.627 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:12.663 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:12.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:12.665 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:16:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:16:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:16:12.749 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:16:12.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:12.790 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:12 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2355229267' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:16:12.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:12.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:12.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:12.843 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T22:16:12.907 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:12.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:12.951 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:12.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:13.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:12 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2355229267' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:16:13.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:12 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2355229267' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:16:13.051 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:13.089 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:13.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:13.191 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:13.224 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:13.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:13.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:13.240 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:16:13.326 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:13.327 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 71 16 103899 103883 91.4241 69.4375 0.00652605 0.0109329 2024-03-11T22:16:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:13.371 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:13.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:13.472 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:13.515 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:13.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:13.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:13.750 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:13.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:13.763 INFO:teuthology.orchestra.run.smithi045.stderr:set pool 1 pg_num to 31 2024-03-11T22:16:13.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:13.892 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:13.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:13.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:13.994 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:14.002 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:13 smithi144 ceph-mon[33984]: pgmap v326: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 891 B/s rd, 71 MiB/s wr, 1.13k op/s; 6756/303480 objects misplaced (2.226%); 13 MiB/s, 200 objects/s recovering 2024-03-11T22:16:14.003 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:13 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:16:14.003 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:13 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:16:14.003 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:13 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:14.003 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:13 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2482113086' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-11T22:16:14.003 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:13 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2355229267' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:16:14.003 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:13 smithi144 ceph-mon[33984]: osdmap e201: 8 total, 8 up, 8 in 2024-03-11T22:16:14.003 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:13 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-11T22:16:14.014 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[39715]: pgmap v326: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 891 B/s rd, 71 MiB/s wr, 1.13k op/s; 6756/303480 objects misplaced (2.226%); 13 MiB/s, 200 objects/s recovering 2024-03-11T22:16:14.014 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:16:14.014 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:16:14.014 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:14.014 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2482113086' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-11T22:16:14.014 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2355229267' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:16:14.014 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[39715]: osdmap e201: 8 total, 8 up, 8 in 2024-03-11T22:16:14.014 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-11T22:16:14.014 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[30178]: pgmap v326: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 891 B/s rd, 71 MiB/s wr, 1.13k op/s; 6756/303480 objects misplaced (2.226%); 13 MiB/s, 200 objects/s recovering 2024-03-11T22:16:14.015 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:16:14.015 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:16:14.015 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:14.015 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2482113086' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-11T22:16:14.015 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2355229267' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:16:14.015 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[30178]: osdmap e201: 8 total, 8 up, 8 in 2024-03-11T22:16:14.015 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:13 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-11T22:16:14.028 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:14.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:14.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:14.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:14.171 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:14.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:14.272 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:14.308 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:14.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:14.326 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 72 16 105101 105085 91.1976 75.125 0.0316319 0.0109594 2024-03-11T22:16:14.410 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:14.454 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:14.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:14.555 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:14.586 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:14.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:14.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:14.688 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:14.725 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:14.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:14.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:14.873 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:14.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:14.976 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:14.996 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[30178]: 3.4 scrub starts 2024-03-11T22:16:14.996 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[30178]: 3.4 scrub ok 2024-03-11T22:16:14.996 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-03-11T22:16:14.996 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[30178]: osdmap e202: 8 total, 8 up, 8 in 2024-03-11T22:16:14.996 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 0]}]: dispatch 2024-03-11T22:16:14.996 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 2]}]: dispatch 2024-03-11T22:16:14.996 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [1, 6]}]: dispatch 2024-03-11T22:16:14.996 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-03-11T22:16:14.996 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[30178]: pgmap v329: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 695 op/s; 6756/303480 objects misplaced (2.226%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T22:16:14.996 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-11T22:16:14.996 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:16:14.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[39715]: 3.4 scrub starts 2024-03-11T22:16:14.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[39715]: 3.4 scrub ok 2024-03-11T22:16:14.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-03-11T22:16:14.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[39715]: osdmap e202: 8 total, 8 up, 8 in 2024-03-11T22:16:14.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 0]}]: dispatch 2024-03-11T22:16:14.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 2]}]: dispatch 2024-03-11T22:16:14.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [1, 6]}]: dispatch 2024-03-11T22:16:14.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-03-11T22:16:14.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[39715]: pgmap v329: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 695 op/s; 6756/303480 objects misplaced (2.226%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T22:16:14.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-11T22:16:14.997 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:16:15.025 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:15.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:15.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:15.043 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:14 smithi144 ceph-mon[33984]: 3.4 scrub starts 2024-03-11T22:16:15.043 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:14 smithi144 ceph-mon[33984]: 3.4 scrub ok 2024-03-11T22:16:15.043 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:14 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-03-11T22:16:15.043 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:14 smithi144 ceph-mon[33984]: osdmap e202: 8 total, 8 up, 8 in 2024-03-11T22:16:15.043 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 0]}]: dispatch 2024-03-11T22:16:15.043 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 2]}]: dispatch 2024-03-11T22:16:15.044 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [1, 6]}]: dispatch 2024-03-11T22:16:15.044 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-03-11T22:16:15.044 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:14 smithi144 ceph-mon[33984]: pgmap v329: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 166 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 695 op/s; 6756/303480 objects misplaced (2.226%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T22:16:15.044 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-11T22:16:15.044 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:16:15.127 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:15.164 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:15.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:15.266 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:15.304 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:15.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:15.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:15.327 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 73 16 106256 106240 90.9367 72.1875 0.0115408 0.0109914 2024-03-11T22:16:15.403 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:16:15.403 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:16:15.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:15.450 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:15.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:15.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:15.597 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:15.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:15.698 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:15.713 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:16:15.732 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:15.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:15.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:16:15.753 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:16:15.804 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:16:15.834 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:15.859 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:16:15.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 0]}]': finished 2024-03-11T22:16:15.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 2]}]': finished 2024-03-11T22:16:15.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [1, 6]}]': finished 2024-03-11T22:16:15.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-03-11T22:16:15.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-11T22:16:15.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[39715]: osdmap e203: 8 total, 8 up, 8 in 2024-03-11T22:16:15.860 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/584634969' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:15.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 0]}]': finished 2024-03-11T22:16:15.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 2]}]': finished 2024-03-11T22:16:15.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [1, 6]}]': finished 2024-03-11T22:16:15.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-03-11T22:16:15.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-11T22:16:15.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[30178]: osdmap e203: 8 total, 8 up, 8 in 2024-03-11T22:16:15.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:15 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/584634969' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:15.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:15.869 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:15.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:15.909 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:16:15.957 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:16:15.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:16.002 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:16:16.007 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:16.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:16.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:16.042 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:16:16.043 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 0]}]': finished 2024-03-11T22:16:16.044 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1", "id": [0, 2]}]': finished 2024-03-11T22:16:16.044 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.14", "id": [1, 6]}]': finished 2024-03-11T22:16:16.044 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-03-11T22:16:16.044 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-11T22:16:16.044 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:15 smithi144 ceph-mon[33984]: osdmap e203: 8 total, 8 up, 8 in 2024-03-11T22:16:16.044 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:15 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/584634969' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:16:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:16:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:16:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:16:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:16:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:16:16.050 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:16:16.050 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:16:16.093 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T22:16:16.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:16.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:16.151 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:16.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:16.252 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:16.294 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:16.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:16.329 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 74 16 107346 107330 90.6263 68.125 0.00955693 0.0110295 2024-03-11T22:16:16.344 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:16:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:16:16.395 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:16.441 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:16.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:16.518 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:16:16.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:16.566 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:16.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:16.667 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:16.704 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:16.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:16.806 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:16.843 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:16.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:16.945 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:16.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:16.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:16.998 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:16 smithi144 ceph-mon[33984]: osdmap e204: 8 total, 8 up, 8 in 2024-03-11T22:16:16.998 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:16 smithi144 ceph-mon[33984]: 1.5 scrub starts 2024-03-11T22:16:16.998 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:16 smithi144 ceph-mon[33984]: 1.5 scrub ok 2024-03-11T22:16:16.998 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:16 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:16.998 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:16 smithi144 ceph-mon[33984]: pgmap v332: 178 pgs: 1 active+recovering+undersized+remapped, 8 activating, 8 peering, 1 active+remapped+backfilling, 7 unknown, 153 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 6/320193 objects degraded (0.002%); 5590/320193 objects misplaced (1.746%); 20 MiB/s, 313 objects/s recovering 2024-03-11T22:16:16.998 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-11T22:16:16.998 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:16 smithi144 ceph-mon[33984]: 1.d scrub starts 2024-03-11T22:16:16.998 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:16 smithi144 ceph-mon[33984]: 1.d scrub ok 2024-03-11T22:16:17.030 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[39715]: osdmap e204: 8 total, 8 up, 8 in 2024-03-11T22:16:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[39715]: 1.5 scrub starts 2024-03-11T22:16:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[39715]: 1.5 scrub ok 2024-03-11T22:16:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[39715]: pgmap v332: 178 pgs: 1 active+recovering+undersized+remapped, 8 activating, 8 peering, 1 active+remapped+backfilling, 7 unknown, 153 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 6/320193 objects degraded (0.002%); 5590/320193 objects misplaced (1.746%); 20 MiB/s, 313 objects/s recovering 2024-03-11T22:16:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-11T22:16:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[39715]: 1.d scrub starts 2024-03-11T22:16:17.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[39715]: 1.d scrub ok 2024-03-11T22:16:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[30178]: osdmap e204: 8 total, 8 up, 8 in 2024-03-11T22:16:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[30178]: 1.5 scrub starts 2024-03-11T22:16:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[30178]: 1.5 scrub ok 2024-03-11T22:16:17.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:17.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[30178]: pgmap v332: 178 pgs: 1 active+recovering+undersized+remapped, 8 activating, 8 peering, 1 active+remapped+backfilling, 7 unknown, 153 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 6/320193 objects degraded (0.002%); 5590/320193 objects misplaced (1.746%); 20 MiB/s, 313 objects/s recovering 2024-03-11T22:16:17.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-11T22:16:17.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[30178]: 1.d scrub starts 2024-03-11T22:16:17.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:16 smithi045 ceph-mon[30178]: 1.d scrub ok 2024-03-11T22:16:17.082 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:17.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:17.125 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:17.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:17.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:17.267 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:17.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:17.329 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 75 16 108503 108487 90.3821 72.3125 0.00364394 0.0110578 2024-03-11T22:16:17.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:17.410 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:17.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:17.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:17.550 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:17.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:17.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:17.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:17.692 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:17.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:17.793 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:17.837 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:17.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:17.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:17.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:17.941 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:17.987 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:17.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:17.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:18.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:17 smithi045 ceph-mon[30178]: 1.15 scrub starts 2024-03-11T22:16:18.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:17 smithi045 ceph-mon[30178]: 1.15 scrub ok 2024-03-11T22:16:18.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:17 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-11T22:16:18.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:17 smithi045 ceph-mon[30178]: osdmap e205: 8 total, 8 up, 8 in 2024-03-11T22:16:18.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:17 smithi045 ceph-mon[30178]: 1.b scrub starts 2024-03-11T22:16:18.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:17 smithi045 ceph-mon[30178]: 1.b scrub ok 2024-03-11T22:16:18.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:17 smithi045 ceph-mon[39715]: 1.15 scrub starts 2024-03-11T22:16:18.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:17 smithi045 ceph-mon[39715]: 1.15 scrub ok 2024-03-11T22:16:18.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:17 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-11T22:16:18.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:17 smithi045 ceph-mon[39715]: osdmap e205: 8 total, 8 up, 8 in 2024-03-11T22:16:18.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:17 smithi045 ceph-mon[39715]: 1.b scrub starts 2024-03-11T22:16:18.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:17 smithi045 ceph-mon[39715]: 1.b scrub ok 2024-03-11T22:16:18.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:18.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:17 smithi144 ceph-mon[33984]: 1.15 scrub starts 2024-03-11T22:16:18.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:17 smithi144 ceph-mon[33984]: 1.15 scrub ok 2024-03-11T22:16:18.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:17 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-11T22:16:18.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:17 smithi144 ceph-mon[33984]: osdmap e205: 8 total, 8 up, 8 in 2024-03-11T22:16:18.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:17 smithi144 ceph-mon[33984]: 1.b scrub starts 2024-03-11T22:16:18.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:17 smithi144 ceph-mon[33984]: 1.b scrub ok 2024-03-11T22:16:18.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:18.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:18.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:18.238 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:18.280 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:18.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:18.329 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 76 15 109671 109656 90.1542 73.0625 0.0136439 0.0110871 2024-03-11T22:16:18.382 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:18.423 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:18.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:18.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:18.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:18.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:18.567 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:18.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:18.633 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:16:18.633 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-03-11T22:16:18.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-03-11T22:16:18.633 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:16:18.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.633 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:27626 7.0 7:0e450ccf:::benchmark_data_smithi045_83007_object27625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:16:18.634 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:55.717490+0000", 2024-03-11T22:16:18.634 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 322.91162312400002, 2024-03-11T22:16:18.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13484212200000001, 2024-03-11T22:16:18.634 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.634 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.634 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:16:18.634 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:16:18.634 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:16:18.634 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 27626 2024-03-11T22:16:18.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.634 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.634 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717490+0000", 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717490+0000", 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717492+0000", 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3859999999999999e-06 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717531+0000", 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9270000000000002e-05 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717544+0000", 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3094e-05 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717550+0000", 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6899999999999997e-06 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.636 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717606+0000", 2024-03-11T22:16:18.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5828000000000001e-05 2024-03-11T22:16:18.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717750+0000", 2024-03-11T22:16:18.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014452900000000001 2024-03-11T22:16:18.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:16:18.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.717910+0000", 2024-03-11T22:16:18.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015962800000000001 2024-03-11T22:16:18.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:16:18.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718911+0000", 2024-03-11T22:16:18.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010012249999999999 2024-03-11T22:16:18.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.719301+0000", 2024-03-11T22:16:18.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00039031800000000001 2024-03-11T22:16:18.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.852224+0000", 2024-03-11T22:16:18.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13292285600000001 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.852278+0000", 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3783000000000003e-05 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.852333+0000", 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4515000000000002e-05 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:28347 7.0 7:043ba8fd:::benchmark_data_smithi045_83007_object28346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:56.330173+0000", 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 322.29894022899998, 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098482391000000002, 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 28347 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330173+0000", 2024-03-11T22:16:18.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330173+0000", 2024-03-11T22:16:18.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330175+0000", 2024-03-11T22:16:18.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.455e-06 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330221+0000", 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6217000000000002e-05 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330235+0000", 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3811e-05 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.643 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330241+0000", 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9610000000000002e-06 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330285+0000", 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4215999999999998e-05 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330392+0000", 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000107118 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330505+0000", 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000112728 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.332109+0000", 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016036710000000001 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.332155+0000", 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6718999999999998e-05 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.428535+0000", 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.096379861999999997 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.428599+0000", 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4277999999999997e-05 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.428656+0000", 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6354999999999998e-05 2024-03-11T22:16:18.646 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.647 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.647 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.647 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.647 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.647 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:28348 7.0 7:02caa2fd:::benchmark_data_smithi045_83007_object28347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:16:18.647 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:56.330255+0000", 2024-03-11T22:16:18.647 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 322.29885826700001, 2024-03-11T22:16:18.647 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098596250999999996, 2024-03-11T22:16:18.647 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.647 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.647 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:16:18.647 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:16:18.647 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 28348 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330255+0000", 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330255+0000", 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.648 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330256+0000", 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2900000000000004e-07 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330287+0000", 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1269999999999997e-05 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330294+0000", 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3679999999999996e-06 2024-03-11T22:16:18.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.330299+0000", 2024-03-11T22:16:18.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1110000000000001e-06 2024-03-11T22:16:18.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.331401+0000", 2024-03-11T22:16:18.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001102331 2024-03-11T22:16:18.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.331519+0000", 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000117959 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.331601+0000", 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2250999999999996e-05 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.332252+0000", 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00065089 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.651 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.332734+0000", 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00048246299999999998 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.428756+0000", 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.096021624 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.428812+0000", 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5971000000000003e-05 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.428852+0000", 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9483999999999997e-05 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:204827 7.0 7:0f3828dd:::benchmark_data_smithi045_83007_object86356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:44.301177+0000", 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 154.327936497, 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.118022245, 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.653 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 204827 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.301177+0000", 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.301177+0000", 2024-03-11T22:16:18.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.301178+0000", 2024-03-11T22:16:18.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.139e-06 2024-03-11T22:16:18.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.301182+0000", 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1119999999999998e-06 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.301193+0000", 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.048e-05 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.301199+0000", 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9689999999999999e-06 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.301248+0000", 2024-03-11T22:16:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9289000000000002e-05 2024-03-11T22:16:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.301340+0000", 2024-03-11T22:16:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1781999999999996e-05 2024-03-11T22:16:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:16:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.301416+0000", 2024-03-11T22:16:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5937000000000002e-05 2024-03-11T22:16:18.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.303608+0000", 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0021920849999999999 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.303920+0000", 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000312417 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.419114+0000", 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.115193928 2024-03-11T22:16:18.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.419154+0000", 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0178000000000001e-05 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.419199+0000", 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4929000000000003e-05 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:207221 7.0 7:0f46885f:::benchmark_data_smithi045_83007_object88750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:48.160031+0000", 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 150.46908255700001, 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.097113327999999999, 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 207221 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.660 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160031+0000", 2024-03-11T22:16:18.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160031+0000", 2024-03-11T22:16:18.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160033+0000", 2024-03-11T22:16:18.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5850000000000001e-06 2024-03-11T22:16:18.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.663 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160037+0000", 2024-03-11T22:16:18.663 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.386e-06 2024-03-11T22:16:18.663 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.663 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.663 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160047+0000", 2024-03-11T22:16:18.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7480000000000005e-06 2024-03-11T22:16:18.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160053+0000", 2024-03-11T22:16:18.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0730000000000001e-06 2024-03-11T22:16:18.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160097+0000", 2024-03-11T22:16:18.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4515999999999999e-05 2024-03-11T22:16:18.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160187+0000", 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9312000000000001e-05 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160283+0000", 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6297000000000003e-05 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.162909+0000", 2024-03-11T22:16:18.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026264719999999999 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.163383+0000", 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00047353800000000002 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.257057+0000", 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.093674161000000006 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.257107+0000", 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9756000000000003e-05 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.257144+0000", 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7484000000000002e-05 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229166 7.0 7:0c994632:::benchmark_data_smithi045_83007_object110695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:10.404154+0000", 2024-03-11T22:16:18.667 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 128.224960012, 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101259895, 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229166 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.404154+0000", 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.404154+0000", 2024-03-11T22:16:18.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.404155+0000", 2024-03-11T22:16:18.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.516e-06 2024-03-11T22:16:18.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.404161+0000", 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5790000000000003e-06 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.404176+0000", 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5058000000000001e-05 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.404184+0000", 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2700000000000004e-06 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.404240+0000", 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5890000000000002e-05 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.404328+0000", 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7724000000000002e-05 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.404401+0000", 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3699e-05 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.406918+0000", 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0025166239999999999 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.407876+0000", 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00095795600000000004 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.505339+0000", 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.097463584000000006 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.505380+0000", 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0292e-05 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.505413+0000", 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3702999999999999e-05 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229386 7.0 7:0557f109:::benchmark_data_smithi045_83007_object110915:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:16:18.673 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:10.796254+0000", 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 127.832859324, 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11522273800000001, 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229386 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.796254+0000", 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.796254+0000", 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.796255+0000", 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5899999999999995e-07 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.796259+0000", 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.352e-06 2024-03-11T22:16:18.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.796268+0000", 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4309999999999994e-06 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.796274+0000", 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6549999999999998e-06 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.803425+0000", 2024-03-11T22:16:18.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0071509679999999997 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.803509+0000", 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3652000000000005e-05 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,3", 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.803577+0000", 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8096999999999995e-05 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.805124+0000", 2024-03-11T22:16:18.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0015470130000000001 2024-03-11T22:16:18.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.678 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:16:18.678 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.805179+0000", 2024-03-11T22:16:18.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4613e-05 2024-03-11T22:16:18.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.678 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.678 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.911412+0000", 2024-03-11T22:16:18.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106233618 2024-03-11T22:16:18.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.911448+0000", 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5423000000000003e-05 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.911477+0000", 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9156999999999999e-05 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.679 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:58717 8.8 8:1e0c84a1:::benchmark_data_smithi045_115206_object58716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:15:32.696685+0000", 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 45.932429020999997, 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.094041894000000001, 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 58717 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.696685+0000", 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.696685+0000", 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.696692+0000", 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.985e-06 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.696771+0000", 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9296999999999997e-05 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.696786+0000", 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5611999999999999e-05 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.696794+0000", 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6079999999999998e-06 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.682 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.696849+0000", 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4432999999999999e-05 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.696976+0000", 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000127675 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.697122+0000", 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014588200000000001 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.701236+0000", 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0041142840000000002 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.701249+0000", 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2169e-05 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.790651+0000", 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.089402166000000005 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.790694+0000", 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3306999999999997e-05 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:32.790726+0000", 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2475999999999999e-05 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "MOSDPGPush(8.c 197/194 [PushOp(8:3e72bd81:::benchmark_data_smithi045_115206_object602:head, version: 179'39, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:3e72bd81:::benchmark_data_smithi045_115206_object602:head@179'39, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T22:16:18.685 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:15:55.341935+0000", 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 23.287178238999999, 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102231086, 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:55.341935+0000", 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:55.341935+0000", 2024-03-11T22:16:18.686 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:55.341936+0000", 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7700000000000004e-07 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:55.345245+0000", 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0033088409999999999 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:55.345262+0000", 2024-03-11T22:16:18.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7481000000000001e-05 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:55.345277+0000", 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5007999999999999e-05 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:55.442263+0000", 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.096985063999999996 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:55.442290+0000", 2024-03-11T22:16:18.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7251e-05 2024-03-11T22:16:18.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.689 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.689 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:55.444166+0000", 2024-03-11T22:16:18.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018766639999999999 2024-03-11T22:16:18.689 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.689 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.689 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.689 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:94691 8.8 8:18f56882:::benchmark_data_smithi045_115206_object94690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-11T22:16:18.689 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:04.534048+0000", 2024-03-11T22:16:18.689 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 14.095065707, 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13062267799999999, 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 94691 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.534048+0000", 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.534048+0000", 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.534049+0000", 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.248e-06 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.534095+0000", 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5497999999999997e-05 2024-03-11T22:16:18.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.534105+0000", 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0634e-05 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.534113+0000", 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3309999999999998e-06 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.534163+0000", 2024-03-11T22:16:18.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0052000000000001e-05 2024-03-11T22:16:18.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.534256+0000", 2024-03-11T22:16:18.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3660999999999997e-05 2024-03-11T22:16:18.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:16:18.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.534354+0000", 2024-03-11T22:16:18.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7429999999999994e-05 2024-03-11T22:16:18.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.536768+0000", 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024146599999999999 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.537032+0000", 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026350000000000001 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.664596+0000", 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.12756441700000001 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.694 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.664641+0000", 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4347000000000001e-05 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.664671+0000", 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9899999999999998e-05 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:94725 8.c e200/194 8:3abe960d:::benchmark_data_smithi045_115206_object94724:head v 200'5980, mlcod=0'0)", 2024-03-11T22:16:18.695 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:04.725559+0000", 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.903555007, 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.116193543, 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.725559+0000", 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.725559+0000", 2024-03-11T22:16:18.696 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.725559+0000", 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8400000000000003e-07 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.725614+0000", 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4716000000000003e-05 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.725625+0000", 2024-03-11T22:16:18.697 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0953e-05 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.725632+0000", 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1799999999999999e-06 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.740891+0000", 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.015258895999999999 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.698 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.740916+0000", 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4403999999999999e-05 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.841072+0000", 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100156972 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.841752+0000", 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00067963799999999999 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.699 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:94741 8.c e200/194 8:30c4a46b:::benchmark_data_smithi045_115206_object94740:head v 200'5981, mlcod=0'0)", 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:04.729041+0000", 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.900072537, 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.120639102, 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.729041+0000", 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.700 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.729041+0000", 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.729042+0000", 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8899999999999999e-07 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.729069+0000", 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6900999999999999e-05 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.701 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.729077+0000", 2024-03-11T22:16:18.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4200000000000007e-06 2024-03-11T22:16:18.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.702 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.702 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.702 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.729083+0000", 2024-03-11T22:16:18.702 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5459999999999999e-06 2024-03-11T22:16:18.702 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.702 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.702 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:18.704 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.748617+0000", 2024-03-11T22:16:18.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.019534068000000002 2024-03-11T22:16:18.704 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.704 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.704 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.704 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.748642+0000", 2024-03-11T22:16:18.704 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5805000000000001e-05 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.849638+0000", 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100995325 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.849680+0000", 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2447999999999997e-05 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.705 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.706 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.706 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:94785 8.4 8:2df8970a:::benchmark_data_smithi045_115206_object94784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-11T22:16:18.706 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:04.744693+0000", 2024-03-11T22:16:18.706 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.884420464, 2024-03-11T22:16:18.706 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.097173099999999998, 2024-03-11T22:16:18.706 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.706 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.706 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:16:18.706 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:16:18.706 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:16:18.706 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 94785 2024-03-11T22:16:18.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.707 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.707 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.707 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.707 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.744693+0000", 2024-03-11T22:16:18.707 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.707 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.744693+0000", 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.744694+0000", 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0999999999999999e-07 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.744720+0000", 2024-03-11T22:16:18.708 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6686000000000002e-05 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.744727+0000", 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8430000000000004e-06 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.744732+0000", 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1699999999999996e-06 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.748569+0000", 2024-03-11T22:16:18.709 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0038366630000000001 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.748699+0000", 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012982400000000001 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,4", 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.748857+0000", 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015844200000000001 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.710 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.754337+0000", 2024-03-11T22:16:18.711 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0054800630000000003 2024-03-11T22:16:18.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.711 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.754372+0000", 2024-03-11T22:16:18.711 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4382999999999999e-05 2024-03-11T22:16:18.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.711 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:16:18.711 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.841764+0000", 2024-03-11T22:16:18.711 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.087392194000000006 2024-03-11T22:16:18.711 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.711 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.841814+0000", 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0547999999999998e-05 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.841866+0000", 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1774000000000003e-05 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.712 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:94792 8.4 8:2160101a:::benchmark_data_smithi045_115206_object94791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-11T22:16:18.713 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:04.751515+0000", 2024-03-11T22:16:18.713 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.877598945999999, 2024-03-11T22:16:18.713 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.105010566, 2024-03-11T22:16:18.713 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.713 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.713 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:16:18.713 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:16:18.713 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:16:18.713 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 94792 2024-03-11T22:16:18.713 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.713 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.713 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.713 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751515+0000", 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751515+0000", 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751515+0000", 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1699999999999998e-07 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.714 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751552+0000", 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7258999999999997e-05 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751559+0000", 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9970000000000001e-06 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751565+0000", 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6339999999999998e-06 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.715 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751632+0000", 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6670999999999999e-05 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751832+0000", 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000200569 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,4", 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751973+0000", 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014118400000000001 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.716 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.797936+0000", 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.045962517000000001 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.797979+0000", 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3492999999999998e-05 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.856463+0000", 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.058483452999999998 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.717 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.718 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.718 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.856499+0000", 2024-03-11T22:16:18.718 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6013000000000002e-05 2024-03-11T22:16:18.718 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.718 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.718 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.718 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.856525+0000", 2024-03-11T22:16:18.718 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6259e-05 2024-03-11T22:16:18.718 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.718 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.718 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.718 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.718 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:94793 8.4 8:2b3ce712:::benchmark_data_smithi045_115206_object94792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:04.751574+0000", 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.877539598, 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10504740999999999, 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 94793 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.719 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751574+0000", 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751574+0000", 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751574+0000", 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2100000000000002e-07 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.720 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751608+0000", 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3782999999999998e-05 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751614+0000", 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0780000000000001e-06 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.751620+0000", 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7679999999999997e-06 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.721 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.754400+0000", 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0027796700000000001 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.754547+0000", 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000147093 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,4", 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.754949+0000", 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00040239200000000001 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.722 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.798017+0000", 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.043067399999999999 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.798059+0000", 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2290999999999999e-05 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.856531+0000", 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.058471750000000003 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.723 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.856584+0000", 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3399000000000002e-05 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.856621+0000", 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7364999999999999e-05 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:94778 8.c e200/194 8:34765f69:::benchmark_data_smithi045_115206_object94777:head v 200'5982, mlcod=0'0)", 2024-03-11T22:16:18.724 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:04.757147+0000", 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.871966332, 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.099502357999999999, 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.757147+0000", 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.757147+0000", 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.725 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.726 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.726 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.757148+0000", 2024-03-11T22:16:18.726 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9999999999999996e-07 2024-03-11T22:16:18.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.726 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.726 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.757191+0000", 2024-03-11T22:16:18.726 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3325999999999998e-05 2024-03-11T22:16:18.726 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.726 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.726 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.726 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.795782+0000", 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.038590669000000001 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.795791+0000", 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9290000000000004e-06 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.798086+0000", 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0022955139999999998 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.727 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.798111+0000", 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4471999999999999e-05 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.856623+0000", 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.058512461000000002 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:04.856650+0000", 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6186999999999998e-05 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.728 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:94939 8.c e200/194 8:33ba10e7:::benchmark_data_smithi045_115206_object94938:head v 200'5993, mlcod=0'0)", 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:05.137619+0000", 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.491495022000001, 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.094054339000000001, 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.137619+0000", 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.729 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.137619+0000", 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.137619+0000", 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7e-07 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.137667+0000", 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8287e-05 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.730 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.137684+0000", 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6759e-05 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.137689+0000", 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9919999999999998e-06 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.137729+0000", 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9549999999999999e-05 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.731 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.137745+0000", 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6419e-05 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.231632+0000", 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.093886977999999996 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.231673+0000", 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0883999999999997e-05 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.732 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:95006 8.c e200/194 8:34da5432:::benchmark_data_smithi045_115206_object95005:head v 200'6000, mlcod=0'0)", 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:05.162853+0000", 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 13.466260245999999, 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.098340194000000006, 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.162853+0000", 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.733 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.734 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.734 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.734 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.162853+0000", 2024-03-11T22:16:18.734 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.734 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.734 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.734 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.734 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.162854+0000", 2024-03-11T22:16:18.734 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2099999999999996e-07 2024-03-11T22:16:18.734 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.734 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.734 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.162908+0000", 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4239e-05 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.227607+0000", 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.064698961999999999 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.227633+0000", 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6169999999999998e-05 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.735 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.242994+0000", 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.015360888 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.243024+0000", 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0224000000000001e-05 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.261147+0000", 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.018122416999999998 2024-03-11T22:16:18.736 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.261194+0000", 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6672999999999999e-05 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:96699 8.c e200/194 8:3bf99845:::benchmark_data_smithi045_115206_object96698:head v 200'6095, mlcod=0'0)", 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:06.737961+0000", 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.891152404, 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10593997099999999, 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.737 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.737961+0000", 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.737961+0000", 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.738 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.737962+0000", 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2999999999999999e-07 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.738015+0000", 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3072999999999998e-05 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.738034+0000", 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8558000000000001e-05 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.739 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.738041+0000", 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5310000000000004e-06 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.740848+0000", 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0028071870000000001 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.740865+0000", 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6350000000000001e-05 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.740 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.843727+0000", 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102862541 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.843901+0000", 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017390099999999999 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.741 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:96708 8.c e200/194 8:314e47f3:::benchmark_data_smithi045_115206_object96707:head v 200'6096, mlcod=0'0)", 2024-03-11T22:16:18.742 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:06.741841+0000", 2024-03-11T22:16:18.742 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 11.887272691, 2024-03-11T22:16:18.742 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.108162407, 2024-03-11T22:16:18.742 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:18.742 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:18.742 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:18.742 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.742 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:18.742 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.741841+0000", 2024-03-11T22:16:18.742 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.742 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.742 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.742 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.741841+0000", 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.741842+0000", 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5700000000000002e-07 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.741892+0000", 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0098e-05 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.743 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:18.744 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.826441+0000", 2024-03-11T22:16:18.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.084548789999999999 2024-03-11T22:16:18.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.744 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:18.744 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.826457+0000", 2024-03-11T22:16:18.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6075000000000001e-05 2024-03-11T22:16:18.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.744 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.744 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:18.744 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.830769+0000", 2024-03-11T22:16:18.744 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0043121629999999999 2024-03-11T22:16:18.744 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.830793+0000", 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3815000000000002e-05 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.849936+0000", 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.019142985000000001 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:06.850003+0000", 2024-03-11T22:16:18.745 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7724000000000004e-05 2024-03-11T22:16:18.746 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.746 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.746 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.746 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:18.746 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:18.746 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:16:18.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:18.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:18.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:18.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:18.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:18.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:18.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:18.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:19.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:19.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:19.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:18 smithi144 ceph-mon[33984]: 6.14 scrub starts 2024-03-11T22:16:19.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:18 smithi144 ceph-mon[33984]: 6.14 scrub ok 2024-03-11T22:16:19.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:18 smithi144 ceph-mon[33984]: pgmap v334: 178 pgs: 1 active+recovering+undersized+remapped, 8 activating, 8 peering, 1 active+remapped+backfilling, 7 unknown, 153 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 6/320193 objects degraded (0.002%); 5590/320193 objects misplaced (1.746%); 18 MiB/s, 282 objects/s recovering 2024-03-11T22:16:19.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:18 smithi144 ceph-mon[33984]: osdmap e206: 8 total, 8 up, 8 in 2024-03-11T22:16:19.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:18 smithi144 ceph-mon[33984]: 5.c scrub starts 2024-03-11T22:16:19.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:18 smithi144 ceph-mon[33984]: 5.c scrub ok 2024-03-11T22:16:19.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:19.170 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:19.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:19.272 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:19.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:18 smithi045 ceph-mon[39715]: 6.14 scrub starts 2024-03-11T22:16:19.280 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:18 smithi045 ceph-mon[39715]: 6.14 scrub ok 2024-03-11T22:16:19.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:18 smithi045 ceph-mon[39715]: pgmap v334: 178 pgs: 1 active+recovering+undersized+remapped, 8 activating, 8 peering, 1 active+remapped+backfilling, 7 unknown, 153 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 6/320193 objects degraded (0.002%); 5590/320193 objects misplaced (1.746%); 18 MiB/s, 282 objects/s recovering 2024-03-11T22:16:19.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:18 smithi045 ceph-mon[39715]: osdmap e206: 8 total, 8 up, 8 in 2024-03-11T22:16:19.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:18 smithi045 ceph-mon[39715]: 5.c scrub starts 2024-03-11T22:16:19.281 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:18 smithi045 ceph-mon[39715]: 5.c scrub ok 2024-03-11T22:16:19.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:18 smithi045 ceph-mon[30178]: 6.14 scrub starts 2024-03-11T22:16:19.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:18 smithi045 ceph-mon[30178]: 6.14 scrub ok 2024-03-11T22:16:19.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:18 smithi045 ceph-mon[30178]: pgmap v334: 178 pgs: 1 active+recovering+undersized+remapped, 8 activating, 8 peering, 1 active+remapped+backfilling, 7 unknown, 153 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 6/320193 objects degraded (0.002%); 5590/320193 objects misplaced (1.746%); 18 MiB/s, 282 objects/s recovering 2024-03-11T22:16:19.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:18 smithi045 ceph-mon[30178]: osdmap e206: 8 total, 8 up, 8 in 2024-03-11T22:16:19.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:18 smithi045 ceph-mon[30178]: 5.c scrub starts 2024-03-11T22:16:19.281 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:18 smithi045 ceph-mon[30178]: 5.c scrub ok 2024-03-11T22:16:19.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:19.303 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:19.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:19.329 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 77 15 110721 110706 89.8356 65.625 0.00464155 0.0111216 2024-03-11T22:16:19.404 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:19.430 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:19.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:19.531 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:19.554 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:19.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:19.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:19.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:19.687 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:19.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:19.788 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:19.811 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:19.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:19.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:19.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:16:19.913 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:19.938 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:19.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:19.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:19.969 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T22:16:20.040 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:20.054 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:16:20.054 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:16:20.054 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-11T22:16:20.055 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T22:16:20.055 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T22:16:20.055 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-11T22:16:20.076 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:20.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:20.178 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:20.190 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:20 smithi045 ceph-mon[30178]: osdmap e207: 8 total, 8 up, 8 in 2024-03-11T22:16:20.190 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:20 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:16:20.190 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:20 smithi045 ceph-mon[30178]: 5.d scrub starts 2024-03-11T22:16:20.205 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:20.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:20.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:20.298 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:16:20.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:20.330 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 78 16 110883 110867 88.8122 10.0625 0.0170106 0.0112537 2024-03-11T22:16:20.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:20 smithi144 ceph-mon[33984]: osdmap e207: 8 total, 8 up, 8 in 2024-03-11T22:16:20.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:20 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:16:20.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:20 smithi144 ceph-mon[33984]: 5.d scrub starts 2024-03-11T22:16:20.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:20 smithi144 ceph-mon[33984]: 5.d scrub ok 2024-03-11T22:16:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:20.350 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:20.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:20.410 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:20.451 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:20.481 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:20.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:20.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:20 smithi045 ceph-mon[39715]: osdmap e207: 8 total, 8 up, 8 in 2024-03-11T22:16:20.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:20 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:16:20.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:20 smithi045 ceph-mon[39715]: 5.d scrub starts 2024-03-11T22:16:20.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:20 smithi045 ceph-mon[39715]: 5.d scrub ok 2024-03-11T22:16:20.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:20 smithi045 ceph-mon[30178]: 5.d scrub ok 2024-03-11T22:16:20.583 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:20.629 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:20.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:20.729 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:20.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:20.769 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:20.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:20.870 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:20.903 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:20.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:20.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:21.006 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:21.037 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:21.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:21.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:21.063 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:16:21.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:21.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:21.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:21.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:21.223 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:16:21.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:21.326 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:21.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:21.330 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 79 15 111979 111964 88.5559 68.5625 0.0200141 0.0112868 2024-03-11T22:16:21.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:21.467 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:21.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:21.527 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:21 smithi144 ceph-mon[33984]: pgmap v337: 178 pgs: 1 active+recovering+undersized+remapped, 8 activating, 8 peering, 1 active+remapped+backfilling, 7 unknown, 153 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 34 MiB/s wr, 545 op/s; 6/320193 objects degraded (0.002%); 5590/320193 objects misplaced (1.746%); 1.2 MiB/s, 19 objects/s recovering 2024-03-11T22:16:21.527 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:21 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:16:21.527 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:21 smithi144 ceph-mon[33984]: osdmap e208: 8 total, 8 up, 8 in 2024-03-11T22:16:21.527 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:21 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:16:21.527 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:21 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:16:21.527 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:21 smithi144 ceph-mon[33984]: 8.a scrub starts 2024-03-11T22:16:21.527 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:21 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/650145896' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:21.527 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:21 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:21.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[30178]: pgmap v337: 178 pgs: 1 active+recovering+undersized+remapped, 8 activating, 8 peering, 1 active+remapped+backfilling, 7 unknown, 153 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 34 MiB/s wr, 545 op/s; 6/320193 objects degraded (0.002%); 5590/320193 objects misplaced (1.746%); 1.2 MiB/s, 19 objects/s recovering 2024-03-11T22:16:21.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:16:21.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[30178]: osdmap e208: 8 total, 8 up, 8 in 2024-03-11T22:16:21.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:16:21.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:16:21.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[30178]: 8.a scrub starts 2024-03-11T22:16:21.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/650145896' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:21.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:21.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[39715]: pgmap v337: 178 pgs: 1 active+recovering+undersized+remapped, 8 activating, 8 peering, 1 active+remapped+backfilling, 7 unknown, 153 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 34 MiB/s wr, 545 op/s; 6/320193 objects degraded (0.002%); 5590/320193 objects misplaced (1.746%); 1.2 MiB/s, 19 objects/s recovering 2024-03-11T22:16:21.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:16:21.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[39715]: osdmap e208: 8 total, 8 up, 8 in 2024-03-11T22:16:21.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:16:21.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:16:21.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[39715]: 8.a scrub starts 2024-03-11T22:16:21.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/650145896' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:21.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:21 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:21.568 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:21.608 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:21.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:21.710 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:21.754 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:21.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:21.857 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:21.895 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:21.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:21.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:22.036 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:22.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:22.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:22.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:22.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:22.279 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:22.318 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:22.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:22.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:22.330 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:16:22.331830+0000 min lat: 0.00125214 max lat: 2.01083 avg lat: 0.0113127 2024-03-11T22:16:22.330 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:16:22.330 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 80 16 113143 113127 88.3575 72.6875 0.0378851 0.0113127 2024-03-11T22:16:22.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:22 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:16:22.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:22 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:16:22.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:22 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:16:22.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:22 smithi144 ceph-mon[33984]: osdmap e209: 8 total, 8 up, 8 in 2024-03-11T22:16:22.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:22 smithi144 ceph-mon[33984]: 1.1a scrub starts 2024-03-11T22:16:22.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:22 smithi144 ceph-mon[33984]: 1.1a scrub ok 2024-03-11T22:16:22.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:22 smithi144 ceph-mon[33984]: 5.8 scrub starts 2024-03-11T22:16:22.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:22 smithi144 ceph-mon[33984]: 5.8 scrub ok 2024-03-11T22:16:22.421 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:22.458 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:22.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:22.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:16:22.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:16:22.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:16:22.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[30178]: osdmap e209: 8 total, 8 up, 8 in 2024-03-11T22:16:22.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[30178]: 1.1a scrub starts 2024-03-11T22:16:22.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[30178]: 1.1a scrub ok 2024-03-11T22:16:22.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[30178]: 5.8 scrub starts 2024-03-11T22:16:22.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[30178]: 5.8 scrub ok 2024-03-11T22:16:22.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:16:22.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:16:22.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:16:22.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[39715]: osdmap e209: 8 total, 8 up, 8 in 2024-03-11T22:16:22.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[39715]: 1.1a scrub starts 2024-03-11T22:16:22.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[39715]: 1.1a scrub ok 2024-03-11T22:16:22.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[39715]: 5.8 scrub starts 2024-03-11T22:16:22.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-mon[39715]: 5.8 scrub ok 2024-03-11T22:16:22.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:22.601 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:22.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:22.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:22.703 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:22.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:22.762 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:22.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:22.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:22.877 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:16:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:16:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:16:22.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:22.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:22.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:23.041 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:23.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:23.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:23.186 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:23.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:23.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:23.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:23.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:23.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:23.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:23.330 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 81 16 114294 114278 88.1547 71.9375 0.00460546 0.0113367 2024-03-11T22:16:23.405 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:16:23.427 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:23.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[30178]: 1.1c scrub starts 2024-03-11T22:16:23.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[30178]: 1.1c scrub ok 2024-03-11T22:16:23.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[30178]: pgmap v340: 178 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 9 peering, 1 active+undersized+remapped, 166 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 6.5 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 4573/334038 objects misplaced (1.369%); 16 MiB/s, 258 objects/s recovering 2024-03-11T22:16:23.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[30178]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:16:23.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[30178]: 1.16 scrub starts 2024-03-11T22:16:23.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[30178]: 1.16 scrub ok 2024-03-11T22:16:23.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[30178]: 8.a scrub ok 2024-03-11T22:16:23.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[30178]: 1.6 scrub starts 2024-03-11T22:16:23.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[30178]: 1.6 scrub ok 2024-03-11T22:16:23.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[39715]: 1.1c scrub starts 2024-03-11T22:16:23.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[39715]: 1.1c scrub ok 2024-03-11T22:16:23.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[39715]: pgmap v340: 178 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 9 peering, 1 active+undersized+remapped, 166 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 6.5 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 4573/334038 objects misplaced (1.369%); 16 MiB/s, 258 objects/s recovering 2024-03-11T22:16:23.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[39715]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:16:23.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[39715]: 1.16 scrub starts 2024-03-11T22:16:23.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[39715]: 1.16 scrub ok 2024-03-11T22:16:23.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[39715]: 8.a scrub ok 2024-03-11T22:16:23.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[39715]: 1.6 scrub starts 2024-03-11T22:16:23.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:23 smithi045 ceph-mon[39715]: 1.6 scrub ok 2024-03-11T22:16:23.451 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:16:23.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:23.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:23.485 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:16:23.495 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:23 smithi144 ceph-mon[33984]: 1.1c scrub starts 2024-03-11T22:16:23.495 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:23 smithi144 ceph-mon[33984]: 1.1c scrub ok 2024-03-11T22:16:23.495 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:23 smithi144 ceph-mon[33984]: pgmap v340: 178 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 9 peering, 1 active+undersized+remapped, 166 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 6.5 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 4573/334038 objects misplaced (1.369%); 16 MiB/s, 258 objects/s recovering 2024-03-11T22:16:23.496 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:23 smithi144 ceph-mon[33984]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:16:23.496 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:23 smithi144 ceph-mon[33984]: 1.16 scrub starts 2024-03-11T22:16:23.496 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:23 smithi144 ceph-mon[33984]: 1.16 scrub ok 2024-03-11T22:16:23.496 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:23 smithi144 ceph-mon[33984]: 8.a scrub ok 2024-03-11T22:16:23.496 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:23 smithi144 ceph-mon[33984]: 1.6 scrub starts 2024-03-11T22:16:23.496 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:23 smithi144 ceph-mon[33984]: 1.6 scrub ok 2024-03-11T22:16:23.524 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:16:23.568 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:16:23.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:23.611 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:16:23.616 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:23.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:23.652 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:16:23.691 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:16:23.703 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:16:23.703 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:16:23.703 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:16:23.703 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:16:23.703 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:16:23.704 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:16:23.704 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:16:23.704 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:16:23.718 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:23.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:23.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:23.864 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:23.914 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:23.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:24.015 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:24.056 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:24.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:24.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:24.157 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:24.191 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:16:24.191 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:16:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:24.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:24.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:24.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:16:24.296 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:24.329 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:24.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:24.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:24.331 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 82 16 115497 115481 87.9956 75.1875 0.00529927 0.011359 2024-03-11T22:16:24.405 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:24 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:16:24.405 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:24 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:16:24.405 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:24 smithi144 ceph-mon[33984]: osdmap e210: 8 total, 8 up, 8 in 2024-03-11T22:16:24.405 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:24 smithi144 ceph-mon[33984]: 1.1d scrub starts 2024-03-11T22:16:24.405 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:24 smithi144 ceph-mon[33984]: 1.1d scrub ok 2024-03-11T22:16:24.405 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:24 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/167442172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:24.406 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:24 smithi144 ceph-mon[33984]: 3.13 scrub starts 2024-03-11T22:16:24.406 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:24 smithi144 ceph-mon[33984]: 3.13 scrub ok 2024-03-11T22:16:24.406 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:24 smithi144 ceph-mon[33984]: 1.7 scrub starts 2024-03-11T22:16:24.406 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:24 smithi144 ceph-mon[33984]: 1.7 scrub ok 2024-03-11T22:16:24.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:24.482 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:24.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:24.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:16:24.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:16:24.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[30178]: osdmap e210: 8 total, 8 up, 8 in 2024-03-11T22:16:24.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[30178]: 1.1d scrub starts 2024-03-11T22:16:24.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[30178]: 1.1d scrub ok 2024-03-11T22:16:24.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/167442172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:24.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[30178]: 3.13 scrub starts 2024-03-11T22:16:24.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[30178]: 3.13 scrub ok 2024-03-11T22:16:24.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[30178]: 1.7 scrub starts 2024-03-11T22:16:24.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[30178]: 1.7 scrub ok 2024-03-11T22:16:24.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:16:24.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:16:24.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[39715]: osdmap e210: 8 total, 8 up, 8 in 2024-03-11T22:16:24.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[39715]: 1.1d scrub starts 2024-03-11T22:16:24.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[39715]: 1.1d scrub ok 2024-03-11T22:16:24.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/167442172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:24.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[39715]: 3.13 scrub starts 2024-03-11T22:16:24.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[39715]: 3.13 scrub ok 2024-03-11T22:16:24.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[39715]: 1.7 scrub starts 2024-03-11T22:16:24.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:24 smithi045 ceph-mon[39715]: 1.7 scrub ok 2024-03-11T22:16:24.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:24.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:24.643 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:24.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:24.742 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:24.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:24.802 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:24.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:24.903 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:24.939 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:24.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:25.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:25.083 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T22:16:25.087 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:25.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:25.144 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:16:25.144 INFO:teuthology.orchestra.run.smithi144.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T22:16:25.144 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:16:25.188 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:25.223 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:25.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:25.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:25.325 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:25.331 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 83 16 116705 116689 87.8448 75.5 0.0104062 0.0113791 2024-03-11T22:16:25.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:25 smithi144 ceph-mon[33984]: pgmap v342: 178 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 9 peering, 1 active+undersized+remapped, 166 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 5.2 KiB/s rd, 58 MiB/s wr, 930 op/s; 4573/334038 objects misplaced (1.369%); 13 MiB/s, 207 objects/s recovering 2024-03-11T22:16:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:25.362 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:25.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:25.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:25.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:25.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:25.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:25.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:25 smithi045 ceph-mon[30178]: pgmap v342: 178 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 9 peering, 1 active+undersized+remapped, 166 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 5.2 KiB/s rd, 58 MiB/s wr, 930 op/s; 4573/334038 objects misplaced (1.369%); 13 MiB/s, 207 objects/s recovering 2024-03-11T22:16:25.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:25 smithi045 ceph-mon[39715]: pgmap v342: 178 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 9 peering, 1 active+undersized+remapped, 166 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 5.2 KiB/s rd, 58 MiB/s wr, 930 op/s; 4573/334038 objects misplaced (1.369%); 13 MiB/s, 207 objects/s recovering 2024-03-11T22:16:25.603 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:25.639 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:25.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:25.740 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:25.776 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:25.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:25.877 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:25.914 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:25.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:25.983 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:16:26.016 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:26.056 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:26.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:26.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:26.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:26.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:26.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:26.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:26.201 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:16:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:16:26.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:26.323 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:26.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:26.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:26.331 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 84 16 117870 117854 87.6657 72.8125 0.00240179 0.0113983 2024-03-11T22:16:26.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:26.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:26.467 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:26.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:26.478 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:26 smithi144 ceph-mon[33984]: 1.13 scrub starts 2024-03-11T22:16:26.479 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:26 smithi144 ceph-mon[33984]: 1.13 scrub ok 2024-03-11T22:16:26.479 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:26 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:16:26.479 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:26 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:16:26.479 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:26 smithi144 ceph-mon[33984]: 6.17 scrub starts 2024-03-11T22:16:26.479 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:26 smithi144 ceph-mon[33984]: 6.17 scrub ok 2024-03-11T22:16:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:26 smithi045 ceph-mon[39715]: 1.13 scrub starts 2024-03-11T22:16:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:26 smithi045 ceph-mon[39715]: 1.13 scrub ok 2024-03-11T22:16:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:26 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:16:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:26 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:16:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:26 smithi045 ceph-mon[39715]: 6.17 scrub starts 2024-03-11T22:16:26.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:26 smithi045 ceph-mon[39715]: 6.17 scrub ok 2024-03-11T22:16:26.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:26 smithi045 ceph-mon[30178]: 1.13 scrub starts 2024-03-11T22:16:26.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:26 smithi045 ceph-mon[30178]: 1.13 scrub ok 2024-03-11T22:16:26.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:26 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:16:26.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:26 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:16:26.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:26 smithi045 ceph-mon[30178]: 6.17 scrub starts 2024-03-11T22:16:26.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:26 smithi045 ceph-mon[30178]: 6.17 scrub ok 2024-03-11T22:16:26.568 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:26.608 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:26.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:26.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:26.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:16:26.710 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:26.744 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:26.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:26.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:26.846 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:26.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:26.883 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:26.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:26.984 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:27.018 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:27.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:27.119 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:27.152 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:27.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:27.254 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:27.259 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:27.289 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:27.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:27.333 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 85 16 119106 119090 87.5415 77.25 0.00915261 0.0114185 2024-03-11T22:16:27.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:27.417 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:27.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:27.519 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:27.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:27.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[39715]: pgmap v343: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 109 MiB/s wr, 1.75k op/s; 4476/351744 objects misplaced (1.273%); 12 MiB/s, 188 objects/s recovering 2024-03-11T22:16:27.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[39715]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-11T22:16:27.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:16:27.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[39715]: osdmap e211: 8 total, 8 up, 8 in 2024-03-11T22:16:27.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[39715]: 2.1 scrub starts 2024-03-11T22:16:27.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[39715]: 2.1 scrub ok 2024-03-11T22:16:27.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:27.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[30178]: pgmap v343: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 109 MiB/s wr, 1.75k op/s; 4476/351744 objects misplaced (1.273%); 12 MiB/s, 188 objects/s recovering 2024-03-11T22:16:27.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[30178]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-11T22:16:27.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:16:27.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[30178]: osdmap e211: 8 total, 8 up, 8 in 2024-03-11T22:16:27.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[30178]: 2.1 scrub starts 2024-03-11T22:16:27.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:27 smithi045 ceph-mon[30178]: 2.1 scrub ok 2024-03-11T22:16:27.536 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:27.536 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:27 smithi144 ceph-mon[33984]: pgmap v343: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 109 MiB/s wr, 1.75k op/s; 4476/351744 objects misplaced (1.273%); 12 MiB/s, 188 objects/s recovering 2024-03-11T22:16:27.536 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:27 smithi144 ceph-mon[33984]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-11T22:16:27.536 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:27 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:16:27.536 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:27 smithi144 ceph-mon[33984]: osdmap e211: 8 total, 8 up, 8 in 2024-03-11T22:16:27.536 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:27 smithi144 ceph-mon[33984]: 2.1 scrub starts 2024-03-11T22:16:27.536 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:27 smithi144 ceph-mon[33984]: 2.1 scrub ok 2024-03-11T22:16:27.551 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:27.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:27.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:27.693 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:27.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:27.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:27.794 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:27.827 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:27.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:27.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:27.964 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:27.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:27.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:28.056 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:16:28.056 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:16:28.066 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:28.105 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:28.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:28.207 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:28.245 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:28.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:28.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:28.334 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 86 16 120244 120228 87.3506 71.125 0.0108402 0.0114434 2024-03-11T22:16:28.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:28 smithi144 ceph-mon[33984]: 5.1f scrub starts 2024-03-11T22:16:28.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:28 smithi144 ceph-mon[33984]: 5.1f scrub ok 2024-03-11T22:16:28.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:28 smithi144 ceph-mon[33984]: 4.d scrub starts 2024-03-11T22:16:28.345 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:28 smithi144 ceph-mon[33984]: 4.d scrub ok 2024-03-11T22:16:28.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:28.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:28.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:28.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:28.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:28.531 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:28.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:28.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:28.669 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:28.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:28.771 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:28 smithi045 ceph-mon[39715]: 5.1f scrub starts 2024-03-11T22:16:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:28 smithi045 ceph-mon[39715]: 5.1f scrub ok 2024-03-11T22:16:28.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:28 smithi045 ceph-mon[39715]: 4.d scrub starts 2024-03-11T22:16:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:28 smithi045 ceph-mon[39715]: 4.d scrub ok 2024-03-11T22:16:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:28 smithi045 ceph-mon[30178]: 5.1f scrub starts 2024-03-11T22:16:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:28 smithi045 ceph-mon[30178]: 5.1f scrub ok 2024-03-11T22:16:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:28 smithi045 ceph-mon[30178]: 4.d scrub starts 2024-03-11T22:16:28.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:28 smithi045 ceph-mon[30178]: 4.d scrub ok 2024-03-11T22:16:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:28.799 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:28.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:28.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:28.926 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:28.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:29.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:29.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:29.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:29.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:29.171 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:29.203 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:29.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:29.305 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:29.334 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 87 16 121130 121114 86.983 55.375 0.148904 0.0114865 2024-03-11T22:16:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:29.335 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:29.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:29.436 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:29.466 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:29.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:29.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:29.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:29 smithi045 ceph-mon[39715]: 4.18 deep-scrub starts 2024-03-11T22:16:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:29 smithi045 ceph-mon[39715]: 4.18 deep-scrub ok 2024-03-11T22:16:29.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:29 smithi045 ceph-mon[39715]: pgmap v345: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 53 MiB/s wr, 843 op/s; 4476/351744 objects misplaced (1.273%); 872 KiB/s, 13 objects/s recovering 2024-03-11T22:16:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:29 smithi045 ceph-mon[30178]: 4.18 deep-scrub starts 2024-03-11T22:16:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:29 smithi045 ceph-mon[30178]: 4.18 deep-scrub ok 2024-03-11T22:16:29.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:29 smithi045 ceph-mon[30178]: pgmap v345: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 53 MiB/s wr, 843 op/s; 4476/351744 objects misplaced (1.273%); 872 KiB/s, 13 objects/s recovering 2024-03-11T22:16:29.567 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:29.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:29 smithi144 ceph-mon[33984]: 4.18 deep-scrub starts 2024-03-11T22:16:29.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:29 smithi144 ceph-mon[33984]: 4.18 deep-scrub ok 2024-03-11T22:16:29.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:29 smithi144 ceph-mon[33984]: pgmap v345: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 53 MiB/s wr, 843 op/s; 4476/351744 objects misplaced (1.273%); 872 KiB/s, 13 objects/s recovering 2024-03-11T22:16:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:29.602 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:29.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:29.703 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:29.731 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:29.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:29.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:29.832 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:29.862 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:29.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:29.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:30.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:30.001 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:30.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:30.103 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:30.138 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:30.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:30.239 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:30.278 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:30.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:30.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:30.334 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 88 16 122302 122286 86.8269 73.25 0.00832853 0.0115119 2024-03-11T22:16:30.379 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:30.411 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:30.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:30.513 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:30.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:30 smithi045 ceph-mon[39715]: 5.9 scrub starts 2024-03-11T22:16:30.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:30 smithi045 ceph-mon[39715]: 5.9 scrub ok 2024-03-11T22:16:30.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:16:30.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:16:30.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:30 smithi045 ceph-mon[30178]: 5.9 scrub starts 2024-03-11T22:16:30.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:30 smithi045 ceph-mon[30178]: 5.9 scrub ok 2024-03-11T22:16:30.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:16:30.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:16:30.536 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:30.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:30.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:30.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:30 smithi144 ceph-mon[33984]: 5.9 scrub starts 2024-03-11T22:16:30.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:30 smithi144 ceph-mon[33984]: 5.9 scrub ok 2024-03-11T22:16:30.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:16:30.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:16:30.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:30.676 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:30.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:30.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:30.778 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:30.813 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:30.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:30.915 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:30.929 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:16:30.929 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:16:30.929 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-11T22:16:30.929 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-03-11T22:16:30.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-11T22:16:30.947 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:30.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:30.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:31.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:31.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:31.112 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:31.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:31.208 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T22:16:31.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:31.262 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:31.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:31.334 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 89 16 123229 123213 86.5023 57.9375 0.00553416 0.0115499 2024-03-11T22:16:31.364 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:31.390 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:31.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:31.492 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:31.512 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:16:31.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:31.531 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:31.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:31.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:31 smithi144 ceph-mon[33984]: pgmap v346: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 46 MiB/s wr, 735 op/s; 4476/351744 objects misplaced (1.273%); 760 KiB/s, 12 objects/s recovering 2024-03-11T22:16:31.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:31 smithi144 ceph-mon[33984]: 8.1 scrub starts 2024-03-11T22:16:31.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:31 smithi144 ceph-mon[33984]: 6.12 scrub starts 2024-03-11T22:16:31.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:31 smithi144 ceph-mon[33984]: 6.12 scrub ok 2024-03-11T22:16:31.632 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:31.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:31.666 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:31.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:31.767 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:31.780 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:31 smithi045 ceph-mon[30178]: pgmap v346: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 46 MiB/s wr, 735 op/s; 4476/351744 objects misplaced (1.273%); 760 KiB/s, 12 objects/s recovering 2024-03-11T22:16:31.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:31 smithi045 ceph-mon[30178]: 8.1 scrub starts 2024-03-11T22:16:31.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:31 smithi045 ceph-mon[30178]: 6.12 scrub starts 2024-03-11T22:16:31.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:31 smithi045 ceph-mon[30178]: 6.12 scrub ok 2024-03-11T22:16:31.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:31 smithi045 ceph-mon[39715]: pgmap v346: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 46 MiB/s wr, 735 op/s; 4476/351744 objects misplaced (1.273%); 760 KiB/s, 12 objects/s recovering 2024-03-11T22:16:31.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:31 smithi045 ceph-mon[39715]: 8.1 scrub starts 2024-03-11T22:16:31.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:31 smithi045 ceph-mon[39715]: 6.12 scrub starts 2024-03-11T22:16:31.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:31 smithi045 ceph-mon[39715]: 6.12 scrub ok 2024-03-11T22:16:31.793 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:31.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:31.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:31.942 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:31.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:32.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:32.079 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:32.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:32.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:32.180 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:32.207 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:32.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:32.309 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:32.318 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:16:32.333 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:32.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:32.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:32.335 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Total time run: 90.0072 2024-03-11T22:16:32.335 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Total writes made: 124115 2024-03-11T22:16:32.335 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Write size: 65536 2024-03-11T22:16:32.335 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Object size: 65536 2024-03-11T22:16:32.335 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Bandwidth (MB/sec): 86.1841 2024-03-11T22:16:32.335 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev Bandwidth: 34.6918 2024-03-11T22:16:32.335 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max bandwidth (MB/sec): 194.562 2024-03-11T22:16:32.335 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min bandwidth (MB/sec): 2.125 2024-03-11T22:16:32.335 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Average IOPS: 1378 2024-03-11T22:16:32.335 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev IOPS: 555.069 2024-03-11T22:16:32.336 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max IOPS: 3113 2024-03-11T22:16:32.336 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min IOPS: 34 2024-03-11T22:16:32.336 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Average Latency(s): 0.0115988 2024-03-11T22:16:32.336 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev Latency(s): 0.0264051 2024-03-11T22:16:32.336 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max latency(s): 2.01083 2024-03-11T22:16:32.336 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min latency(s): 0.00125214 2024-03-11T22:16:32.338 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T22:16:32.435 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:32.469 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:32.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:32.488 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:32 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1336825289' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:16:32.488 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:32 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1336825289' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:16:32.548 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:32 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1336825289' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:16:32.571 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:32.606 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:32.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:32.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:32.747 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:32.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:32.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:32.780 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:16:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:16:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:16:32.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:32.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:32.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:32.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:32.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:33.014 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:33.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:33.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:33.116 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:33.149 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:33.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:33.251 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:33.281 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:33.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:33.383 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:33.419 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:33.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:33.521 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:33.557 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:33.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:33 smithi045 ceph-mon[30178]: pgmap v347: 178 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 176 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 4382/368421 objects misplaced (1.189%); 1.3 MiB/s, 21 objects/s recovering 2024-03-11T22:16:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:33 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:33.563 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:33 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1336825289' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:16:33.564 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:33 smithi045 ceph-mon[30178]: osdmap e212: 8 total, 8 up, 8 in 2024-03-11T22:16:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:33 smithi045 ceph-mon[39715]: pgmap v347: 178 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 176 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 4382/368421 objects misplaced (1.189%); 1.3 MiB/s, 21 objects/s recovering 2024-03-11T22:16:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:33 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:33 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1336825289' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:16:33.564 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:33 smithi045 ceph-mon[39715]: osdmap e212: 8 total, 8 up, 8 in 2024-03-11T22:16:33.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:33 smithi144 ceph-mon[33984]: pgmap v347: 178 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 176 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 4382/368421 objects misplaced (1.189%); 1.3 MiB/s, 21 objects/s recovering 2024-03-11T22:16:33.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:33 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:33.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:33 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1336825289' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:16:33.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:33 smithi144 ceph-mon[33984]: osdmap e212: 8 total, 8 up, 8 in 2024-03-11T22:16:33.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:33.693 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:33.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:33.794 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:33.837 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:33.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:33.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:33.975 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:33.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:34.077 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:34.113 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:34.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:34.257 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:34.290 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:34.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:34.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:34.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:34.431 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:34.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:34.505 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:16:34.505 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:16:34.532 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:34.568 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:34.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:34.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:34.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:34.711 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:34.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:34.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:34.835 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:34.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:34.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:34.937 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:34.963 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:34.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:35.065 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:35.099 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:35.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:35.200 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:35.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:35.226 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:35.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:35.327 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:35.366 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:35.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:35.416 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T22:16:35.468 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:35.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:35.510 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:35.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:35.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:35 smithi144 ceph-mon[33984]: pgmap v349: 178 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 176 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 695 op/s; 4382/368421 objects misplaced (1.189%); 752 KiB/s, 11 objects/s recovering 2024-03-11T22:16:35.611 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:35.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:35.648 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:35.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:35.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:35.780 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:35 smithi045 ceph-mon[39715]: pgmap v349: 178 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 176 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 695 op/s; 4382/368421 objects misplaced (1.189%); 752 KiB/s, 11 objects/s recovering 2024-03-11T22:16:35.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:35 smithi045 ceph-mon[30178]: pgmap v349: 178 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 176 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 695 op/s; 4382/368421 objects misplaced (1.189%); 752 KiB/s, 11 objects/s recovering 2024-03-11T22:16:35.789 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:35.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:35.795 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:16:35.890 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:35.929 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:35.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:35.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:36.030 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:36.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:36.067 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:36.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:36.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:36.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:36.215 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:36.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:36.262 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:16:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:16:36.316 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:36.350 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:36.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:36.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:36.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:36.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:36.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:36.630 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:36.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:36.731 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:36.753 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:36.766 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:36.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:36.868 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:36.903 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:36.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:37.005 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:37.041 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:37.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:37.143 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:37.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:37.177 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:37.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:37.278 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:37.319 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:37.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:37.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:37.344 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:37 smithi144 ceph-mon[33984]: pgmap v350: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 1.11k op/s; 4301/360927 objects misplaced (1.192%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:16:37.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:37.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:37.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:37.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:37.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:37 smithi045 ceph-mon[30178]: pgmap v350: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 1.11k op/s; 4301/360927 objects misplaced (1.192%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:16:37.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:37.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:37 smithi045 ceph-mon[39715]: pgmap v350: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 1.11k op/s; 4301/360927 objects misplaced (1.192%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:16:37.562 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:37.589 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:37.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:37.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:37.691 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:37.729 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:37.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:37.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:37.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:37.832 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:16:37.832 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:16:37.863 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:37.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:37.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:37.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:37.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:16:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:37.992 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:37.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:38.093 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:38.125 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:38.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:38.226 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:38.251 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:38.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:38.352 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:38.409 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:38.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:38.507 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T22:16:38.511 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:38.583 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:38.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:38.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:38.685 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:38.713 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:16:38.725 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:38.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:38.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:38.758 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:16:38.798 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:16:38.827 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:38.842 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:16:38.861 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:16:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:38.873 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:38.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:38.884 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:16:38.928 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:16:38.972 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:16:38.974 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:39.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:39.011 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:39.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:39.012 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:16:39.016 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:16:39.016 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:16:39.016 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:16:39.016 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:16:39.017 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:16:39.017 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:16:39.017 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:16:39.017 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:16:39.112 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:39.151 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:39.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:39.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:39.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:39.293 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:39.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:39.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:39.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:39.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:39.525 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:39 smithi144 ceph-mon[33984]: pgmap v351: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 1.11k op/s; 4301/360927 objects misplaced (1.192%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:16:39.525 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:39 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3428250956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:39.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:39 smithi045 ceph-mon[39715]: pgmap v351: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 1.11k op/s; 4301/360927 objects misplaced (1.192%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:16:39.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:39 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3428250956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:39.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:39 smithi045 ceph-mon[30178]: pgmap v351: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 1.11k op/s; 4301/360927 objects misplaced (1.192%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:16:39.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:39 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3428250956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:39.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:39.571 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:39.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:39.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:39.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:39.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:39.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:39.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:39.855 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:39.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:39.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:39.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:39.997 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:39.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:39.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:40.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:40.133 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:40.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:40.235 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:40.270 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:40.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:40.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:40.372 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:40.407 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:40.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:40.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:40.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:40.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:40.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:40.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:40.685 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:40.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:40.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:40.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:40.825 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:40.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:40.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:40.903 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:40.927 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:40.971 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:16:40.971 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-03-11T22:16:40.971 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-03-11T22:16:40.971 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-03-11T22:16:40.971 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:16:40.981 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:40.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:40.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:41.085 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:41.120 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:41.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:41.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:41.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:41.264 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:41.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:41.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:41.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:41.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:41.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:41.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:41.510 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:41 smithi045 ceph-mon[30178]: pgmap v352: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 1.11k op/s; 4301/360927 objects misplaced (1.192%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:16:41.510 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:41 smithi045 ceph-mon[39715]: pgmap v352: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 1.11k op/s; 4301/360927 objects misplaced (1.192%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:16:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:41.556 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:41.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:41.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:41 smithi144 ceph-mon[33984]: pgmap v352: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 1.11k op/s; 4301/360927 objects misplaced (1.192%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:16:41.653 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T22:16:41.658 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:41.738 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:41.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:41.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:41.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:41.878 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:41.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:41.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:41.942 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:16:41.980 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:42.017 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:42.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:42.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:42.035 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:16:42.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:42.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:42.177 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:42.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:42.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:42.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:42.315 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:42.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:42.415 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:42.465 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:42.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:42.567 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:42.582 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:16:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:16:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:16:42.595 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:42.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:42.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:42.740 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:42.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:42.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:42.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:42.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:42.985 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:43.020 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:43.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:43.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:43.121 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:43.153 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:43.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:43.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:43.300 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:43.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:43.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:43.351 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:43.402 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:43.432 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:43.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:43.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:43.497 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:43 smithi045 ceph-mon[39715]: pgmap v353: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 9.4 MiB/s wr, 1.17k op/s; 4207/342432 objects misplaced (1.229%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:16:43.498 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:43 smithi045 ceph-mon[30178]: pgmap v353: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 9.4 MiB/s wr, 1.17k op/s; 4207/342432 objects misplaced (1.229%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:16:43.534 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:43.562 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:43.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:43.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:43 smithi144 ceph-mon[33984]: pgmap v353: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 9.4 MiB/s wr, 1.17k op/s; 4207/342432 objects misplaced (1.229%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:16:43.665 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:43.694 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:43.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:43.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:43.796 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:43.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:43.830 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:43.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:43.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:43.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:43.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:44.069 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:44.105 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:44.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:44.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:44.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:44.247 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:44.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:44.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:44.281 INFO:teuthology.orchestra.run.smithi045.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-11T22:16:44.293 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:16:44.349 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:44.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:44.383 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:44.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:44.484 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:44.516 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:44.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:44.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:44 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2817997064' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:16:44.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:44 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/864095855' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:44.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:44 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:44.608 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:16:44.608 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-03-11T22:16:44.608 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-03-11T22:16:44.608 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:16:44.608 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.608 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:158976 7.8 e137/99 7:1dedfb11:::benchmark_data_smithi045_83007_object40505:head v 137'10058, mlcod=137'10052)", 2024-03-11T22:16:44.608 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:54.947094+0000", 2024-03-11T22:16:44.608 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 229.65870075199999, 2024-03-11T22:16:44.608 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.013789272, 2024-03-11T22:16:44.608 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.608 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.608 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.608 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.947094+0000", 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.947094+0000", 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.947095+0000", 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.077e-06 2024-03-11T22:16:44.609 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.947102+0000", 2024-03-11T22:16:44.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3420000000000002e-06 2024-03-11T22:16:44.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.947117+0000", 2024-03-11T22:16:44.610 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4756999999999999e-05 2024-03-11T22:16:44.610 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.610 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.610 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.610 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.947125+0000", 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8739999999999995e-06 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.948744+0000", 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016188680000000001 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.948762+0000", 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8309999999999999e-05 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.611 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.960760+0000", 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011997933000000001 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:54.960883+0000", 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012311099999999999 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:190833 7.8 e142/99 7:10f5ead6:::benchmark_data_smithi045_83007_object72362:head v 142'12079, mlcod=142'12077)", 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:27.176748+0000", 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 197.42904659199999, 2024-03-11T22:16:44.612 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.014695094000000001, 2024-03-11T22:16:44.613 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.613 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.613 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.613 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176748+0000", 2024-03-11T22:16:44.613 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.613 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.613 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.613 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176748+0000", 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176749+0000", 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3900000000000004e-07 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176757+0000", 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5390000000000007e-06 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.614 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176766+0000", 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6750000000000008e-06 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.176772+0000", 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9209999999999997e-06 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.178072+0000", 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013003120000000001 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.615 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.178095+0000", 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2648000000000001e-05 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.191322+0000", 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.013227536999999999 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:27.191443+0000", 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000120723 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.616 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(osd.0.0:50549 7.6 e149/99 7:62e1388f:::benchmark_data_smithi045_83007_object85115:head v 149'12975, mlcod=148'12957)", 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:37.650539+0000", 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 186.95525516000001, 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.015179632, 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.650539+0000", 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.617 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.650539+0000", 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.650540+0000", 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7500000000000001e-07 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.650544+0000", 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8829999999999999e-06 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.618 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.650556+0000", 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2069000000000001e-05 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.650561+0000", 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5679999999999999e-06 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.652614+0000", 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0020531809999999998 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.619 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.652633+0000", 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8715999999999998e-05 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.665665+0000", 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.013032263000000001 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.665719+0000", 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3576999999999999e-05 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.620 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.621 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.621 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.621 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.621 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(osd.0.0:50551 7.6 e149/99 7:62e1d3df:::benchmark_data_smithi045_83007_object105290:head v 149'12977, mlcod=148'12957)", 2024-03-11T22:16:44.621 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:37.657006+0000", 2024-03-11T22:16:44.621 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 186.94878822800001, 2024-03-11T22:16:44.621 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.015661405, 2024-03-11T22:16:44.621 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:44.622 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.622 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.622 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.622 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.657006+0000", 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.657006+0000", 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.657007+0000", 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8199999999999998e-07 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.623 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.657025+0000", 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7638e-05 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.657047+0000", 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1888000000000001e-05 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.657055+0000", 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9519999999999994e-06 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.624 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.661076+0000", 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0040210289999999997 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.661102+0000", 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6061e-05 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.672556+0000", 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011454717999999999 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.625 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.626 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.672668+0000", 2024-03-11T22:16:44.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011123700000000001 2024-03-11T22:16:44.626 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.626 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.626 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.626 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.626 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.626 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(osd.0.0:50553 7.6 e149/99 7:62e5aa41:::benchmark_data_smithi045_83007_object85724:head v 149'12978, mlcod=148'12957)", 2024-03-11T22:16:44.626 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:37.657562+0000", 2024-03-11T22:16:44.626 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 186.94823237099999, 2024-03-11T22:16:44.626 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.017575137000000001, 2024-03-11T22:16:44.626 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.626 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.657562+0000", 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.657562+0000", 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.657563+0000", 2024-03-11T22:16:44.627 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0299999999999999e-07 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.657570+0000", 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.391e-06 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.657578+0000", 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8250000000000005e-06 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.628 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.657584+0000", 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1129999999999997e-06 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.663388+0000", 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0058038100000000004 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.663406+0000", 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8159999999999999e-05 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.629 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.675071+0000", 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011665294 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.675137+0000", 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5940999999999998e-05 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(osd.0.0:50555 7.6 e149/99 7:62e5d22f:::benchmark_data_smithi045_83007_object99602:head v 149'12979, mlcod=148'12957)", 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:37.660317+0000", 2024-03-11T22:16:44.630 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 186.94547774899999, 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.014873516999999999, 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.660317+0000", 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.660317+0000", 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.631 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.660317+0000", 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0299999999999998e-07 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.660843+0000", 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00052568199999999997 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.660946+0000", 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000103222 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.632 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.660955+0000", 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.0450000000000006e-06 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.665816+0000", 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.004860212 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.665839+0000", 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3529000000000001e-05 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.633 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.675140+0000", 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0093007320000000008 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.675190+0000", 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0392000000000001e-05 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:199703 7.6 e149/99 7:6e2d6afe:::benchmark_data_smithi045_83007_object81232:head v 149'12999, mlcod=149'12996)", 2024-03-11T22:16:44.634 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:37.866290+0000", 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 186.739504434, 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.027544747000000001, 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.866290+0000", 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.866290+0000", 2024-03-11T22:16:44.635 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.866291+0000", 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0860000000000001e-06 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.866301+0000", 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0087e-05 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.866318+0000", 2024-03-11T22:16:44.636 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6291e-05 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.866325+0000", 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8520000000000004e-06 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.866377+0000", 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1776e-05 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.866400+0000", 2024-03-11T22:16:44.637 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3122e-05 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.893788+0000", 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.027387274999999999 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.893835+0000", 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7258000000000001e-05 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.638 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:199715 7.6 e149/99 7:64c47080:::benchmark_data_smithi045_83007_object81244:head v 149'13000, mlcod=149'12997)", 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:37.869911+0000", 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 186.73588342799999, 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.023956990000000001, 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.869911+0000", 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.639 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.869911+0000", 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.869912+0000", 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1319999999999999e-06 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.869921+0000", 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7790000000000001e-06 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.640 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.869940+0000", 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9123999999999999e-05 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.869949+0000", 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.0100000000000001e-06 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.870093+0000", 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000143762 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.641 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.870110+0000", 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7020999999999998e-05 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.893839+0000", 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.023728885000000002 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:37.893868+0000", 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9277000000000001e-05 2024-03-11T22:16:44.642 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.643 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.643 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.643 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.643 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.643 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:206904 7.6 e150/99 7:67ac4335:::benchmark_data_smithi045_83007_object88433:head v 150'14260, mlcod=150'14258)", 2024-03-11T22:16:44.643 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:47.665172+0000", 2024-03-11T22:16:44.643 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 176.94062267499999, 2024-03-11T22:16:44.643 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.017462976000000002, 2024-03-11T22:16:44.643 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.643 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.643 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.665172+0000", 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.665172+0000", 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.665173+0000", 2024-03-11T22:16:44.644 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8100000000000002e-07 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.665179+0000", 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8240000000000002e-06 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.665189+0000", 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.2099999999999999e-06 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.665194+0000", 2024-03-11T22:16:44.645 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4469999999999997e-06 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.665246+0000", 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1598999999999998e-05 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.665273+0000", 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7503000000000001e-05 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.682603+0000", 2024-03-11T22:16:44.646 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.017329517999999999 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.682635+0000", 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2193999999999997e-05 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:207126 7.6 e150/99 7:6e2dc143:::benchmark_data_smithi045_83007_object88655:head v 150'14298, mlcod=150'14294)", 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:47.973713+0000", 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 176.63208140399999, 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.01405913, 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.647 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.973713+0000", 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.973713+0000", 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.973714+0000", 2024-03-11T22:16:44.648 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2799999999999995e-07 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.973720+0000", 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9560000000000002e-06 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.973731+0000", 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1234e-05 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.649 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.973737+0000", 2024-03-11T22:16:44.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1850000000000001e-06 2024-03-11T22:16:44.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.973793+0000", 2024-03-11T22:16:44.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5834000000000001e-05 2024-03-11T22:16:44.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.650 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.650 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.973813+0000", 2024-03-11T22:16:44.650 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9394000000000001e-05 2024-03-11T22:16:44.650 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.650 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.987738+0000", 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.013925306 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:47.987772+0000", 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4393000000000001e-05 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:57315 8.d e190/175 8:b8542098:::benchmark_data_smithi045_115206_object57314:head v 190'3671, mlcod=190'3667)", 2024-03-11T22:16:44.651 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:15:31.740530+0000", 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 72.865264013000001, 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.015238109999999999, 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:31.740530+0000", 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:31.740530+0000", 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.652 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:31.740531+0000", 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7800000000000002e-07 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:31.740552+0000", 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.09e-05 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:31.740559+0000", 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1829999999999997e-06 2024-03-11T22:16:44.653 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:31.740565+0000", 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7769999999999999e-06 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:31.745329+0000", 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0047640039999999996 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:31.745353+0000", 2024-03-11T22:16:44.654 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4498000000000002e-05 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:31.755677+0000", 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010323961 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:31.755769+0000", 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1409000000000005e-05 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.655 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.656 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:66785 8.5 e190/175 8:a1343342:::benchmark_data_smithi045_115206_object66784:head v 190'4145, mlcod=190'4142)", 2024-03-11T22:16:44.656 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:15:38.418495+0000", 2024-03-11T22:16:44.656 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 66.187299349, 2024-03-11T22:16:44.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.015308802, 2024-03-11T22:16:44.656 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.656 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.656 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.656 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.656 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.418495+0000", 2024-03-11T22:16:44.656 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.656 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.656 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.418495+0000", 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.418496+0000", 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1399999999999996e-07 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.418518+0000", 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.243e-05 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.657 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.418533+0000", 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.455e-05 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.418541+0000", 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0530000000000007e-06 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.418611+0000", 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9509000000000001e-05 2024-03-11T22:16:44.658 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.418629+0000", 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8992999999999999e-05 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.433751+0000", 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.015121674 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.433804+0000", 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2778999999999997e-05 2024-03-11T22:16:44.659 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:66786 8.5 e190/175 8:a630a951:::benchmark_data_smithi045_115206_object66785:head v 190'4146, mlcod=190'4142)", 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:15:38.420233+0000", 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 66.185561469999996, 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.015243771, 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.660 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420233+0000", 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420233+0000", 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420238+0000", 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4789999999999998e-06 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.661 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420261+0000", 2024-03-11T22:16:44.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3003999999999999e-05 2024-03-11T22:16:44.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420282+0000", 2024-03-11T22:16:44.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1889999999999999e-05 2024-03-11T22:16:44.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.662 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.662 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.662 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420289+0000", 2024-03-11T22:16:44.662 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7950000000000002e-06 2024-03-11T22:16:44.662 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.663 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:44 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2817997064' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:16:44.663 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:44 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/864095855' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:44.663 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:44 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:44.664 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:44 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2817997064' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:16:44.664 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:44 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/864095855' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:44.664 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:44 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:16:44.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.664 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.664 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.423347+0000", 2024-03-11T22:16:44.664 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0030574640000000002 2024-03-11T22:16:44.664 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.664 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.423367+0000", 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0681000000000001e-05 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.435410+0000", 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012042367999999999 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.435477+0000", 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7089999999999996e-05 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.665 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:66787 8.d e190/175 8:bdcc8ddf:::benchmark_data_smithi045_115206_object66786:head v 190'4232, mlcod=190'4229)", 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:15:38.420743+0000", 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 66.185051513999994, 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.014662552000000001, 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420743+0000", 2024-03-11T22:16:44.666 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420743+0000", 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420744+0000", 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3499999999999996e-07 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420766+0000", 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2574000000000001e-05 2024-03-11T22:16:44.667 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420775+0000", 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.268e-06 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420782+0000", 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9419999999999998e-06 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420848+0000", 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5195000000000002e-05 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.668 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.420867+0000", 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9279999999999998e-05 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.435327+0000", 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.014460274 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:38.435406+0000", 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8384e-05 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.669 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.670 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.670 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:87247 8.d e199/194 8:bee693ad:::benchmark_data_smithi045_115206_object87246:head v 199'5475, mlcod=199'5471)", 2024-03-11T22:16:44.670 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:15:57.691414+0000", 2024-03-11T22:16:44.670 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 46.914380757000004, 2024-03-11T22:16:44.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.014231109, 2024-03-11T22:16:44.670 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.670 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.670 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.670 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.670 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.670 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:57.691414+0000", 2024-03-11T22:16:44.670 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:57.691414+0000", 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:57.691414+0000", 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6300000000000001e-07 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:57.691435+0000", 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1222000000000001e-05 2024-03-11T22:16:44.671 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:57.691442+0000", 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7039999999999997e-06 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:57.691448+0000", 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.806e-06 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:57.694772+0000", 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0033237470000000002 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.672 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:57.694823+0000", 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1099e-05 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:57.705465+0000", 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010642057 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:57.705645+0000", 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000180111 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.673 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:87963 8.d e199/194 8:b66b9500:::benchmark_data_smithi045_115206_object87962:head v 199'5521, mlcod=199'5517)", 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:15:58.310476+0000", 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 46.295318436000002, 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.019126150000000001, 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.310476+0000", 2024-03-11T22:16:44.674 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.310476+0000", 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.310477+0000", 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8700000000000004e-07 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.675 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.310553+0000", 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5753999999999993e-05 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.310563+0000", 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0086e-05 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.310571+0000", 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3040000000000005e-06 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.676 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.310731+0000", 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000159412 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.310891+0000", 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016013 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.329398+0000", 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.018507276999999999 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.677 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.329602+0000", 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00020430000000000001 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:87968 8.5 e199/175 8:ae4e82fe:::benchmark_data_smithi045_115206_object87967:head v 199'5471, mlcod=199'5469)", 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:15:58.313146+0000", 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 46.292648110000002, 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.01988407, 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.678 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.313146+0000", 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.313146+0000", 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.313148+0000", 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1230000000000001e-06 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.679 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.313175+0000", 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7826000000000001e-05 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.313190+0000", 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.469e-05 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.313199+0000", 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3719999999999994e-06 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.680 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.313469+0000", 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026969800000000002 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.313514+0000", 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4776e-05 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.332890+0000", 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.019376233999999999 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.681 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.682 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.333030+0000", 2024-03-11T22:16:44.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014035100000000001 2024-03-11T22:16:44.682 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.682 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.682 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.682 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.682 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.682 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:87973 8.5 e199/175 8:a23d2180:::benchmark_data_smithi045_115206_object87972:head v 199'5472, mlcod=199'5469)", 2024-03-11T22:16:44.682 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:15:58.317212+0000", 2024-03-11T22:16:44.682 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 46.288582337000001, 2024-03-11T22:16:44.682 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.015849650999999999, 2024-03-11T22:16:44.682 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.682 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.317212+0000", 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.317212+0000", 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.317213+0000", 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.113e-06 2024-03-11T22:16:44.683 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.317237+0000", 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4189000000000002e-05 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.317251+0000", 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3199e-05 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.317259+0000", 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8349999999999993e-06 2024-03-11T22:16:44.684 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.320007+0000", 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0027474520000000001 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.320068+0000", 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0924000000000001e-05 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.333037+0000", 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.012969339 2024-03-11T22:16:44.685 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.686 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.686 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.686 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:15:58.333062+0000", 2024-03-11T22:16:44.686 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4600000000000002e-05 2024-03-11T22:16:44.686 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.686 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.686 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.686 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.686 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.686 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:93325 8.d e200/194 8:b9fa7811:::benchmark_data_smithi045_115206_object93324:head v 200'5870, mlcod=200'5867)", 2024-03-11T22:16:44.686 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:03.196095+0000", 2024-03-11T22:16:44.686 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 41.409699729000003, 2024-03-11T22:16:44.686 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.01611698, 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.196095+0000", 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.196095+0000", 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.687 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.196096+0000", 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2240000000000001e-06 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.196123+0000", 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6942999999999999e-05 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.196143+0000", 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9715e-05 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.688 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.196152+0000", 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.2140000000000002e-06 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.196344+0000", 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00019206200000000001 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.196395+0000", 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1409999999999997e-05 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.689 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.212161+0000", 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.015765477999999999 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.212212+0000", 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0933999999999997e-05 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:95546 8.5 e200/175 8:ae2f22db:::benchmark_data_smithi045_115206_object95545:head v 200'5987, mlcod=200'5983)", 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:05.638742+0000", 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 38.967052754999997, 2024-03-11T22:16:44.690 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.013688684, 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.638742+0000", 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.638742+0000", 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.691 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.638742+0000", 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4499999999999998e-07 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.638762+0000", 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0183999999999999e-05 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.638769+0000", 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.55e-06 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.692 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.638774+0000", 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9030000000000003e-06 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.641384+0000", 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026105529999999998 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.641497+0000", 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000112351 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.693 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.652247+0000", 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010750641 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:05.652430+0000", 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018315700000000001 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:16:44.694 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:16:44.696 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:44.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:44.798 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:44.830 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:44.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:44.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:44.932 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:44.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:44.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:44.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:45.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:45.114 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:45.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:45.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:45.215 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:45.248 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:45.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:45.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:45.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:45.395 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:45.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:45.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T22:16:45.496 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:45.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:45.527 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:45.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:45.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:45 smithi144 ceph-mon[33984]: pgmap v354: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 947 B/s rd, 7.9 MiB/s wr, 983 op/s; 4207/342432 objects misplaced (1.229%); 942 KiB/s, 14 objects/s recovering 2024-03-11T22:16:45.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:45 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:45.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:45 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2817997064' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:16:45.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:45 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:16:45.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:45 smithi144 ceph-mon[33984]: osdmap e213: 8 total, 8 up, 8 in 2024-03-11T22:16:45.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:16:45.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:16:45.627 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:45.678 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:45.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:45.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[39715]: pgmap v354: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 947 B/s rd, 7.9 MiB/s wr, 983 op/s; 4207/342432 objects misplaced (1.229%); 942 KiB/s, 14 objects/s recovering 2024-03-11T22:16:45.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:45.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2817997064' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:16:45.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:16:45.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[39715]: osdmap e213: 8 total, 8 up, 8 in 2024-03-11T22:16:45.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:16:45.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:16:45.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[30178]: pgmap v354: 178 pgs: 1 active+recovering+undersized+remapped, 177 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 947 B/s rd, 7.9 MiB/s wr, 983 op/s; 4207/342432 objects misplaced (1.229%); 942 KiB/s, 14 objects/s recovering 2024-03-11T22:16:45.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:45.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2817997064' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:16:45.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:16:45.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[30178]: osdmap e213: 8 total, 8 up, 8 in 2024-03-11T22:16:45.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:16:45.734 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:16:45.779 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:45.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:45.818 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:45.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:45.919 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:45.951 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:45.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:46.053 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:46.092 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:46.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:46.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:46.194 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:46.225 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:46.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:46.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:46.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:46.344 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:16:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:16:46.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:46.370 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:46.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:46.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:46.507 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:46.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:46.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:46.609 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:46.648 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:46.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:46.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:46.750 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:46.789 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:46.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:46.891 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:46.910 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:46.922 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:46.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:46.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:46.967 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:16:47.009 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:16:47.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:47.049 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:16:47.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:47.077 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:47.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:47.089 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:16:47.130 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:16:47.171 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:16:47.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:47.211 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:16:47.217 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:47.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:47.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:47.252 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:16:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:16:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:16:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:16:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:16:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:16:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:16:47.258 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:16:47.259 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:16:47.319 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:47.359 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:47.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:47.460 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:47.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:47.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:47 smithi045 ceph-mon[39715]: pgmap v356: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 87/323364 objects degraded (0.027%); 4014/323364 objects misplaced (1.241%); 1.2 MiB/s, 19 objects/s recovering 2024-03-11T22:16:47.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:47 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2517900145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:47.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:47 smithi045 ceph-mon[39715]: Health check failed: Degraded data redundancy: 87/323364 objects degraded (0.027%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:16:47.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:47.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:47 smithi045 ceph-mon[30178]: pgmap v356: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 87/323364 objects degraded (0.027%); 4014/323364 objects misplaced (1.241%); 1.2 MiB/s, 19 objects/s recovering 2024-03-11T22:16:47.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2517900145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:47.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:47 smithi045 ceph-mon[30178]: Health check failed: Degraded data redundancy: 87/323364 objects degraded (0.027%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:16:47.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:47.493 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:47.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:47.594 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:47.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:47.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:47 smithi144 ceph-mon[33984]: pgmap v356: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 87/323364 objects degraded (0.027%); 4014/323364 objects misplaced (1.241%); 1.2 MiB/s, 19 objects/s recovering 2024-03-11T22:16:47.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:47 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2517900145' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:16:47.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:47 smithi144 ceph-mon[33984]: Health check failed: Degraded data redundancy: 87/323364 objects degraded (0.027%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:16:47.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:47.633 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:47.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:47.734 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:47.767 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:47.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:47.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:47.911 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:47.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:47.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:48.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:48.055 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:48.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:48.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:16:48.156 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:48.215 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:48.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:48.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:48.317 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:48.345 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:48.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:48.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:48.447 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:48.477 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:48.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:48.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:48.579 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:48.607 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:48.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:48.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:48.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:48.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:48.747 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:48.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:48.848 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:48.876 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:48.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:48.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:49.023 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:49.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:49.123 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:49.152 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:49.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:49.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:49.254 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:49.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:49.291 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:49.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:49.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:49.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:49 smithi045 ceph-mon[30178]: pgmap v357: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 87/323364 objects degraded (0.027%); 4014/323364 objects misplaced (1.241%); 1.2 MiB/s, 19 objects/s recovering 2024-03-11T22:16:49.424 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:49 smithi045 ceph-mon[39715]: pgmap v357: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 87/323364 objects degraded (0.027%); 4014/323364 objects misplaced (1.241%); 1.2 MiB/s, 19 objects/s recovering 2024-03-11T22:16:49.438 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:49.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:49.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:49.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:49 smithi144 ceph-mon[33984]: pgmap v357: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 87/323364 objects degraded (0.027%); 4014/323364 objects misplaced (1.241%); 1.2 MiB/s, 19 objects/s recovering 2024-03-11T22:16:49.614 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:49.645 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:49.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:49.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:49.747 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:49.778 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:49.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:49.792 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:49.815 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:16:49.815 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:16:49.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:49.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:49.932 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:49.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:50.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:50.072 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:50.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:50.174 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:50.206 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:50.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:50.307 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:50.344 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:50.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:50.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:50.446 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:50.483 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:50.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:50.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:50.585 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:50.641 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:50.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:50.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:50.647 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T22:16:50.743 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:50.798 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:50.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:50.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:50.985 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:50.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:50.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:51.087 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:51.119 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:51.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:51.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:51.156 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:16:51.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:51.259 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:51.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:51.360 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:51.391 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:51.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:51.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:51.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:51 smithi045 ceph-mon[39715]: pgmap v358: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 87/323364 objects degraded (0.027%); 4014/323364 objects misplaced (1.241%); 1.2 MiB/s, 19 objects/s recovering 2024-03-11T22:16:51.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:51 smithi045 ceph-mon[30178]: pgmap v358: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 87/323364 objects degraded (0.027%); 4014/323364 objects misplaced (1.241%); 1.2 MiB/s, 19 objects/s recovering 2024-03-11T22:16:51.544 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:51.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:51.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:51.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:51 smithi144 ceph-mon[33984]: pgmap v358: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 87/323364 objects degraded (0.027%); 4014/323364 objects misplaced (1.241%); 1.2 MiB/s, 19 objects/s recovering 2024-03-11T22:16:51.598 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:16:51.598 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:16:51.598 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-03-11T22:16:51.598 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T22:16:51.598 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T22:16:51.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-03-11T22:16:51.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:51.696 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:51.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:51.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:51.797 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:51.825 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:51.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:51.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:51.927 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:51.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:51.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:51.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:52.071 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:52.101 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:52.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:52.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:52.158 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:52.203 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:52.232 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:52.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:52.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:52.360 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:52 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 361/304737 objects degraded (0.118%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:16:52.360 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:52 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 361/304737 objects degraded (0.118%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:16:52.376 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:52.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:52.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:52.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:52.531 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:52.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:52.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:52 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 361/304737 objects degraded (0.118%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:16:52.632 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:52.646 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:16:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:16:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:16:52.667 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:52.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:52.770 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:52.802 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:52.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:52.904 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:52.937 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:52.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:52.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:53.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:53.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:53.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:53.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:53.181 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:53.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:53.217 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:53.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:53.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:53.360 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:53.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:53.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:53.466 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:53.466 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:16:53.507 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:53.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:53.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:53.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:53 smithi045 ceph-mon[39715]: pgmap v359: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 361/304737 objects degraded (0.118%); 3632/304737 objects misplaced (1.192%); 1.3 MiB/s, 21 objects/s recovering 2024-03-11T22:16:53.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:53 smithi045 ceph-mon[30178]: pgmap v359: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 361/304737 objects degraded (0.118%); 3632/304737 objects misplaced (1.192%); 1.3 MiB/s, 21 objects/s recovering 2024-03-11T22:16:53.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:53 smithi144 ceph-mon[33984]: pgmap v359: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 361/304737 objects degraded (0.118%); 3632/304737 objects misplaced (1.192%); 1.3 MiB/s, 21 objects/s recovering 2024-03-11T22:16:53.609 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:53.651 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:53.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:53.752 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:53.791 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:53.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:53.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:53.893 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:53.924 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:53.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:54.025 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:54.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:54.056 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:54.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:54.157 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:54.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:54.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:54.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:54.295 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:16:54.297 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:54.344 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:54.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:54.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:54.446 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:54.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:54.483 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:54.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:54.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:54.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:54 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1437929046' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:16:54.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:54 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:16:54.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:54 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:16:54.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:54.623 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:54.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:54.638 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:54 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1437929046' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:16:54.638 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:54 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:16:54.638 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:54 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:16:54.638 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:54 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1437929046' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:16:54.638 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:54 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:16:54.639 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:54 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:16:54.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:54.768 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:54.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:54.869 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:54.902 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:54.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:54.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:55.004 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:55.039 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:55.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:55.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:55.141 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:55.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:55.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:55.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:55.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:55.294 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:55 smithi045 ceph-mon[30178]: pgmap v360: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 361/304737 objects degraded (0.118%); 3632/304737 objects misplaced (1.192%); 1.3 MiB/s, 21 objects/s recovering 2024-03-11T22:16:55.319 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:55.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:55.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:55.421 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:55.454 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:55.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:55.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:55.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:55 smithi045 ceph-mon[39715]: pgmap v360: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 361/304737 objects degraded (0.118%); 3632/304737 objects misplaced (1.192%); 1.3 MiB/s, 21 objects/s recovering 2024-03-11T22:16:55.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:55 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:55.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:55 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1437929046' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:16:55.574 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:55 smithi045 ceph-mon[39715]: osdmap e214: 8 total, 8 up, 8 in 2024-03-11T22:16:55.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:55 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:55.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:55 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1437929046' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:16:55.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:55 smithi045 ceph-mon[30178]: osdmap e214: 8 total, 8 up, 8 in 2024-03-11T22:16:55.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:55 smithi144 ceph-mon[33984]: pgmap v360: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 361/304737 objects degraded (0.118%); 3632/304737 objects misplaced (1.192%); 1.3 MiB/s, 21 objects/s recovering 2024-03-11T22:16:55.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:55 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:16:55.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:55 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1437929046' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:16:55.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:55 smithi144 ceph-mon[33984]: osdmap e214: 8 total, 8 up, 8 in 2024-03-11T22:16:55.599 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:55.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:55.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:55.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:55.742 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:55.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:55.843 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:55.871 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:55.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:55.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:55.973 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:56.014 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:56.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:56.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:56.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:56.133 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:16:56.134 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:16:56.157 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:56.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:56.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:56.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:56.283 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:56.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:56.384 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:56.415 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:56.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:56.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:56.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:56 smithi045 ceph-mon[30178]: 5.2 scrub starts 2024-03-11T22:16:56.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:56 smithi045 ceph-mon[30178]: 5.2 scrub ok 2024-03-11T22:16:56.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:56 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:16:56.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:56 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:16:56.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:56 smithi045 ceph-mon[30178]: 8.1 scrub starts 2024-03-11T22:16:56.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:56 smithi045 ceph-mon[39715]: 5.2 scrub starts 2024-03-11T22:16:56.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:56 smithi045 ceph-mon[39715]: 5.2 scrub ok 2024-03-11T22:16:56.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:56 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:16:56.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:56 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:16:56.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:56 smithi045 ceph-mon[39715]: 8.1 scrub starts 2024-03-11T22:16:56.553 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:56.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:56.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:56.594 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:16:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:16:56.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:56 smithi144 ceph-mon[33984]: 5.2 scrub starts 2024-03-11T22:16:56.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:56 smithi144 ceph-mon[33984]: 5.2 scrub ok 2024-03-11T22:16:56.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:56 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:16:56.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:56 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:16:56.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:56 smithi144 ceph-mon[33984]: 8.1 scrub starts 2024-03-11T22:16:56.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:56.691 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:56.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:56.793 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:56.836 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:56.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:56.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:56.938 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:56.977 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:56.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:56.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:57.079 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:57.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:16:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:57.116 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:57.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:57.217 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:57.249 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:57.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:57.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:57.351 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:57.385 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:57.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:57.486 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:57.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:57.518 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:57.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:57.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:57.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:57 smithi045 ceph-mon[39715]: pgmap v362: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 690/285522 objects degraded (0.242%); 3202/285522 objects misplaced (1.121%); 1.3 MiB/s, 20 objects/s recovering 2024-03-11T22:16:57.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:57 smithi045 ceph-mon[39715]: osdmap e215: 8 total, 8 up, 8 in 2024-03-11T22:16:57.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:57.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:57 smithi045 ceph-mon[30178]: pgmap v362: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 690/285522 objects degraded (0.242%); 3202/285522 objects misplaced (1.121%); 1.3 MiB/s, 20 objects/s recovering 2024-03-11T22:16:57.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:57 smithi045 ceph-mon[30178]: osdmap e215: 8 total, 8 up, 8 in 2024-03-11T22:16:57.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:16:57.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:57 smithi144 ceph-mon[33984]: pgmap v362: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 690/285522 objects degraded (0.242%); 3202/285522 objects misplaced (1.121%); 1.3 MiB/s, 20 objects/s recovering 2024-03-11T22:16:57.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:57 smithi144 ceph-mon[33984]: osdmap e215: 8 total, 8 up, 8 in 2024-03-11T22:16:57.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:57.665 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:57.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:57.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:57.811 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:57.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:57.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:57.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:57.950 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:57.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:58.052 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:16:58.092 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:58.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:16:58.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T22:16:58.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:58.246 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:58.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:58.349 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:58.376 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:58.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:58.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:58 smithi045 ceph-mon[30178]: 5.b scrub starts 2024-03-11T22:16:58.379 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:58 smithi045 ceph-mon[30178]: 5.b scrub ok 2024-03-11T22:16:58.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:58 smithi045 ceph-mon[39715]: 5.b scrub starts 2024-03-11T22:16:58.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:58 smithi045 ceph-mon[39715]: 5.b scrub ok 2024-03-11T22:16:58.478 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:58.513 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:58.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:58.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:58 smithi144 ceph-mon[33984]: 5.b scrub starts 2024-03-11T22:16:58.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:58 smithi144 ceph-mon[33984]: 5.b scrub ok 2024-03-11T22:16:58.614 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:58.651 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:58.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:58.752 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:16:58.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:58.786 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:58.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:16:58.887 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:58.920 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:58.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:58.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:59.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:16:59.059 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:59.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:59.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:16:59.161 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:16:59.193 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:59.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:16:59.294 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:59.327 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:59.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:59.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:16:59.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:16:59.469 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:59.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:59.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:16:59.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:59 smithi045 ceph-mon[39715]: 8.1 scrub ok 2024-03-11T22:16:59.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:16:59 smithi045 ceph-mon[39715]: pgmap v364: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 690/285522 objects degraded (0.242%); 3202/285522 objects misplaced (1.121%); 1.6 MiB/s, 26 objects/s recovering 2024-03-11T22:16:59.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:59 smithi045 ceph-mon[30178]: 8.1 scrub ok 2024-03-11T22:16:59.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:16:59 smithi045 ceph-mon[30178]: pgmap v364: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 690/285522 objects degraded (0.242%); 3202/285522 objects misplaced (1.121%); 1.6 MiB/s, 26 objects/s recovering 2024-03-11T22:16:59.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:16:59.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:59 smithi144 ceph-mon[33984]: 8.1 scrub ok 2024-03-11T22:16:59.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:16:59 smithi144 ceph-mon[33984]: pgmap v364: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 690/285522 objects degraded (0.242%); 3202/285522 objects misplaced (1.121%); 1.6 MiB/s, 26 objects/s recovering 2024-03-11T22:16:59.609 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:16:59.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:59.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:16:59.711 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:16:59.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:59.793 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:59.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:16:59.894 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:16:59.922 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:16:59.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:16:59.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:00.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:00.063 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:00.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:00.164 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:00.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:00.201 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:00.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:00.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:00.331 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:00.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:00.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:00.433 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:00.471 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:00.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:00.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:00.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:00.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:00.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:00.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:00.755 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:00.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:00.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:00.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:00.897 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:00.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:00.998 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:01.035 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:01.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:01.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:01.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:01.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:01.171 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:01.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:01.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:01.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:01.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:01.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:01.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:01.432 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:01.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:01.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:01.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:17:01.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:17:01.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[39715]: pgmap v365: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 800 op/s; 690/285522 objects degraded (0.242%); 3202/285522 objects misplaced (1.121%); 808 KiB/s, 12 objects/s recovering 2024-03-11T22:17:01.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[39715]: 8.3 scrub starts 2024-03-11T22:17:01.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[39715]: 1.14 scrub starts 2024-03-11T22:17:01.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[39715]: 1.14 scrub ok 2024-03-11T22:17:01.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 690/285522 objects degraded (0.242%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:01.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:17:01.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:17:01.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[30178]: pgmap v365: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 800 op/s; 690/285522 objects degraded (0.242%); 3202/285522 objects misplaced (1.121%); 808 KiB/s, 12 objects/s recovering 2024-03-11T22:17:01.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[30178]: 8.3 scrub starts 2024-03-11T22:17:01.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[30178]: 1.14 scrub starts 2024-03-11T22:17:01.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[30178]: 1.14 scrub ok 2024-03-11T22:17:01.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:01 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 690/285522 objects degraded (0.242%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:01.534 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:01.564 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:01.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:01.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:17:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:17:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:01 smithi144 ceph-mon[33984]: pgmap v365: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 800 op/s; 690/285522 objects degraded (0.242%); 3202/285522 objects misplaced (1.121%); 808 KiB/s, 12 objects/s recovering 2024-03-11T22:17:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:01 smithi144 ceph-mon[33984]: 8.3 scrub starts 2024-03-11T22:17:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:01 smithi144 ceph-mon[33984]: 1.14 scrub starts 2024-03-11T22:17:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:01 smithi144 ceph-mon[33984]: 1.14 scrub ok 2024-03-11T22:17:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:01 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 690/285522 objects degraded (0.242%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:01.667 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:01.699 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:01.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:01.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:01.836 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:01.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:01.938 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:01.962 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:01.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:01.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:02.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:02.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:02.102 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:02.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:02.203 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:02.238 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:02.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:02.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:02.340 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:02.375 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:02.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:02.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:02.477 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:02.500 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:02.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:02.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:02.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:02 smithi045 ceph-mon[39715]: 4.5 scrub starts 2024-03-11T22:17:02.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:02 smithi045 ceph-mon[39715]: 4.5 scrub ok 2024-03-11T22:17:02.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:02 smithi045 ceph-mon[39715]: osdmap e216: 8 total, 8 up, 8 in 2024-03-11T22:17:02.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:02 smithi045 ceph-mon[30178]: 4.5 scrub starts 2024-03-11T22:17:02.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:02 smithi045 ceph-mon[30178]: 4.5 scrub ok 2024-03-11T22:17:02.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:02 smithi045 ceph-mon[30178]: osdmap e216: 8 total, 8 up, 8 in 2024-03-11T22:17:02.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:02 smithi144 ceph-mon[33984]: 4.5 scrub starts 2024-03-11T22:17:02.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:02 smithi144 ceph-mon[33984]: 4.5 scrub ok 2024-03-11T22:17:02.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:02 smithi144 ceph-mon[33984]: osdmap e216: 8 total, 8 up, 8 in 2024-03-11T22:17:02.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:02.649 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:02.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:02.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:02.790 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:02.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:02.891 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:02.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:02.925 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:02.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:03.026 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:03.030 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:17:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:17:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:17:03.058 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:03.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:03.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:03.160 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:03.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:03.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:03.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:03.301 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:03.324 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:03.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:03.426 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:03.460 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:03.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:03.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:03 smithi045 ceph-mon[39715]: pgmap v367: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.58k op/s; 944/267321 objects degraded (0.353%); 2827/267321 objects misplaced (1.058%); 1.8 MiB/s, 28 objects/s recovering 2024-03-11T22:17:03.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:03 smithi045 ceph-mon[39715]: 8.3 scrub ok 2024-03-11T22:17:03.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:03 smithi045 ceph-mon[30178]: pgmap v367: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.58k op/s; 944/267321 objects degraded (0.353%); 2827/267321 objects misplaced (1.058%); 1.8 MiB/s, 28 objects/s recovering 2024-03-11T22:17:03.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:03 smithi045 ceph-mon[30178]: 8.3 scrub ok 2024-03-11T22:17:03.562 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:03.594 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:03.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:03.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:03 smithi144 ceph-mon[33984]: pgmap v367: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.58k op/s; 944/267321 objects degraded (0.353%); 2827/267321 objects misplaced (1.058%); 1.8 MiB/s, 28 objects/s recovering 2024-03-11T22:17:03.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:03 smithi144 ceph-mon[33984]: 8.3 scrub ok 2024-03-11T22:17:03.696 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:03.719 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:03.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:03.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:03.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:03.862 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:03.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:03.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:04.001 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:04.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:04.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:04.125 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:04.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:04.227 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:04.261 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:04.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:04.363 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:04.396 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:04.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:04.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:04.499 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:04.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:04.523 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:04.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:04.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:04 smithi045 ceph-mon[39715]: 5.a scrub starts 2024-03-11T22:17:04.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:04 smithi045 ceph-mon[39715]: 5.a scrub ok 2024-03-11T22:17:04.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:04 smithi045 ceph-mon[30178]: 5.a scrub starts 2024-03-11T22:17:04.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:04 smithi045 ceph-mon[30178]: 5.a scrub ok 2024-03-11T22:17:04.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:04 smithi144 ceph-mon[33984]: 5.a scrub starts 2024-03-11T22:17:04.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:04 smithi144 ceph-mon[33984]: 5.a scrub ok 2024-03-11T22:17:04.624 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:04.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:04.662 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:04.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:04.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:04.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:04.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:04.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:04.933 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:04.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:04.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:05.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:05.076 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:05.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:05.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:05.202 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:05.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:05.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:05.340 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:05.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:05.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:05.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:05.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:05.484 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:05.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:05.495 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:05 smithi144 ceph-mon[33984]: 4.a scrub starts 2024-03-11T22:17:05.495 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:05 smithi144 ceph-mon[33984]: 4.a scrub ok 2024-03-11T22:17:05.495 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:05 smithi144 ceph-mon[33984]: pgmap v368: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 758 op/s; 944/267321 objects degraded (0.353%); 2827/267321 objects misplaced (1.058%); 968 KiB/s, 15 objects/s recovering 2024-03-11T22:17:05.495 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:05 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:17:05.496 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:05 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:17:05.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:05 smithi045 ceph-mon[39715]: 4.a scrub starts 2024-03-11T22:17:05.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:05 smithi045 ceph-mon[39715]: 4.a scrub ok 2024-03-11T22:17:05.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:05 smithi045 ceph-mon[39715]: pgmap v368: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 758 op/s; 944/267321 objects degraded (0.353%); 2827/267321 objects misplaced (1.058%); 968 KiB/s, 15 objects/s recovering 2024-03-11T22:17:05.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:05 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:17:05.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:05 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:17:05.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:05 smithi045 ceph-mon[30178]: 4.a scrub starts 2024-03-11T22:17:05.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:05 smithi045 ceph-mon[30178]: 4.a scrub ok 2024-03-11T22:17:05.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:05 smithi045 ceph-mon[30178]: pgmap v368: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 758 op/s; 944/267321 objects degraded (0.353%); 2827/267321 objects misplaced (1.058%); 968 KiB/s, 15 objects/s recovering 2024-03-11T22:17:05.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:05 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:17:05.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:05 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:17:05.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:05.620 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:05.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:05.721 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:05.758 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:05.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:05.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:05.860 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:05.895 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:05.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:05.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:05.997 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:06.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:06.033 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:06.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:06.134 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:06.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:06.169 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:06.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:06.270 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:06.293 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:06.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:06.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:06.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:06.432 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:06.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:06.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:06.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:06 smithi045 ceph-mon[30178]: osdmap e217: 8 total, 8 up, 8 in 2024-03-11T22:17:06.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:06 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 944/267321 objects degraded (0.353%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:06.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:06 smithi045 ceph-mon[30178]: osdmap e218: 8 total, 8 up, 8 in 2024-03-11T22:17:06.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:06 smithi045 ceph-mon[39715]: osdmap e217: 8 total, 8 up, 8 in 2024-03-11T22:17:06.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:06 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 944/267321 objects degraded (0.353%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:06.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:06 smithi045 ceph-mon[39715]: osdmap e218: 8 total, 8 up, 8 in 2024-03-11T22:17:06.534 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:06.566 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:06.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:06.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:06 smithi144 ceph-mon[33984]: osdmap e217: 8 total, 8 up, 8 in 2024-03-11T22:17:06.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:06 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 944/267321 objects degraded (0.353%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:06.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:06 smithi144 ceph-mon[33984]: osdmap e218: 8 total, 8 up, 8 in 2024-03-11T22:17:06.595 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:17:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:17:06.668 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:06.703 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:06.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:06.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:06.805 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:06.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:06.837 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:06.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:06.938 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:06.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:06.969 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:06.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:07.070 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:07.107 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:07.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:07.209 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:07.242 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:07.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:07.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:07.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:07.383 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:07.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:07.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:07.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:07.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:07 smithi045 ceph-mon[39715]: 6.3 scrub starts 2024-03-11T22:17:07.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:07 smithi045 ceph-mon[39715]: pgmap v371: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.83k op/s; 1133/252615 objects degraded (0.449%); 2535/252615 objects misplaced (1.004%); 2.3 MiB/s, 37 objects/s recovering 2024-03-11T22:17:07.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:07 smithi045 ceph-mon[39715]: 6.3 scrub ok 2024-03-11T22:17:07.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:07.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:07 smithi045 ceph-mon[30178]: 6.3 scrub starts 2024-03-11T22:17:07.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:07 smithi045 ceph-mon[30178]: pgmap v371: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.83k op/s; 1133/252615 objects degraded (0.449%); 2535/252615 objects misplaced (1.004%); 2.3 MiB/s, 37 objects/s recovering 2024-03-11T22:17:07.476 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:07 smithi045 ceph-mon[30178]: 6.3 scrub ok 2024-03-11T22:17:07.485 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:07.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:07.517 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:07.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:07.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:07.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:07 smithi144 ceph-mon[33984]: 6.3 scrub starts 2024-03-11T22:17:07.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:07 smithi144 ceph-mon[33984]: pgmap v371: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.83k op/s; 1133/252615 objects degraded (0.449%); 2535/252615 objects misplaced (1.004%); 2.3 MiB/s, 37 objects/s recovering 2024-03-11T22:17:07.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:07 smithi144 ceph-mon[33984]: 6.3 scrub ok 2024-03-11T22:17:07.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:07.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:07.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:07.777 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:07.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:07.879 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:07.907 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:07.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:08.008 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:08.037 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:08.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:08.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:08.139 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:08.163 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:08.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:08.264 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:08.302 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:08.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:08.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:08.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:08.431 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:08.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:08.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:08 smithi045 ceph-mon[39715]: 6.5 scrub starts 2024-03-11T22:17:08.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:08 smithi045 ceph-mon[39715]: 6.5 scrub ok 2024-03-11T22:17:08.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:08 smithi045 ceph-mon[30178]: 6.5 scrub starts 2024-03-11T22:17:08.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:08 smithi045 ceph-mon[30178]: 6.5 scrub ok 2024-03-11T22:17:08.535 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:08.570 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:08.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:08.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:08 smithi144 ceph-mon[33984]: 6.5 scrub starts 2024-03-11T22:17:08.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:08 smithi144 ceph-mon[33984]: 6.5 scrub ok 2024-03-11T22:17:08.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:08.694 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:08.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:08.796 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:08.823 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:08.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:08.843 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:17:08.843 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T22:17:08.843 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:17:08.925 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:08.957 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:08.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:09.059 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:09.094 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:09.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:09.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:09.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:09.226 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:09.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:09.327 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:09.363 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:09.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:09.465 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:09.489 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:09.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:09.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:09 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:17:09.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:09 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:17:09.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:09 smithi045 ceph-mon[30178]: pgmap v372: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 729 B/s rd, 0 B/s wr, 699 op/s; 1133/252615 objects degraded (0.449%); 2535/252615 objects misplaced (1.004%); 939 KiB/s, 14 objects/s recovering 2024-03-11T22:17:09.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:09 smithi045 ceph-mon[30178]: osdmap e219: 8 total, 8 up, 8 in 2024-03-11T22:17:09.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:09 smithi045 ceph-mon[30178]: 6.f scrub starts 2024-03-11T22:17:09.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:09 smithi045 ceph-mon[30178]: 6.f scrub ok 2024-03-11T22:17:09.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:09 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:17:09.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:09 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:17:09.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:09 smithi045 ceph-mon[39715]: pgmap v372: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 729 B/s rd, 0 B/s wr, 699 op/s; 1133/252615 objects degraded (0.449%); 2535/252615 objects misplaced (1.004%); 939 KiB/s, 14 objects/s recovering 2024-03-11T22:17:09.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:09 smithi045 ceph-mon[39715]: osdmap e219: 8 total, 8 up, 8 in 2024-03-11T22:17:09.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:09 smithi045 ceph-mon[39715]: 6.f scrub starts 2024-03-11T22:17:09.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:09 smithi045 ceph-mon[39715]: 6.f scrub ok 2024-03-11T22:17:09.591 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:09.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:09 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:17:09.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:09 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:17:09.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:09 smithi144 ceph-mon[33984]: pgmap v372: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 729 B/s rd, 0 B/s wr, 699 op/s; 1133/252615 objects degraded (0.449%); 2535/252615 objects misplaced (1.004%); 939 KiB/s, 14 objects/s recovering 2024-03-11T22:17:09.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:09 smithi144 ceph-mon[33984]: osdmap e219: 8 total, 8 up, 8 in 2024-03-11T22:17:09.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:09 smithi144 ceph-mon[33984]: 6.f scrub starts 2024-03-11T22:17:09.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:09 smithi144 ceph-mon[33984]: 6.f scrub ok 2024-03-11T22:17:09.628 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:09.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:09.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:09.760 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:09.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:09.862 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:09.893 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:09.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:09.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:09.995 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:10.073 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:10.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:10.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:10.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:10.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:10.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:10.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:10.300 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:10.331 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:10.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:10.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:10.433 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:10.466 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:10.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:10.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:10 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:17:10.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:10 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:17:10.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:10 smithi045 ceph-mon[39715]: 4.7 scrub starts 2024-03-11T22:17:10.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:10 smithi045 ceph-mon[39715]: 4.7 scrub ok 2024-03-11T22:17:10.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:10 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:17:10.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:10 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:17:10.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:10 smithi045 ceph-mon[30178]: 4.7 scrub starts 2024-03-11T22:17:10.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:10 smithi045 ceph-mon[30178]: 4.7 scrub ok 2024-03-11T22:17:10.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:10.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:10 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:17:10.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:10 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:17:10.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:10 smithi144 ceph-mon[33984]: 4.7 scrub starts 2024-03-11T22:17:10.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:10 smithi144 ceph-mon[33984]: 4.7 scrub ok 2024-03-11T22:17:10.607 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:10.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:10.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:10.709 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:10.733 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:10.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:10.834 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:10.864 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:10.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:10.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:10.965 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:10.999 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:10.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:11.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:11.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:11.140 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:11.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:11.241 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:11.277 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:11.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:11.378 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:11.404 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:11.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:11.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:11.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:11.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[39715]: pgmap v374: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 817 op/s; 1133/252615 objects degraded (0.449%); 2535/252615 objects misplaced (1.004%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:17:11.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[39715]: osdmap e220: 8 total, 8 up, 8 in 2024-03-11T22:17:11.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[39715]: 8.7 scrub starts 2024-03-11T22:17:11.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[39715]: 4.1c scrub starts 2024-03-11T22:17:11.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[39715]: 4.1c scrub ok 2024-03-11T22:17:11.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 1133/252615 objects degraded (0.449%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:11.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[39715]: osdmap e221: 8 total, 8 up, 8 in 2024-03-11T22:17:11.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[30178]: pgmap v374: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 817 op/s; 1133/252615 objects degraded (0.449%); 2535/252615 objects misplaced (1.004%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:17:11.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[30178]: osdmap e220: 8 total, 8 up, 8 in 2024-03-11T22:17:11.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[30178]: 8.7 scrub starts 2024-03-11T22:17:11.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[30178]: 4.1c scrub starts 2024-03-11T22:17:11.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[30178]: 4.1c scrub ok 2024-03-11T22:17:11.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 1133/252615 objects degraded (0.449%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:11.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:11 smithi045 ceph-mon[30178]: osdmap e221: 8 total, 8 up, 8 in 2024-03-11T22:17:11.543 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:11.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:11.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:11 smithi144 ceph-mon[33984]: pgmap v374: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 817 op/s; 1133/252615 objects degraded (0.449%); 2535/252615 objects misplaced (1.004%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:17:11.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:11 smithi144 ceph-mon[33984]: osdmap e220: 8 total, 8 up, 8 in 2024-03-11T22:17:11.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:11 smithi144 ceph-mon[33984]: 8.7 scrub starts 2024-03-11T22:17:11.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:11 smithi144 ceph-mon[33984]: 4.1c scrub starts 2024-03-11T22:17:11.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:11 smithi144 ceph-mon[33984]: 4.1c scrub ok 2024-03-11T22:17:11.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:11 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 1133/252615 objects degraded (0.449%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:11.599 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:11 smithi144 ceph-mon[33984]: osdmap e221: 8 total, 8 up, 8 in 2024-03-11T22:17:11.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:11.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:11.669 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:11.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:11.770 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:11.803 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:11.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:11.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:11.905 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:11.930 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:11.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:12.032 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:12.065 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:12.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:12.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:12.167 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:12.204 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:12.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:12.306 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:12.342 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:12.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:12.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:12.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:12.480 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:12.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:12.485 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:12 smithi045 ceph-mon[39715]: 8.9 scrub starts 2024-03-11T22:17:12.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:12 smithi045 ceph-mon[30178]: 8.9 scrub starts 2024-03-11T22:17:12.583 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:12.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:12 smithi144 ceph-mon[33984]: 8.9 scrub starts 2024-03-11T22:17:12.617 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:12.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:12.688 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:12.699 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:12.719 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:12.748 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:12.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:12.781 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:17:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:17:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:17:12.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:12.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:12.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:12.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:13.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:13.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:13.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:13.187 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:13.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:13.288 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:13.319 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:13.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:13.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:13.421 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:13.455 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:13.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:13.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:13.510 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:13 smithi045 ceph-mon[39715]: 8.5 scrub starts 2024-03-11T22:17:13.511 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:13 smithi045 ceph-mon[39715]: pgmap v377: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 885 op/s; 1327/236679 objects degraded (0.561%); 2217/236679 objects misplaced (0.937%); 1.3 MiB/s, 20 objects/s recovering 2024-03-11T22:17:13.511 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:13 smithi045 ceph-mon[39715]: 8.9 scrub ok 2024-03-11T22:17:13.511 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:13 smithi045 ceph-mon[39715]: 8.7 scrub ok 2024-03-11T22:17:13.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:13 smithi045 ceph-mon[30178]: 8.5 scrub starts 2024-03-11T22:17:13.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:13 smithi045 ceph-mon[30178]: pgmap v377: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 885 op/s; 1327/236679 objects degraded (0.561%); 2217/236679 objects misplaced (0.937%); 1.3 MiB/s, 20 objects/s recovering 2024-03-11T22:17:13.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:13 smithi045 ceph-mon[30178]: 8.9 scrub ok 2024-03-11T22:17:13.511 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:13 smithi045 ceph-mon[30178]: 8.7 scrub ok 2024-03-11T22:17:13.557 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:13.586 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:13.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:13.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:13 smithi144 ceph-mon[33984]: 8.5 scrub starts 2024-03-11T22:17:13.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:13 smithi144 ceph-mon[33984]: pgmap v377: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 885 op/s; 1327/236679 objects degraded (0.561%); 2217/236679 objects misplaced (0.937%); 1.3 MiB/s, 20 objects/s recovering 2024-03-11T22:17:13.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:13 smithi144 ceph-mon[33984]: 8.9 scrub ok 2024-03-11T22:17:13.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:13 smithi144 ceph-mon[33984]: 8.7 scrub ok 2024-03-11T22:17:13.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:13.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:13.740 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:13.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:13.841 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:13.873 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:13.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:13.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:13.975 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:14.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:14.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:14.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:14.051 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:17:14.051 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:17:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:17:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:17:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:17:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:17:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:17:14.052 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:17:14.054 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:17:14.054 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:17:14.054 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:17:14.054 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:17:14.054 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:17:14.055 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:17:14.055 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:17:14.055 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:17:14.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:14.159 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:14.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:14.261 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:14.294 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:14.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:14.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:14.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:14.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:14.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:14 smithi045 ceph-mon[39715]: 5.19 scrub starts 2024-03-11T22:17:14.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:14 smithi045 ceph-mon[39715]: 5.19 scrub ok 2024-03-11T22:17:14.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:14 smithi045 ceph-mon[39715]: 8.5 scrub ok 2024-03-11T22:17:14.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:14 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2562257975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:17:14.478 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:14 smithi045 ceph-mon[30178]: 5.19 scrub starts 2024-03-11T22:17:14.478 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:14 smithi045 ceph-mon[30178]: 5.19 scrub ok 2024-03-11T22:17:14.478 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:14 smithi045 ceph-mon[30178]: 8.5 scrub ok 2024-03-11T22:17:14.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:14 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2562257975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:17:14.538 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:14.571 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:14.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:14.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:14.594 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:14 smithi144 ceph-mon[33984]: 5.19 scrub starts 2024-03-11T22:17:14.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:14 smithi144 ceph-mon[33984]: 5.19 scrub ok 2024-03-11T22:17:14.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:14 smithi144 ceph-mon[33984]: 8.5 scrub ok 2024-03-11T22:17:14.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:14 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2562257975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:17:14.673 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:14.709 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:14.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:14.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:14.810 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:14.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:14.837 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:14.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:14.938 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:14.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:14.982 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:14.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:15.083 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:15.126 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:15.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:15.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:15.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:15.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:15.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:15.370 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:15.402 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:15.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:15.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:15.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:15 smithi045 ceph-mon[39715]: pgmap v378: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 885 op/s; 1327/236679 objects degraded (0.561%); 2217/236679 objects misplaced (0.937%); 1.3 MiB/s, 20 objects/s recovering 2024-03-11T22:17:15.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:17:15.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:15 smithi045 ceph-mon[30178]: pgmap v378: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 885 op/s; 1327/236679 objects degraded (0.561%); 2217/236679 objects misplaced (0.937%); 1.3 MiB/s, 20 objects/s recovering 2024-03-11T22:17:15.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:17:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:15.545 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:15.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:15.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:15 smithi144 ceph-mon[33984]: pgmap v378: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 176 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 885 op/s; 1327/236679 objects degraded (0.561%); 2217/236679 objects misplaced (0.937%); 1.3 MiB/s, 20 objects/s recovering 2024-03-11T22:17:15.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:17:15.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:15.685 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:15.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:15.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:15.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:15.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:15.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:15.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:15.980 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:15.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:16.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:16.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:16.127 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:16.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:16.228 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:16.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:16.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:16.371 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:16.399 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:16.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:16.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:16.501 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:16.530 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:16.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:16.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:16.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:16 smithi045 ceph-mon[30178]: 1.f scrub starts 2024-03-11T22:17:16.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:16 smithi045 ceph-mon[30178]: 1.f scrub ok 2024-03-11T22:17:16.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:16 smithi045 ceph-mon[30178]: 1.3 scrub starts 2024-03-11T22:17:16.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:16 smithi045 ceph-mon[30178]: 1.3 scrub ok 2024-03-11T22:17:16.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:16 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 1327/236679 objects degraded (0.561%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:16.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:16 smithi045 ceph-mon[39715]: 1.f scrub starts 2024-03-11T22:17:16.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:16 smithi045 ceph-mon[39715]: 1.f scrub ok 2024-03-11T22:17:16.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:16 smithi045 ceph-mon[39715]: 1.3 scrub starts 2024-03-11T22:17:16.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:16 smithi045 ceph-mon[39715]: 1.3 scrub ok 2024-03-11T22:17:16.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:16 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 1327/236679 objects degraded (0.561%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:16.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:16 smithi144 ceph-mon[33984]: 1.f scrub starts 2024-03-11T22:17:16.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:16 smithi144 ceph-mon[33984]: 1.f scrub ok 2024-03-11T22:17:16.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:16 smithi144 ceph-mon[33984]: 1.3 scrub starts 2024-03-11T22:17:16.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:16 smithi144 ceph-mon[33984]: 1.3 scrub ok 2024-03-11T22:17:16.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:16 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 1327/236679 objects degraded (0.561%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:16.595 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:17:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:17:16.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:16.673 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:16.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:16.775 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:16.806 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:16.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:16.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:16.946 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:16.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:16.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:17.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:17.089 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:17.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:17.191 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:17.222 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:17.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:17.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:17.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:17.348 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:17.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:17.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:17.450 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:17.480 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:17.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:17.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:17.531 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:17:17.531 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:17:17.531 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-11T22:17:17.531 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-11T22:17:17.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:17:17.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:17.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:17 smithi045 ceph-mon[39715]: pgmap v379: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s; 1327/223410 objects degraded (0.594%); 2217/223410 objects misplaced (0.992%); 993 KiB/s, 15 objects/s recovering 2024-03-11T22:17:17.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:17.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:17 smithi045 ceph-mon[30178]: pgmap v379: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s; 1327/223410 objects degraded (0.594%); 2217/223410 objects misplaced (0.992%); 993 KiB/s, 15 objects/s recovering 2024-03-11T22:17:17.550 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:17.551 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:17 smithi144 ceph-mon[33984]: pgmap v379: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s; 1327/223410 objects degraded (0.594%); 2217/223410 objects misplaced (0.992%); 993 KiB/s, 15 objects/s recovering 2024-03-11T22:17:17.582 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:17.606 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:17.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:17.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:17.766 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:17.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:17.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:17.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:17.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:18.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:18.055 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:18.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:18.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:18.156 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:18.190 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:18.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:18.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:18.332 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:18.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:18.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:18.434 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:18.458 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:18.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:18.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:18 smithi045 ceph-mon[39715]: 3.0 scrub starts 2024-03-11T22:17:18.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:18 smithi045 ceph-mon[39715]: 3.0 scrub ok 2024-03-11T22:17:18.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:18 smithi045 ceph-mon[30178]: 3.0 scrub starts 2024-03-11T22:17:18.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:18 smithi045 ceph-mon[30178]: 3.0 scrub ok 2024-03-11T22:17:18.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:18.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:18 smithi144 ceph-mon[33984]: 3.0 scrub starts 2024-03-11T22:17:18.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:18 smithi144 ceph-mon[33984]: 3.0 scrub ok 2024-03-11T22:17:18.599 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:18.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:18.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:18.701 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:18.734 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:18.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:18.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:18.836 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:18.868 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:18.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:18.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:18.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:19.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:19.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:19.120 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:19.151 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:19.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:19.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:19.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:19.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:19.295 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:19.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:19.396 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:19.424 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:19.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:19.526 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:19.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:19 smithi045 ceph-mon[30178]: pgmap v380: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 1560/218703 objects degraded (0.713%); 1840/218703 objects misplaced (0.841%); 2.1 MiB/s, 33 objects/s recovering 2024-03-11T22:17:19.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:19 smithi045 ceph-mon[39715]: pgmap v380: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 1560/218703 objects degraded (0.713%); 1840/218703 objects misplaced (0.841%); 2.1 MiB/s, 33 objects/s recovering 2024-03-11T22:17:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:19.562 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:19.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:19.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:19 smithi144 ceph-mon[33984]: pgmap v380: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 1560/218703 objects degraded (0.713%); 1840/218703 objects misplaced (0.841%); 2.1 MiB/s, 33 objects/s recovering 2024-03-11T22:17:19.663 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:19.690 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:19.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:19.697 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:19.717 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:17:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-03-11T22:17:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-03-11T22:17:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:17:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205570 7.6 7:6c76be4d:::benchmark_data_smithi045_83007_object87099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.334799+0000", 2024-03-11T22:17:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 214.37733726499999, 2024-03-11T22:17:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.254940903, 2024-03-11T22:17:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.717 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205570 2024-03-11T22:17:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334799+0000", 2024-03-11T22:17:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334799+0000", 2024-03-11T22:17:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334800+0000", 2024-03-11T22:17:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9999999999999995e-07 2024-03-11T22:17:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334805+0000", 2024-03-11T22:17:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1220000000000003e-06 2024-03-11T22:17:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334818+0000", 2024-03-11T22:17:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2605e-05 2024-03-11T22:17:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.334827+0000", 2024-03-11T22:17:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6980000000000004e-06 2024-03-11T22:17:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.442726+0000", 2024-03-11T22:17:19.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.107898972 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.442893+0000", 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016715600000000001 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.449827+0000", 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0069343119999999998 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.463002+0000", 2024-03-11T22:17:19.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013174859000000001 2024-03-11T22:17:19.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.463047+0000", 2024-03-11T22:17:19.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5019999999999999e-05 2024-03-11T22:17:19.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.589496+0000", 2024-03-11T22:17:19.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.126448703 2024-03-11T22:17:19.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.589674+0000", 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000178324 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.589740+0000", 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6131999999999996e-05 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.723 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205573 7.6 7:6e2605c0:::benchmark_data_smithi045_83007_object87102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.339661+0000", 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 214.37247542700001, 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26130369399999998, 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205573 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339661+0000", 2024-03-11T22:17:19.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339661+0000", 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339662+0000", 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5700000000000002e-07 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339666+0000", 2024-03-11T22:17:19.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.715e-06 2024-03-11T22:17:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339674+0000", 2024-03-11T22:17:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1249999999999993e-06 2024-03-11T22:17:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.339679+0000", 2024-03-11T22:17:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1529999999999996e-06 2024-03-11T22:17:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.463078+0000", 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.123399414 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.463200+0000", 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012128700000000001 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.467183+0000", 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0039832870000000003 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.473615+0000", 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0064318769999999999 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.473658+0000", 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3689000000000001e-05 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.600799+0000", 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.12714049199999999 2024-03-11T22:17:19.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.600900+0000", 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000101374 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.600965+0000", 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4523999999999994e-05 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.729 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205765 7.6 7:61c98817:::benchmark_data_smithi045_83007_object87294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.659350+0000", 2024-03-11T22:17:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 214.05278672, 2024-03-11T22:17:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26587694699999997, 2024-03-11T22:17:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205765 2024-03-11T22:17:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659350+0000", 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659350+0000", 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659351+0000", 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0299999999999998e-07 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659355+0000", 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0879999999999997e-06 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659368+0000", 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3246e-05 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659375+0000", 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2620000000000001e-06 2024-03-11T22:17:19.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.659433+0000", 2024-03-11T22:17:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7500000000000002e-05 2024-03-11T22:17:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.660319+0000", 2024-03-11T22:17:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00088605900000000002 2024-03-11T22:17:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.660450+0000", 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000131118 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.663683+0000", 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032326999999999998 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.667411+0000", 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037283849999999999 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925108+0000", 2024-03-11T22:17:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25769741899999998 2024-03-11T22:17:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925161+0000", 2024-03-11T22:17:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2249000000000001e-05 2024-03-11T22:17:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925227+0000", 2024-03-11T22:17:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6118000000000006e-05 2024-03-11T22:17:19.735 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(osd.2.0:40363 7.e e150/99 7:7ad23d92:::benchmark_data_smithi045_83007_object117059:head v 150'14406, mlcod=150'14402)", 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.670957+0000", 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 214.04118000400001, 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.254146123, 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.670957+0000", 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.670957+0000", 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.670957+0000", 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.97e-07 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.737 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.670960+0000", 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2030000000000001e-06 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.670965+0000", 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2259999999999996e-06 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.670969+0000", 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9079999999999997e-06 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.671761+0000", 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00079232900000000004 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.671779+0000", 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7638999999999998e-05 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925059+0000", 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25328029099999999 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925103+0000", 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.333e-05 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205777 7.6 7:6d0e94e9:::benchmark_data_smithi045_83007_object87306:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.674945+0000", 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 214.03719195799999, 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26027302000000002, 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.740 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205777 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.674945+0000", 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.674945+0000", 2024-03-11T22:17:19.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.674945+0000", 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1399999999999997e-07 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.674948+0000", 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3780000000000001e-06 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.674955+0000", 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8730000000000001e-06 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.674961+0000", 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3290000000000001e-06 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678647+0000", 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003686553 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678780+0000", 2024-03-11T22:17:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000133101 2024-03-11T22:17:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678913+0000", 2024-03-11T22:17:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013320899999999999 2024-03-11T22:17:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.744 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925270+0000", 2024-03-11T22:17:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24635607000000001 2024-03-11T22:17:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.928772+0000", 2024-03-11T22:17:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035028960000000001 2024-03-11T22:17:19.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935100+0000", 2024-03-11T22:17:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0063271990000000004 2024-03-11T22:17:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935156+0000", 2024-03-11T22:17:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6638000000000002e-05 2024-03-11T22:17:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935218+0000", 2024-03-11T22:17:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1359999999999995e-05 2024-03-11T22:17:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(osd.2.0:40365 7.e e150/99 7:7ad5e5b2:::benchmark_data_smithi045_83007_object116423:head v 150'14408, mlcod=150'14402)", 2024-03-11T22:17:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.682871+0000", 2024-03-11T22:17:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 214.02926511199999, 2024-03-11T22:17:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25217707900000003, 2024-03-11T22:17:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.747 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.682871+0000", 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.682871+0000", 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.682872+0000", 2024-03-11T22:17:19.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8100000000000002e-07 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.682879+0000", 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6279999999999999e-06 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.682890+0000", 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0725e-05 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.682897+0000", 2024-03-11T22:17:19.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1999999999999997e-06 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925128+0000", 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.242230959 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.925146+0000", 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8527999999999999e-05 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.750 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935008+0000", 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0098619789999999999 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935049+0000", 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0379000000000001e-05 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(osd.2.0:40367 7.e e150/99 7:7ad6d4df:::benchmark_data_smithi045_83007_object106301:head v 150'14409, mlcod=150'14402)", 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.685109+0000", 2024-03-11T22:17:19.751 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 214.02702713299999, 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26332421700000003, 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685109+0000", 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685109+0000", 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685110+0000", 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1999999999999998e-07 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685117+0000", 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5400000000000001e-06 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685128+0000", 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1573000000000001e-05 2024-03-11T22:17:19.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685136+0000", 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4739999999999999e-06 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.928850+0000", 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24371410600000001 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.928871+0000", 2024-03-11T22:17:19.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1254e-05 2024-03-11T22:17:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.948384+0000", 2024-03-11T22:17:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.019512390000000001 2024-03-11T22:17:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.948434+0000", 2024-03-11T22:17:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9960000000000003e-05 2024-03-11T22:17:19.755 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:205777 7.6 e150/99 ondisk, result = 0)", 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.685801+0000", 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 214.026335805, 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26263702300000002, 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685801+0000", 2024-03-11T22:17:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685801+0000", 2024-03-11T22:17:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685801+0000", 2024-03-11T22:17:19.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.92e-07 2024-03-11T22:17:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.685806+0000", 2024-03-11T22:17:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9679999999999997e-06 2024-03-11T22:17:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.824171+0000", 2024-03-11T22:17:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.138364602 2024-03-11T22:17:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.758 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.824182+0000", 2024-03-11T22:17:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0991e-05 2024-03-11T22:17:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935084+0000", 2024-03-11T22:17:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.110902444 2024-03-11T22:17:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.935093+0000", 2024-03-11T22:17:19.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3829999999999992e-06 2024-03-11T22:17:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.948438+0000", 2024-03-11T22:17:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013344943 2024-03-11T22:17:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206092 7.6 7:6f97db3b:::benchmark_data_smithi045_83007_object87621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.364084+0000", 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 213.34805229400001, 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26853602199999999, 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206092 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.364084+0000", 2024-03-11T22:17:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.364084+0000", 2024-03-11T22:17:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.364085+0000", 2024-03-11T22:17:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6499999999999998e-07 2024-03-11T22:17:19.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.364089+0000", 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6509999999999999e-06 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.364100+0000", 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1148999999999999e-05 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.364108+0000", 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8499999999999994e-06 2024-03-11T22:17:19.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.619709+0000", 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25560139399999998 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.619854+0000", 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014506 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.620091+0000", 2024-03-11T22:17:19.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023680199999999999 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.623283+0000", 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031920569999999999 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.623326+0000", 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3340000000000002e-05 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632383+0000", 2024-03-11T22:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090561749999999996 2024-03-11T22:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632555+0000", 2024-03-11T22:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017295 2024-03-11T22:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632620+0000", 2024-03-11T22:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4828999999999999e-05 2024-03-11T22:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.766 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206102 7.6 7:6abe9cdb:::benchmark_data_smithi045_83007_object87631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.381022+0000", 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 213.331114939, 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25920192800000003, 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206102 2024-03-11T22:17:19.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.381022+0000", 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.381022+0000", 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.381022+0000", 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4499999999999997e-07 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.381025+0000", 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8459999999999999e-06 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.381031+0000", 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0630000000000002e-06 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.381036+0000", 2024-03-11T22:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8400000000000002e-06 2024-03-11T22:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.623360+0000", 2024-03-11T22:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24232437500000001 2024-03-11T22:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.623487+0000", 2024-03-11T22:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012716800000000001 2024-03-11T22:17:19.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.623637+0000", 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014924599999999999 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632284+0000", 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0086475739999999999 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.637029+0000", 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0047448339999999999 2024-03-11T22:17:19.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.640103+0000", 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030742759999999999 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.640160+0000", 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6295999999999998e-05 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.640224+0000", 2024-03-11T22:17:19.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3965000000000002e-05 2024-03-11T22:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206106 7.6 7:6eb77cd5:::benchmark_data_smithi045_83007_object87635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.386909+0000", 2024-03-11T22:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 213.32522759899999, 2024-03-11T22:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25651542799999999, 2024-03-11T22:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.773 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206106 2024-03-11T22:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.774 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.386909+0000", 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.386909+0000", 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.386911+0000", 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.525e-06 2024-03-11T22:17:19.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.386915+0000", 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3030000000000001e-06 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.386926+0000", 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0937e-05 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.386933+0000", 2024-03-11T22:17:19.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5399999999999998e-06 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.626396+0000", 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.239463062 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.626515+0000", 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011841300000000001 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.777 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.626649+0000", 2024-03-11T22:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000133819 2024-03-11T22:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632331+0000", 2024-03-11T22:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0056820280000000004 2024-03-11T22:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.637035+0000", 2024-03-11T22:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0047048860000000001 2024-03-11T22:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.643259+0000", 2024-03-11T22:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0062232880000000004 2024-03-11T22:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.643311+0000", 2024-03-11T22:17:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2222000000000002e-05 2024-03-11T22:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.643424+0000", 2024-03-11T22:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113405 2024-03-11T22:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206109 7.6 7:639211c5:::benchmark_data_smithi045_83007_object87638:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.390165+0000", 2024-03-11T22:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 213.321971338, 2024-03-11T22:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26705835500000003, 2024-03-11T22:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206109 2024-03-11T22:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.781 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.390165+0000", 2024-03-11T22:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.390165+0000", 2024-03-11T22:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.782 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.390166+0000", 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1179999999999999e-06 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.390171+0000", 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5229999999999997e-06 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.390182+0000", 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0787e-05 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.783 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.390189+0000", 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2289999999999997e-06 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.629400+0000", 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.239211216 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.629516+0000", 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011550899999999999 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.629641+0000", 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000125625 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632369+0000", 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027274249999999999 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.637041+0000", 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046722300000000003 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657111+0000", 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.020069908000000001 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657163+0000", 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2206999999999997e-05 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657224+0000", 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0578000000000002e-05 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.786 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206113 7.6 7:6bba99fc:::benchmark_data_smithi045_83007_object87642:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.394088+0000", 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 213.318048551, 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26328315299999999, 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206113 2024-03-11T22:17:19.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.394088+0000", 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.394088+0000", 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.394089+0000", 2024-03-11T22:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.389e-06 2024-03-11T22:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.394094+0000", 2024-03-11T22:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5789999999999997e-06 2024-03-11T22:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.394105+0000", 2024-03-11T22:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1455e-05 2024-03-11T22:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.394113+0000", 2024-03-11T22:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5650000000000004e-06 2024-03-11T22:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632727+0000", 2024-03-11T22:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23861432799999999 2024-03-11T22:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.632903+0000", 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000176007 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.633016+0000", 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011265 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.640264+0000", 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072476320000000004 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.651405+0000", 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011141751 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657270+0000", 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058646929999999998 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657319+0000", 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9270000000000001e-05 2024-03-11T22:17:19.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657371+0000", 2024-03-11T22:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1833999999999999e-05 2024-03-11T22:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.793 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206115 7.6 7:6c309bcc:::benchmark_data_smithi045_83007_object87644:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.396871+0000", 2024-03-11T22:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 213.31526601100001, 2024-03-11T22:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26066324600000002, 2024-03-11T22:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206115 2024-03-11T22:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.794 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.396871+0000", 2024-03-11T22:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.396871+0000", 2024-03-11T22:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.396872+0000", 2024-03-11T22:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.327e-06 2024-03-11T22:17:19.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.396895+0000", 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3008999999999999e-05 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.396907+0000", 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1812e-05 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.396914+0000", 2024-03-11T22:17:19.796 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7670000000000003e-06 2024-03-11T22:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.637102+0000", 2024-03-11T22:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.240187228 2024-03-11T22:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.637231+0000", 2024-03-11T22:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012963100000000001 2024-03-11T22:17:19.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.637349+0000", 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011807900000000001 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.640299+0000", 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029497719999999998 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.651411+0000", 2024-03-11T22:17:19.798 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011112037 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657418+0000", 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060068689999999998 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657475+0000", 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6654999999999999e-05 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.657534+0000", 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.906e-05 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206260 7.6 7:64cdc02f:::benchmark_data_smithi045_83007_object87789:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.716010+0000", 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 212.996126836, 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.35534831, 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.800 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206260 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.716010+0000", 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.716010+0000", 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.801 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.716010+0000", 2024-03-11T22:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7700000000000005e-07 2024-03-11T22:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.716013+0000", 2024-03-11T22:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7829999999999999e-06 2024-03-11T22:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.716019+0000", 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0190000000000003e-06 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.716024+0000", 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2209999999999996e-06 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.929286+0000", 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.213261531 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.929393+0000", 2024-03-11T22:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010679300000000001 2024-03-11T22:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.929510+0000", 2024-03-11T22:17:19.804 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116945 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.939058+0000", 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0095488250000000004 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.943333+0000", 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042741690000000004 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.805 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071236+0000", 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.127903724 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071295+0000", 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9200000000000002e-05 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071358+0000", 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2623e-05 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.806 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206278 7.6 7:606b27ef:::benchmark_data_smithi045_83007_object87807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.743193+0000", 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 212.968943811, 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.32832548299999997, 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206278 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.807 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.743193+0000", 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.743193+0000", 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.743195+0000", 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.73e-06 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.743200+0000", 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2750000000000003e-06 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.743216+0000", 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6545000000000002e-05 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.743226+0000", 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2480000000000002e-06 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.933884+0000", 2024-03-11T22:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19065871100000001 2024-03-11T22:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.934001+0000", 2024-03-11T22:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116561 2024-03-11T22:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.934132+0000", 2024-03-11T22:17:19.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013122899999999999 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.943338+0000", 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0092062749999999999 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.943410+0000", 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1379999999999998e-05 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071427+0000", 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.12801737199999999 2024-03-11T22:17:19.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071480+0000", 2024-03-11T22:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2843999999999998e-05 2024-03-11T22:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071518+0000", 2024-03-11T22:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8312999999999997e-05 2024-03-11T22:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206302 7.6 7:6f8f4a0d:::benchmark_data_smithi045_83007_object87831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.783991+0000", 2024-03-11T22:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 212.92814600200001, 2024-03-11T22:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.287799886, 2024-03-11T22:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.813 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206302 2024-03-11T22:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.783991+0000", 2024-03-11T22:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.814 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.783991+0000", 2024-03-11T22:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.783992+0000", 2024-03-11T22:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1170000000000001e-06 2024-03-11T22:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.783996+0000", 2024-03-11T22:17:19.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3950000000000003e-06 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.784005+0000", 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0969999999999995e-06 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.784012+0000", 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1740000000000003e-06 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.939154+0000", 2024-03-11T22:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15514196999999999 2024-03-11T22:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.939302+0000", 2024-03-11T22:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014793200000000001 2024-03-11T22:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.939424+0000", 2024-03-11T22:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012194899999999999 2024-03-11T22:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.943452+0000", 2024-03-11T22:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040274509999999996 2024-03-11T22:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.818 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.943496+0000", 2024-03-11T22:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3856999999999997e-05 2024-03-11T22:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071598+0000", 2024-03-11T22:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.128102418 2024-03-11T22:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071737+0000", 2024-03-11T22:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000139281 2024-03-11T22:17:19.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:47.071790+0000", 2024-03-11T22:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3245000000000003e-05 2024-03-11T22:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:207163 7.6 7:6ad72b27:::benchmark_data_smithi045_83007_object88692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:48.007054+0000", 2024-03-11T22:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 211.70508241799999, 2024-03-11T22:17:19.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.28364153800000003, 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 207163 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.007054+0000", 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.007054+0000", 2024-03-11T22:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.007055+0000", 2024-03-11T22:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6600000000000005e-07 2024-03-11T22:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.007059+0000", 2024-03-11T22:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1760000000000003e-06 2024-03-11T22:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.007070+0000", 2024-03-11T22:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1046000000000001e-05 2024-03-11T22:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.007078+0000", 2024-03-11T22:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.418e-06 2024-03-11T22:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.167390+0000", 2024-03-11T22:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16031189300000001 2024-03-11T22:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.167479+0000", 2024-03-11T22:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9093999999999998e-05 2024-03-11T22:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.167545+0000", 2024-03-11T22:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5897999999999998e-05 2024-03-11T22:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.176768+0000", 2024-03-11T22:17:19.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0092230179999999995 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.176786+0000", 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8175000000000001e-05 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.289974+0000", 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.113188146 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.827 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.290618+0000", 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00064369999999999998 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.290696+0000", 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8108000000000005e-05 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:207176 7.6 7:633abc52:::benchmark_data_smithi045_83007_object88705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:48.023879+0000", 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 211.68825786799999, 2024-03-11T22:17:19.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26985240199999999, 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 207176 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.023879+0000", 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.023879+0000", 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.023880+0000", 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5850000000000001e-06 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.023884+0000", 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7589999999999999e-06 2024-03-11T22:17:19.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.023902+0000", 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7965000000000002e-05 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.023910+0000", 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5950000000000001e-06 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.170249+0000", 2024-03-11T22:17:19.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.146339513 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.170383+0000", 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000133657 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.170518+0000", 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000135053 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.176823+0000", 2024-03-11T22:17:19.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0063052079999999996 2024-03-11T22:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.176903+0000", 2024-03-11T22:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0298999999999998e-05 2024-03-11T22:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.293591+0000", 2024-03-11T22:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.116688002 2024-03-11T22:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.293670+0000", 2024-03-11T22:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8276999999999996e-05 2024-03-11T22:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.293731+0000", 2024-03-11T22:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1488999999999994e-05 2024-03-11T22:17:19.834 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:207181 7.6 7:6f47bc4a:::benchmark_data_smithi045_83007_object88710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:48.028352+0000", 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 211.68378455800001, 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26552772899999999, 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:19.835 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 207181 2024-03-11T22:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.028352+0000", 2024-03-11T22:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:19.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.028352+0000", 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.028353+0000", 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.46e-06 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.028358+0000", 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9840000000000001e-06 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.028373+0000", 2024-03-11T22:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4144e-05 2024-03-11T22:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.028380+0000", 2024-03-11T22:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5850000000000002e-06 2024-03-11T22:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.172534+0000", 2024-03-11T22:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14415375999999999 2024-03-11T22:17:19.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.172651+0000", 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117262 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.172793+0000", 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014129100000000001 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.176888+0000", 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.004095975 2024-03-11T22:17:19.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.176910+0000", 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2014999999999998e-05 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.293777+0000", 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.116866316 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.293823+0000", 2024-03-11T22:17:19.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6659000000000002e-05 2024-03-11T22:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:48.293880+0000", 2024-03-11T22:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6277999999999999e-05 2024-03-11T22:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:19.841 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:19.842 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:17:19.853 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:19.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:19.955 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:19.987 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:19.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:19.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:20.089 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:20.124 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:20.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:20.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:20.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:20.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:20.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:20.301 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:17:20.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:20.410 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:20.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:20.511 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:20.553 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:20.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:20.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:20.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:20 smithi144 ceph-mon[33984]: 5.10 scrub starts 2024-03-11T22:17:20.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:20 smithi144 ceph-mon[33984]: 5.10 scrub ok 2024-03-11T22:17:20.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:20 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1394352314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:17:20.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:20 smithi144 ceph-mon[33984]: 6.1d scrub starts 2024-03-11T22:17:20.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:20 smithi144 ceph-mon[33984]: 6.1d scrub ok 2024-03-11T22:17:20.655 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:20.688 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:20.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:20.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:20 smithi045 ceph-mon[30178]: 5.10 scrub starts 2024-03-11T22:17:20.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:20 smithi045 ceph-mon[30178]: 5.10 scrub ok 2024-03-11T22:17:20.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:20 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1394352314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:17:20.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:20 smithi045 ceph-mon[30178]: 6.1d scrub starts 2024-03-11T22:17:20.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:20 smithi045 ceph-mon[30178]: 6.1d scrub ok 2024-03-11T22:17:20.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:20 smithi045 ceph-mon[39715]: 5.10 scrub starts 2024-03-11T22:17:20.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:20 smithi045 ceph-mon[39715]: 5.10 scrub ok 2024-03-11T22:17:20.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:20 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1394352314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:17:20.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:20 smithi045 ceph-mon[39715]: 6.1d scrub starts 2024-03-11T22:17:20.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:20 smithi045 ceph-mon[39715]: 6.1d scrub ok 2024-03-11T22:17:20.789 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:20.833 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:20.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:20.936 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:20.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:17:20.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:20.978 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:20.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:21.079 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:21.118 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:21.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:21.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:21.277 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:21.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:21.378 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:21.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:21.434 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:21.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:21.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:21.573 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:21.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:21.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:21.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:21 smithi144 ceph-mon[33984]: pgmap v381: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s; 1560/218703 objects degraded (0.713%); 1840/218703 objects misplaced (0.841%); 1.7 MiB/s, 26 objects/s recovering 2024-03-11T22:17:21.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:21 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:17:21.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:21 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1394352314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:17:21.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:21 smithi144 ceph-mon[33984]: osdmap e222: 8 total, 8 up, 8 in 2024-03-11T22:17:21.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:21 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:21.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:21 smithi144 ceph-mon[33984]: 3.e scrub starts 2024-03-11T22:17:21.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:21 smithi144 ceph-mon[33984]: 3.e scrub ok 2024-03-11T22:17:21.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:21 smithi144 ceph-mon[33984]: 6.8 deep-scrub starts 2024-03-11T22:17:21.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:21 smithi144 ceph-mon[33984]: 6.8 deep-scrub ok 2024-03-11T22:17:21.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:21 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 1560/218703 objects degraded (0.713%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:21.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:21 smithi144 ceph-mon[33984]: osdmap e223: 8 total, 8 up, 8 in 2024-03-11T22:17:21.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:21.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:21.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:21.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:21.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[39715]: pgmap v381: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s; 1560/218703 objects degraded (0.713%); 1840/218703 objects misplaced (0.841%); 1.7 MiB/s, 26 objects/s recovering 2024-03-11T22:17:21.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:17:21.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1394352314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:17:21.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[39715]: osdmap e222: 8 total, 8 up, 8 in 2024-03-11T22:17:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[39715]: 3.e scrub starts 2024-03-11T22:17:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[39715]: 3.e scrub ok 2024-03-11T22:17:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[39715]: 6.8 deep-scrub starts 2024-03-11T22:17:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[39715]: 6.8 deep-scrub ok 2024-03-11T22:17:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 1560/218703 objects degraded (0.713%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[39715]: osdmap e223: 8 total, 8 up, 8 in 2024-03-11T22:17:21.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[30178]: pgmap v381: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s; 1560/218703 objects degraded (0.713%); 1840/218703 objects misplaced (0.841%); 1.7 MiB/s, 26 objects/s recovering 2024-03-11T22:17:21.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:17:21.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1394352314' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:17:21.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[30178]: osdmap e222: 8 total, 8 up, 8 in 2024-03-11T22:17:21.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:17:21.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[30178]: 3.e scrub starts 2024-03-11T22:17:21.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[30178]: 3.e scrub ok 2024-03-11T22:17:21.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[30178]: 6.8 deep-scrub starts 2024-03-11T22:17:21.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[30178]: 6.8 deep-scrub ok 2024-03-11T22:17:21.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 1560/218703 objects degraded (0.713%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:21.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:21 smithi045 ceph-mon[30178]: osdmap e223: 8 total, 8 up, 8 in 2024-03-11T22:17:21.818 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:21.856 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:21.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:21.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:21.958 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:21.989 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:21.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:21.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:22.091 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:22.125 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:22.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:22.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:22.227 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:22.258 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:22.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:22.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:22.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:22.397 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:22.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:22.498 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:22.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T22:17:22.530 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:22.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:22.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:22.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:22 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:22.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:22 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:17:22.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:22 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:17:22.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:22 smithi045 ceph-mon[39715]: 3.9 scrub starts 2024-03-11T22:17:22.548 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:22 smithi045 ceph-mon[39715]: 3.9 scrub ok 2024-03-11T22:17:22.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:22 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:22.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:22 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:17:22.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:22 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:17:22.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:22 smithi045 ceph-mon[30178]: 3.9 scrub starts 2024-03-11T22:17:22.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:22 smithi045 ceph-mon[30178]: 3.9 scrub ok 2024-03-11T22:17:22.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:22 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:17:22.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:22 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:17:22.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:22 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:17:22.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:22 smithi144 ceph-mon[33984]: 3.9 scrub starts 2024-03-11T22:17:22.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:22 smithi144 ceph-mon[33984]: 3.9 scrub ok 2024-03-11T22:17:22.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:22.682 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:22.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:22.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:22.784 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:22.815 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:22.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:22.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:22.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:22.925 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:17:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:17:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:17:22.983 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:22.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:22.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:23.085 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:23.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:23.122 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:23.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:23.223 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:23.254 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:23.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:23.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:23.356 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:23.397 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:23.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:23.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:23.499 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:23.530 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:23.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:23.623 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:23.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:23.698 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:23.719 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:23.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:23.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:23.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:23 smithi045 ceph-mon[39715]: pgmap v384: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 1769/201324 objects degraded (0.879%); 1490/201324 objects misplaced (0.740%); 2.2 MiB/s, 35 objects/s recovering 2024-03-11T22:17:23.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:23 smithi045 ceph-mon[39715]: 4.4 scrub starts 2024-03-11T22:17:23.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:23 smithi045 ceph-mon[39715]: 4.4 scrub ok 2024-03-11T22:17:23.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:23 smithi045 ceph-mon[39715]: 5.4 scrub starts 2024-03-11T22:17:23.823 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:23 smithi045 ceph-mon[39715]: 5.4 scrub ok 2024-03-11T22:17:23.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:23 smithi045 ceph-mon[30178]: pgmap v384: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 1769/201324 objects degraded (0.879%); 1490/201324 objects misplaced (0.740%); 2.2 MiB/s, 35 objects/s recovering 2024-03-11T22:17:23.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:23 smithi045 ceph-mon[30178]: 4.4 scrub starts 2024-03-11T22:17:23.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:23 smithi045 ceph-mon[30178]: 4.4 scrub ok 2024-03-11T22:17:23.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:23 smithi045 ceph-mon[30178]: 5.4 scrub starts 2024-03-11T22:17:23.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:23 smithi045 ceph-mon[30178]: 5.4 scrub ok 2024-03-11T22:17:23.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:23.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:23.972 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:23.996 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:23.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:23.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:24.024 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:17:24.028 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:17:24.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:23 smithi144 ceph-mon[33984]: pgmap v384: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 1769/201324 objects degraded (0.879%); 1490/201324 objects misplaced (0.740%); 2.2 MiB/s, 35 objects/s recovering 2024-03-11T22:17:24.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:23 smithi144 ceph-mon[33984]: 4.4 scrub starts 2024-03-11T22:17:24.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:23 smithi144 ceph-mon[33984]: 4.4 scrub ok 2024-03-11T22:17:24.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:23 smithi144 ceph-mon[33984]: 5.4 scrub starts 2024-03-11T22:17:24.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:23 smithi144 ceph-mon[33984]: 5.4 scrub ok 2024-03-11T22:17:24.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:24.140 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:24.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:24.241 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:24.273 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:24.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:24.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:24.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:24.415 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:24.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:24.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:24.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:24.553 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:24.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:24.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:24.655 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:24.688 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:24.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:24.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:24.789 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:24.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:24.825 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:24.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:24.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:24.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:24.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:24.987 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:17:25.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:24 smithi045 ceph-mon[39715]: 4.2 scrub starts 2024-03-11T22:17:25.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:24 smithi045 ceph-mon[39715]: 4.2 scrub ok 2024-03-11T22:17:25.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:24 smithi045 ceph-mon[30178]: 4.2 scrub starts 2024-03-11T22:17:25.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:24 smithi045 ceph-mon[30178]: 4.2 scrub ok 2024-03-11T22:17:25.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:25.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:24 smithi144 ceph-mon[33984]: 4.2 scrub starts 2024-03-11T22:17:25.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:24 smithi144 ceph-mon[33984]: 4.2 scrub ok 2024-03-11T22:17:25.118 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:25.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:25.220 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:25.253 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:25.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:25.356 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:25.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:25.418 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:25.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:25.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:25.558 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:25.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:25.630 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-03-11T22:17:22.214089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4385446260,"num_objects":67108,"num_object_clones":0,"num_object_copies":201324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1769,"num_objects_misplaced":1490,"num_objects_unfound":0,"num_objects_dirty":67108,"num_whiteouts":0,"num_read":1190,"num_read_kb":942,"num_write":181757,"num_write_kb":7943406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7837,"num_bytes_recovered":509871897,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70454,"ondisk_log_size":70454,"up":534,"acting":533,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":536,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15650920,"kb_used_data":13595104,"kb_used_omap":12,"kb_used_meta":2055475,"kb_avail":734310296,"statfs":{"total":767960285184,"available":751933743104,"internally_reserved":0,"allocated":13921386496,"data_stored":13918381090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":2104806963},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,5,6,5,5,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-772341760,"num_objects":-11785,"num_object_clones":0,"num_object_copies":-35355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":442,"num_objects_misplaced":-727,"num_objects_unfound":0,"num_objects_dirty":-11785,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11785,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":285,"num_bytes_recovered":18677760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002784"},"pg_stats":[{"pgid":"8.0","version":"222'11207","reported_seq":11312,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.840809+0000","last_change":"2024-03-11T22:14:52.240741+0000","last_active":"2024-03-11T22:17:20.840809+0000","last_peered":"2024-03-11T22:17:20.840809+0000","last_clean":"2024-03-11T22:17:20.840809+0000","last_became_active":"2024-03-11T22:14:52.240283+0000","last_became_peered":"2024-03-11T22:14:52.240283+0000","last_unstale":"2024-03-11T22:17:20.840809+0000","last_undegraded":"2024-03-11T22:17:20.840809+0000","last_fullsized":"2024-03-11T22:17:20.840809+0000","mapping_epoch":175,"log_start":"211'7377","ondisk_log_start":"211'7377","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3830,"log_dups_size":0,"ondisk_log_size":3830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274923520,"num_objects":4195,"num_object_clones":0,"num_object_copies":12585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11207,"num_write_kb":492864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":139,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.308704+0000","last_change":"2024-03-11T22:14:52.237967+0000","last_active":"2024-03-11T22:17:20.308704+0000","last_peered":"2024-03-11T22:17:20.308704+0000","last_clean":"2024-03-11T22:17:20.308704+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:17:20.308704+0000","last_undegraded":"2024-03-11T22:17:20.308704+0000","last_fullsized":"2024-03-11T22:17:20.308704+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:14:24.847623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:24.847623+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":346,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.307114+0000","last_change":"2024-03-11T22:14:12.801583+0000","last_active":"2024-03-11T22:17:20.307114+0000","last_peered":"2024-03-11T22:17:20.307114+0000","last_clean":"2024-03-11T22:17:20.307114+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:17:20.307114+0000","last_undegraded":"2024-03-11T22:17:20.307114+0000","last_fullsized":"2024-03-11T22:17:20.307114+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:12.801470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:12.801470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":469,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.295899+0000","last_change":"2024-03-11T22:16:19.736198+0000","last_active":"2024-03-11T22:17:20.295899+0000","last_peered":"2024-03-11T22:17:20.295899+0000","last_clean":"2024-03-11T22:17:20.295899+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:17:20.295899+0000","last_undegraded":"2024-03-11T22:17:20.295899+0000","last_fullsized":"2024-03-11T22:17:20.295899+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:16:19.736028+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:09:38.769825+0000","last_clean_scrub_stamp":"2024-03-11T22:16:19.736028+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:26.416651+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":113,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.314028+0000","last_change":"2024-03-11T22:15:40.342976+0000","last_active":"2024-03-11T22:17:20.314028+0000","last_peered":"2024-03-11T22:17:20.314028+0000","last_clean":"2024-03-11T22:17:20.314028+0000","last_became_active":"2024-03-11T22:14:52.243690+0000","last_became_peered":"2024-03-11T22:14:52.243690+0000","last_unstale":"2024-03-11T22:17:20.314028+0000","last_undegraded":"2024-03-11T22:17:20.314028+0000","last_fullsized":"2024-03-11T22:17:20.314028+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:40.342855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:40.342855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":623,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.304920+0000","last_change":"2024-03-11T22:16:15.808286+0000","last_active":"2024-03-11T22:17:20.304920+0000","last_peered":"2024-03-11T22:17:20.304920+0000","last_clean":"2024-03-11T22:17:20.304920+0000","last_became_active":"2024-03-11T22:16:15.807919+0000","last_became_peered":"2024-03-11T22:16:15.807919+0000","last_unstale":"2024-03-11T22:17:20.304920+0000","last_undegraded":"2024-03-11T22:17:20.304920+0000","last_fullsized":"2024-03-11T22:17:20.304920+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:10.906610+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:10.906610+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:23.676227+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"222'11452","reported_seq":11684,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.632746+0000","last_change":"2024-03-11T22:16:57.831247+0000","last_active":"2024-03-11T22:17:20.632746+0000","last_peered":"2024-03-11T22:17:20.632746+0000","last_clean":"2024-03-11T22:17:20.632746+0000","last_became_active":"2024-03-11T22:14:52.244799+0000","last_became_peered":"2024-03-11T22:14:52.244799+0000","last_unstale":"2024-03-11T22:17:20.632746+0000","last_undegraded":"2024-03-11T22:17:20.632746+0000","last_fullsized":"2024-03-11T22:17:20.632746+0000","mapping_epoch":175,"log_start":"211'7588","ondisk_log_start":"211'7588","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'9805","last_scrub_stamp":"2024-03-11T22:16:57.830919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:16:57.830919+0000","objects_scrubbed":5831,"log_size":3864,"log_dups_size":0,"ondisk_log_size":3864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:16.450126+0000","scrub_duration":2357,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268042240,"num_objects":4090,"num_object_clones":0,"num_object_copies":12270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11452,"num_write_kb":497344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":168,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.295743+0000","last_change":"2024-03-11T22:16:27.753585+0000","last_active":"2024-03-11T22:17:20.295743+0000","last_peered":"2024-03-11T22:17:20.295743+0000","last_clean":"2024-03-11T22:17:20.295743+0000","last_became_active":"2024-03-11T22:14:52.239649+0000","last_became_peered":"2024-03-11T22:14:52.239649+0000","last_unstale":"2024-03-11T22:17:20.295743+0000","last_undegraded":"2024-03-11T22:17:20.295743+0000","last_fullsized":"2024-03-11T22:17:20.295743+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-11T22:16:27.753301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:16:27.753301+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:32.089452+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":361,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.204282+0000","last_change":"2024-03-11T22:15:42.136659+0000","last_active":"2024-03-11T22:17:11.204282+0000","last_peered":"2024-03-11T22:17:11.204282+0000","last_clean":"2024-03-11T22:17:11.204282+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:17:11.204282+0000","last_undegraded":"2024-03-11T22:17:11.204282+0000","last_fullsized":"2024-03-11T22:17:11.204282+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:15:42.136540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":346,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.313545+0000","last_change":"2024-03-11T22:16:18.291183+0000","last_active":"2024-03-11T22:17:20.313545+0000","last_peered":"2024-03-11T22:17:20.313545+0000","last_clean":"2024-03-11T22:17:20.313545+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:17:20.313545+0000","last_undegraded":"2024-03-11T22:17:20.313545+0000","last_fullsized":"2024-03-11T22:17:20.313545+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:16:18.291063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:16:18.291063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:31.729079+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":350,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.294740+0000","last_change":"2024-03-11T22:17:08.849845+0000","last_active":"2024-03-11T22:17:20.294740+0000","last_peered":"2024-03-11T22:17:20.294740+0000","last_clean":"2024-03-11T22:17:20.294740+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:17:20.294740+0000","last_undegraded":"2024-03-11T22:17:20.294740+0000","last_fullsized":"2024-03-11T22:17:20.294740+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:08.849692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:17:08.849692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:35.070401+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":614,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299391+0000","last_change":"2024-03-11T22:16:15.801127+0000","last_active":"2024-03-11T22:17:20.299391+0000","last_peered":"2024-03-11T22:17:20.299391+0000","last_clean":"2024-03-11T22:17:20.299391+0000","last_became_active":"2024-03-11T22:16:15.800725+0000","last_became_peered":"2024-03-11T22:16:15.800725+0000","last_unstale":"2024-03-11T22:17:20.299391+0000","last_undegraded":"2024-03-11T22:17:20.299391+0000","last_fullsized":"2024-03-11T22:17:20.299391+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:11.673454+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:11.673454+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:24.691150+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"223'11308","reported_seq":11415,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.211186+0000","last_change":"2024-03-11T22:14:52.243433+0000","last_active":"2024-03-11T22:17:21.211186+0000","last_peered":"2024-03-11T22:17:21.211186+0000","last_clean":"2024-03-11T22:17:21.211186+0000","last_became_active":"2024-03-11T22:14:52.243038+0000","last_became_peered":"2024-03-11T22:14:52.243038+0000","last_unstale":"2024-03-11T22:17:21.211186+0000","last_undegraded":"2024-03-11T22:17:21.211186+0000","last_fullsized":"2024-03-11T22:17:21.211186+0000","mapping_epoch":175,"log_start":"206'6824","ondisk_log_start":"206'6824","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267517952,"num_objects":4082,"num_object_clones":0,"num_object_copies":12246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11308,"num_write_kb":492480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":149,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.205273+0000","last_change":"2024-03-11T22:15:22.254990+0000","last_active":"2024-03-11T22:17:11.205273+0000","last_peered":"2024-03-11T22:17:11.205273+0000","last_clean":"2024-03-11T22:17:11.205273+0000","last_became_active":"2024-03-11T22:14:52.240437+0000","last_became_peered":"2024-03-11T22:14:52.240437+0000","last_unstale":"2024-03-11T22:17:11.205273+0000","last_undegraded":"2024-03-11T22:17:11.205273+0000","last_fullsized":"2024-03-11T22:17:11.205273+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:15:22.254869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:22.254869+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":114,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.295042+0000","last_change":"2024-03-11T22:15:41.871554+0000","last_active":"2024-03-11T22:17:20.295042+0000","last_peered":"2024-03-11T22:17:20.295042+0000","last_clean":"2024-03-11T22:17:20.295042+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:17:20.295042+0000","last_undegraded":"2024-03-11T22:17:20.295042+0000","last_fullsized":"2024-03-11T22:17:20.295042+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:41.870596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:41.870596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":338,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.300872+0000","last_change":"2024-03-11T22:14:26.672389+0000","last_active":"2024-03-11T22:17:20.300872+0000","last_peered":"2024-03-11T22:17:20.300872+0000","last_clean":"2024-03-11T22:17:20.300872+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:17:20.300872+0000","last_undegraded":"2024-03-11T22:17:20.300872+0000","last_fullsized":"2024-03-11T22:17:20.300872+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:26.672153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.672153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":326,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.308169+0000","last_change":"2024-03-11T22:13:17.533805+0000","last_active":"2024-03-11T22:17:20.308169+0000","last_peered":"2024-03-11T22:17:20.308169+0000","last_clean":"2024-03-11T22:17:20.308169+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:17:20.308169+0000","last_undegraded":"2024-03-11T22:17:20.308169+0000","last_fullsized":"2024-03-11T22:17:20.308169+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:17.533607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:17.533607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"93'130","reported_seq":643,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.312886+0000","last_change":"2024-03-11T22:16:17.287773+0000","last_active":"2024-03-11T22:17:20.312886+0000","last_peered":"2024-03-11T22:17:20.312886+0000","last_clean":"2024-03-11T22:17:20.312886+0000","last_became_active":"2024-03-11T22:16:15.803198+0000","last_became_peered":"2024-03-11T22:16:15.803198+0000","last_unstale":"2024-03-11T22:17:20.312886+0000","last_undegraded":"2024-03-11T22:17:20.312886+0000","last_fullsized":"2024-03-11T22:17:20.312886+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:16:17.287503+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:16:17.287503+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:32.456308+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3","version":"222'11030","reported_seq":11198,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.616275+0000","last_change":"2024-03-11T22:17:02.663466+0000","last_active":"2024-03-11T22:17:20.616275+0000","last_peered":"2024-03-11T22:17:20.616275+0000","last_clean":"2024-03-11T22:17:20.616275+0000","last_became_active":"2024-03-11T22:14:52.245192+0000","last_became_peered":"2024-03-11T22:14:52.245192+0000","last_unstale":"2024-03-11T22:17:20.616275+0000","last_undegraded":"2024-03-11T22:17:20.616275+0000","last_fullsized":"2024-03-11T22:17:20.616275+0000","mapping_epoch":175,"log_start":"210'7141","ondisk_log_start":"210'7141","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'9910","last_scrub_stamp":"2024-03-11T22:17:02.662660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:17:02.662660+0000","objects_scrubbed":5367,"log_size":3889,"log_dups_size":0,"ondisk_log_size":3889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:17.887320+0000","scrub_duration":2248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274071552,"num_objects":4182,"num_object_clones":0,"num_object_copies":12546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11030,"num_write_kb":486784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":375,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.313598+0000","last_change":"2024-03-11T22:15:31.357450+0000","last_active":"2024-03-11T22:17:20.313598+0000","last_peered":"2024-03-11T22:17:20.313598+0000","last_clean":"2024-03-11T22:17:20.313598+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:17:20.313598+0000","last_undegraded":"2024-03-11T22:17:20.313598+0000","last_fullsized":"2024-03-11T22:17:20.313598+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:15:31.357146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:31.357146+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":353,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.306372+0000","last_change":"2024-03-11T22:15:51.183637+0000","last_active":"2024-03-11T22:17:20.306372+0000","last_peered":"2024-03-11T22:17:20.306372+0000","last_clean":"2024-03-11T22:17:20.306372+0000","last_became_active":"2024-03-11T22:15:51.182962+0000","last_became_peered":"2024-03-11T22:15:51.182962+0000","last_unstale":"2024-03-11T22:17:20.306372+0000","last_undegraded":"2024-03-11T22:17:20.306372+0000","last_fullsized":"2024-03-11T22:17:20.306372+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:38.013058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:38.013058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":169,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299941+0000","last_change":"2024-03-11T22:15:39.663447+0000","last_active":"2024-03-11T22:17:20.299941+0000","last_peered":"2024-03-11T22:17:20.299941+0000","last_clean":"2024-03-11T22:17:20.299941+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:17:20.299941+0000","last_undegraded":"2024-03-11T22:17:20.299941+0000","last_fullsized":"2024-03-11T22:17:20.299941+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:15:39.663324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:39.663324+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":334,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299625+0000","last_change":"2024-03-11T22:15:14.620431+0000","last_active":"2024-03-11T22:17:20.299625+0000","last_peered":"2024-03-11T22:17:20.299625+0000","last_clean":"2024-03-11T22:17:20.299625+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:17:20.299625+0000","last_undegraded":"2024-03-11T22:17:20.299625+0000","last_fullsized":"2024-03-11T22:17:20.299625+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.620337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.620337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":614,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299306+0000","last_change":"2024-03-11T22:16:15.798685+0000","last_active":"2024-03-11T22:17:20.299306+0000","last_peered":"2024-03-11T22:17:20.299306+0000","last_clean":"2024-03-11T22:17:20.299306+0000","last_became_active":"2024-03-11T22:16:15.798293+0000","last_became_peered":"2024-03-11T22:16:15.798293+0000","last_unstale":"2024-03-11T22:17:20.299306+0000","last_undegraded":"2024-03-11T22:17:20.299306+0000","last_fullsized":"2024-03-11T22:17:20.299306+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:12.712874+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:12.712874+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:24.691165+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.c","version":"222'11403","reported_seq":15126,"reported_epoch":223,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T22:17:21.248875+0000","last_change":"2024-03-11T22:16:44.434421+0000","last_active":"2024-03-11T22:17:21.248875+0000","last_peered":"2024-03-11T22:17:21.248875+0000","last_clean":"2024-03-11T22:15:50.311709+0000","last_became_active":"2024-03-11T22:15:52.324400+0000","last_became_peered":"2024-03-11T22:15:52.324400+0000","last_unstale":"2024-03-11T22:17:21.248875+0000","last_undegraded":"2024-03-11T22:16:44.431040+0000","last_fullsized":"2024-03-11T22:15:52.170552+0000","mapping_epoch":194,"log_start":"206'6964","ondisk_log_start":"206'6964","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278986752,"num_objects":4257,"num_object_clones":0,"num_object_copies":12771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1769,"num_objects_misplaced":1490,"num_objects_unfound":0,"num_objects_dirty":4257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11403,"num_write_kb":501120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1806,"num_bytes_recovered":118358016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3","objects":998},{"shards":"0,3,7","objects":1490}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":125,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.305313+0000","last_change":"2024-03-11T22:16:15.819368+0000","last_active":"2024-03-11T22:17:20.305313+0000","last_peered":"2024-03-11T22:17:20.305313+0000","last_clean":"2024-03-11T22:17:20.305313+0000","last_became_active":"2024-03-11T22:16:15.819008+0000","last_became_peered":"2024-03-11T22:16:15.819008+0000","last_unstale":"2024-03-11T22:17:20.305313+0000","last_undegraded":"2024-03-11T22:17:20.305313+0000","last_fullsized":"2024-03-11T22:17:20.305313+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.293977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.293977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":402,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.306467+0000","last_change":"2024-03-11T22:15:51.206019+0000","last_active":"2024-03-11T22:17:20.306467+0000","last_peered":"2024-03-11T22:17:20.306467+0000","last_clean":"2024-03-11T22:17:20.306467+0000","last_became_active":"2024-03-11T22:15:51.183718+0000","last_became_peered":"2024-03-11T22:15:51.183718+0000","last_unstale":"2024-03-11T22:17:20.306467+0000","last_undegraded":"2024-03-11T22:17:20.306467+0000","last_fullsized":"2024-03-11T22:17:20.306467+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:15:33.497732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:33.497732+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":353,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.307085+0000","last_change":"2024-03-11T22:15:51.187210+0000","last_active":"2024-03-11T22:17:20.307085+0000","last_peered":"2024-03-11T22:17:20.307085+0000","last_clean":"2024-03-11T22:17:20.307085+0000","last_became_active":"2024-03-11T22:15:51.186074+0000","last_became_peered":"2024-03-11T22:15:51.186074+0000","last_unstale":"2024-03-11T22:17:20.307085+0000","last_undegraded":"2024-03-11T22:17:20.307085+0000","last_fullsized":"2024-03-11T22:17:20.307085+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:26.481122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:26.481122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":113,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.306060+0000","last_change":"2024-03-11T22:15:12.339167+0000","last_active":"2024-03-11T22:17:20.306060+0000","last_peered":"2024-03-11T22:17:20.306060+0000","last_clean":"2024-03-11T22:17:20.306060+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:17:20.306060+0000","last_undegraded":"2024-03-11T22:17:20.306060+0000","last_fullsized":"2024-03-11T22:17:20.306060+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:12.339050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:12.339050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":567,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.296323+0000","last_change":"2024-03-11T22:16:15.823345+0000","last_active":"2024-03-11T22:17:20.296323+0000","last_peered":"2024-03-11T22:17:20.296323+0000","last_clean":"2024-03-11T22:17:20.296323+0000","last_became_active":"2024-03-11T22:16:15.800213+0000","last_became_peered":"2024-03-11T22:16:15.800213+0000","last_unstale":"2024-03-11T22:17:20.296323+0000","last_undegraded":"2024-03-11T22:17:20.296323+0000","last_fullsized":"2024-03-11T22:17:20.296323+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:16:15.823236+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:16:15.823236+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:39.514612+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.d","version":"222'11447","reported_seq":23508,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.313226+0000","last_change":"2024-03-11T22:16:20.231578+0000","last_active":"2024-03-11T22:17:20.313226+0000","last_peered":"2024-03-11T22:17:20.313226+0000","last_clean":"2024-03-11T22:17:20.313226+0000","last_became_active":"2024-03-11T22:16:20.224100+0000","last_became_peered":"2024-03-11T22:16:20.224100+0000","last_unstale":"2024-03-11T22:17:20.313226+0000","last_undegraded":"2024-03-11T22:17:20.313226+0000","last_fullsized":"2024-03-11T22:17:20.313226+0000","mapping_epoch":207,"log_start":"210'7417","ondisk_log_start":"210'7417","created":175,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4030,"log_dups_size":0,"ondisk_log_size":4030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274464792,"num_objects":4189,"num_object_clones":0,"num_object_copies":12567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11447,"num_write_kb":500289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5970,"num_bytes_recovered":391249920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":453,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.307194+0000","last_change":"2024-03-11T22:15:31.519229+0000","last_active":"2024-03-11T22:17:20.307194+0000","last_peered":"2024-03-11T22:17:20.307194+0000","last_clean":"2024-03-11T22:17:20.307194+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:17:20.307194+0000","last_undegraded":"2024-03-11T22:17:20.307194+0000","last_fullsized":"2024-03-11T22:17:20.307194+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:15:31.513336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:31.513336+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":153,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299480+0000","last_change":"2024-03-11T22:15:36.626072+0000","last_active":"2024-03-11T22:17:20.299480+0000","last_peered":"2024-03-11T22:17:20.299480+0000","last_clean":"2024-03-11T22:17:20.299480+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:17:20.299480+0000","last_undegraded":"2024-03-11T22:17:20.299480+0000","last_fullsized":"2024-03-11T22:17:20.299480+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:15:36.625916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:36.625916+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":350,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.213180+0000","last_change":"2024-03-11T22:15:08.820614+0000","last_active":"2024-03-11T22:17:21.213180+0000","last_peered":"2024-03-11T22:17:21.213180+0000","last_clean":"2024-03-11T22:17:21.213180+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:17:21.213180+0000","last_undegraded":"2024-03-11T22:17:21.213180+0000","last_fullsized":"2024-03-11T22:17:21.213180+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:15:08.820296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:08.820296+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":291,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.205299+0000","last_change":"2024-03-11T22:17:06.233833+0000","last_active":"2024-03-11T22:17:11.205299+0000","last_peered":"2024-03-11T22:17:11.205299+0000","last_clean":"2024-03-11T22:17:11.205299+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:17:11.205299+0000","last_undegraded":"2024-03-11T22:17:11.205299+0000","last_fullsized":"2024-03-11T22:17:11.205299+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:06.233710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:17:06.233710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:12.125843+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":631,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.308881+0000","last_change":"2024-03-11T22:15:54.193467+0000","last_active":"2024-03-11T22:17:20.308881+0000","last_peered":"2024-03-11T22:17:20.308881+0000","last_clean":"2024-03-11T22:17:20.308881+0000","last_became_active":"2024-03-11T22:15:54.193112+0000","last_became_peered":"2024-03-11T22:15:54.193112+0000","last_unstale":"2024-03-11T22:17:20.308881+0000","last_undegraded":"2024-03-11T22:17:20.308881+0000","last_fullsized":"2024-03-11T22:17:20.308881+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.243267+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.243267+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"223'11265","reported_seq":11372,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.239392+0000","last_change":"2024-03-11T22:14:52.242543+0000","last_active":"2024-03-11T22:17:21.239392+0000","last_peered":"2024-03-11T22:17:21.239392+0000","last_clean":"2024-03-11T22:17:21.239392+0000","last_became_active":"2024-03-11T22:14:52.240224+0000","last_became_peered":"2024-03-11T22:14:52.240224+0000","last_unstale":"2024-03-11T22:17:21.239392+0000","last_undegraded":"2024-03-11T22:17:21.239392+0000","last_fullsized":"2024-03-11T22:17:21.239392+0000","mapping_epoch":175,"log_start":"206'6821","ondisk_log_start":"206'6821","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4444,"log_dups_size":0,"ondisk_log_size":4444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271646720,"num_objects":4145,"num_object_clones":0,"num_object_copies":12435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11265,"num_write_kb":493120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":455,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.208113+0000","last_change":"2024-03-11T22:14:27.105480+0000","last_active":"2024-03-11T22:17:21.208113+0000","last_peered":"2024-03-11T22:17:21.208113+0000","last_clean":"2024-03-11T22:17:21.208113+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:17:21.208113+0000","last_undegraded":"2024-03-11T22:17:21.208113+0000","last_fullsized":"2024-03-11T22:17:21.208113+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:14:27.105394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:27.105394+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":366,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.312653+0000","last_change":"2024-03-11T22:15:33.327216+0000","last_active":"2024-03-11T22:17:20.312653+0000","last_peered":"2024-03-11T22:17:20.312653+0000","last_clean":"2024-03-11T22:17:20.312653+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:17:20.312653+0000","last_undegraded":"2024-03-11T22:17:20.312653+0000","last_fullsized":"2024-03-11T22:17:20.312653+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:15:33.327074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:33.327074+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":355,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299895+0000","last_change":"2024-03-11T22:15:24.618272+0000","last_active":"2024-03-11T22:17:20.299895+0000","last_peered":"2024-03-11T22:17:20.299895+0000","last_clean":"2024-03-11T22:17:20.299895+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:17:20.299895+0000","last_undegraded":"2024-03-11T22:17:20.299895+0000","last_fullsized":"2024-03-11T22:17:20.299895+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:24.618157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:24.618157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":337,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.208030+0000","last_change":"2024-03-11T22:15:26.900176+0000","last_active":"2024-03-11T22:17:21.208030+0000","last_peered":"2024-03-11T22:17:21.208030+0000","last_clean":"2024-03-11T22:17:21.208030+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:17:21.208030+0000","last_undegraded":"2024-03-11T22:17:21.208030+0000","last_fullsized":"2024-03-11T22:17:21.208030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:26.899954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:26.899954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":530,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.206638+0000","last_change":"2024-03-11T22:16:24.042466+0000","last_active":"2024-03-11T22:17:11.206638+0000","last_peered":"2024-03-11T22:17:11.206638+0000","last_clean":"2024-03-11T22:17:11.206638+0000","last_became_active":"2024-03-11T22:16:15.809129+0000","last_became_peered":"2024-03-11T22:16:15.809129+0000","last_unstale":"2024-03-11T22:17:11.206638+0000","last_undegraded":"2024-03-11T22:17:11.206638+0000","last_fullsized":"2024-03-11T22:17:11.206638+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:16:24.042352+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:16:24.042352+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:44.877464+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.f","version":"222'11422","reported_seq":11527,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.827213+0000","last_change":"2024-03-11T22:14:52.238556+0000","last_active":"2024-03-11T22:17:20.827213+0000","last_peered":"2024-03-11T22:17:20.827213+0000","last_clean":"2024-03-11T22:17:20.827213+0000","last_became_active":"2024-03-11T22:14:52.238241+0000","last_became_peered":"2024-03-11T22:14:52.238241+0000","last_unstale":"2024-03-11T22:17:20.827213+0000","last_undegraded":"2024-03-11T22:17:20.827213+0000","last_fullsized":"2024-03-11T22:17:20.827213+0000","mapping_epoch":175,"log_start":"211'7583","ondisk_log_start":"211'7583","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3839,"log_dups_size":0,"ondisk_log_size":3839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264896512,"num_objects":4042,"num_object_clones":0,"num_object_copies":12126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":494848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":353,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.204178+0000","last_change":"2024-03-11T22:16:55.173139+0000","last_active":"2024-03-11T22:17:11.204178+0000","last_peered":"2024-03-11T22:17:11.204178+0000","last_clean":"2024-03-11T22:17:11.204178+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:17:11.204178+0000","last_undegraded":"2024-03-11T22:17:11.204178+0000","last_fullsized":"2024-03-11T22:17:11.204178+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:16:55.173033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:16:55.173033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:14.487034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":381,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.207790+0000","last_change":"2024-03-11T22:15:22.846514+0000","last_active":"2024-03-11T22:17:21.207790+0000","last_peered":"2024-03-11T22:17:21.207790+0000","last_clean":"2024-03-11T22:17:21.207790+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:17:21.207790+0000","last_undegraded":"2024-03-11T22:17:21.207790+0000","last_fullsized":"2024-03-11T22:17:21.207790+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:15:22.846352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:22.846352+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":355,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.313363+0000","last_change":"2024-03-11T22:16:13.310674+0000","last_active":"2024-03-11T22:17:20.313363+0000","last_peered":"2024-03-11T22:17:20.313363+0000","last_clean":"2024-03-11T22:17:20.313363+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:17:20.313363+0000","last_undegraded":"2024-03-11T22:17:20.313363+0000","last_fullsized":"2024-03-11T22:17:20.313363+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:16:13.310557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:16:13.310557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":343,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.312962+0000","last_change":"2024-03-11T22:15:21.454660+0000","last_active":"2024-03-11T22:17:20.312962+0000","last_peered":"2024-03-11T22:17:20.312962+0000","last_clean":"2024-03-11T22:17:20.312962+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:17:20.312962+0000","last_undegraded":"2024-03-11T22:17:20.312962+0000","last_fullsized":"2024-03-11T22:17:20.312962+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:21.454272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:21.454272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":521,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.205378+0000","last_change":"2024-03-11T22:16:23.065850+0000","last_active":"2024-03-11T22:17:11.205378+0000","last_peered":"2024-03-11T22:17:11.205378+0000","last_clean":"2024-03-11T22:17:11.205378+0000","last_became_active":"2024-03-11T22:16:15.810945+0000","last_became_peered":"2024-03-11T22:16:15.810945+0000","last_unstale":"2024-03-11T22:17:11.205378+0000","last_undegraded":"2024-03-11T22:17:11.205378+0000","last_fullsized":"2024-03-11T22:17:11.205378+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:16:23.065734+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:16:23.065734+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:35.194525+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"222'11488","reported_seq":11672,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.645921+0000","last_change":"2024-03-11T22:16:22.575339+0000","last_active":"2024-03-11T22:17:20.645921+0000","last_peered":"2024-03-11T22:17:20.645921+0000","last_clean":"2024-03-11T22:17:20.645921+0000","last_became_active":"2024-03-11T22:14:52.246614+0000","last_became_peered":"2024-03-11T22:14:52.246614+0000","last_unstale":"2024-03-11T22:17:20.645921+0000","last_undegraded":"2024-03-11T22:17:20.645921+0000","last_fullsized":"2024-03-11T22:17:20.645921+0000","mapping_epoch":175,"log_start":"211'7639","ondisk_log_start":"211'7639","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'7188","last_scrub_stamp":"2024-03-11T22:16:22.574914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:16:22.574914+0000","objects_scrubbed":7123,"log_size":3849,"log_dups_size":0,"ondisk_log_size":3849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:43.278607+0000","scrub_duration":2249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276561920,"num_objects":4220,"num_object_clones":0,"num_object_copies":12660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11488,"num_write_kb":502656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":339,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.301927+0000","last_change":"2024-03-11T22:14:29.697272+0000","last_active":"2024-03-11T22:17:20.301927+0000","last_peered":"2024-03-11T22:17:20.301927+0000","last_clean":"2024-03-11T22:17:20.301927+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:17:20.301927+0000","last_undegraded":"2024-03-11T22:17:20.301927+0000","last_fullsized":"2024-03-11T22:17:20.301927+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:29.697144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:29.697144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":371,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.212934+0000","last_change":"2024-03-11T22:15:45.150431+0000","last_active":"2024-03-11T22:17:21.212934+0000","last_peered":"2024-03-11T22:17:21.212934+0000","last_clean":"2024-03-11T22:17:21.212934+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:17:21.212934+0000","last_undegraded":"2024-03-11T22:17:21.212934+0000","last_fullsized":"2024-03-11T22:17:21.212934+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:15:45.150297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:45.150297+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":864,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.312568+0000","last_change":"2024-03-11T22:15:38.304410+0000","last_active":"2024-03-11T22:17:20.312568+0000","last_peered":"2024-03-11T22:17:20.312568+0000","last_clean":"2024-03-11T22:17:20.312568+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:17:20.312568+0000","last_undegraded":"2024-03-11T22:17:20.312568+0000","last_fullsized":"2024-03-11T22:17:20.312568+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:15:38.304286+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:15:38.304286+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":494,"num_read_kb":489,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":349,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.228402+0000","last_change":"2024-03-11T22:14:35.025497+0000","last_active":"2024-03-11T22:17:21.228402+0000","last_peered":"2024-03-11T22:17:21.228402+0000","last_clean":"2024-03-11T22:17:21.228402+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:17:21.228402+0000","last_undegraded":"2024-03-11T22:17:21.228402+0000","last_fullsized":"2024-03-11T22:17:21.228402+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:35.025383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:35.025383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":351,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.313150+0000","last_change":"2024-03-11T22:15:44.401163+0000","last_active":"2024-03-11T22:17:20.313150+0000","last_peered":"2024-03-11T22:17:20.313150+0000","last_clean":"2024-03-11T22:17:20.313150+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:17:20.313150+0000","last_undegraded":"2024-03-11T22:17:20.313150+0000","last_fullsized":"2024-03-11T22:17:20.313150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:44.401045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:44.401045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":630,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.313095+0000","last_change":"2024-03-11T22:17:15.606781+0000","last_active":"2024-03-11T22:17:20.313095+0000","last_peered":"2024-03-11T22:17:20.313095+0000","last_clean":"2024-03-11T22:17:20.313095+0000","last_became_active":"2024-03-11T22:14:38.164101+0000","last_became_peered":"2024-03-11T22:14:38.164101+0000","last_unstale":"2024-03-11T22:17:20.313095+0000","last_undegraded":"2024-03-11T22:17:20.313095+0000","last_fullsized":"2024-03-11T22:17:20.313095+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:17:15.603876+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:17:15.603876+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:43.017145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"221'11123","reported_seq":11282,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.228707+0000","last_change":"2024-03-11T22:17:12.413518+0000","last_active":"2024-03-11T22:17:20.228707+0000","last_peered":"2024-03-11T22:17:20.228707+0000","last_clean":"2024-03-11T22:17:20.228707+0000","last_became_active":"2024-03-11T22:14:52.240210+0000","last_became_peered":"2024-03-11T22:14:52.240210+0000","last_unstale":"2024-03-11T22:17:20.228707+0000","last_undegraded":"2024-03-11T22:17:20.228707+0000","last_fullsized":"2024-03-11T22:17:20.228707+0000","mapping_epoch":175,"log_start":"203'6615","ondisk_log_start":"203'6615","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'10565","last_scrub_stamp":"2024-03-11T22:17:12.413223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:17:12.413223+0000","objects_scrubbed":4772,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:36.605418+0000","scrub_duration":1249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273350656,"num_objects":4171,"num_object_clones":0,"num_object_copies":12513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11123,"num_write_kb":489408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":299,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.297162+0000","last_change":"2024-03-11T22:15:51.185575+0000","last_active":"2024-03-11T22:17:20.297162+0000","last_peered":"2024-03-11T22:17:20.297162+0000","last_clean":"2024-03-11T22:17:20.297162+0000","last_became_active":"2024-03-11T22:15:51.185218+0000","last_became_peered":"2024-03-11T22:15:51.185218+0000","last_unstale":"2024-03-11T22:17:20.297162+0000","last_undegraded":"2024-03-11T22:17:20.297162+0000","last_fullsized":"2024-03-11T22:17:20.297162+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:30.829461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:15:30.829461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":485,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:13.449909+0000","last_change":"2024-03-11T22:17:01.129168+0000","last_active":"2024-03-11T22:17:13.449909+0000","last_peered":"2024-03-11T22:17:13.449909+0000","last_clean":"2024-03-11T22:17:13.449909+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:17:13.449909+0000","last_undegraded":"2024-03-11T22:17:13.449909+0000","last_fullsized":"2024-03-11T22:17:13.449909+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-11T22:17:01.129013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:17:01.129013+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:08.001948+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":347,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.304988+0000","last_change":"2024-03-11T22:13:33.460800+0000","last_active":"2024-03-11T22:17:20.304988+0000","last_peered":"2024-03-11T22:17:20.304988+0000","last_clean":"2024-03-11T22:17:20.304988+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:17:20.304988+0000","last_undegraded":"2024-03-11T22:17:20.304988+0000","last_fullsized":"2024-03-11T22:17:20.304988+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:33.460308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:13:33.460308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":633,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.208121+0000","last_change":"2024-03-11T22:15:54.203009+0000","last_active":"2024-03-11T22:17:21.208121+0000","last_peered":"2024-03-11T22:17:21.208121+0000","last_clean":"2024-03-11T22:17:21.208121+0000","last_became_active":"2024-03-11T22:15:54.199718+0000","last_became_peered":"2024-03-11T22:15:54.199718+0000","last_unstale":"2024-03-11T22:17:21.208121+0000","last_undegraded":"2024-03-11T22:17:21.208121+0000","last_fullsized":"2024-03-11T22:17:21.208121+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:03.739634+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:15:03.739634+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":335,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.300756+0000","last_change":"2024-03-11T22:13:34.478753+0000","last_active":"2024-03-11T22:17:20.300756+0000","last_peered":"2024-03-11T22:17:20.300756+0000","last_clean":"2024-03-11T22:17:20.300756+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:17:20.300756+0000","last_undegraded":"2024-03-11T22:17:20.300756+0000","last_fullsized":"2024-03-11T22:17:20.300756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:13:34.478634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:13:34.478634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"222'11043","reported_seq":11148,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.842376+0000","last_change":"2024-03-11T22:14:52.240355+0000","last_active":"2024-03-11T22:17:20.842376+0000","last_peered":"2024-03-11T22:17:20.842376+0000","last_clean":"2024-03-11T22:17:20.842376+0000","last_became_active":"2024-03-11T22:14:52.238824+0000","last_became_peered":"2024-03-11T22:14:52.238824+0000","last_unstale":"2024-03-11T22:17:20.842376+0000","last_undegraded":"2024-03-11T22:17:20.842376+0000","last_fullsized":"2024-03-11T22:17:20.842376+0000","mapping_epoch":175,"log_start":"210'7169","ondisk_log_start":"210'7169","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":3874,"log_dups_size":0,"ondisk_log_size":3874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271646720,"num_objects":4145,"num_object_clones":0,"num_object_copies":12435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11043,"num_write_kb":486016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":355,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.295207+0000","last_change":"2024-03-11T22:15:44.873768+0000","last_active":"2024-03-11T22:17:20.295207+0000","last_peered":"2024-03-11T22:17:20.295207+0000","last_clean":"2024-03-11T22:17:20.295207+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:17:20.295207+0000","last_undegraded":"2024-03-11T22:17:20.295207+0000","last_fullsized":"2024-03-11T22:17:20.295207+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:44.873386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:44.873386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":386,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.313747+0000","last_change":"2024-03-11T22:17:09.617759+0000","last_active":"2024-03-11T22:17:20.313747+0000","last_peered":"2024-03-11T22:17:20.313747+0000","last_clean":"2024-03-11T22:17:20.313747+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:17:20.313747+0000","last_undegraded":"2024-03-11T22:17:20.313747+0000","last_fullsized":"2024-03-11T22:17:20.313747+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'13","last_scrub_stamp":"2024-03-11T22:17:09.617517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:17:09.617517+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:28.838181+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":380,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.295188+0000","last_change":"2024-03-11T22:16:26.732561+0000","last_active":"2024-03-11T22:17:20.295188+0000","last_peered":"2024-03-11T22:17:20.295188+0000","last_clean":"2024-03-11T22:17:20.295188+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:17:20.295188+0000","last_undegraded":"2024-03-11T22:17:20.295188+0000","last_fullsized":"2024-03-11T22:17:20.295188+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:16:26.732291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:16:26.732291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:32.316601+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":363,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.312717+0000","last_change":"2024-03-11T22:17:17.685400+0000","last_active":"2024-03-11T22:17:20.312717+0000","last_peered":"2024-03-11T22:17:20.312717+0000","last_clean":"2024-03-11T22:17:20.312717+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:17:20.312717+0000","last_undegraded":"2024-03-11T22:17:20.312717+0000","last_fullsized":"2024-03-11T22:17:20.312717+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:17.685173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:17:17.685173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:39.627634+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":291,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.205378+0000","last_change":"2024-03-11T22:17:07.216002+0000","last_active":"2024-03-11T22:17:11.205378+0000","last_peered":"2024-03-11T22:17:11.205378+0000","last_clean":"2024-03-11T22:17:11.205378+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:17:11.205378+0000","last_undegraded":"2024-03-11T22:17:11.205378+0000","last_fullsized":"2024-03-11T22:17:11.205378+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:07.215787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:17:07.215787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:13.875132+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":640,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.211982+0000","last_change":"2024-03-11T22:15:54.187363+0000","last_active":"2024-03-11T22:17:21.211982+0000","last_peered":"2024-03-11T22:17:21.211982+0000","last_clean":"2024-03-11T22:17:21.211982+0000","last_became_active":"2024-03-11T22:15:54.186991+0000","last_became_peered":"2024-03-11T22:15:54.186991+0000","last_unstale":"2024-03-11T22:17:21.211982+0000","last_undegraded":"2024-03-11T22:17:21.211982+0000","last_fullsized":"2024-03-11T22:17:21.211982+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:04.770858+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:04.770858+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"222'11461","reported_seq":11566,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299516+0000","last_change":"2024-03-11T22:14:52.243442+0000","last_active":"2024-03-11T22:17:20.299516+0000","last_peered":"2024-03-11T22:17:20.299516+0000","last_clean":"2024-03-11T22:17:20.299516+0000","last_became_active":"2024-03-11T22:14:52.243066+0000","last_became_peered":"2024-03-11T22:14:52.243066+0000","last_unstale":"2024-03-11T22:17:20.299516+0000","last_undegraded":"2024-03-11T22:17:20.299516+0000","last_fullsized":"2024-03-11T22:17:20.299516+0000","mapping_epoch":175,"log_start":"210'7449","ondisk_log_start":"210'7449","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4012,"log_dups_size":0,"ondisk_log_size":4012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275447808,"num_objects":4203,"num_object_clones":0,"num_object_copies":12609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":501248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":349,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.210924+0000","last_change":"2024-03-11T22:15:30.922162+0000","last_active":"2024-03-11T22:17:21.210924+0000","last_peered":"2024-03-11T22:17:21.210924+0000","last_clean":"2024-03-11T22:17:21.210924+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:17:21.210924+0000","last_undegraded":"2024-03-11T22:17:21.210924+0000","last_fullsized":"2024-03-11T22:17:21.210924+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:30.922001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:30.922001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":505,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.313449+0000","last_change":"2024-03-11T22:15:11.547339+0000","last_active":"2024-03-11T22:17:20.313449+0000","last_peered":"2024-03-11T22:17:20.313449+0000","last_clean":"2024-03-11T22:17:20.313449+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:17:20.313449+0000","last_undegraded":"2024-03-11T22:17:20.313449+0000","last_fullsized":"2024-03-11T22:17:20.313449+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-11T22:15:11.547182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:11.547182+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":116,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.207702+0000","last_change":"2024-03-11T22:15:24.899285+0000","last_active":"2024-03-11T22:17:21.207702+0000","last_peered":"2024-03-11T22:17:21.207702+0000","last_clean":"2024-03-11T22:17:21.207702+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:17:21.207702+0000","last_undegraded":"2024-03-11T22:17:21.207702+0000","last_fullsized":"2024-03-11T22:17:21.207702+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:24.899168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:24.899168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":359,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299564+0000","last_change":"2024-03-11T22:14:16.662735+0000","last_active":"2024-03-11T22:17:20.299564+0000","last_peered":"2024-03-11T22:17:20.299564+0000","last_clean":"2024-03-11T22:17:20.299564+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:17:20.299564+0000","last_undegraded":"2024-03-11T22:17:20.299564+0000","last_fullsized":"2024-03-11T22:17:20.299564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:14:16.662596+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:14:16.662596+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":341,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.307052+0000","last_change":"2024-03-11T22:15:51.192142+0000","last_active":"2024-03-11T22:17:20.307052+0000","last_peered":"2024-03-11T22:17:20.307052+0000","last_clean":"2024-03-11T22:17:20.307052+0000","last_became_active":"2024-03-11T22:15:51.191481+0000","last_became_peered":"2024-03-11T22:15:51.191481+0000","last_unstale":"2024-03-11T22:17:20.307052+0000","last_undegraded":"2024-03-11T22:17:20.307052+0000","last_fullsized":"2024-03-11T22:17:20.307052+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:32.518054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:15:32.518054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":621,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299371+0000","last_change":"2024-03-11T22:15:06.673117+0000","last_active":"2024-03-11T22:17:20.299371+0000","last_peered":"2024-03-11T22:17:20.299371+0000","last_clean":"2024-03-11T22:17:20.299371+0000","last_became_active":"2024-03-11T22:14:38.164859+0000","last_became_peered":"2024-03-11T22:14:38.164859+0000","last_unstale":"2024-03-11T22:17:20.299371+0000","last_undegraded":"2024-03-11T22:17:20.299371+0000","last_fullsized":"2024-03-11T22:17:20.299371+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.672938+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.672938+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"222'11302","reported_seq":11462,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.625825+0000","last_change":"2024-03-11T22:17:12.534954+0000","last_active":"2024-03-11T22:17:20.625825+0000","last_peered":"2024-03-11T22:17:20.625825+0000","last_clean":"2024-03-11T22:17:20.625825+0000","last_became_active":"2024-03-11T22:14:52.242455+0000","last_became_peered":"2024-03-11T22:14:52.242455+0000","last_unstale":"2024-03-11T22:17:20.625825+0000","last_undegraded":"2024-03-11T22:17:20.625825+0000","last_fullsized":"2024-03-11T22:17:20.625825+0000","mapping_epoch":175,"log_start":"210'7360","ondisk_log_start":"210'7360","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'10700","last_scrub_stamp":"2024-03-11T22:17:12.534657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:17:12.534657+0000","objects_scrubbed":4923,"log_size":3942,"log_dups_size":0,"ondisk_log_size":3942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:30.556659+0000","scrub_duration":1914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279445504,"num_objects":4264,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11302,"num_write_kb":498112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":376,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.208021+0000","last_change":"2024-03-11T22:15:11.902221+0000","last_active":"2024-03-11T22:17:21.208021+0000","last_peered":"2024-03-11T22:17:21.208021+0000","last_clean":"2024-03-11T22:17:21.208021+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:17:21.208021+0000","last_undegraded":"2024-03-11T22:17:21.208021+0000","last_fullsized":"2024-03-11T22:17:21.208021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:15:11.902080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:11.902080+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":346,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.300142+0000","last_change":"2024-03-11T22:14:37.615454+0000","last_active":"2024-03-11T22:17:20.300142+0000","last_peered":"2024-03-11T22:17:20.300142+0000","last_clean":"2024-03-11T22:17:20.300142+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:17:20.300142+0000","last_undegraded":"2024-03-11T22:17:20.300142+0000","last_fullsized":"2024-03-11T22:17:20.300142+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:37.615375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:37.615375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":355,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.294375+0000","last_change":"2024-03-11T22:17:03.730273+0000","last_active":"2024-03-11T22:17:20.294375+0000","last_peered":"2024-03-11T22:17:20.294375+0000","last_clean":"2024-03-11T22:17:20.294375+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:17:20.294375+0000","last_undegraded":"2024-03-11T22:17:20.294375+0000","last_fullsized":"2024-03-11T22:17:20.294375+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:03.730140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:17:03.730140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:06.300746+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":343,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.228685+0000","last_change":"2024-03-11T22:15:51.185584+0000","last_active":"2024-03-11T22:17:21.228685+0000","last_peered":"2024-03-11T22:17:21.228685+0000","last_clean":"2024-03-11T22:17:21.228685+0000","last_became_active":"2024-03-11T22:15:51.184473+0000","last_became_peered":"2024-03-11T22:15:51.184473+0000","last_unstale":"2024-03-11T22:17:21.228685+0000","last_undegraded":"2024-03-11T22:17:21.228685+0000","last_fullsized":"2024-03-11T22:17:21.228685+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:36.007981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:36.007981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"93'130","reported_seq":451,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.305380+0000","last_change":"2024-03-11T22:16:15.801234+0000","last_active":"2024-03-11T22:17:20.305380+0000","last_peered":"2024-03-11T22:17:20.305380+0000","last_clean":"2024-03-11T22:17:20.305380+0000","last_became_active":"2024-03-11T22:16:15.800914+0000","last_became_peered":"2024-03-11T22:16:15.800914+0000","last_unstale":"2024-03-11T22:17:20.305380+0000","last_undegraded":"2024-03-11T22:17:20.305380+0000","last_fullsized":"2024-03-11T22:17:20.305380+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:09.205116+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:09.205116+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:23.676248+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"223'11602","reported_seq":11709,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.259163+0000","last_change":"2024-03-11T22:14:52.243118+0000","last_active":"2024-03-11T22:17:21.259163+0000","last_peered":"2024-03-11T22:17:21.259163+0000","last_clean":"2024-03-11T22:17:21.259163+0000","last_became_active":"2024-03-11T22:14:52.242807+0000","last_became_peered":"2024-03-11T22:14:52.242807+0000","last_unstale":"2024-03-11T22:17:21.259163+0000","last_undegraded":"2024-03-11T22:17:21.259163+0000","last_fullsized":"2024-03-11T22:17:21.259163+0000","mapping_epoch":175,"log_start":"209'7180","ondisk_log_start":"209'7180","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4422,"log_dups_size":0,"ondisk_log_size":4422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270270464,"num_objects":4124,"num_object_clones":0,"num_object_copies":12372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11602,"num_write_kb":503232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":414,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.204195+0000","last_change":"2024-03-11T22:17:04.185636+0000","last_active":"2024-03-11T22:17:11.204195+0000","last_peered":"2024-03-11T22:17:11.204195+0000","last_clean":"2024-03-11T22:17:11.204195+0000","last_became_active":"2024-03-11T22:15:51.190082+0000","last_became_peered":"2024-03-11T22:15:51.190082+0000","last_unstale":"2024-03-11T22:17:11.204195+0000","last_undegraded":"2024-03-11T22:17:11.204195+0000","last_fullsized":"2024-03-11T22:17:11.204195+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-11T22:17:04.185469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:17:04.185469+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:27.803690+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":292,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.300422+0000","last_change":"2024-03-11T22:15:51.186287+0000","last_active":"2024-03-11T22:17:20.300422+0000","last_peered":"2024-03-11T22:17:20.300422+0000","last_clean":"2024-03-11T22:17:20.300422+0000","last_became_active":"2024-03-11T22:15:51.185946+0000","last_became_peered":"2024-03-11T22:15:51.185946+0000","last_unstale":"2024-03-11T22:17:20.300422+0000","last_undegraded":"2024-03-11T22:17:20.300422+0000","last_fullsized":"2024-03-11T22:17:20.300422+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.649172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.649172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":354,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.294317+0000","last_change":"2024-03-11T22:16:57.628174+0000","last_active":"2024-03-11T22:17:20.294317+0000","last_peered":"2024-03-11T22:17:20.294317+0000","last_clean":"2024-03-11T22:17:20.294317+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:17:20.294317+0000","last_undegraded":"2024-03-11T22:17:20.294317+0000","last_fullsized":"2024-03-11T22:17:20.294317+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:16:57.628059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.710319+0000","last_clean_scrub_stamp":"2024-03-11T22:16:57.628059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:06.507290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":299,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.294109+0000","last_change":"2024-03-11T22:15:51.186904+0000","last_active":"2024-03-11T22:17:20.294109+0000","last_peered":"2024-03-11T22:17:20.294109+0000","last_clean":"2024-03-11T22:17:20.294109+0000","last_became_active":"2024-03-11T22:15:51.186315+0000","last_became_peered":"2024-03-11T22:15:51.186315+0000","last_unstale":"2024-03-11T22:17:20.294109+0000","last_undegraded":"2024-03-11T22:17:20.294109+0000","last_fullsized":"2024-03-11T22:17:20.294109+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:37.928924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:37.928924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"93'130","reported_seq":524,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:15.165918+0000","last_change":"2024-03-11T22:17:15.165999+0000","last_active":"2024-03-11T22:17:15.165918+0000","last_peered":"2024-03-11T22:17:15.165918+0000","last_clean":"2024-03-11T22:17:15.165918+0000","last_became_active":"2024-03-11T22:14:43.195433+0000","last_became_peered":"2024-03-11T22:14:43.195433+0000","last_unstale":"2024-03-11T22:17:15.165918+0000","last_undegraded":"2024-03-11T22:17:15.165918+0000","last_fullsized":"2024-03-11T22:17:15.165918+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:17:15.165750+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:17:15.165750+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:28.337433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"221'11244","reported_seq":11403,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.214052+0000","last_change":"2024-03-11T22:17:13.297486+0000","last_active":"2024-03-11T22:17:20.214052+0000","last_peered":"2024-03-11T22:17:20.214052+0000","last_clean":"2024-03-11T22:17:20.214052+0000","last_became_active":"2024-03-11T22:14:52.240614+0000","last_became_peered":"2024-03-11T22:14:52.240614+0000","last_unstale":"2024-03-11T22:17:20.214052+0000","last_undegraded":"2024-03-11T22:17:20.214052+0000","last_fullsized":"2024-03-11T22:17:20.214052+0000","mapping_epoch":175,"log_start":"205'6772","ondisk_log_start":"205'6772","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'10706","last_scrub_stamp":"2024-03-11T22:17:13.297308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:17:13.297308+0000","objects_scrubbed":4816,"log_size":4472,"log_dups_size":0,"ondisk_log_size":4472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:39.477873+0000","scrub_duration":1129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276955136,"num_objects":4226,"num_object_clones":0,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11244,"num_write_kb":495040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":355,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.293924+0000","last_change":"2024-03-11T22:16:21.725946+0000","last_active":"2024-03-11T22:17:20.293924+0000","last_peered":"2024-03-11T22:17:20.293924+0000","last_clean":"2024-03-11T22:17:20.293924+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:17:20.293924+0000","last_undegraded":"2024-03-11T22:17:20.293924+0000","last_fullsized":"2024-03-11T22:17:20.293924+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:16:21.725824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:16:21.725824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:44.948740+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":162,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299666+0000","last_change":"2024-03-11T22:15:11.678247+0000","last_active":"2024-03-11T22:17:20.299666+0000","last_peered":"2024-03-11T22:17:20.299666+0000","last_clean":"2024-03-11T22:17:20.299666+0000","last_became_active":"2024-03-11T22:14:52.247400+0000","last_became_peered":"2024-03-11T22:14:52.247400+0000","last_unstale":"2024-03-11T22:17:20.299666+0000","last_undegraded":"2024-03-11T22:17:20.299666+0000","last_fullsized":"2024-03-11T22:17:20.299666+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:15:11.678114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:11.678114+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":123,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.314200+0000","last_change":"2024-03-11T22:15:51.187301+0000","last_active":"2024-03-11T22:17:20.314200+0000","last_peered":"2024-03-11T22:17:20.314200+0000","last_clean":"2024-03-11T22:17:20.314200+0000","last_became_active":"2024-03-11T22:15:51.186974+0000","last_became_peered":"2024-03-11T22:15:51.186974+0000","last_unstale":"2024-03-11T22:17:20.314200+0000","last_undegraded":"2024-03-11T22:17:20.314200+0000","last_fullsized":"2024-03-11T22:17:20.314200+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:08.477912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:08.477912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":346,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.307499+0000","last_change":"2024-03-11T22:15:51.203126+0000","last_active":"2024-03-11T22:17:20.307499+0000","last_peered":"2024-03-11T22:17:20.307499+0000","last_clean":"2024-03-11T22:17:20.307499+0000","last_became_active":"2024-03-11T22:15:51.191693+0000","last_became_peered":"2024-03-11T22:15:51.191693+0000","last_unstale":"2024-03-11T22:17:20.307499+0000","last_undegraded":"2024-03-11T22:17:20.307499+0000","last_fullsized":"2024-03-11T22:17:20.307499+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:15:40.678629+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:15:40.678629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"93'130","reported_seq":642,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.306482+0000","last_change":"2024-03-11T22:16:15.819440+0000","last_active":"2024-03-11T22:17:20.306482+0000","last_peered":"2024-03-11T22:17:20.306482+0000","last_clean":"2024-03-11T22:17:20.306482+0000","last_became_active":"2024-03-11T22:16:15.816005+0000","last_became_peered":"2024-03-11T22:16:15.816005+0000","last_unstale":"2024-03-11T22:17:20.306482+0000","last_undegraded":"2024-03-11T22:17:20.306482+0000","last_fullsized":"2024-03-11T22:17:20.306482+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:05.248940+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.248940+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:23.676202+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"222'11524","reported_seq":11629,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.323269+0000","last_change":"2024-03-11T22:14:52.249598+0000","last_active":"2024-03-11T22:17:20.323269+0000","last_peered":"2024-03-11T22:17:20.323269+0000","last_clean":"2024-03-11T22:17:20.323269+0000","last_became_active":"2024-03-11T22:14:52.248793+0000","last_became_peered":"2024-03-11T22:14:52.248793+0000","last_unstale":"2024-03-11T22:17:20.323269+0000","last_undegraded":"2024-03-11T22:17:20.323269+0000","last_fullsized":"2024-03-11T22:17:20.323269+0000","mapping_epoch":175,"log_start":"210'7501","ondisk_log_start":"210'7501","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:51.211031+0000","last_clean_scrub_stamp":"2024-03-11T22:14:51.211031+0000","objects_scrubbed":0,"log_size":4023,"log_dups_size":0,"ondisk_log_size":4023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286785536,"num_objects":4376,"num_object_clones":0,"num_object_copies":13128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11524,"num_write_kb":508800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"194'10","reported_seq":409,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:13.449693+0000","last_change":"2024-03-11T22:16:29.101786+0000","last_active":"2024-03-11T22:17:13.449693+0000","last_peered":"2024-03-11T22:17:13.449693+0000","last_clean":"2024-03-11T22:17:13.449693+0000","last_became_active":"2024-03-11T22:15:51.190539+0000","last_became_peered":"2024-03-11T22:15:51.190539+0000","last_unstale":"2024-03-11T22:17:13.449693+0000","last_undegraded":"2024-03-11T22:17:13.449693+0000","last_fullsized":"2024-03-11T22:17:13.449693+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'10","last_scrub_stamp":"2024-03-11T22:16:29.101689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:16:29.101689+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:45.775174+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"91'22","reported_seq":405,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299415+0000","last_change":"2024-03-11T22:15:51.200837+0000","last_active":"2024-03-11T22:17:20.299415+0000","last_peered":"2024-03-11T22:17:20.299415+0000","last_clean":"2024-03-11T22:17:20.299415+0000","last_became_active":"2024-03-11T22:15:51.187016+0000","last_became_peered":"2024-03-11T22:15:51.187016+0000","last_unstale":"2024-03-11T22:17:20.299415+0000","last_undegraded":"2024-03-11T22:17:20.299415+0000","last_fullsized":"2024-03-11T22:17:20.299415+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'22","last_scrub_stamp":"2024-03-11T22:15:25.618938+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:15:25.618938+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":137,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.208592+0000","last_change":"2024-03-11T22:15:51.196449+0000","last_active":"2024-03-11T22:17:21.208592+0000","last_peered":"2024-03-11T22:17:21.208592+0000","last_clean":"2024-03-11T22:17:21.208592+0000","last_became_active":"2024-03-11T22:15:51.184840+0000","last_became_peered":"2024-03-11T22:15:51.184840+0000","last_unstale":"2024-03-11T22:17:21.208592+0000","last_undegraded":"2024-03-11T22:17:21.208592+0000","last_fullsized":"2024-03-11T22:17:21.208592+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:15:15.946740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:15.946740+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":334,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299820+0000","last_change":"2024-03-11T22:15:21.641164+0000","last_active":"2024-03-11T22:17:20.299820+0000","last_peered":"2024-03-11T22:17:20.299820+0000","last_clean":"2024-03-11T22:17:20.299820+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:17:20.299820+0000","last_undegraded":"2024-03-11T22:17:20.299820+0000","last_fullsized":"2024-03-11T22:17:20.299820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:21.641053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:21.641053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"93'130","reported_seq":643,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.313282+0000","last_change":"2024-03-11T22:16:16.271580+0000","last_active":"2024-03-11T22:17:20.313282+0000","last_peered":"2024-03-11T22:17:20.313282+0000","last_clean":"2024-03-11T22:17:20.313282+0000","last_became_active":"2024-03-11T22:16:15.801588+0000","last_became_peered":"2024-03-11T22:16:15.801588+0000","last_unstale":"2024-03-11T22:17:20.313282+0000","last_undegraded":"2024-03-11T22:17:20.313282+0000","last_fullsized":"2024-03-11T22:17:20.313282+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:16:16.271418+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:16:16.271418+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:44.790126+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"75'6","reported_seq":365,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.307341+0000","last_change":"2024-03-11T22:15:51.197712+0000","last_active":"2024-03-11T22:17:20.307341+0000","last_peered":"2024-03-11T22:17:20.307341+0000","last_clean":"2024-03-11T22:17:20.307341+0000","last_became_active":"2024-03-11T22:15:51.186719+0000","last_became_peered":"2024-03-11T22:15:51.186719+0000","last_unstale":"2024-03-11T22:17:20.307341+0000","last_undegraded":"2024-03-11T22:17:20.307341+0000","last_fullsized":"2024-03-11T22:17:20.307341+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:15:27.448002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:27.448002+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":352,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.203784+0000","last_change":"2024-03-11T22:15:35.101348+0000","last_active":"2024-03-11T22:17:11.203784+0000","last_peered":"2024-03-11T22:17:11.203784+0000","last_clean":"2024-03-11T22:17:11.203784+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:17:11.203784+0000","last_undegraded":"2024-03-11T22:17:11.203784+0000","last_fullsized":"2024-03-11T22:17:11.203784+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:35.101214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:15:35.101214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":346,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.300639+0000","last_change":"2024-03-11T22:15:41.760176+0000","last_active":"2024-03-11T22:17:20.300639+0000","last_peered":"2024-03-11T22:17:20.300639+0000","last_clean":"2024-03-11T22:17:20.300639+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:17:20.300639+0000","last_undegraded":"2024-03-11T22:17:20.300639+0000","last_fullsized":"2024-03-11T22:17:20.300639+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:41.760044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:07.195289+0000","last_clean_scrub_stamp":"2024-03-11T22:15:41.760044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":282,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.207118+0000","last_change":"2024-03-11T22:15:39.142832+0000","last_active":"2024-03-11T22:17:11.207118+0000","last_peered":"2024-03-11T22:17:11.207118+0000","last_clean":"2024-03-11T22:17:11.207118+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:17:11.207118+0000","last_undegraded":"2024-03-11T22:17:11.207118+0000","last_fullsized":"2024-03-11T22:17:11.207118+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:39.142690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:15:39.142690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"93'130","reported_seq":631,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299286+0000","last_change":"2024-03-11T22:15:05.687130+0000","last_active":"2024-03-11T22:17:20.299286+0000","last_peered":"2024-03-11T22:17:20.299286+0000","last_clean":"2024-03-11T22:17:20.299286+0000","last_became_active":"2024-03-11T22:14:43.194413+0000","last_became_peered":"2024-03-11T22:14:43.194413+0000","last_unstale":"2024-03-11T22:17:20.299286+0000","last_undegraded":"2024-03-11T22:17:20.299286+0000","last_fullsized":"2024-03-11T22:17:20.299286+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:05.687010+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:05.687010+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"93'130","reported_seq":638,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.312060+0000","last_change":"2024-03-11T22:16:25.363344+0000","last_active":"2024-03-11T22:17:20.312060+0000","last_peered":"2024-03-11T22:17:20.312060+0000","last_clean":"2024-03-11T22:17:20.312060+0000","last_became_active":"2024-03-11T22:14:43.198216+0000","last_became_peered":"2024-03-11T22:14:43.198216+0000","last_unstale":"2024-03-11T22:17:20.312060+0000","last_undegraded":"2024-03-11T22:17:20.312060+0000","last_fullsized":"2024-03-11T22:17:20.312060+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:16:25.363230+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:16:25.363230+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:54.284245+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"93'13","reported_seq":380,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.222494+0000","last_change":"2024-03-11T22:15:51.198003+0000","last_active":"2024-03-11T22:17:21.222494+0000","last_peered":"2024-03-11T22:17:21.222494+0000","last_clean":"2024-03-11T22:17:21.222494+0000","last_became_active":"2024-03-11T22:15:51.183890+0000","last_became_peered":"2024-03-11T22:15:51.183890+0000","last_unstale":"2024-03-11T22:17:21.222494+0000","last_undegraded":"2024-03-11T22:17:21.222494+0000","last_fullsized":"2024-03-11T22:17:21.222494+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:14:26.127737+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.127737+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":115,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.211264+0000","last_change":"2024-03-11T22:15:43.115122+0000","last_active":"2024-03-11T22:17:11.211264+0000","last_peered":"2024-03-11T22:17:11.211264+0000","last_clean":"2024-03-11T22:17:11.211264+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:17:11.211264+0000","last_undegraded":"2024-03-11T22:17:11.211264+0000","last_fullsized":"2024-03-11T22:17:11.211264+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:43.114976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:43.114976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":344,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.307934+0000","last_change":"2024-03-11T22:15:51.187935+0000","last_active":"2024-03-11T22:17:20.307934+0000","last_peered":"2024-03-11T22:17:20.307934+0000","last_clean":"2024-03-11T22:17:20.307934+0000","last_became_active":"2024-03-11T22:15:51.187605+0000","last_became_peered":"2024-03-11T22:15:51.187605+0000","last_unstale":"2024-03-11T22:17:20.307934+0000","last_undegraded":"2024-03-11T22:17:20.307934+0000","last_fullsized":"2024-03-11T22:17:20.307934+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.850128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.850128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":350,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.297062+0000","last_change":"2024-03-11T22:16:17.832501+0000","last_active":"2024-03-11T22:17:20.297062+0000","last_peered":"2024-03-11T22:17:20.297062+0000","last_clean":"2024-03-11T22:17:20.297062+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:17:20.297062+0000","last_undegraded":"2024-03-11T22:17:20.297062+0000","last_fullsized":"2024-03-11T22:17:20.297062+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:16:17.831896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:16:17.831896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:26.169342+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"188'13","reported_seq":370,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.300574+0000","last_change":"2024-03-11T22:15:29.636731+0000","last_active":"2024-03-11T22:17:20.300574+0000","last_peered":"2024-03-11T22:17:20.300574+0000","last_clean":"2024-03-11T22:17:20.300574+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:17:20.300574+0000","last_undegraded":"2024-03-11T22:17:20.300574+0000","last_fullsized":"2024-03-11T22:17:20.300574+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'13","last_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.636639+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":354,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.211458+0000","last_change":"2024-03-11T22:15:51.182980+0000","last_active":"2024-03-11T22:17:21.211458+0000","last_peered":"2024-03-11T22:17:21.211458+0000","last_clean":"2024-03-11T22:17:21.211458+0000","last_became_active":"2024-03-11T22:15:51.182535+0000","last_became_peered":"2024-03-11T22:15:51.182535+0000","last_unstale":"2024-03-11T22:17:21.211458+0000","last_undegraded":"2024-03-11T22:17:21.211458+0000","last_fullsized":"2024-03-11T22:17:21.211458+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:33.012017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:33.012017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"204'10","reported_seq":477,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.308460+0000","last_change":"2024-03-11T22:16:15.835401+0000","last_active":"2024-03-11T22:17:20.308460+0000","last_peered":"2024-03-11T22:17:20.308460+0000","last_clean":"2024-03-11T22:17:20.308460+0000","last_became_active":"2024-03-11T22:16:15.823853+0000","last_became_peered":"2024-03-11T22:16:15.823853+0000","last_unstale":"2024-03-11T22:17:20.308460+0000","last_undegraded":"2024-03-11T22:17:20.308460+0000","last_fullsized":"2024-03-11T22:17:20.308460+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:14:26.011319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:26.011319+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":121,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.294579+0000","last_change":"2024-03-11T22:16:25.805142+0000","last_active":"2024-03-11T22:17:20.294579+0000","last_peered":"2024-03-11T22:17:20.294579+0000","last_clean":"2024-03-11T22:17:20.294579+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:17:20.294579+0000","last_undegraded":"2024-03-11T22:17:20.294579+0000","last_fullsized":"2024-03-11T22:17:20.294579+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:16:25.805037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:16:25.805037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:41.014097+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"93'130","reported_seq":624,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299497+0000","last_change":"2024-03-11T22:15:07.646561+0000","last_active":"2024-03-11T22:17:20.299497+0000","last_peered":"2024-03-11T22:17:20.299497+0000","last_clean":"2024-03-11T22:17:20.299497+0000","last_became_active":"2024-03-11T22:14:43.194350+0000","last_became_peered":"2024-03-11T22:14:43.194350+0000","last_unstale":"2024-03-11T22:17:20.299497+0000","last_undegraded":"2024-03-11T22:17:20.299497+0000","last_fullsized":"2024-03-11T22:17:20.299497+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:07.646448+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:15:07.646448+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":375,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.308367+0000","last_change":"2024-03-11T22:16:16.055622+0000","last_active":"2024-03-11T22:17:20.308367+0000","last_peered":"2024-03-11T22:17:20.308367+0000","last_clean":"2024-03-11T22:17:20.308367+0000","last_became_active":"2024-03-11T22:16:15.815814+0000","last_became_peered":"2024-03-11T22:16:15.815814+0000","last_unstale":"2024-03-11T22:17:20.308367+0000","last_undegraded":"2024-03-11T22:17:20.308367+0000","last_fullsized":"2024-03-11T22:17:20.308367+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:15:29.460297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.460297+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":131,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.294223+0000","last_change":"2024-03-11T22:16:23.773081+0000","last_active":"2024-03-11T22:17:20.294223+0000","last_peered":"2024-03-11T22:17:20.294223+0000","last_clean":"2024-03-11T22:17:20.294223+0000","last_became_active":"2024-03-11T22:15:51.186355+0000","last_became_peered":"2024-03-11T22:15:51.186355+0000","last_unstale":"2024-03-11T22:17:20.294223+0000","last_undegraded":"2024-03-11T22:17:20.294223+0000","last_fullsized":"2024-03-11T22:17:20.294223+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:16:23.772977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:16:23.772977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:33.738280+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":460,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299801+0000","last_change":"2024-03-11T22:15:31.661867+0000","last_active":"2024-03-11T22:17:20.299801+0000","last_peered":"2024-03-11T22:17:20.299801+0000","last_clean":"2024-03-11T22:17:20.299801+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:17:20.299801+0000","last_undegraded":"2024-03-11T22:17:20.299801+0000","last_fullsized":"2024-03-11T22:17:20.299801+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:15:31.661660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:31.661660+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":334,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.212825+0000","last_change":"2024-03-11T22:15:51.189902+0000","last_active":"2024-03-11T22:17:21.212825+0000","last_peered":"2024-03-11T22:17:21.212825+0000","last_clean":"2024-03-11T22:17:21.212825+0000","last_became_active":"2024-03-11T22:15:51.188000+0000","last_became_peered":"2024-03-11T22:15:51.188000+0000","last_unstale":"2024-03-11T22:17:21.212825+0000","last_undegraded":"2024-03-11T22:17:21.212825+0000","last_fullsized":"2024-03-11T22:17:21.212825+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:14.969572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:14.969572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"93'130","reported_seq":582,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.228087+0000","last_change":"2024-03-11T22:15:06.779041+0000","last_active":"2024-03-11T22:17:21.228087+0000","last_peered":"2024-03-11T22:17:21.228087+0000","last_clean":"2024-03-11T22:17:21.228087+0000","last_became_active":"2024-03-11T22:14:43.191297+0000","last_became_peered":"2024-03-11T22:14:43.191297+0000","last_unstale":"2024-03-11T22:17:21.228087+0000","last_undegraded":"2024-03-11T22:17:21.228087+0000","last_fullsized":"2024-03-11T22:17:21.228087+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:06.778786+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:06.778786+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"93'130","reported_seq":485,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.313854+0000","last_change":"2024-03-11T22:16:22.390420+0000","last_active":"2024-03-11T22:17:20.313854+0000","last_peered":"2024-03-11T22:17:20.313854+0000","last_clean":"2024-03-11T22:17:20.313854+0000","last_became_active":"2024-03-11T22:16:21.239384+0000","last_became_peered":"2024-03-11T22:16:21.239384+0000","last_unstale":"2024-03-11T22:17:20.313854+0000","last_undegraded":"2024-03-11T22:17:20.313854+0000","last_fullsized":"2024-03-11T22:17:20.313854+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:16:22.390294+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:16:22.390294+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:24.009281+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":149,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.205200+0000","last_change":"2024-03-11T22:15:44.126540+0000","last_active":"2024-03-11T22:17:11.205200+0000","last_peered":"2024-03-11T22:17:11.205200+0000","last_clean":"2024-03-11T22:17:11.205200+0000","last_became_active":"2024-03-11T22:14:52.239810+0000","last_became_peered":"2024-03-11T22:14:52.239810+0000","last_unstale":"2024-03-11T22:17:11.205200+0000","last_undegraded":"2024-03-11T22:17:11.205200+0000","last_fullsized":"2024-03-11T22:17:11.205200+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:15:44.126413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:44.126413+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":130,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.204795+0000","last_change":"2024-03-11T22:15:51.191241+0000","last_active":"2024-03-11T22:17:11.204795+0000","last_peered":"2024-03-11T22:17:11.204795+0000","last_clean":"2024-03-11T22:17:11.204795+0000","last_became_active":"2024-03-11T22:15:51.185312+0000","last_became_peered":"2024-03-11T22:15:51.185312+0000","last_unstale":"2024-03-11T22:17:11.204795+0000","last_undegraded":"2024-03-11T22:17:11.204795+0000","last_fullsized":"2024-03-11T22:17:11.204795+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:15:26.154167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:26.154167+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":346,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.312577+0000","last_change":"2024-03-11T22:15:32.310641+0000","last_active":"2024-03-11T22:17:20.312577+0000","last_peered":"2024-03-11T22:17:20.312577+0000","last_clean":"2024-03-11T22:17:20.312577+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:17:20.312577+0000","last_undegraded":"2024-03-11T22:17:20.312577+0000","last_fullsized":"2024-03-11T22:17:20.312577+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:32.310504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:32.310504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":326,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.305068+0000","last_change":"2024-03-11T22:12:46.492608+0000","last_active":"2024-03-11T22:17:20.305068+0000","last_peered":"2024-03-11T22:17:20.305068+0000","last_clean":"2024-03-11T22:17:20.305068+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:17:20.305068+0000","last_undegraded":"2024-03-11T22:17:20.305068+0000","last_fullsized":"2024-03-11T22:17:20.305068+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:46.492494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:46.492494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"93'130","reported_seq":496,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.300335+0000","last_change":"2024-03-11T22:16:21.220580+0000","last_active":"2024-03-11T22:17:20.300335+0000","last_peered":"2024-03-11T22:17:20.300335+0000","last_clean":"2024-03-11T22:17:20.300335+0000","last_became_active":"2024-03-11T22:16:21.220243+0000","last_became_peered":"2024-03-11T22:16:21.220243+0000","last_unstale":"2024-03-11T22:17:20.300335+0000","last_undegraded":"2024-03-11T22:17:20.300335+0000","last_fullsized":"2024-03-11T22:17:20.300335+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:08.189043+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:08.189043+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:22.688668+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":366,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.312679+0000","last_change":"2024-03-11T22:15:24.432542+0000","last_active":"2024-03-11T22:17:20.312679+0000","last_peered":"2024-03-11T22:17:20.312679+0000","last_clean":"2024-03-11T22:17:20.312679+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:17:20.312679+0000","last_undegraded":"2024-03-11T22:17:20.312679+0000","last_fullsized":"2024-03-11T22:17:20.312679+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:15:24.432264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:24.432264+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":283,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.306291+0000","last_change":"2024-03-11T22:15:51.190683+0000","last_active":"2024-03-11T22:17:20.306291+0000","last_peered":"2024-03-11T22:17:20.306291+0000","last_clean":"2024-03-11T22:17:20.306291+0000","last_became_active":"2024-03-11T22:15:51.189292+0000","last_became_peered":"2024-03-11T22:15:51.189292+0000","last_unstale":"2024-03-11T22:17:20.306291+0000","last_undegraded":"2024-03-11T22:17:20.306291+0000","last_fullsized":"2024-03-11T22:17:20.306291+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:21.856963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:21.856963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":338,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.307314+0000","last_change":"2024-03-11T22:14:15.858384+0000","last_active":"2024-03-11T22:17:20.307314+0000","last_peered":"2024-03-11T22:17:20.307314+0000","last_clean":"2024-03-11T22:17:20.307314+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:17:20.307314+0000","last_undegraded":"2024-03-11T22:17:20.307314+0000","last_fullsized":"2024-03-11T22:17:20.307314+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:15.857082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:14:15.857082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":336,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.207892+0000","last_change":"2024-03-11T22:15:10.896807+0000","last_active":"2024-03-11T22:17:21.207892+0000","last_peered":"2024-03-11T22:17:21.207892+0000","last_clean":"2024-03-11T22:17:21.207892+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:17:21.207892+0000","last_undegraded":"2024-03-11T22:17:21.207892+0000","last_fullsized":"2024-03-11T22:17:21.207892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:10.896676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:10.896676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"93'130","reported_seq":633,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.296587+0000","last_change":"2024-03-11T22:17:00.669192+0000","last_active":"2024-03-11T22:17:20.296587+0000","last_peered":"2024-03-11T22:17:20.296587+0000","last_clean":"2024-03-11T22:17:20.296587+0000","last_became_active":"2024-03-11T22:14:43.194955+0000","last_became_peered":"2024-03-11T22:14:43.194955+0000","last_unstale":"2024-03-11T22:17:20.296587+0000","last_undegraded":"2024-03-11T22:17:20.296587+0000","last_fullsized":"2024-03-11T22:17:20.296587+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:17:00.668878+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:17:00.668878+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:01.120969+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":371,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.305485+0000","last_change":"2024-03-11T22:15:51.204960+0000","last_active":"2024-03-11T22:17:20.305485+0000","last_peered":"2024-03-11T22:17:20.305485+0000","last_clean":"2024-03-11T22:17:20.305485+0000","last_became_active":"2024-03-11T22:15:51.191282+0000","last_became_peered":"2024-03-11T22:15:51.191282+0000","last_unstale":"2024-03-11T22:17:20.305485+0000","last_undegraded":"2024-03-11T22:17:20.305485+0000","last_fullsized":"2024-03-11T22:17:20.305485+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:13:18.486039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:18.486039+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":360,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.300533+0000","last_change":"2024-03-11T22:15:51.184445+0000","last_active":"2024-03-11T22:17:20.300533+0000","last_peered":"2024-03-11T22:17:20.300533+0000","last_clean":"2024-03-11T22:17:20.300533+0000","last_became_active":"2024-03-11T22:15:51.184115+0000","last_became_peered":"2024-03-11T22:15:51.184115+0000","last_unstale":"2024-03-11T22:17:20.300533+0000","last_undegraded":"2024-03-11T22:17:20.300533+0000","last_fullsized":"2024-03-11T22:17:20.300533+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:45.801375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:45.801375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":122,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:19.160820+0000","last_change":"2024-03-11T22:17:19.160859+0000","last_active":"2024-03-11T22:17:19.160820+0000","last_peered":"2024-03-11T22:17:19.160820+0000","last_clean":"2024-03-11T22:17:19.160820+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:17:19.160820+0000","last_undegraded":"2024-03-11T22:17:19.160820+0000","last_fullsized":"2024-03-11T22:17:19.160820+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:19.160748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:17:19.160748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:43.465766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":334,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.307593+0000","last_change":"2024-03-11T22:14:23.930193+0000","last_active":"2024-03-11T22:17:20.307593+0000","last_peered":"2024-03-11T22:17:20.307593+0000","last_clean":"2024-03-11T22:17:20.307593+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:17:20.307593+0000","last_undegraded":"2024-03-11T22:17:20.307593+0000","last_fullsized":"2024-03-11T22:17:20.307593+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.930091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.930091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"93'130","reported_seq":569,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299398+0000","last_change":"2024-03-11T22:16:21.235774+0000","last_active":"2024-03-11T22:17:20.299398+0000","last_peered":"2024-03-11T22:17:20.299398+0000","last_clean":"2024-03-11T22:17:20.299398+0000","last_became_active":"2024-03-11T22:16:21.235365+0000","last_became_peered":"2024-03-11T22:16:21.235365+0000","last_unstale":"2024-03-11T22:17:20.299398+0000","last_undegraded":"2024-03-11T22:17:20.299398+0000","last_fullsized":"2024-03-11T22:17:20.299398+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:16:16.793538+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:16:16.793538+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":360,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.306271+0000","last_change":"2024-03-11T22:13:31.491151+0000","last_active":"2024-03-11T22:17:20.306271+0000","last_peered":"2024-03-11T22:17:20.306271+0000","last_clean":"2024-03-11T22:17:20.306271+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:17:20.306271+0000","last_undegraded":"2024-03-11T22:17:20.306271+0000","last_fullsized":"2024-03-11T22:17:20.306271+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-11T22:13:31.490878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:13:31.490878+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":348,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.208695+0000","last_change":"2024-03-11T22:15:14.878681+0000","last_active":"2024-03-11T22:17:21.208695+0000","last_peered":"2024-03-11T22:17:21.208695+0000","last_clean":"2024-03-11T22:17:21.208695+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:17:21.208695+0000","last_undegraded":"2024-03-11T22:17:21.208695+0000","last_fullsized":"2024-03-11T22:17:21.208695+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.878563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.878563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":158,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.311779+0000","last_change":"2024-03-11T22:15:43.405176+0000","last_active":"2024-03-11T22:17:20.311779+0000","last_peered":"2024-03-11T22:17:20.311779+0000","last_clean":"2024-03-11T22:17:20.311779+0000","last_became_active":"2024-03-11T22:14:08.442596+0000","last_became_peered":"2024-03-11T22:14:08.442596+0000","last_unstale":"2024-03-11T22:17:20.311779+0000","last_undegraded":"2024-03-11T22:17:20.311779+0000","last_fullsized":"2024-03-11T22:17:20.311779+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:43.405052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:43.405052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":292,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.294580+0000","last_change":"2024-03-11T22:16:30.666597+0000","last_active":"2024-03-11T22:17:20.294580+0000","last_peered":"2024-03-11T22:17:20.294580+0000","last_clean":"2024-03-11T22:17:20.294580+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:17:20.294580+0000","last_undegraded":"2024-03-11T22:17:20.294580+0000","last_fullsized":"2024-03-11T22:17:20.294580+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:16:30.666054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:16:30.666054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:31.101163+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1a","version":"93'130","reported_seq":486,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.311627+0000","last_change":"2024-03-11T22:16:21.403212+0000","last_active":"2024-03-11T22:17:20.311627+0000","last_peered":"2024-03-11T22:17:20.311627+0000","last_clean":"2024-03-11T22:17:20.311627+0000","last_became_active":"2024-03-11T22:16:21.235879+0000","last_became_peered":"2024-03-11T22:16:21.235879+0000","last_unstale":"2024-03-11T22:17:20.311627+0000","last_undegraded":"2024-03-11T22:17:20.311627+0000","last_fullsized":"2024-03-11T22:17:20.311627+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:16:21.402845+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:16:21.402845+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:29.595450+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":368,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.206736+0000","last_change":"2024-03-11T22:15:29.232861+0000","last_active":"2024-03-11T22:17:11.206736+0000","last_peered":"2024-03-11T22:17:11.206736+0000","last_clean":"2024-03-11T22:17:11.206736+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:17:11.206736+0000","last_undegraded":"2024-03-11T22:17:11.206736+0000","last_fullsized":"2024-03-11T22:17:11.206736+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:15:29.232725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.232725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":346,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.306098+0000","last_change":"2024-03-11T22:14:07.822714+0000","last_active":"2024-03-11T22:17:20.306098+0000","last_peered":"2024-03-11T22:17:20.306098+0000","last_clean":"2024-03-11T22:17:20.306098+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:17:20.306098+0000","last_undegraded":"2024-03-11T22:17:20.306098+0000","last_fullsized":"2024-03-11T22:17:20.306098+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:07.822036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:07.822036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":349,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.219901+0000","last_change":"2024-03-11T22:15:44.071290+0000","last_active":"2024-03-11T22:17:21.219901+0000","last_peered":"2024-03-11T22:17:21.219901+0000","last_clean":"2024-03-11T22:17:21.219901+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:17:21.219901+0000","last_undegraded":"2024-03-11T22:17:21.219901+0000","last_fullsized":"2024-03-11T22:17:21.219901+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:44.070947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:44.070947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":350,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.313847+0000","last_change":"2024-03-11T22:17:19.752891+0000","last_active":"2024-03-11T22:17:20.313847+0000","last_peered":"2024-03-11T22:17:20.313847+0000","last_clean":"2024-03-11T22:17:20.313847+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:17:20.313847+0000","last_undegraded":"2024-03-11T22:17:20.313847+0000","last_fullsized":"2024-03-11T22:17:20.313847+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:19.752761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:17:19.752761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:37.869724+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1b","version":"93'130","reported_seq":552,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299847+0000","last_change":"2024-03-11T22:16:21.219301+0000","last_active":"2024-03-11T22:17:20.299847+0000","last_peered":"2024-03-11T22:17:20.299847+0000","last_clean":"2024-03-11T22:17:20.299847+0000","last_became_active":"2024-03-11T22:16:21.218899+0000","last_became_peered":"2024-03-11T22:16:21.218899+0000","last_unstale":"2024-03-11T22:17:20.299847+0000","last_undegraded":"2024-03-11T22:17:20.299847+0000","last_fullsized":"2024-03-11T22:17:20.299847+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:04.424114+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:04.424114+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:22.688640+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":378,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.213333+0000","last_change":"2024-03-11T22:15:51.200779+0000","last_active":"2024-03-11T22:17:21.213333+0000","last_peered":"2024-03-11T22:17:21.213333+0000","last_clean":"2024-03-11T22:17:21.213333+0000","last_became_active":"2024-03-11T22:15:51.188441+0000","last_became_peered":"2024-03-11T22:15:51.188441+0000","last_unstale":"2024-03-11T22:17:21.213333+0000","last_undegraded":"2024-03-11T22:17:21.213333+0000","last_fullsized":"2024-03-11T22:17:21.213333+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:15:19.958690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:19.958690+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":355,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.311975+0000","last_change":"2024-03-11T22:15:37.329154+0000","last_active":"2024-03-11T22:17:20.311975+0000","last_peered":"2024-03-11T22:17:20.311975+0000","last_clean":"2024-03-11T22:17:20.311975+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:17:20.311975+0000","last_undegraded":"2024-03-11T22:17:20.311975+0000","last_fullsized":"2024-03-11T22:17:20.311975+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:15:37.329026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:37.329026+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"194'9","reported_seq":481,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:13.449960+0000","last_change":"2024-03-11T22:16:27.105037+0000","last_active":"2024-03-11T22:17:13.449960+0000","last_peered":"2024-03-11T22:17:13.449960+0000","last_clean":"2024-03-11T22:17:13.449960+0000","last_became_active":"2024-03-11T22:15:51.188803+0000","last_became_peered":"2024-03-11T22:15:51.188803+0000","last_unstale":"2024-03-11T22:17:13.449960+0000","last_undegraded":"2024-03-11T22:17:13.449960+0000","last_fullsized":"2024-03-11T22:17:13.449960+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'9","last_scrub_stamp":"2024-03-11T22:16:27.104921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:16:27.104921+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:32.124021+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":295,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299937+0000","last_change":"2024-03-11T22:15:33.601133+0000","last_active":"2024-03-11T22:17:20.299937+0000","last_peered":"2024-03-11T22:17:20.299937+0000","last_clean":"2024-03-11T22:17:20.299937+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:17:20.299937+0000","last_undegraded":"2024-03-11T22:17:20.299937+0000","last_fullsized":"2024-03-11T22:17:20.299937+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:15:33.601006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:33.601006+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.18","version":"93'130","reported_seq":627,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299031+0000","last_change":"2024-03-11T22:16:21.223678+0000","last_active":"2024-03-11T22:17:20.299031+0000","last_peered":"2024-03-11T22:17:20.299031+0000","last_clean":"2024-03-11T22:17:20.299031+0000","last_became_active":"2024-03-11T22:16:21.223269+0000","last_became_peered":"2024-03-11T22:16:21.223269+0000","last_unstale":"2024-03-11T22:17:20.299031+0000","last_undegraded":"2024-03-11T22:17:20.299031+0000","last_fullsized":"2024-03-11T22:17:20.299031+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:11.673454+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:11.673454+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:22.688691+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":365,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.306219+0000","last_change":"2024-03-11T22:14:27.998984+0000","last_active":"2024-03-11T22:17:20.306219+0000","last_peered":"2024-03-11T22:17:20.306219+0000","last_clean":"2024-03-11T22:17:20.306219+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:17:20.306219+0000","last_undegraded":"2024-03-11T22:17:20.306219+0000","last_fullsized":"2024-03-11T22:17:20.306219+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:14:27.998819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:27.998819+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":113,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.312423+0000","last_change":"2024-03-11T22:15:23.422984+0000","last_active":"2024-03-11T22:17:20.312423+0000","last_peered":"2024-03-11T22:17:20.312423+0000","last_clean":"2024-03-11T22:17:20.312423+0000","last_became_active":"2024-03-11T22:14:52.245199+0000","last_became_peered":"2024-03-11T22:14:52.245199+0000","last_unstale":"2024-03-11T22:17:20.312423+0000","last_undegraded":"2024-03-11T22:17:20.312423+0000","last_fullsized":"2024-03-11T22:17:20.312423+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:23.418102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:23.418102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":104,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.308613+0000","last_change":"2024-03-11T22:14:52.233517+0000","last_active":"2024-03-11T22:17:20.308613+0000","last_peered":"2024-03-11T22:17:20.308613+0000","last_clean":"2024-03-11T22:17:20.308613+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:17:20.308613+0000","last_undegraded":"2024-03-11T22:17:20.308613+0000","last_fullsized":"2024-03-11T22:17:20.308613+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.849679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.849679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":327,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.306930+0000","last_change":"2024-03-11T22:12:44.464428+0000","last_active":"2024-03-11T22:17:20.306930+0000","last_peered":"2024-03-11T22:17:20.306930+0000","last_clean":"2024-03-11T22:17:20.306930+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:17:20.306930+0000","last_undegraded":"2024-03-11T22:17:20.306930+0000","last_fullsized":"2024-03-11T22:17:20.306930+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:44.464308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:44.464308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.19","version":"93'130","reported_seq":631,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.304670+0000","last_change":"2024-03-11T22:16:21.237684+0000","last_active":"2024-03-11T22:17:20.304670+0000","last_peered":"2024-03-11T22:17:20.304670+0000","last_clean":"2024-03-11T22:17:20.304670+0000","last_became_active":"2024-03-11T22:16:21.237274+0000","last_became_peered":"2024-03-11T22:16:21.237274+0000","last_unstale":"2024-03-11T22:17:20.304670+0000","last_undegraded":"2024-03-11T22:17:20.304670+0000","last_fullsized":"2024-03-11T22:17:20.304670+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:14:10.906610+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:14:10.906610+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:24.717189+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":391,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.296773+0000","last_change":"2024-03-11T22:17:10.839559+0000","last_active":"2024-03-11T22:17:20.296773+0000","last_peered":"2024-03-11T22:17:20.296773+0000","last_clean":"2024-03-11T22:17:20.296773+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:17:20.296773+0000","last_undegraded":"2024-03-11T22:17:20.296773+0000","last_fullsized":"2024-03-11T22:17:20.296773+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:17:10.838976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:17:10.838976+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:14.723513+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":354,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.207891+0000","last_change":"2024-03-11T22:15:51.186643+0000","last_active":"2024-03-11T22:17:21.207891+0000","last_peered":"2024-03-11T22:17:21.207891+0000","last_clean":"2024-03-11T22:17:21.207891+0000","last_became_active":"2024-03-11T22:15:51.186230+0000","last_became_peered":"2024-03-11T22:15:51.186230+0000","last_unstale":"2024-03-11T22:17:21.207891+0000","last_undegraded":"2024-03-11T22:17:21.207891+0000","last_fullsized":"2024-03-11T22:17:21.207891+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:32.050761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:32.050761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":346,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.313693+0000","last_change":"2024-03-11T22:15:41.407553+0000","last_active":"2024-03-11T22:17:20.313693+0000","last_peered":"2024-03-11T22:17:20.313693+0000","last_clean":"2024-03-11T22:17:20.313693+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:17:20.313693+0000","last_undegraded":"2024-03-11T22:17:20.313693+0000","last_fullsized":"2024-03-11T22:17:20.313693+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:41.407438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:41.407438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":111,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.205331+0000","last_change":"2024-03-11T22:15:28.217629+0000","last_active":"2024-03-11T22:17:11.205331+0000","last_peered":"2024-03-11T22:17:11.205331+0000","last_clean":"2024-03-11T22:17:11.205331+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:17:11.205331+0000","last_undegraded":"2024-03-11T22:17:11.205331+0000","last_fullsized":"2024-03-11T22:17:11.205331+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:28.217499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:28.217499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1e","version":"93'130","reported_seq":458,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.304933+0000","last_change":"2024-03-11T22:16:21.225690+0000","last_active":"2024-03-11T22:17:20.304933+0000","last_peered":"2024-03-11T22:17:20.304933+0000","last_clean":"2024-03-11T22:17:20.304933+0000","last_became_active":"2024-03-11T22:16:21.225270+0000","last_became_peered":"2024-03-11T22:16:21.225270+0000","last_unstale":"2024-03-11T22:17:20.304933+0000","last_undegraded":"2024-03-11T22:17:20.304933+0000","last_fullsized":"2024-03-11T22:17:20.304933+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:15:09.205116+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:15:09.205116+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:24.717164+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"95'7","reported_seq":178,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.307006+0000","last_change":"2024-03-11T22:14:52.242050+0000","last_active":"2024-03-11T22:17:20.307006+0000","last_peered":"2024-03-11T22:17:20.307006+0000","last_clean":"2024-03-11T22:17:20.307006+0000","last_became_active":"2024-03-11T22:14:52.238411+0000","last_became_peered":"2024-03-11T22:14:52.238411+0000","last_unstale":"2024-03-11T22:17:20.307006+0000","last_undegraded":"2024-03-11T22:17:20.307006+0000","last_fullsized":"2024-03-11T22:17:20.307006+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'7","last_scrub_stamp":"2024-03-11T22:13:32.041219+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:13:32.041219+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":346,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.300057+0000","last_change":"2024-03-11T22:14:30.716953+0000","last_active":"2024-03-11T22:17:20.300057+0000","last_peered":"2024-03-11T22:17:20.300057+0000","last_clean":"2024-03-11T22:17:20.300057+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:17:20.300057+0000","last_undegraded":"2024-03-11T22:17:20.300057+0000","last_fullsized":"2024-03-11T22:17:20.300057+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:14:30.716844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":115,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.311818+0000","last_change":"2024-03-11T22:15:28.434583+0000","last_active":"2024-03-11T22:17:20.311818+0000","last_peered":"2024-03-11T22:17:20.311818+0000","last_clean":"2024-03-11T22:17:20.311818+0000","last_became_active":"2024-03-11T22:14:52.239391+0000","last_became_peered":"2024-03-11T22:14:52.239391+0000","last_unstale":"2024-03-11T22:17:20.311818+0000","last_undegraded":"2024-03-11T22:17:20.311818+0000","last_fullsized":"2024-03-11T22:17:20.311818+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_clean_scrub_stamp":"2024-03-11T22:15:28.434334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":334,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299571+0000","last_change":"2024-03-11T22:14:22.631560+0000","last_active":"2024-03-11T22:17:20.299571+0000","last_peered":"2024-03-11T22:17:20.299571+0000","last_clean":"2024-03-11T22:17:20.299571+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:17:20.299571+0000","last_undegraded":"2024-03-11T22:17:20.299571+0000","last_fullsized":"2024-03-11T22:17:20.299571+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:22.631456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:22.631456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"75'9","reported_seq":147,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.305148+0000","last_change":"2024-03-11T22:15:18.401675+0000","last_active":"2024-03-11T22:17:20.305148+0000","last_peered":"2024-03-11T22:17:20.305148+0000","last_clean":"2024-03-11T22:17:20.305148+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:17:20.305148+0000","last_undegraded":"2024-03-11T22:17:20.305148+0000","last_fullsized":"2024-03-11T22:17:20.305148+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:15:18.401549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:18.401549+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":354,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299713+0000","last_change":"2024-03-11T22:15:15.590670+0000","last_active":"2024-03-11T22:17:20.299713+0000","last_peered":"2024-03-11T22:17:20.299713+0000","last_clean":"2024-03-11T22:17:20.299713+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:17:20.299713+0000","last_undegraded":"2024-03-11T22:17:20.299713+0000","last_fullsized":"2024-03-11T22:17:20.299713+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:15.590551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:15.590551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":344,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299951+0000","last_change":"2024-03-11T22:15:51.185431+0000","last_active":"2024-03-11T22:17:20.299951+0000","last_peered":"2024-03-11T22:17:20.299951+0000","last_clean":"2024-03-11T22:17:20.299951+0000","last_became_active":"2024-03-11T22:15:51.184944+0000","last_became_peered":"2024-03-11T22:15:51.184944+0000","last_unstale":"2024-03-11T22:17:20.299951+0000","last_undegraded":"2024-03-11T22:17:20.299951+0000","last_fullsized":"2024-03-11T22:17:20.299951+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:28.674962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:14:28.674962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":336,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.208260+0000","last_change":"2024-03-11T22:15:20.947535+0000","last_active":"2024-03-11T22:17:21.208260+0000","last_peered":"2024-03-11T22:17:21.208260+0000","last_clean":"2024-03-11T22:17:21.208260+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:17:21.208260+0000","last_undegraded":"2024-03-11T22:17:21.208260+0000","last_fullsized":"2024-03-11T22:17:21.208260+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:20.947405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:20.947405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1c","version":"93'130","reported_seq":559,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.204556+0000","last_change":"2024-03-11T22:16:22.100316+0000","last_active":"2024-03-11T22:17:11.204556+0000","last_peered":"2024-03-11T22:17:11.204556+0000","last_clean":"2024-03-11T22:17:11.204556+0000","last_became_active":"2024-03-11T22:16:21.226659+0000","last_became_peered":"2024-03-11T22:16:21.226659+0000","last_unstale":"2024-03-11T22:17:11.204556+0000","last_undegraded":"2024-03-11T22:17:11.204556+0000","last_fullsized":"2024-03-11T22:17:11.204556+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:16:22.100172+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:16:22.100172+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:49.164455+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":359,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.306223+0000","last_change":"2024-03-11T22:12:49.510144+0000","last_active":"2024-03-11T22:17:20.306223+0000","last_peered":"2024-03-11T22:17:20.306223+0000","last_clean":"2024-03-11T22:17:20.306223+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:17:20.306223+0000","last_undegraded":"2024-03-11T22:17:20.306223+0000","last_fullsized":"2024-03-11T22:17:20.306223+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:12:49.509860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:12:49.509860+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":353,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.306443+0000","last_change":"2024-03-11T22:13:19.463087+0000","last_active":"2024-03-11T22:17:20.306443+0000","last_peered":"2024-03-11T22:17:20.306443+0000","last_clean":"2024-03-11T22:17:20.306443+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:17:20.306443+0000","last_undegraded":"2024-03-11T22:17:20.306443+0000","last_fullsized":"2024-03-11T22:17:20.306443+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:13:19.462955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:13:19.462955+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":111,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.203882+0000","last_change":"2024-03-11T22:15:16.288357+0000","last_active":"2024-03-11T22:17:11.203882+0000","last_peered":"2024-03-11T22:17:11.203882+0000","last_clean":"2024-03-11T22:17:11.203882+0000","last_became_active":"2024-03-11T22:14:52.247689+0000","last_became_peered":"2024-03-11T22:14:52.247689+0000","last_unstale":"2024-03-11T22:17:11.203882+0000","last_undegraded":"2024-03-11T22:17:11.203882+0000","last_fullsized":"2024-03-11T22:17:11.203882+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:16.288232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:16.288232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":332,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.307407+0000","last_change":"2024-03-11T22:15:51.191909+0000","last_active":"2024-03-11T22:17:20.307407+0000","last_peered":"2024-03-11T22:17:20.307407+0000","last_clean":"2024-03-11T22:17:20.307407+0000","last_became_active":"2024-03-11T22:15:51.191529+0000","last_became_peered":"2024-03-11T22:15:51.191529+0000","last_unstale":"2024-03-11T22:17:20.307407+0000","last_undegraded":"2024-03-11T22:17:20.307407+0000","last_fullsized":"2024-03-11T22:17:20.307407+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:12:50.533067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:12:50.533067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"93'130","reported_seq":650,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.313384+0000","last_change":"2024-03-11T22:16:23.333589+0000","last_active":"2024-03-11T22:17:20.313384+0000","last_peered":"2024-03-11T22:17:20.313384+0000","last_clean":"2024-03-11T22:17:20.313384+0000","last_became_active":"2024-03-11T22:16:21.245542+0000","last_became_peered":"2024-03-11T22:16:21.245542+0000","last_unstale":"2024-03-11T22:17:20.313384+0000","last_undegraded":"2024-03-11T22:17:20.313384+0000","last_fullsized":"2024-03-11T22:17:20.313384+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:16:23.333459+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:16:23.333459+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:29.981207+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"211'13","reported_seq":150,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:11.204064+0000","last_change":"2024-03-11T22:16:28.105325+0000","last_active":"2024-03-11T22:17:11.204064+0000","last_peered":"2024-03-11T22:17:11.204064+0000","last_clean":"2024-03-11T22:17:11.204064+0000","last_became_active":"2024-03-11T22:14:52.247411+0000","last_became_peered":"2024-03-11T22:14:52.247411+0000","last_unstale":"2024-03-11T22:17:11.204064+0000","last_undegraded":"2024-03-11T22:17:11.204064+0000","last_fullsized":"2024-03-11T22:17:11.204064+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'13","last_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:16:28.105152+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:17:49.199814+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":349,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-11T22:17:21.208430+0000","last_change":"2024-03-11T22:14:28.062521+0000","last_active":"2024-03-11T22:17:21.208430+0000","last_peered":"2024-03-11T22:17:21.208430+0000","last_clean":"2024-03-11T22:17:21.208430+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:17:21.208430+0000","last_undegraded":"2024-03-11T22:17:21.208430+0000","last_fullsized":"2024-03-11T22:17:21.208430+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:28.062439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:28.062439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":53,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-11T22:17:13.155017+0000","last_change":"2024-03-11T22:17:13.155061+0000","last_active":"2024-03-11T22:17:13.155017+0000","last_peered":"2024-03-11T22:17:13.155017+0000","last_clean":"2024-03-11T22:17:13.155017+0000","last_became_active":"2024-03-11T22:16:15.810413+0000","last_became_peered":"2024-03-11T22:16:15.810413+0000","last_unstale":"2024-03-11T22:17:13.155017+0000","last_undegraded":"2024-03-11T22:17:13.155017+0000","last_fullsized":"2024-03-11T22:17:13.155017+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:13.154925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:17:13.154925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:18:15.027068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":112,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-11T22:17:20.299402+0000","last_change":"2024-03-11T22:15:12.647958+0000","last_active":"2024-03-11T22:17:20.299402+0000","last_peered":"2024-03-11T22:17:20.299402+0000","last_clean":"2024-03-11T22:17:20.299402+0000","last_became_active":"2024-03-11T22:14:52.247273+0000","last_became_peered":"2024-03-11T22:14:52.247273+0000","last_unstale":"2024-03-11T22:17:20.299402+0000","last_undegraded":"2024-03-11T22:17:20.299402+0000","last_fullsized":"2024-03-11T22:17:20.299402+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:12.647884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:12.647884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4385013784,"num_objects":66911,"num_object_clones":0,"num_object_copies":200733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1769,"num_objects_misplaced":1490,"num_objects_unfound":0,"num_objects_dirty":66911,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181321,"num_write_kb":7943361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7776,"num_bytes_recovered":509607936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13756411904,"data_stored":13756399688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65921,"ondisk_log_size":65921,"up":48,"acting":47,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":494,"num_read_kb":489,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":426528,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4030,"ondisk_log_size":4030,"up":93,"acting":93,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":688,"num_read_kb":445,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005166,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1660700,"kb_used_data":1480980,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92084452,"statfs":{"total":95995035648,"available":94294478848,"internally_reserved":0,"allocated":1516523520,"data_stored":1516186186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":184023499},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332211,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1914612,"kb_used_data":1622444,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91830540,"statfs":{"total":95995035648,"available":94034472960,"internally_reserved":0,"allocated":1661382656,"data_stored":1661021212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":299170246},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574989,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2174828,"kb_used_data":1900644,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":91570324,"statfs":{"total":95995035648,"available":93768011776,"internally_reserved":0,"allocated":1946259456,"data_stored":1945841139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738515,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1929548,"kb_used_data":1605376,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91815604,"statfs":{"total":95995035648,"available":94019178496,"internally_reserved":0,"allocated":1643905024,"data_stored":1643530868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,1,3,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411464,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1875192,"kb_used_data":1627376,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91869960,"statfs":{"total":95995035648,"available":94074839040,"internally_reserved":0,"allocated":1666433024,"data_stored":1666068349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1890080,"kb_used_data":1610388,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91855072,"statfs":{"total":95995035648,"available":94059593728,"internally_reserved":0,"allocated":1649037312,"data_stored":1648643660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":286390727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920894,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1561480,"kb_used_data":1381696,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92183672,"statfs":{"total":95995035648,"available":94396080128,"internally_reserved":0,"allocated":1414856704,"data_stored":1414474584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":184023494},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659255,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2644480,"kb_used_data":2366200,"kb_used_omap":1,"kb_used_meta":278270,"kb_avail":91100672,"statfs":{"total":95995035648,"available":93287088128,"internally_reserved":0,"allocated":2422988800,"data_stored":2422615092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":284948934},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1622343680,"data_stored":1622343680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1912340480,"data_stored":1912340480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1640042496,"data_stored":1640038424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1643970560,"data_stored":1643970560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1389297664,"data_stored":1389297664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2393313280,"data_stored":2393309208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1656623104,"data_stored":1656619032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1498480640,"data_stored":1498480640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:17:25.637 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-03-11T22:17:25.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-03-11T22:17:25.659 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:25.690 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:25.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:25.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:25.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:25.833 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:25.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:25.934 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:25.969 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:25.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:25.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:25.982 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:25 smithi045 ceph-mon[39715]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:17:25.982 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:25 smithi045 ceph-mon[39715]: pgmap v385: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 920 op/s; 1769/201324 objects degraded (0.879%); 1490/201324 objects misplaced (0.740%); 2.2 MiB/s, 35 objects/s recovering 2024-03-11T22:17:25.982 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:25 smithi045 ceph-mon[39715]: 5.12 scrub starts 2024-03-11T22:17:25.982 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:25 smithi045 ceph-mon[39715]: 5.12 scrub ok 2024-03-11T22:17:25.983 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:25 smithi045 ceph-mon[30178]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:17:25.983 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:25 smithi045 ceph-mon[30178]: pgmap v385: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 920 op/s; 1769/201324 objects degraded (0.879%); 1490/201324 objects misplaced (0.740%); 2.2 MiB/s, 35 objects/s recovering 2024-03-11T22:17:25.983 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:25 smithi045 ceph-mon[30178]: 5.12 scrub starts 2024-03-11T22:17:25.983 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:25 smithi045 ceph-mon[30178]: 5.12 scrub ok 2024-03-11T22:17:26.070 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:26.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:25 smithi144 ceph-mon[33984]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:17:26.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:25 smithi144 ceph-mon[33984]: pgmap v385: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 920 op/s; 1769/201324 objects degraded (0.879%); 1490/201324 objects misplaced (0.740%); 2.2 MiB/s, 35 objects/s recovering 2024-03-11T22:17:26.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:25 smithi144 ceph-mon[33984]: 5.12 scrub starts 2024-03-11T22:17:26.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:25 smithi144 ceph-mon[33984]: 5.12 scrub ok 2024-03-11T22:17:26.103 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:26.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:26.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:26.205 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:26.238 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:26.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:26.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:26.340 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:26.375 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:26.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:26.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:26.476 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:26.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:26.511 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:26.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:26.595 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:17:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:17:26.612 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:26.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:26.650 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:26.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:26.750 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:26.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:26.786 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:26.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:26.887 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:26.922 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:26.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:27.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:27.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:26 smithi045 ceph-mon[30178]: 4.f deep-scrub starts 2024-03-11T22:17:27.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:26 smithi045 ceph-mon[30178]: 4.f deep-scrub ok 2024-03-11T22:17:27.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:26 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 1769/201324 objects degraded (0.879%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:27.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:26 smithi045 ceph-mon[39715]: 4.f deep-scrub starts 2024-03-11T22:17:27.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:26 smithi045 ceph-mon[39715]: 4.f deep-scrub ok 2024-03-11T22:17:27.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:26 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 1769/201324 objects degraded (0.879%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:27.060 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:27.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:27.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:27.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:26 smithi144 ceph-mon[33984]: 4.f deep-scrub starts 2024-03-11T22:17:27.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:26 smithi144 ceph-mon[33984]: 4.f deep-scrub ok 2024-03-11T22:17:27.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:26 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 1769/201324 objects degraded (0.879%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:27.161 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:27.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:27.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:27.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:27.355 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:27.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:27.457 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:27.492 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:27.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:27.594 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:27.626 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:27.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:27.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:27.766 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:27.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:27.866 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:27.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:27.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[39715]: pgmap v386: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 1769/187257 objects degraded (0.945%); 1490/187257 objects misplaced (0.796%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:17:27.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[39715]: 3.1a scrub starts 2024-03-11T22:17:27.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[39715]: 3.1a scrub ok 2024-03-11T22:17:27.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[39715]: 5.6 scrub starts 2024-03-11T22:17:27.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[39715]: 5.6 scrub ok 2024-03-11T22:17:27.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[39715]: 4.e scrub starts 2024-03-11T22:17:27.896 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[39715]: 4.e scrub ok 2024-03-11T22:17:27.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:27.896 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[30178]: pgmap v386: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 1769/187257 objects degraded (0.945%); 1490/187257 objects misplaced (0.796%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:17:27.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[30178]: 3.1a scrub starts 2024-03-11T22:17:27.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[30178]: 3.1a scrub ok 2024-03-11T22:17:27.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[30178]: 5.6 scrub starts 2024-03-11T22:17:27.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[30178]: 5.6 scrub ok 2024-03-11T22:17:27.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[30178]: 4.e scrub starts 2024-03-11T22:17:27.897 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:27 smithi045 ceph-mon[30178]: 4.e scrub ok 2024-03-11T22:17:27.898 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:27.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:28.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:28.041 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:28.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:28.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:28.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:28.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:27 smithi144 ceph-mon[33984]: pgmap v386: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 1769/187257 objects degraded (0.945%); 1490/187257 objects misplaced (0.796%); 1.1 MiB/s, 17 objects/s recovering 2024-03-11T22:17:28.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:27 smithi144 ceph-mon[33984]: 3.1a scrub starts 2024-03-11T22:17:28.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:27 smithi144 ceph-mon[33984]: 3.1a scrub ok 2024-03-11T22:17:28.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:27 smithi144 ceph-mon[33984]: 5.6 scrub starts 2024-03-11T22:17:28.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:27 smithi144 ceph-mon[33984]: 5.6 scrub ok 2024-03-11T22:17:28.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:27 smithi144 ceph-mon[33984]: 4.e scrub starts 2024-03-11T22:17:28.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:27 smithi144 ceph-mon[33984]: 4.e scrub ok 2024-03-11T22:17:28.143 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:28.173 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:28.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:28.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:28.275 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:28.306 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:28.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:28.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:28.408 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:28.434 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:28.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:28.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:28.578 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:28.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:28.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:28.680 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:28.705 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:28.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:28.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:28.729 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:17:28.729 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:17:28.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.729 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:191364 8.a 8:5f1b220b:::benchmark_data_smithi045_115206_object67246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-03-11T22:17:28.729 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:28.610255+0000", 2024-03-11T22:17:28.729 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.014418199, 2024-03-11T22:17:28.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.014443378, 2024-03-11T22:17:28.729 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:28.730 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:17:28.730 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:28.730 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:17:28.730 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:17:28.730 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 191364 2024-03-11T22:17:28.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.730 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:28.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:28.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610255+0000", 2024-03-11T22:17:28.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:28.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:28.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610255+0000", 2024-03-11T22:17:28.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:28.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:28.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610257+0000", 2024-03-11T22:17:28.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5549999999999999e-06 2024-03-11T22:17:28.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:28.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610261+0000", 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1790000000000001e-06 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610272+0000", 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1532000000000001e-05 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610280+0000", 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3470000000000001e-06 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610784+0000", 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00050383600000000002 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610899+0000", 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011565699999999999 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.611008+0000", 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000109141 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:28.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.616473+0000", 2024-03-11T22:17:28.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0054646979999999996 2024-03-11T22:17:28.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:28.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.616486+0000", 2024-03-11T22:17:28.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2886e-05 2024-03-11T22:17:28.734 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:28.734 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:28.734 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:28.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.735 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:191368 8.a 8:5db2f00c:::benchmark_data_smithi045_115206_object67250:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-03-11T22:17:28.735 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:28.610710+0000", 2024-03-11T22:17:28.735 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.013962949000000001, 2024-03-11T22:17:28.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.014046708, 2024-03-11T22:17:28.735 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:28.735 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:17:28.735 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:28.735 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:17:28.735 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:17:28.735 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 191368 2024-03-11T22:17:28.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.735 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:28.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.736 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:28.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610710+0000", 2024-03-11T22:17:28.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:28.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.736 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:28.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610710+0000", 2024-03-11T22:17:28.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:28.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.736 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:28.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610711+0000", 2024-03-11T22:17:28.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0200000000000001e-07 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610715+0000", 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7560000000000001e-06 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610726+0000", 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1113e-05 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:28.737 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.610733+0000", 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4059999999999999e-06 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.613076+0000", 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023429330000000002 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.613194+0000", 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117199 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.613310+0000", 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116826 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.616494+0000", 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031835470000000001 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.619826+0000", 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033318029999999999 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:28.739 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:191370 8.a 8:50ef37b2:::benchmark_data_smithi045_115206_object67252:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:28.616561+0000", 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0081125380000000007, 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0082676759999999998, 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 191370 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.740 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:28.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:28.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.616561+0000", 2024-03-11T22:17:28.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:28.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:28.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.616561+0000", 2024-03-11T22:17:28.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:28.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:28.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.616562+0000", 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4410000000000001e-06 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.616567+0000", 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6720000000000003e-06 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.616583+0000", 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5571000000000001e-05 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.616745+0000", 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016272300000000001 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.617345+0000", 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00060000500000000003 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.617470+0000", 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012462500000000001 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.617592+0000", 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012205999999999999 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.619783+0000", 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021913800000000002 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.622820+0000", 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030368869999999998 2024-03-11T22:17:28.744 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:191377 8.a 8:535c33a0:::benchmark_data_smithi045_115206_object67259:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:28.619803+0000", 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0048708140000000002, 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0050919370000000004, 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 191377 2024-03-11T22:17:28.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.619803+0000", 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.619803+0000", 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.619804+0000", 2024-03-11T22:17:28.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2899999999999999e-06 2024-03-11T22:17:28.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:28.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.619810+0000", 2024-03-11T22:17:28.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8449999999999999e-06 2024-03-11T22:17:28.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:28.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.619822+0000", 2024-03-11T22:17:28.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2407e-05 2024-03-11T22:17:28.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.619839+0000", 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6827999999999999e-05 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.620141+0000", 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00030158499999999999 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.620258+0000", 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011788799999999999 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.620378+0000", 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119845 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.622642+0000", 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0022637680000000002 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:28.622872+0000", 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023017799999999999 2024-03-11T22:17:28.749 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:28.750 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:28.750 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:28.750 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:28.750 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:17:28.750 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 4 2024-03-11T22:17:28.750 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:17:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:28 smithi045 ceph-mon[39715]: 5.1a scrub starts 2024-03-11T22:17:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:28 smithi045 ceph-mon[39715]: 5.1a scrub ok 2024-03-11T22:17:28.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:28 smithi045 ceph-mon[39715]: 6.14 deep-scrub starts 2024-03-11T22:17:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:28 smithi045 ceph-mon[39715]: 6.14 deep-scrub ok 2024-03-11T22:17:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:28 smithi045 ceph-mon[39715]: 5.18 scrub starts 2024-03-11T22:17:28.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:28 smithi045 ceph-mon[39715]: 5.18 scrub ok 2024-03-11T22:17:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:28 smithi045 ceph-mon[30178]: 5.1a scrub starts 2024-03-11T22:17:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:28 smithi045 ceph-mon[30178]: 5.1a scrub ok 2024-03-11T22:17:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:28 smithi045 ceph-mon[30178]: 6.14 deep-scrub starts 2024-03-11T22:17:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:28 smithi045 ceph-mon[30178]: 6.14 deep-scrub ok 2024-03-11T22:17:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:28 smithi045 ceph-mon[30178]: 5.18 scrub starts 2024-03-11T22:17:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:28 smithi045 ceph-mon[30178]: 5.18 scrub ok 2024-03-11T22:17:28.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:28.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:28.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:28.986 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:28.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:29.026 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:29.089 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:28 smithi144 ceph-mon[33984]: 5.1a scrub starts 2024-03-11T22:17:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:28 smithi144 ceph-mon[33984]: 5.1a scrub ok 2024-03-11T22:17:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:28 smithi144 ceph-mon[33984]: 6.14 deep-scrub starts 2024-03-11T22:17:29.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:28 smithi144 ceph-mon[33984]: 6.14 deep-scrub ok 2024-03-11T22:17:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:28 smithi144 ceph-mon[33984]: 5.18 scrub starts 2024-03-11T22:17:29.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:28 smithi144 ceph-mon[33984]: 5.18 scrub ok 2024-03-11T22:17:29.123 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:29.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:29.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:29.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:29.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:29.259 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:29.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:29.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:29.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:29.410 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:29.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:29.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:29.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:29.553 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:29.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:29.654 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:29.687 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:29.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:29.777 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:17:29.789 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:29.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:29 smithi045 ceph-mon[30178]: pgmap v387: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 2020/182598 objects degraded (1.106%); 1082/182598 objects misplaced (0.593%); 2.3 MiB/s, 37 objects/s recovering 2024-03-11T22:17:29.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:29 smithi045 ceph-mon[30178]: 5.d scrub starts 2024-03-11T22:17:29.823 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:29 smithi045 ceph-mon[30178]: 5.d scrub ok 2024-03-11T22:17:29.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:29 smithi045 ceph-mon[39715]: pgmap v387: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 2020/182598 objects degraded (1.106%); 1082/182598 objects misplaced (0.593%); 2.3 MiB/s, 37 objects/s recovering 2024-03-11T22:17:29.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:29 smithi045 ceph-mon[39715]: 5.d scrub starts 2024-03-11T22:17:29.824 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:29 smithi045 ceph-mon[39715]: 5.d scrub ok 2024-03-11T22:17:29.824 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:17:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:29.830 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:29.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:29.863 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:17:29.908 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:17:29.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:29.956 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:17:29.975 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:29.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:29.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:29.998 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:17:30.044 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:17:30.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:30.082 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:17:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:17:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:17:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:17:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:17:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:17:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:17:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:17:30.093 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:17:30.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:29 smithi144 ceph-mon[33984]: pgmap v387: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 2020/182598 objects degraded (1.106%); 1082/182598 objects misplaced (0.593%); 2.3 MiB/s, 37 objects/s recovering 2024-03-11T22:17:30.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:29 smithi144 ceph-mon[33984]: 5.d scrub starts 2024-03-11T22:17:30.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:29 smithi144 ceph-mon[33984]: 5.d scrub ok 2024-03-11T22:17:30.116 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:30.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:30.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:30.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:30.256 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:30.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:30.359 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:30.388 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:30.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:30.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T22:17:30.489 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:30.541 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:30.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:30.642 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:30.687 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:30.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:30.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:30.789 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:30.821 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:30.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:30.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:30.922 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:30.952 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:30.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:30.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:31.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:30 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2710782241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:17:31.031 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:30 smithi045 ceph-mon[30178]: 5.11 scrub starts 2024-03-11T22:17:31.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:30 smithi045 ceph-mon[30178]: 5.11 scrub ok 2024-03-11T22:17:31.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:17:31.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:17:31.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:30 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2710782241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:17:31.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:30 smithi045 ceph-mon[39715]: 5.11 scrub starts 2024-03-11T22:17:31.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:30 smithi045 ceph-mon[39715]: 5.11 scrub ok 2024-03-11T22:17:31.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:17:31.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:17:31.054 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:31.088 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:31.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:31.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:31.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:30 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2710782241' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:17:31.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:30 smithi144 ceph-mon[33984]: 5.11 scrub starts 2024-03-11T22:17:31.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:30 smithi144 ceph-mon[33984]: 5.11 scrub ok 2024-03-11T22:17:31.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:17:31.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:17:31.191 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:31.224 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:31.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:31.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:31.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:31.363 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:31.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:31.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:31.502 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:31.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:31.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:31.586 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:31.604 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:31.636 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:31.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:31.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:31.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:31.782 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:31.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:31.883 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:31.916 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:31.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:32.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:31 smithi045 ceph-mon[39715]: pgmap v388: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s; 2020/182598 objects degraded (1.106%); 1082/182598 objects misplaced (0.593%); 1.9 MiB/s, 30 objects/s recovering 2024-03-11T22:17:32.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:31 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:17:32.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:31 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:17:32.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:31 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 2020/182598 objects degraded (1.106%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:32.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:31 smithi045 ceph-mon[39715]: osdmap e224: 8 total, 8 up, 8 in 2024-03-11T22:17:32.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:31 smithi045 ceph-mon[30178]: pgmap v388: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s; 2020/182598 objects degraded (1.106%); 1082/182598 objects misplaced (0.593%); 1.9 MiB/s, 30 objects/s recovering 2024-03-11T22:17:32.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:31 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:17:32.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:31 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:17:32.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:31 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 2020/182598 objects degraded (1.106%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:32.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:31 smithi045 ceph-mon[30178]: osdmap e224: 8 total, 8 up, 8 in 2024-03-11T22:17:32.017 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:32.042 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:32.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:32.076 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:32.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:31 smithi144 ceph-mon[33984]: pgmap v388: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s; 2020/182598 objects degraded (1.106%); 1082/182598 objects misplaced (0.593%); 1.9 MiB/s, 30 objects/s recovering 2024-03-11T22:17:32.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:31 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:17:32.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:31 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:17:32.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:31 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 2020/182598 objects degraded (1.106%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:32.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:31 smithi144 ceph-mon[33984]: osdmap e224: 8 total, 8 up, 8 in 2024-03-11T22:17:32.144 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:32.197 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:32.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:32.298 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:32.333 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:32.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:32.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:32.434 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:32.464 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:32.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:32.566 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:32.596 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:32.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:32.600 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:17:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:17:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:17:32.699 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:32.731 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:32.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:32.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:32.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:32.863 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:32.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:32.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:33.004 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:33.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:33.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:33.142 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:33.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:33.244 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:33.278 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:33.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:33.380 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:33.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:33.415 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:33.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:33.516 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:33.546 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:33.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:33.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:33.648 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:33.680 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:33.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:33.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:33.782 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:33.814 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:33.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:33.916 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:33.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:33.949 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:33.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:34.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:33 smithi045 ceph-mon[39715]: pgmap v390: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2284/163194 objects degraded (1.400%); 654/163194 objects misplaced (0.401%); 2.0 MiB/s, 32 objects/s recovering 2024-03-11T22:17:34.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:33 smithi045 ceph-mon[30178]: pgmap v390: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2284/163194 objects degraded (1.400%); 654/163194 objects misplaced (0.401%); 2.0 MiB/s, 32 objects/s recovering 2024-03-11T22:17:34.050 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:34.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:17:34.079 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:34.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:34.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:33 smithi144 ceph-mon[33984]: pgmap v390: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2284/163194 objects degraded (1.400%); 654/163194 objects misplaced (0.401%); 2.0 MiB/s, 32 objects/s recovering 2024-03-11T22:17:34.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:34.220 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:34.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:34.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:34.322 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:34.358 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:34.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:34.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:34.460 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:34.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:34.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:34.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:34.596 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:34.635 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:34.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:34.737 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:34.755 INFO:teuthology.orchestra.run.smithi045.stderr:set pool 1 pg_num to 21 2024-03-11T22:17:34.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:34 smithi045 ceph-mon[30178]: 6.12 scrub starts 2024-03-11T22:17:34.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:34 smithi045 ceph-mon[30178]: 6.12 scrub ok 2024-03-11T22:17:34.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:34 smithi045 ceph-mon[30178]: 3.10 scrub starts 2024-03-11T22:17:34.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:34 smithi045 ceph-mon[30178]: 3.10 scrub ok 2024-03-11T22:17:34.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:34 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1529894043' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T22:17:34.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:34 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T22:17:34.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:34 smithi045 ceph-mon[39715]: 6.12 scrub starts 2024-03-11T22:17:34.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:34 smithi045 ceph-mon[39715]: 6.12 scrub ok 2024-03-11T22:17:34.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:34 smithi045 ceph-mon[39715]: 3.10 scrub starts 2024-03-11T22:17:34.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:34 smithi045 ceph-mon[39715]: 3.10 scrub ok 2024-03-11T22:17:34.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:34 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1529894043' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T22:17:34.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:34 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T22:17:34.775 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:34.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:34.878 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:34.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:34.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:35.014 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:35.047 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:35.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:35.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:35.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:34 smithi144 ceph-mon[33984]: 6.12 scrub starts 2024-03-11T22:17:35.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:34 smithi144 ceph-mon[33984]: 6.12 scrub ok 2024-03-11T22:17:35.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:34 smithi144 ceph-mon[33984]: 3.10 scrub starts 2024-03-11T22:17:35.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:34 smithi144 ceph-mon[33984]: 3.10 scrub ok 2024-03-11T22:17:35.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:34 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1529894043' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T22:17:35.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:34 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-11T22:17:35.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:35.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:35.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:35.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:35.300 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:35.333 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:35.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:35.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:35.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:35.472 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:35.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:35.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:35.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:35.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:35.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:35.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:35.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:35.758 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:35.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:35.859 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:35.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:35 smithi045 ceph-mon[39715]: pgmap v391: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2284/163194 objects degraded (1.400%); 654/163194 objects misplaced (0.401%); 2.0 MiB/s, 32 objects/s recovering 2024-03-11T22:17:35.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:35 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-11T22:17:35.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:35 smithi045 ceph-mon[39715]: osdmap e225: 8 total, 8 up, 8 in 2024-03-11T22:17:35.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:35 smithi045 ceph-mon[39715]: 6.15 scrub starts 2024-03-11T22:17:35.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:35 smithi045 ceph-mon[39715]: 6.15 scrub ok 2024-03-11T22:17:35.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:35 smithi045 ceph-mon[30178]: pgmap v391: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2284/163194 objects degraded (1.400%); 654/163194 objects misplaced (0.401%); 2.0 MiB/s, 32 objects/s recovering 2024-03-11T22:17:35.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:35 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-11T22:17:35.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:35 smithi045 ceph-mon[30178]: osdmap e225: 8 total, 8 up, 8 in 2024-03-11T22:17:35.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:35 smithi045 ceph-mon[30178]: 6.15 scrub starts 2024-03-11T22:17:35.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:35 smithi045 ceph-mon[30178]: 6.15 scrub ok 2024-03-11T22:17:35.895 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:35.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:35.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:35.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:35.997 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:35 smithi144 ceph-mon[33984]: pgmap v391: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s; 2284/163194 objects degraded (1.400%); 654/163194 objects misplaced (0.401%); 2.0 MiB/s, 32 objects/s recovering 2024-03-11T22:17:35.997 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:35 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-11T22:17:35.997 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:35 smithi144 ceph-mon[33984]: osdmap e225: 8 total, 8 up, 8 in 2024-03-11T22:17:35.997 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:35 smithi144 ceph-mon[33984]: 6.15 scrub starts 2024-03-11T22:17:35.997 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:35 smithi144 ceph-mon[33984]: 6.15 scrub ok 2024-03-11T22:17:36.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:36.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:36.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:36.145 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:36.182 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:36.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:36.283 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:36.317 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:36.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:36.345 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:17:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:17:36.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:36.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:36.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:36.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:36.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:36.605 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:36.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:36.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:36.742 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:36.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:36.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:36.844 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:36.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:36 smithi144 ceph-mon[33984]: 4.12 deep-scrub starts 2024-03-11T22:17:36.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:36 smithi144 ceph-mon[33984]: 4.12 deep-scrub ok 2024-03-11T22:17:36.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:36 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 2284/163194 objects degraded (1.400%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:36.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:36 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T22:17:36.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:36 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1451365524' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:17:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:36.883 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:36.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:36.984 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:37.023 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:37.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:36 smithi045 ceph-mon[39715]: 4.12 deep-scrub starts 2024-03-11T22:17:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:36 smithi045 ceph-mon[39715]: 4.12 deep-scrub ok 2024-03-11T22:17:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:36 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 2284/163194 objects degraded (1.400%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:36 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T22:17:37.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:36 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1451365524' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:17:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:36 smithi045 ceph-mon[30178]: 4.12 deep-scrub starts 2024-03-11T22:17:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:36 smithi045 ceph-mon[30178]: 4.12 deep-scrub ok 2024-03-11T22:17:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:36 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 2284/163194 objects degraded (1.400%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:36 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-11T22:17:37.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:36 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1451365524' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:17:37.124 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:37.144 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:37.156 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:37.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:37.222 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:17:37.258 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:37.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:37.289 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:37.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:37.389 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:37.424 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:37.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:37.526 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:37.560 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:37.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:37.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:37.662 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:37.701 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:37.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:37.802 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:37.837 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:37.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:37.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:37.938 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:37.974 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:37.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:38.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:38.031 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[39715]: pgmap v393: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 2284/148386 objects degraded (1.539%); 654/148386 objects misplaced (0.441%); 1.3 MiB/s, 20 objects/s recovering 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[39715]: 3.19 deep-scrub starts 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[39715]: 3.19 deep-scrub ok 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1451365524' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[39715]: osdmap e226: 8 total, 8 up, 8 in 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[30178]: pgmap v393: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 2284/148386 objects degraded (1.539%); 654/148386 objects misplaced (0.441%); 1.3 MiB/s, 20 objects/s recovering 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[30178]: 3.19 deep-scrub starts 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[30178]: 3.19 deep-scrub ok 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1451365524' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:17:38.032 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:37 smithi045 ceph-mon[30178]: osdmap e226: 8 total, 8 up, 8 in 2024-03-11T22:17:38.036 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:17:38.036 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:17:38.076 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:37 smithi144 ceph-mon[33984]: pgmap v393: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 2284/148386 objects degraded (1.539%); 654/148386 objects misplaced (0.441%); 1.3 MiB/s, 20 objects/s recovering 2024-03-11T22:17:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:37 smithi144 ceph-mon[33984]: 3.19 deep-scrub starts 2024-03-11T22:17:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:37 smithi144 ceph-mon[33984]: 3.19 deep-scrub ok 2024-03-11T22:17:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:37 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:17:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:37 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-11T22:17:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:37 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1451365524' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:17:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:37 smithi144 ceph-mon[33984]: osdmap e226: 8 total, 8 up, 8 in 2024-03-11T22:17:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:38.116 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:38.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:38.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:38.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:38.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:38.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:38.366 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:38.400 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:38.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:38.502 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:38.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:38.537 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:38.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:38.638 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:38.669 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:38.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:38.771 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:38.810 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:38.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:38.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:38.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:38.948 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:38.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:38.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:39.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:39.090 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:39.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:39.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:39.192 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:39.214 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:39.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:39.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:39.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:39.354 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:39.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:39.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:39.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:39.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:39.493 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:39.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:39.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:39 smithi045 ceph-mon[39715]: pgmap v395: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 734 B/s rd, 0 B/s wr, 1.17k op/s; 2500/143670 objects degraded (1.740%); 249/143670 objects misplaced (0.173%); 3.2 MiB/s, 50 objects/s recovering 2024-03-11T22:17:39.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:39 smithi045 ceph-mon[39715]: osdmap e227: 8 total, 8 up, 8 in 2024-03-11T22:17:39.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:39 smithi045 ceph-mon[30178]: pgmap v395: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 734 B/s rd, 0 B/s wr, 1.17k op/s; 2500/143670 objects degraded (1.740%); 249/143670 objects misplaced (0.173%); 3.2 MiB/s, 50 objects/s recovering 2024-03-11T22:17:39.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:39 smithi045 ceph-mon[30178]: osdmap e227: 8 total, 8 up, 8 in 2024-03-11T22:17:39.594 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:39.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:39 smithi144 ceph-mon[33984]: pgmap v395: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 734 B/s rd, 0 B/s wr, 1.17k op/s; 2500/143670 objects degraded (1.740%); 249/143670 objects misplaced (0.173%); 3.2 MiB/s, 50 objects/s recovering 2024-03-11T22:17:39.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:39 smithi144 ceph-mon[33984]: osdmap e227: 8 total, 8 up, 8 in 2024-03-11T22:17:39.629 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:39.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:39.731 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:39.764 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:39.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:39.867 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:39.899 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:39.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:40.000 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:40.032 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:40.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:40.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:40.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:40.170 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:40.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:40.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:40.272 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:40.295 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:40.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:40.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:40.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:40.434 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:40.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:40.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:40.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:40 smithi045 ceph-mon[30178]: osdmap e228: 8 total, 8 up, 8 in 2024-03-11T22:17:40.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:40 smithi045 ceph-mon[39715]: osdmap e228: 8 total, 8 up, 8 in 2024-03-11T22:17:40.536 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:40.569 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:40.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:40.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:40 smithi144 ceph-mon[33984]: osdmap e228: 8 total, 8 up, 8 in 2024-03-11T22:17:40.662 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:17:40.698 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:17:40.698 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-11T22:17:40.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T22:17:40.699 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:40.723 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:40.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:40.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:40.871 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:40.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:40.973 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:41.015 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:41.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:41.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:41.117 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:41.140 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:41.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:41.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:41.242 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:41.279 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:41.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:41.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:41.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:41.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:41.404 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:41.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:41.505 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:41.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:41 smithi045 ceph-mon[39715]: pgmap v398: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 934 B/s rd, 0 B/s wr, 1.19k op/s; 2500/143670 objects degraded (1.740%); 249/143670 objects misplaced (0.173%); 2.2 MiB/s, 34 objects/s recovering 2024-03-11T22:17:41.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:41 smithi045 ceph-mon[39715]: osdmap e229: 8 total, 8 up, 8 in 2024-03-11T22:17:41.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:41 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 2500/143670 objects degraded (1.740%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:41.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:41 smithi045 ceph-mon[30178]: pgmap v398: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 934 B/s rd, 0 B/s wr, 1.19k op/s; 2500/143670 objects degraded (1.740%); 249/143670 objects misplaced (0.173%); 2.2 MiB/s, 34 objects/s recovering 2024-03-11T22:17:41.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:41 smithi045 ceph-mon[30178]: osdmap e229: 8 total, 8 up, 8 in 2024-03-11T22:17:41.531 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:41 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 2500/143670 objects degraded (1.740%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:41.542 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:41.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:41.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:41 smithi144 ceph-mon[33984]: pgmap v398: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 934 B/s rd, 0 B/s wr, 1.19k op/s; 2500/143670 objects degraded (1.740%); 249/143670 objects misplaced (0.173%); 2.2 MiB/s, 34 objects/s recovering 2024-03-11T22:17:41.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:41 smithi144 ceph-mon[33984]: osdmap e229: 8 total, 8 up, 8 in 2024-03-11T22:17:41.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:41 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 2500/143670 objects degraded (1.740%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:41.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:41.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:41.678 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:41.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:41.779 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:41.802 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:41.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:41.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:41.904 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:41.936 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:41.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:41.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:42.038 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:42.076 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:42.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:42.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:42.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:42.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:42.302 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:42.333 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:42.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:42.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:42.435 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:42.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:42.471 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:42.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:42.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:42 smithi045 ceph-mon[39715]: osdmap e230: 8 total, 8 up, 8 in 2024-03-11T22:17:42.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:42 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:17:42.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:42 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:17:42.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:42 smithi045 ceph-mon[30178]: osdmap e230: 8 total, 8 up, 8 in 2024-03-11T22:17:42.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:42 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:17:42.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:42 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:17:42.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:42.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:42 smithi144 ceph-mon[33984]: osdmap e230: 8 total, 8 up, 8 in 2024-03-11T22:17:42.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:42 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:17:42.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:42 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:17:42.605 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:42.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:42.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:42.707 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:42.739 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:42.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:42.841 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:42.874 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:42.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:42.974 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:43.001 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:43.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:43.031 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:17:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:17:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:17:43.102 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:43.140 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:43.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:43.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:43.281 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:43.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:43.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:43.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:43.406 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:43.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:43.508 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:43.540 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:43.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:43.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:43 smithi144 ceph-mon[33984]: pgmap v401: 178 pgs: 8 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 168 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 2544/124458 objects degraded (2.044%); 1.8 MiB/s, 51 objects/s recovering 2024-03-11T22:17:43.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:43 smithi144 ceph-mon[33984]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:17:43.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:43 smithi144 ceph-mon[33984]: osdmap e231: 8 total, 8 up, 8 in 2024-03-11T22:17:43.643 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:43.666 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:43.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:43.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:43.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:43.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:43 smithi045 ceph-mon[30178]: pgmap v401: 178 pgs: 8 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 168 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 2544/124458 objects degraded (2.044%); 1.8 MiB/s, 51 objects/s recovering 2024-03-11T22:17:43.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:43 smithi045 ceph-mon[30178]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:17:43.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:43 smithi045 ceph-mon[30178]: osdmap e231: 8 total, 8 up, 8 in 2024-03-11T22:17:43.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:43 smithi045 ceph-mon[39715]: pgmap v401: 178 pgs: 8 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 168 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s; 2544/124458 objects degraded (2.044%); 1.8 MiB/s, 51 objects/s recovering 2024-03-11T22:17:43.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:43 smithi045 ceph-mon[39715]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:17:43.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:43 smithi045 ceph-mon[39715]: osdmap e231: 8 total, 8 up, 8 in 2024-03-11T22:17:43.807 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:43.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:43.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:43.908 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:43.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:43.943 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:43.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:44.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:44.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:44.068 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:44.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:44.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:44.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:44.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:44.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:44.307 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:44.339 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:44.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:44.440 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:44.463 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:44.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:44.564 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:44.601 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:44.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:44.703 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:44.739 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:44.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:44.841 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:44.864 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:44.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:44.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:45.002 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:45.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:45.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:45.104 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:45.134 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:45.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:45.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:45.236 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:45.271 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:45.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:45.373 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:45.406 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:45.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:45.508 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:45.543 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:45.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:45.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:45.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:45 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:17:45.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:45 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:17:45.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:45 smithi144 ceph-mon[33984]: pgmap v403: 178 pgs: 8 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 168 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.29k op/s; 2544/124458 objects degraded (2.044%); 1.5 MiB/s, 41 objects/s recovering 2024-03-11T22:17:45.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:45 smithi144 ceph-mon[33984]: osdmap e232: 8 total, 8 up, 8 in 2024-03-11T22:17:45.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:17:45.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:17:45.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:45.668 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:45.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:45.770 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:45.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:45 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:17:45.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:45 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:17:45.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:45 smithi045 ceph-mon[39715]: pgmap v403: 178 pgs: 8 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 168 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.29k op/s; 2544/124458 objects degraded (2.044%); 1.5 MiB/s, 41 objects/s recovering 2024-03-11T22:17:45.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:45 smithi045 ceph-mon[39715]: osdmap e232: 8 total, 8 up, 8 in 2024-03-11T22:17:45.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:17:45.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:17:45.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:45 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:17:45.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:45 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:17:45.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:45 smithi045 ceph-mon[30178]: pgmap v403: 178 pgs: 8 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 168 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.29k op/s; 2544/124458 objects degraded (2.044%); 1.5 MiB/s, 41 objects/s recovering 2024-03-11T22:17:45.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:45 smithi045 ceph-mon[30178]: osdmap e232: 8 total, 8 up, 8 in 2024-03-11T22:17:45.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:17:45.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:17:45.793 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:45.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:45.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:45.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:45.934 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:45.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:45.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:46.036 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:46.066 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:46.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:46.167 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:46.193 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:46.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:46.294 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:46.333 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:46.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:46.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:46.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:46.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:46.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:46.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:46.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:46 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 2544/124458 objects degraded (2.044%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:46.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-11T22:17:46.595 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:17:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:17:46.602 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:46.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:46.704 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:46.735 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:46.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:46.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:46.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:46 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 2544/124458 objects degraded (2.044%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:46.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-11T22:17:46.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:46 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 2544/124458 objects degraded (2.044%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:46.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-03-11T22:17:46.837 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:46.865 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:46.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:46.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:46.967 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:47.003 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:47.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:47.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:47.106 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:47.138 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:47.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:47.240 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:47.272 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:47.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:47.374 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:47.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:47.418 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:47.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:47.466 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:47.466 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:47 smithi045 ceph-mon[30178]: pgmap v405: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 2.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 2.7 KiB/s rd, 9.2 KiB/s wr, 1.56k op/s; 2544/113373 objects degraded (2.244%); 1.2 MiB/s, 34 objects/s recovering 2024-03-11T22:17:47.466 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:47 smithi045 ceph-mon[30178]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-11T22:17:47.466 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:47 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]': finished 2024-03-11T22:17:47.466 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:47 smithi045 ceph-mon[30178]: osdmap e233: 8 total, 8 up, 8 in 2024-03-11T22:17:47.467 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:47.467 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:47 smithi045 ceph-mon[39715]: pgmap v405: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 2.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 2.7 KiB/s rd, 9.2 KiB/s wr, 1.56k op/s; 2544/113373 objects degraded (2.244%); 1.2 MiB/s, 34 objects/s recovering 2024-03-11T22:17:47.467 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:47 smithi045 ceph-mon[39715]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-11T22:17:47.467 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:47 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]': finished 2024-03-11T22:17:47.467 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:47 smithi045 ceph-mon[39715]: osdmap e233: 8 total, 8 up, 8 in 2024-03-11T22:17:47.519 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:47.549 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:47.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:47.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:47.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:47 smithi144 ceph-mon[33984]: pgmap v405: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 2.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 2.7 KiB/s rd, 9.2 KiB/s wr, 1.56k op/s; 2544/113373 objects degraded (2.244%); 1.2 MiB/s, 34 objects/s recovering 2024-03-11T22:17:47.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:47 smithi144 ceph-mon[33984]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-11T22:17:47.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:47 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "30"}]': finished 2024-03-11T22:17:47.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:47 smithi144 ceph-mon[33984]: osdmap e233: 8 total, 8 up, 8 in 2024-03-11T22:17:47.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:47.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:47.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:47.796 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:47.826 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:47.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:47.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:47.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T22:17:47.927 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:47.954 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:47.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:48.055 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:48.086 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:48.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:48.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:48.188 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:48.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:48.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:48.338 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:48.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:48.363 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:48.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:48.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:48.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:48.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:48.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:48.603 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:48.636 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:48.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:48.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:48.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:48.772 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:48.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:48.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:48.874 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:48.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:17:48.908 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:48.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:49.010 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:49.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:49.040 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:49.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:49.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:49.189 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:49.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:49.291 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:49.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:49.327 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:49.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:49.428 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:49.462 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:49.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:49.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:49.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:49 smithi045 ceph-mon[39715]: pgmap v408: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 2.2 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.9 KiB/s rd, 9.3 KiB/s wr, 840 op/s; 2345/109524 objects degraded (2.141%); 6.6 KiB/s, 33 objects/s recovering 2024-03-11T22:17:49.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:49 smithi045 ceph-mon[39715]: osdmap e234: 8 total, 8 up, 8 in 2024-03-11T22:17:49.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:49 smithi045 ceph-mon[30178]: pgmap v408: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 2.2 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.9 KiB/s rd, 9.3 KiB/s wr, 840 op/s; 2345/109524 objects degraded (2.141%); 6.6 KiB/s, 33 objects/s recovering 2024-03-11T22:17:49.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:49 smithi045 ceph-mon[30178]: osdmap e234: 8 total, 8 up, 8 in 2024-03-11T22:17:49.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:49.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:49 smithi144 ceph-mon[33984]: pgmap v408: 178 pgs: 1 active+recovering+undersized+degraded+remapped, 177 active+clean; 2.2 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.9 KiB/s rd, 9.3 KiB/s wr, 840 op/s; 2345/109524 objects degraded (2.141%); 6.6 KiB/s, 33 objects/s recovering 2024-03-11T22:17:49.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:49 smithi144 ceph-mon[33984]: osdmap e234: 8 total, 8 up, 8 in 2024-03-11T22:17:49.604 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:49.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:49.706 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:49.741 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:49.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:49.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:49.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:49.882 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:49.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:49.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:49.984 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:50.019 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:50.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:50.121 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:50.155 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:50.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:50.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:50.264 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:50 smithi045 ceph-mon[39715]: osdmap e235: 8 total, 8 up, 8 in 2024-03-11T22:17:50.264 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:50 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-11T22:17:50.291 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:50.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:50.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:50.415 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:50.455 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:50.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:50 smithi045 ceph-mon[30178]: osdmap e235: 8 total, 8 up, 8 in 2024-03-11T22:17:50.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:50 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-11T22:17:50.556 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:50.586 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:50.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:50.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:50 smithi144 ceph-mon[33984]: osdmap e235: 8 total, 8 up, 8 in 2024-03-11T22:17:50.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:50 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-03-11T22:17:50.689 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:50.723 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:50.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:50.825 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:50.840 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:50.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:50.938 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:50.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:51.039 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:51.071 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:51.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:51.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:51.173 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:51.203 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:51.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:51.305 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:51.349 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:51.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:51.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:51 smithi045 ceph-mon[30178]: pgmap v410: 177 pgs: 1 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.9 KiB/s rd, 9.3 KiB/s wr, 840 op/s; 2345/109524 objects degraded (2.141%); 6.6 KiB/s, 33 objects/s recovering 2024-03-11T22:17:51.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:51 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]': finished 2024-03-11T22:17:51.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:51 smithi045 ceph-mon[30178]: osdmap e236: 8 total, 8 up, 8 in 2024-03-11T22:17:51.381 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:51 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 2345/109524 objects degraded (2.141%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:51.382 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:51 smithi045 ceph-mon[39715]: pgmap v410: 177 pgs: 1 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.9 KiB/s rd, 9.3 KiB/s wr, 840 op/s; 2345/109524 objects degraded (2.141%); 6.6 KiB/s, 33 objects/s recovering 2024-03-11T22:17:51.382 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:51 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]': finished 2024-03-11T22:17:51.382 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:51 smithi045 ceph-mon[39715]: osdmap e236: 8 total, 8 up, 8 in 2024-03-11T22:17:51.382 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:51 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 2345/109524 objects degraded (2.141%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:51.451 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:51.481 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:51.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:51.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:51.583 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:51.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:51 smithi144 ceph-mon[33984]: pgmap v410: 177 pgs: 1 active+recovering+undersized+degraded+remapped, 176 active+clean; 2.2 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.9 KiB/s rd, 9.3 KiB/s wr, 840 op/s; 2345/109524 objects degraded (2.141%); 6.6 KiB/s, 33 objects/s recovering 2024-03-11T22:17:51.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:51 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "29"}]': finished 2024-03-11T22:17:51.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:51 smithi144 ceph-mon[33984]: osdmap e236: 8 total, 8 up, 8 in 2024-03-11T22:17:51.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:51 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 2345/109524 objects degraded (2.141%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:51.609 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:51.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:51.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:51.713 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:51.744 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:51.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:51.846 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:51.871 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:51.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:51.972 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:52.005 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:52.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:52.106 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:52.132 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:52.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:52.233 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:52.267 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:52.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:52.368 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:52.401 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:52.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:52.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:52.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:52 smithi045 ceph-mon[30178]: osdmap e237: 8 total, 8 up, 8 in 2024-03-11T22:17:52.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:52 smithi045 ceph-mon[39715]: osdmap e237: 8 total, 8 up, 8 in 2024-03-11T22:17:52.542 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:52.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:52.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:52.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:52 smithi144 ceph-mon[33984]: osdmap e237: 8 total, 8 up, 8 in 2024-03-11T22:17:52.644 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:52.677 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:52.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:52.779 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:52.793 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:17:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:17:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:17:52.808 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:52.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:52.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:52.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:52.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:52.957 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:52.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:53.058 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:53.091 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:53.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:53.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:53.155 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:17:53.193 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:53.208 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:17:53.228 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:53.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:53.247 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:17:53.297 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:17:53.330 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:53.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:53 smithi045 ceph-mon[39715]: pgmap v413: 177 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 174 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 2123/91902 objects degraded (2.310%); 0 B/s, 55 objects/s recovering 2024-03-11T22:17:53.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:53 smithi045 ceph-mon[39715]: osdmap e238: 8 total, 8 up, 8 in 2024-03-11T22:17:53.355 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:53 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/327789977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:17:53.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:53 smithi045 ceph-mon[30178]: pgmap v413: 177 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 174 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 2123/91902 objects degraded (2.310%); 0 B/s, 55 objects/s recovering 2024-03-11T22:17:53.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:53 smithi045 ceph-mon[30178]: osdmap e238: 8 total, 8 up, 8 in 2024-03-11T22:17:53.356 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:53 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/327789977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:17:53.356 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:17:53.371 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:53.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:53.397 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:17:53.438 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:17:53.473 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:53.479 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:17:53.486 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:17:53.487 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:17:53.487 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:17:53.487 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:17:53.487 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:17:53.487 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:17:53.487 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:17:53.487 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:17:53.501 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:53.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:53.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:53 smithi144 ceph-mon[33984]: pgmap v413: 177 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 174 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 2123/91902 objects degraded (2.310%); 0 B/s, 55 objects/s recovering 2024-03-11T22:17:53.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:53 smithi144 ceph-mon[33984]: osdmap e238: 8 total, 8 up, 8 in 2024-03-11T22:17:53.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:53 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/327789977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:17:53.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:53.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:53.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:53.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:53.741 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:53.773 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:53.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:53.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:53.875 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:53.910 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:53.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:53.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:54.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:54.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:54.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:54.152 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:54.189 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:54.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:54.290 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:54.333 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:54.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:54.435 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:54.471 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:54.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:54.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:54.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:54 smithi144 ceph-mon[33984]: osdmap e239: 8 total, 8 up, 8 in 2024-03-11T22:17:54.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:54 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-11T22:17:54.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:54.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:54.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:54.625 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:54 smithi045 ceph-mon[30178]: osdmap e239: 8 total, 8 up, 8 in 2024-03-11T22:17:54.626 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:54 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-11T22:17:54.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:54 smithi045 ceph-mon[39715]: osdmap e239: 8 total, 8 up, 8 in 2024-03-11T22:17:54.626 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:54 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-03-11T22:17:54.716 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:54.744 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:54.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:54.846 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:54.881 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:54.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:54.982 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:55.030 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:55.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:55.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:55.132 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:55.163 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:55.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:55.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:55.306 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:55.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:55.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:55.402 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:17:55.407 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:55.437 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:55.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:55.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:55.516 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:55 smithi045 ceph-mon[30178]: pgmap v416: 176 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 174 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 2123/91902 objects degraded (2.310%); 0 B/s, 55 objects/s recovering 2024-03-11T22:17:55.516 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:55 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]': finished 2024-03-11T22:17:55.516 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:55 smithi045 ceph-mon[30178]: osdmap e240: 8 total, 8 up, 8 in 2024-03-11T22:17:55.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:55 smithi045 ceph-mon[39715]: pgmap v416: 176 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 174 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 2123/91902 objects degraded (2.310%); 0 B/s, 55 objects/s recovering 2024-03-11T22:17:55.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:55 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]': finished 2024-03-11T22:17:55.517 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:55 smithi045 ceph-mon[39715]: osdmap e240: 8 total, 8 up, 8 in 2024-03-11T22:17:55.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:55.581 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:55.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:55.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:55 smithi144 ceph-mon[33984]: pgmap v416: 176 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 174 active+clean; 1.9 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 2123/91902 objects degraded (2.310%); 0 B/s, 55 objects/s recovering 2024-03-11T22:17:55.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:55 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "28"}]': finished 2024-03-11T22:17:55.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:55 smithi144 ceph-mon[33984]: osdmap e240: 8 total, 8 up, 8 in 2024-03-11T22:17:55.682 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:55.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:55.717 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:55.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:55.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:55.857 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:55.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:55.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:55.959 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:56.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:56.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:56.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:56.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:56.164 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:56.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:56.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:56.266 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:56.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:56.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:56.408 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:56.437 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:56.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:56.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:56.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:17:56.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:56 smithi045 ceph-mon[39715]: osdmap e241: 8 total, 8 up, 8 in 2024-03-11T22:17:56.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:56 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 2123/91902 objects degraded (2.310%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:56.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:56 smithi045 ceph-mon[39715]: osdmap e242: 8 total, 8 up, 8 in 2024-03-11T22:17:56.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:56 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-11T22:17:56.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:56 smithi045 ceph-mon[30178]: osdmap e241: 8 total, 8 up, 8 in 2024-03-11T22:17:56.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:56 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 2123/91902 objects degraded (2.310%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:56.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:56 smithi045 ceph-mon[30178]: osdmap e242: 8 total, 8 up, 8 in 2024-03-11T22:17:56.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:56 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-11T22:17:56.539 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:56.573 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:56.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:56.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:56 smithi144 ceph-mon[33984]: osdmap e241: 8 total, 8 up, 8 in 2024-03-11T22:17:56.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:56 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 2123/91902 objects degraded (2.310%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:17:56.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:56 smithi144 ceph-mon[33984]: osdmap e242: 8 total, 8 up, 8 in 2024-03-11T22:17:56.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:56 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-03-11T22:17:56.595 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:17:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:17:56.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:56.737 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:56.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:56.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:56.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:56.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:56.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:57.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:57.029 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:57.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:57.032 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:17:57.032 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24326.0:4 3.0 3:00000000::::head [pgnls start_epoch 68] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e68)", 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:08:33.043301+0000", 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 563.76357124100002, 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25786676200000003, 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24326", 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/2228576527", 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 4 2024-03-11T22:17:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043301+0000", 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043301+0000", 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043302+0000", 2024-03-11T22:17:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1209999999999999e-06 2024-03-11T22:17:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043306+0000", 2024-03-11T22:17:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3769999999999998e-06 2024-03-11T22:17:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043314+0000", 2024-03-11T22:17:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0700000000000007e-06 2024-03-11T22:17:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043324+0000", 2024-03-11T22:17:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5529999999999998e-06 2024-03-11T22:17:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043371+0000", 2024-03-11T22:17:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7068999999999999e-05 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.043380+0000", 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5229999999999995e-06 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.300840+0000", 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.257460668 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.300879+0000", 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8714000000000002e-05 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:08:33.301168+0000", 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028866699999999999 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24403.0:109 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e98)", 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:34.741322+0000", 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 442.06554975500001, 2024-03-11T22:17:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.67899541299999999, 2024-03-11T22:17:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:17:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24403", 2024-03-11T22:17:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.144:0/840743586", 2024-03-11T22:17:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 109 2024-03-11T22:17:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741322+0000", 2024-03-11T22:17:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741322+0000", 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741324+0000", 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6279999999999999e-06 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741331+0000", 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8449999999999997e-06 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741340+0000", 2024-03-11T22:17:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6190000000000006e-06 2024-03-11T22:17:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741351+0000", 2024-03-11T22:17:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0372e-05 2024-03-11T22:17:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741431+0000", 2024-03-11T22:17:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0368e-05 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for peered", 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:34.741444+0000", 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2971e-05 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:35.420168+0000", 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.67872427000000002 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:35.420267+0000", 2024-03-11T22:17:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8834999999999998e-05 2024-03-11T22:17:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:35.420318+0000", 2024-03-11T22:17:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0504999999999998e-05 2024-03-11T22:17:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:104387 7.b 7:df0bad5e:::benchmark_data_smithi045_83007_object104386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:17:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:11:59.150464+0000", 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 357.65640797700001, 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15740042900000001, 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 104387 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.150464+0000", 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.150464+0000", 2024-03-11T22:17:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.150466+0000", 2024-03-11T22:17:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.61e-06 2024-03-11T22:17:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.150490+0000", 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3969999999999999e-05 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.150503+0000", 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3179999999999999e-05 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.150512+0000", 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0469999999999999e-06 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.157740+0000", 2024-03-11T22:17:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072285049999999997 2024-03-11T22:17:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.157883+0000", 2024-03-11T22:17:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014308800000000001 2024-03-11T22:17:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:17:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.158034+0000", 2024-03-11T22:17:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015051800000000001 2024-03-11T22:17:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.163788+0000", 2024-03-11T22:17:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0057538609999999999 2024-03-11T22:17:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307382+0000", 2024-03-11T22:17:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14359450400000001 2024-03-11T22:17:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307583+0000", 2024-03-11T22:17:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020014100000000001 2024-03-11T22:17:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307740+0000", 2024-03-11T22:17:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000157353 2024-03-11T22:17:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307865+0000", 2024-03-11T22:17:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000124652 2024-03-11T22:17:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:104397 7.b 7:d952fc32:::benchmark_data_smithi045_83007_object104396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:11:59.155411+0000", 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 357.65146102900002, 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15261744899999999, 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 104397 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.155411+0000", 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.155411+0000", 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.155413+0000", 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6640000000000001e-06 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.155437+0000", 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4524000000000001e-05 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.155452+0000", 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4314999999999999e-05 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.155460+0000", 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5080000000000005e-06 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.160887+0000", 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0054268509999999999 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.161013+0000", 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012621400000000001 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:17:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.161174+0000", 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016122099999999999 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.163920+0000", 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002745482 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307532+0000", 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14361226299999999 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307914+0000", 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00038154600000000002 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307973+0000", 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9379999999999997e-05 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.308028+0000", 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5481e-05 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:104387 7.b e124/99 ondisk, result = 0)", 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:11:59.172734+0000", 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 357.63413811800001, 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13475810999999999, 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.172734+0000", 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.172734+0000", 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.172735+0000", 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4000000000000001e-07 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.172739+0000", 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6079999999999998e-06 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.252931+0000", 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.080191585999999995 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.252947+0000", 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6714e-05 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307373+0000", 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.054425306999999999 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307378+0000", 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2619999999999998e-06 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:11:59.307492+0000", 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113993 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:105476 7.b 7:d7bdd260:::benchmark_data_smithi045_83007_object105475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:00.338837+0000", 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 356.468035171, 2024-03-11T22:17:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13040384299999999, 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 105476 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.338837+0000", 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.338837+0000", 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.338838+0000", 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.226e-06 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.338860+0000", 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1418000000000001e-05 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.338869+0000", 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6390000000000004e-06 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.338876+0000", 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7599999999999997e-06 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.423952+0000", 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.085076499999999999 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.424062+0000", 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000109219 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.424207+0000", 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014562200000000001 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.430914+0000", 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0067071889999999997 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.430958+0000", 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3109999999999999e-05 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.469051+0000", 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038093082 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.469167+0000", 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116786 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.469241+0000", 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3292000000000002e-05 2024-03-11T22:17:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:105477 7.b 7:d80a290b:::benchmark_data_smithi045_83007_object105476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:12:00.339000+0000", 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 356.46787207, 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.130386109, 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 105477 2024-03-11T22:17:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.339000+0000", 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.339000+0000", 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.339001+0000", 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.44e-07 2024-03-11T22:17:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.339020+0000", 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9505999999999999e-05 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.339028+0000", 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8939999999999993e-06 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.339034+0000", 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7790000000000001e-06 2024-03-11T22:17:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.426890+0000", 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.087855723999999996 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.427025+0000", 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013506600000000001 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.428229+0000", 2024-03-11T22:17:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00120411 2024-03-11T22:17:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.431049+0000", 2024-03-11T22:17:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028202689999999998 2024-03-11T22:17:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.431110+0000", 2024-03-11T22:17:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0888999999999999e-05 2024-03-11T22:17:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.469251+0000", 2024-03-11T22:17:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038140652999999997 2024-03-11T22:17:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.469336+0000", 2024-03-11T22:17:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5028e-05 2024-03-11T22:17:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:12:00.469386+0000", 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0547000000000002e-05 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.14790.0:229683 7.d e158/99 7:b3eb5a94:::benchmark_data_smithi045_83007_object111212:head v 158'14757, mlcod=158'14751)", 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:14:11.148214+0000", 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 225.65865794600001, 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14398571299999999, 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148214+0000", 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148214+0000", 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148214+0000", 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5699999999999999e-07 2024-03-11T22:17:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148218+0000", 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1250000000000003e-06 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148230+0000", 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1420000000000001e-05 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148238+0000", 2024-03-11T22:17:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1769999999999998e-06 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148704+0000", 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000466155 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.148733+0000", 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8277e-05 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.292055+0000", 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14332220200000001 2024-03-11T22:17:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.292200+0000", 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0001451 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:94689 8.1 8:8ee448e6:::benchmark_data_smithi045_115206_object94688:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:16:04.529517+0000", 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 112.277354887, 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13979560299999999, 2024-03-11T22:17:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 94689 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.529517+0000", 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.529517+0000", 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.529519+0000", 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.629e-06 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.529538+0000", 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9276e-05 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.529549+0000", 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0817e-05 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.529555+0000", 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5409999999999997e-06 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.529619+0000", 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3609999999999996e-05 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.529761+0000", 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000141772 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,2", 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.529926+0000", 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016482000000000001 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.531993+0000", 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020675519999999998 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.532053+0000", 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0180000000000003e-05 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.669174+0000", 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13712068199999999 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.669246+0000", 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1594999999999995e-05 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.669313+0000", 2024-03-11T22:17:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7129000000000001e-05 2024-03-11T22:17:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:94877 8.1 8:8114391d:::benchmark_data_smithi045_115206_object94876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-11T22:17:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:16:04.900409+0000", 2024-03-11T22:17:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 111.906463221, 2024-03-11T22:17:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.170641037, 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 94877 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900409+0000", 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900409+0000", 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900410+0000", 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.513e-06 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900439+0000", 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8464000000000001e-05 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900454+0000", 2024-03-11T22:17:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5634999999999999e-05 2024-03-11T22:17:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900464+0000", 2024-03-11T22:17:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5580000000000006e-06 2024-03-11T22:17:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900522+0000", 2024-03-11T22:17:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7896999999999998e-05 2024-03-11T22:17:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900663+0000", 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014134 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,2", 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900806+0000", 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000142318 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.916218+0000", 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015412837 2024-03-11T22:17:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.916231+0000", 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3069e-05 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:05.070913+0000", 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.154681875 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:05.070991+0000", 2024-03-11T22:17:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7941000000000005e-05 2024-03-11T22:17:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:05.071050+0000", 2024-03-11T22:17:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.859e-05 2024-03-11T22:17:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:94878 8.1 8:8e81c461:::benchmark_data_smithi045_115206_object94877:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-11T22:17:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:16:04.900484+0000", 2024-03-11T22:17:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 111.90638758599999, 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17088695700000001, 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 94878 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900484+0000", 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900484+0000", 2024-03-11T22:17:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900485+0000", 2024-03-11T22:17:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.06e-07 2024-03-11T22:17:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900507+0000", 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1549e-05 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900513+0000", 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2450000000000003e-06 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.900517+0000", 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.352e-06 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.902698+0000", 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002181161 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.902856+0000", 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015810999999999999 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,2", 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.902996+0000", 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013917199999999999 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.916237+0000", 2024-03-11T22:17:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013241299999999999 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.916271+0000", 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4233999999999998e-05 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:05.071271+0000", 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15499964599999999 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:05.071329+0000", 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8189000000000001e-05 2024-03-11T22:17:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:05.071371+0000", 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2392999999999999e-05 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:94882 8.1 8:8d0e9fad:::benchmark_data_smithi045_115206_object94881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:16:04.902118+0000", 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 111.904754454, 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16941510400000001, 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 94882 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.902118+0000", 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.902118+0000", 2024-03-11T22:17:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.902118+0000", 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2399999999999997e-07 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.902142+0000", 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3264e-05 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.902152+0000", 2024-03-11T22:17:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0519e-05 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.902159+0000", 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1470000000000004e-06 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.904621+0000", 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00246197 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.904712+0000", 2024-03-11T22:17:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1065999999999999e-05 2024-03-11T22:17:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,2", 2024-03-11T22:17:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.904808+0000", 2024-03-11T22:17:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5841999999999995e-05 2024-03-11T22:17:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.916317+0000", 2024-03-11T22:17:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011508760999999999 2024-03-11T22:17:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.918457+0000", 2024-03-11T22:17:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021401419999999998 2024-03-11T22:17:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:05.071421+0000", 2024-03-11T22:17:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15296352599999999 2024-03-11T22:17:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:05.071487+0000", 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6026999999999997e-05 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:05.071533+0000", 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6016000000000001e-05 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:94888 8.1 8:85458edc:::benchmark_data_smithi045_115206_object94887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:16:04.914177+0000", 2024-03-11T22:17:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 111.892695016, 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.157494204, 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 94888 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.914177+0000", 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.914177+0000", 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.914178+0000", 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4300000000000003e-07 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.914200+0000", 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2583999999999999e-05 2024-03-11T22:17:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.914208+0000", 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.481e-06 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.914214+0000", 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8760000000000002e-06 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.916352+0000", 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021380909999999999 2024-03-11T22:17:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.916512+0000", 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000160686 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,2", 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.916683+0000", 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017080099999999999 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.918607+0000", 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019242 2024-03-11T22:17:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.105 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.105 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.105 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:04.919252+0000", 2024-03-11T22:17:57.105 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000645058 2024-03-11T22:17:57.105 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.105 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.105 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.105 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:05.071575+0000", 2024-03-11T22:17:57.105 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15232226200000001 2024-03-11T22:17:57.105 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.105 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.105 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.105 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:05.071633+0000", 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8047000000000002e-05 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:16:05.071671+0000", 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8575000000000003e-05 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24935.0:177382 8.f e221/175 8:f90d6ab3:::benchmark_data_smithi045_115206_object53264:head v 221'11153, mlcod=221'11149)", 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:17.285591+0000", 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 39.521280611000002, 2024-03-11T22:17:57.106 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14602975300000001, 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.285591+0000", 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.285591+0000", 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.107 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.285592+0000", 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0500000000000003e-07 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.285597+0000", 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3630000000000002e-06 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.285608+0000", 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0532e-05 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.108 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.285616+0000", 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8909999999999995e-06 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.285716+0000", 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000100414 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.285737+0000", 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0304000000000001e-05 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.109 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.431579+0000", 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14584291499999999 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.431621+0000", 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1729e-05 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:177386 8.a 8:542eeb83:::benchmark_data_smithi045_115206_object53268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:17.286134+0000", 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 39.520737771, 2024-03-11T22:17:57.110 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.148365633, 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 177386 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286134+0000", 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.111 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286134+0000", 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286135+0000", 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2020000000000001e-06 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286141+0000", 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0579999999999997e-06 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.112 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286152+0000", 2024-03-11T22:17:57.113 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1236999999999999e-05 2024-03-11T22:17:57.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.113 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.113 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.113 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286159+0000", 2024-03-11T22:17:57.113 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6729999999999992e-06 2024-03-11T22:17:57.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.113 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.113 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.113 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286724+0000", 2024-03-11T22:17:57.113 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00056495600000000003 2024-03-11T22:17:57.113 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.113 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286855+0000", 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013016500000000001 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286967+0000", 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000112458 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.431464+0000", 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.144497387 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.114 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.431506+0000", 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1829999999999998e-05 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.434279+0000", 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027732260000000002 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.434453+0000", 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000173628 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.115 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.434500+0000", 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6813000000000001e-05 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:177387 8.1 8:8b3a9a73:::benchmark_data_smithi045_115206_object53269:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:17.286173+0000", 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 39.520698699999997, 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14605842999999999, 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.116 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 177387 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286173+0000", 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286173+0000", 2024-03-11T22:17:57.117 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286174+0000", 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.63e-07 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286176+0000", 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4540000000000001e-06 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286183+0000", 2024-03-11T22:17:57.118 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.365e-06 2024-03-11T22:17:57.119 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.119 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.119 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.119 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286188+0000", 2024-03-11T22:17:57.119 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7169999999999996e-06 2024-03-11T22:17:57.119 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.119 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.119 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.119 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286575+0000", 2024-03-11T22:17:57.119 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00038614399999999999 2024-03-11T22:17:57.119 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.120 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.120 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.120 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286697+0000", 2024-03-11T22:17:57.120 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012237899999999999 2024-03-11T22:17:57.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.120 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.120 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,2", 2024-03-11T22:17:57.120 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.286817+0000", 2024-03-11T22:17:57.120 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000120401 2024-03-11T22:17:57.120 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.120 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.120 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.120 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.428967+0000", 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14214961100000001 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.428980+0000", 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3563999999999999e-05 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.432091+0000", 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031108780000000001 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.121 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.432181+0000", 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9815999999999995e-05 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.432232+0000", 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0637999999999999e-05 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:177392 8.7 8:ea0350b5:::benchmark_data_smithi045_115206_object53274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:17.288071+0000", 2024-03-11T22:17:57.122 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 39.518801486999998, 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15006133699999999, 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 177392 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288071+0000", 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.123 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288071+0000", 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288072+0000", 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.066e-06 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288076+0000", 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5009999999999998e-06 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.124 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288086+0000", 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0353000000000001e-05 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288094+0000", 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4159999999999998e-06 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288143+0000", 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9351999999999998e-05 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.125 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288311+0000", 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016823200000000001 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288487+0000", 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017519500000000001 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.431570+0000", 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.143083601 2024-03-11T22:17:57.126 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.431772+0000", 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020191999999999999 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.435911+0000", 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0041392030000000002 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.438035+0000", 2024-03-11T22:17:57.127 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002123635 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.438132+0000", 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6862999999999997e-05 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.24935.0:177389 8.f e221/175 8:f4a4b2e2:::benchmark_data_smithi045_115206_object53271:head v 221'11154, mlcod=221'11151)", 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:17.288199+0000", 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 39.518672998, 2024-03-11T22:17:57.128 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14345591899999999, 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288199+0000", 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288199+0000", 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.129 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288200+0000", 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3799999999999997e-07 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288204+0000", 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2629999999999997e-06 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288211+0000", 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4490000000000002e-06 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.130 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288217+0000", 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7010000000000001e-06 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.429382+0000", 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14116444 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.429402+0000", 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0908e-05 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.131 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.431624+0000", 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0022217230000000001 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.431655+0000", 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0796999999999997e-05 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.24935.0:177387 8.1 e221/175 ondisk, result = 0)", 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:17.289323+0000", 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 39.517549281999997, 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14291436599999999, 2024-03-11T22:17:57.132 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.289323+0000", 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.289323+0000", 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.289324+0000", 2024-03-11T22:17:57.133 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2500000000000005e-07 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.289328+0000", 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2050000000000004e-06 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.430370+0000", 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14104232799999999 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.430381+0000", 2024-03-11T22:17:57.134 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1092e-05 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.432079+0000", 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016981629999999999 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.432086+0000", 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7549999999999997e-06 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.432237+0000", 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015109800000000001 2024-03-11T22:17:57.135 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:177394 8.1 8:88d46367:::benchmark_data_smithi045_115206_object53276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:17.290212+0000", 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 39.516659953999998, 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.163603893, 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 177394 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.136 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.290212+0000", 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.290212+0000", 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.290213+0000", 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0529999999999999e-06 2024-03-11T22:17:57.137 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.290218+0000", 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.242e-06 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.428415+0000", 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13819659400000001 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.428430+0000", 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5116999999999999e-05 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.429006+0000", 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00057603600000000004 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.429119+0000", 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011312899999999999 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,2", 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.429238+0000", 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011855500000000001 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.139 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.434808+0000", 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0055699449999999998 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.441278+0000", 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0064701180000000004 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.453720+0000", 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012442544999999999 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:17:57.140 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.453775+0000", 2024-03-11T22:17:57.141 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4626000000000002e-05 2024-03-11T22:17:57.141 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:17:57.141 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:17:57.141 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:17:57.141 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.453816+0000", 2024-03-11T22:17:57.141 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0933000000000003e-05 2024-03-11T22:17:57.141 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.141 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.141 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.141 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:17:57.141 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:17:57.141 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:17:57.141 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:57.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:57.178 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:57.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:57.279 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:57.317 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:57.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:57.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:57.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:57.468 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:57.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:57.569 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:57.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:57.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:57 smithi144 ceph-mon[33984]: pgmap v420: 175 pgs: 1 clean+premerge+peered, 1 active+recovering+undersized+degraded+remapped, 173 active+clean; 1.6 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s; 2123/77553 objects degraded (2.737%) 2024-03-11T22:17:57.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:57 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]': finished 2024-03-11T22:17:57.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:57 smithi144 ceph-mon[33984]: osdmap e243: 8 total, 8 up, 8 in 2024-03-11T22:17:57.598 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:57.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:57.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:57.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:57.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:57 smithi045 ceph-mon[30178]: pgmap v420: 175 pgs: 1 clean+premerge+peered, 1 active+recovering+undersized+degraded+remapped, 173 active+clean; 1.6 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s; 2123/77553 objects degraded (2.737%) 2024-03-11T22:17:57.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:57 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]': finished 2024-03-11T22:17:57.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:57 smithi045 ceph-mon[30178]: osdmap e243: 8 total, 8 up, 8 in 2024-03-11T22:17:57.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:17:57.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:57 smithi045 ceph-mon[39715]: pgmap v420: 175 pgs: 1 clean+premerge+peered, 1 active+recovering+undersized+degraded+remapped, 173 active+clean; 1.6 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s; 2123/77553 objects degraded (2.737%) 2024-03-11T22:17:57.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:57 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "27"}]': finished 2024-03-11T22:17:57.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:57 smithi045 ceph-mon[39715]: osdmap e243: 8 total, 8 up, 8 in 2024-03-11T22:17:57.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:17:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:57.723 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:57.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:17:57.824 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:57.855 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:57.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:57.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:57.996 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:57.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:58.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:58.133 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:58.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:58.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:17:58.280 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:58.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:17:58.382 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:17:58.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:58.416 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:58.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:17:58.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:58.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:58.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:58.657 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:58.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:58.687 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:58.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:58.788 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:58.817 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:58.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:58.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:17:58.957 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:58.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:17:59.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:17:59.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:17:59.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:59.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:17:59.198 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:59.235 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:59.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:59.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T22:17:59.337 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:17:59.368 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:59.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:59.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:17:59.470 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:59.503 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:59.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:59.504 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:59 smithi045 ceph-mon[39715]: pgmap v422: 175 pgs: 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 173 active+clean; 1.5 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 1878/72465 objects degraded (2.592%); 0 B/s, 61 objects/s recovering 2024-03-11T22:17:59.504 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:17:59 smithi045 ceph-mon[39715]: osdmap e244: 8 total, 8 up, 8 in 2024-03-11T22:17:59.504 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:59 smithi045 ceph-mon[30178]: pgmap v422: 175 pgs: 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 173 active+clean; 1.5 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 1878/72465 objects degraded (2.592%); 0 B/s, 61 objects/s recovering 2024-03-11T22:17:59.504 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:17:59 smithi045 ceph-mon[30178]: osdmap e244: 8 total, 8 up, 8 in 2024-03-11T22:17:59.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:59 smithi144 ceph-mon[33984]: pgmap v422: 175 pgs: 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 173 active+clean; 1.5 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 1878/72465 objects degraded (2.592%); 0 B/s, 61 objects/s recovering 2024-03-11T22:17:59.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:17:59 smithi144 ceph-mon[33984]: osdmap e244: 8 total, 8 up, 8 in 2024-03-11T22:17:59.604 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:59.641 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:59.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:17:59.744 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:17:59.777 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:59.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:17:59.879 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:17:59.911 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:17:59.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:17:59.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:00.013 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:00.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:00.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:00.192 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:00.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:00.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:00.294 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:00.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:00.328 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:00.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:00.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:00.486 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:00.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:00.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:00 smithi045 ceph-mon[39715]: osdmap e245: 8 total, 8 up, 8 in 2024-03-11T22:18:00.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T22:18:00.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:00 smithi045 ceph-mon[30178]: osdmap e245: 8 total, 8 up, 8 in 2024-03-11T22:18:00.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T22:18:00.589 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:00.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:00 smithi144 ceph-mon[33984]: osdmap e245: 8 total, 8 up, 8 in 2024-03-11T22:18:00.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T22:18:00.625 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:00.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:00.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:00.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:00.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:00.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:00.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:00.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:00.946 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:00.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:01.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:01.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:01.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:01.143 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:18:01.143 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:18:01.143 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 246, 2024-03-11T22:18:01.143 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "f75aafa4-dff2-11ee-95c8-87774f69a715", 2024-03-11T22:18:01.143 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-03-11T22:02:51.075940+0000", 2024-03-11T22:18:01.143 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-03-11T22:18:00.247198+0000", 2024-03-11T22:18:01.143 INFO:teuthology.orchestra.run.smithi045.stdout: "last_up_change": "2024-03-11T22:08:17.992246+0000", 2024-03-11T22:18:01.143 INFO:teuthology.orchestra.run.smithi045.stdout: "last_in_change": "2024-03-11T22:08:00.853801+0000", 2024-03-11T22:18:01.143 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T22:18:01.143 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_num": 5801984, 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_set": [ 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: "noscrub", 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: "pglog_hardlimit", 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: "purged_snapdirs", 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: "recovery_deletes", 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: "sortbitwise" 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_version": 18, 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_snapshot": "", 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_max": 8, 2024-03-11T22:18:01.144 INFO:teuthology.orchestra.run.smithi045.stdout: "max_osd": 8, 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "require_min_compat_client": "luminous", 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "min_compat_client": "luminous", 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "require_osd_release": "squid", 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "allow_crimson": false, 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "pools": [ 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 1, 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": ".mgr", 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:06:12.903386+0000", 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:18:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:18:01.146 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:18:01.146 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:18:01.146 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:18:01.146 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:18:01.146 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:18:01.146 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:18:01.146 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 27, 2024-03-11T22:18:01.146 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 21, 2024-03-11T22:18:01.146 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 21, 2024-03-11T22:18:01.146 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 21, 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 26, 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "1.1b", 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 245, 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 244, 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 244, 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "93'130", 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "93'130" 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "246", 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "246", 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "246", 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:18:01.147 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:18:01.148 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:18:01.149 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: "options": { 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_max": 32, 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_min": 1 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": {} 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.559999942779541, 2024-03-11T22:18:01.150 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.559999942779541, 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2.0699999332427979, 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2.0699999332427979, 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 2, 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "datapool", 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:31.014060+0000", 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 8193, 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T22:18:01.151 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 3, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 3, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 3, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 3, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 3, 2024-03-11T22:18:01.152 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "71", 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 2, 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 71, 2024-03-11T22:18:01.153 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:18:01.154 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:18:01.155 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd": {} 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.9900000095367432, 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.9900000095367432, 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.1699999570846558, 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T22:18:01.156 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 3, 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": ".rgw.root", 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:31.591167+0000", 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:18:01.157 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 32, 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 32, 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 32, 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 32, 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 32, 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:18:01.158 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "69", 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:18:01.159 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:18:01.160 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw": {} 2024-03-11T22:18:01.161 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.5, 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.5, 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2, 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2, 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.162 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 4, 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "default.rgw.log", 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:33.606516+0000", 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:18:01.163 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 32, 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 32, 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 32, 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 32, 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 32, 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "71", 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:18:01.164 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:18:01.165 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:18:01.166 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw": {} 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.8799999952316284, 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.8799999952316284, 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T22:18:01.167 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2.5, 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2.5, 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 5, 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "default.rgw.control", 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:35.315522+0000", 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:18:01.168 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 32, 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 32, 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 32, 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 32, 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 32, 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:18:01.169 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "73", 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:18:01.170 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:18:01.171 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:18:01.172 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw": {} 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.1299999952316284, 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.1299999952316284, 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 1.5, 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 1.5, 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T22:18:01.173 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.174 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 6, 2024-03-11T22:18:01.174 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "default.rgw.meta", 2024-03-11T22:18:01.174 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:37.383856+0000", 2024-03-11T22:18:01.174 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:18:01.174 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 32, 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 32, 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 32, 2024-03-11T22:18:01.175 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 32, 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 32, 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "75", 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:18:01.176 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:18:01.177 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:18:01.178 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: "options": { 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_bias": 4 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw": {} 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.5, 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.5, 2024-03-11T22:18:01.179 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2, 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2, 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 8, 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "unique_pool_1", 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:14:50.877734+0000", 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:18:01.180 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 16, 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 16, 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 16, 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 16, 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 16, 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:18:01.181 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:18:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:18:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:18:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:18:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:18:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "179", 2024-03-11T22:18:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:18:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:18:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:18:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:18:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:18:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:18:01.182 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:18:01.183 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:18:01.184 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "rados": {} 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.5, 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.5, 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2, 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2, 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:18:01.185 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "osds": [ 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 0, 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "55a4e633-f2a6-4d27-b10c-de1c0f87cf32", 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:18:01.186 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 8, 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 228, 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6802", 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6803", 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:18:01.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6804", 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6805", 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.188 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6808", 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6809", 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:18:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6806", 2024-03-11T22:18:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:18:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.190 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6807", 2024-03-11T22:18:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:18:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6803/2216353820", 2024-03-11T22:18:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.45:6805/2216353820", 2024-03-11T22:18:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.45:6809/2216353820", 2024-03-11T22:18:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.45:6807/2216353820", 2024-03-11T22:18:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:18:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:18:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:18:01.191 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 1, 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "60274eeb-591a-43fe-a579-f13d13631e14", 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 13, 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 245, 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:18:01.192 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6810", 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6811", 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:18:01.193 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6812", 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6813", 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:18:01.194 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6816", 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6817", 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:18:01.195 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6814", 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6815", 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6811/1241486091", 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.45:6813/1241486091", 2024-03-11T22:18:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.45:6817/1241486091", 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.45:6815/1241486091", 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 2, 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b", 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:18:01.197 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 18, 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 229, 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6818", 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.198 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6819", 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6820", 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6821", 2024-03-11T22:18:01.199 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6824", 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6825", 2024-03-11T22:18:01.200 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6822", 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6823", 2024-03-11T22:18:01.201 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6819/4251983518", 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.45:6821/4251983518", 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.45:6825/4251983518", 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.45:6823/4251983518", 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.202 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 3, 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "e51a65f3-96f6-478c-8d26-df1ace4a5058", 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 26, 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 242, 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6826", 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6827", 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.204 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6828", 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6829", 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.205 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6832", 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6833", 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.206 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6830", 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6831", 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6827/3178755830", 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.45:6829/3178755830", 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.45:6833/3178755830", 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.45:6831/3178755830", 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:18:01.207 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:18:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:18:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 4, 2024-03-11T22:18:01.208 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "32d0b71f-c696-4677-bdd3-b88a04c97604", 2024-03-11T22:18:01.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:18:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:18:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:18:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 0, 2024-03-11T22:18:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:18:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:18:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 33, 2024-03-11T22:18:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 173, 2024-03-11T22:18:01.209 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6800", 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6801", 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.210 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6802", 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6803", 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6806", 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6807", 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.212 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6804", 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6805", 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.144:6801/912745159", 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.144:6803/912745159", 2024-03-11T22:18:01.213 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.144:6807/912745159", 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.144:6805/912745159", 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 5, 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "f46b91af-d67d-404c-aad1-75348e35d1c4", 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:18:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 41, 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 229, 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6808", 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.215 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6809", 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6810", 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.216 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6811", 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6814", 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.217 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6815", 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6812", 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.218 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6813", 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.144:6809/3162869174", 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.144:6811/3162869174", 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.144:6815/3162869174", 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.144:6813/3162869174", 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.219 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 6, 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "970b128d-cd56-4dff-8345-3196b46c1257", 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 51, 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 229, 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.220 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6816", 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6817", 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6818", 2024-03-11T22:18:01.221 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6819", 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6822", 2024-03-11T22:18:01.222 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6823", 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6820", 2024-03-11T22:18:01.223 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6821", 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.144:6817/3439446288", 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.144:6819/3439446288", 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.144:6823/3439446288", 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.144:6821/3439446288", 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:18:01.224 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 7, 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "40100aaf-830d-4137-b037-5cf167cba308", 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 0.3585662841796875, 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 0, 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 61, 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 73, 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:18:01.225 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:18:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6824", 2024-03-11T22:18:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:18:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6825", 2024-03-11T22:18:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:18:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:18:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.227 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6826", 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6827", 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6830", 2024-03-11T22:18:01.228 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6831", 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6828", 2024-03-11T22:18:01.229 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6829", 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.144:6825/2371064429", 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.144:6827/2371064429", 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.144:6831/2371064429", 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.144:6829/2371064429", 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:18:01.230 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:18:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:18:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:18:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_xinfo": [ 2024-03-11T22:18:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 0, 2024-03-11T22:18:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:18:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:18:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:18:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:18:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:18:01.231 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:17:43.462969+0000", 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 1, 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:16:55.452542+0000", 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 2, 2024-03-11T22:18:01.232 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:16:53.784059+0000", 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 3, 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:18:01.233 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:17:41.544368+0000", 2024-03-11T22:18:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:18:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 4, 2024-03-11T22:18:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:18:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:18:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:18:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:18:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:18:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:17:09.444319+0000", 2024-03-11T22:18:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:18:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 5, 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:17:30.251395+0000", 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 6, 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:18:01.235 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:17:04.394684+0000", 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 7, 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:17:08.125127+0000", 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:18:01.236 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_upmap": [ 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "5.11", 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: "osds": [ 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: 1, 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: 2 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_upmap_items": [ 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "4.9", 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: "mappings": [ 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.237 INFO:teuthology.orchestra.run.smithi045.stdout: "from": 1, 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: "to": 5 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "4.b", 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: "mappings": [ 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: "from": 1, 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: "to": 2 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "4.10", 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "mappings": [ 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "from": 1, 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "to": 2 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "4.14", 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "mappings": [ 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "from": 1, 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: "to": 0 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "5.1", 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "mappings": [ 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "from": 0, 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "to": 2 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "5.14", 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "mappings": [ 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "from": 1, 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: "to": 6 2024-03-11T22:18:01.240 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "5.19", 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "mappings": [ 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "from": 1, 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "to": 6 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_upmap_primaries": [], 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_temp": [ 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "8.c", 2024-03-11T22:18:01.241 INFO:teuthology.orchestra.run.smithi045.stdout: "osds": [ 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: 0, 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: 3 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_temp": [], 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "blocklist": { 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/1901139892": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/1110781721": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6800/2299156110": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/2331445880": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6801/4256719031": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6801/2299156110": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:18:01.242 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6800/3882489981": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/272906373": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/11898002": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/2777740355": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/1903379086": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/926341889": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6800/4256719031": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/2513993410": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/2584267737": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/3509006862": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6801/3882489981": "2024-03-12T22:09:14.021723+0000" 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "range_blocklist": {}, 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profiles": { 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "default": { 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "crush-failure-domain": "osd", 2024-03-11T22:18:01.243 INFO:teuthology.orchestra.run.smithi045.stdout: "k": "2", 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "m": "1", 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "plugin": "jerasure", 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "technique": "reed_sol_van" 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps_queue": [], 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "new_removed_snaps": [], 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "new_purged_snaps": [], 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_node_flags": {}, 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "device_class_flags": {}, 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": { 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode_enabled": false, 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_bucket_count": 0, 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "degraded_stretch_mode": 0, 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "recovering_stretch_mode": 0, 2024-03-11T22:18:01.244 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode_bucket": 0 2024-03-11T22:18:01.245 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:01.245 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:18:01.276 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:01.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:01.378 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:01.409 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:01.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:01.511 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:01.531 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[39715]: pgmap v425: 174 pgs: 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 172 active+clean; 1.5 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 422 op/s; 1878/72465 objects degraded (2.592%); 0 B/s, 60 objects/s recovering 2024-03-11T22:18:01.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T22:18:01.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[39715]: osdmap e246: 8 total, 8 up, 8 in 2024-03-11T22:18:01.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:18:01.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:18:01.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2623583226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T22:18:01.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 1878/72465 objects degraded (2.592%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:01.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[30178]: pgmap v425: 174 pgs: 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 172 active+clean; 1.5 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 422 op/s; 1878/72465 objects degraded (2.592%); 0 B/s, 60 objects/s recovering 2024-03-11T22:18:01.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T22:18:01.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[30178]: osdmap e246: 8 total, 8 up, 8 in 2024-03-11T22:18:01.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:18:01.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:18:01.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2623583226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T22:18:01.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:01 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 1878/72465 objects degraded (2.592%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:01.542 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:01.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:01.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:01 smithi144 ceph-mon[33984]: pgmap v425: 174 pgs: 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 172 active+clean; 1.5 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 422 op/s; 1878/72465 objects degraded (2.592%); 0 B/s, 60 objects/s recovering 2024-03-11T22:18:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T22:18:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:01 smithi144 ceph-mon[33984]: osdmap e246: 8 total, 8 up, 8 in 2024-03-11T22:18:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:18:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:18:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:01 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2623583226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T22:18:01.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:01 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 1878/72465 objects degraded (2.592%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:01.627 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:01.628 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:01.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:01.699 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:01.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:01.801 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:01.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:01.835 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:01.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:01.936 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:01.969 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:01.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:02.071 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:02.107 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:02.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:02.209 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:02.224 INFO:tasks.thrashosds.thrasher:j is {'epoch': 246, 'fsid': 'f75aafa4-dff2-11ee-95c8-87774f69a715', 'created': '2024-03-11T22:02:51.075940+0000', 'modified': '2024-03-11T22:18:00.247198+0000', 'last_up_change': '2024-03-11T22:08:17.992246+0000', 'last_in_change': '2024-03-11T22:08:00.853801+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T22:06:12.903386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 27, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '1.1b', 'ready_epoch': 245, 'last_epoch_started': 244, 'last_epoch_clean': 244, 'source_version': "93'130", 'target_version': "93'130"}, 'last_change': '246', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '246', 'last_force_op_resend_preluminous': '246', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.559999942779541, 'score_stable': 1.559999942779541, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.069999933242798, 'raw_score_stable': 2.069999933242798, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-11T22:08:31.014060+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T22:08:31.591167+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T22:08:33.606516+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T22:08:35.315522+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T22:08:37.383856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-11T22:14:50.877734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}], 'osds': [{'osd': 0, 'uuid': '55a4e633-f2a6-4d27-b10c-de1c0f87cf32', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 228, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6802', 'nonce': 2216353820}, {'type': 'v1', 'addr': '172.21.15.45:6803', 'nonce': 2216353820}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6804', 'nonce': 2216353820}, {'type': 'v1', 'addr': '172.21.15.45:6805', 'nonce': 2216353820}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6808', 'nonce': 2216353820}, {'type': 'v1', 'addr': '172.21.15.45:6809', 'nonce': 2216353820}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6806', 'nonce': 2216353820}, {'type': 'v1', 'addr': '172.21.15.45:6807', 'nonce': 2216353820}]}, 'public_addr': '172.21.15.45:6803/2216353820', 'cluster_addr': '172.21.15.45:6805/2216353820', 'heartbeat_back_addr': '172.21.15.45:6809/2216353820', 'heartbeat_front_addr': '172.21.15.45:6807/2216353820', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '60274eeb-591a-43fe-a579-f13d13631e14', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 245, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6810', 'nonce': 1241486091}, {'type': 'v1', 'addr': '172.21.15.45:6811', 'nonce': 1241486091}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6812', 'nonce': 1241486091}, {'type': 'v1', 'addr': '172.21.15.45:6813', 'nonce': 1241486091}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6816', 'nonce': 1241486091}, {'type': 'v1', 'addr': '172.21.15.45:6817', 'nonce': 1241486091}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6814', 'nonce': 1241486091}, {'type': 'v1', 'addr': '172.21.15.45:6815', 'nonce': 1241486091}]}, 'public_addr': '172.21.15.45:6811/1241486091', 'cluster_addr': '172.21.15.45:6813/1241486091', 'heartbeat_back_addr': '172.21.15.45:6817/1241486091', 'heartbeat_front_addr': '172.21.15.45:6815/1241486091', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 229, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6818', 'nonce': 4251983518}, {'type': 'v1', 'addr': '172.21.15.45:6819', 'nonce': 4251983518}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6820', 'nonce': 4251983518}, {'type': 'v1', 'addr': '172.21.15.45:6821', 'nonce': 4251983518}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6824', 'nonce': 4251983518}, {'type': 'v1', 'addr': '172.21.15.45:6825', 'nonce': 4251983518}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6822', 'nonce': 4251983518}, {'type': 'v1', 'addr': '172.21.15.45:6823', 'nonce': 4251983518}]}, 'public_addr': '172.21.15.45:6819/4251983518', 'cluster_addr': '172.21.15.45:6821/4251983518', 'heartbeat_back_addr': '172.21.15.45:6825/4251983518', 'heartbeat_front_addr': '172.21.15.45:6823/4251983518', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e51a65f3-96f6-478c-8d26-df1ace4a5058', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6826', 'nonce': 3178755830}, {'type': 'v1', 'addr': '172.21.15.45:6827', 'nonce': 3178755830}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6828', 'nonce': 3178755830}, {'type': 'v1', 'addr': '172.21.15.45:6829', 'nonce': 3178755830}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6832', 'nonce': 3178755830}, {'type': 'v1', 'addr': '172.21.15.45:6833', 'nonce': 3178755830}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6830', 'nonce': 3178755830}, {'type': 'v1', 'addr': '172.21.15.45:6831', 'nonce': 3178755830}]}, 'public_addr': '172.21.15.45:6827/3178755830', 'cluster_addr': '172.21.15.45:6829/3178755830', 'heartbeat_back_addr': '172.21.15.45:6833/3178755830', 'heartbeat_front_addr': '172.21.15.45:6831/3178755830', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '32d0b71f-c696-4677-bdd3-b88a04c97604', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 173, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6800', 'nonce': 912745159}, {'type': 'v1', 'addr': '172.21.15.144:6801', 'nonce': 912745159}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6802', 'nonce': 912745159}, {'type': 'v1', 'addr': '172.21.15.144:6803', 'nonce': 912745159}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6806', 'nonce': 912745159}, {'type': 'v1', 'addr': '172.21.15.144:6807', 'nonce': 912745159}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6804', 'nonce': 912745159}, {'type': 'v1', 'addr': '172.21.15.144:6805', 'nonce': 912745159}]}, 'public_addr': '172.21.15.144:6801/912745159', 'cluster_addr': '172.21.15.144:6803/912745159', 'heartbeat_back_addr': '172.21.15.144:6807/912745159', 'heartbeat_front_addr': '172.21.15.144:6805/912745159', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f46b91af-d67d-404c-aad1-75348e35d1c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 229, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6808', 'nonce': 3162869174}, {'type': 'v1', 'addr': '172.21.15.144:6809', 'nonce': 3162869174}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6810', 'nonce': 3162869174}, {'type': 'v1', 'addr': '172.21.15.144:6811', 'nonce': 3162869174}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6814', 'nonce': 3162869174}, {'type': 'v1', 'addr': '172.21.15.144:6815', 'nonce': 3162869174}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6812', 'nonce': 3162869174}, {'type': 'v1', 'addr': '172.21.15.144:6813', 'nonce': 3162869174}]}, 'public_addr': '172.21.15.144:6809/3162869174', 'cluster_addr': '172.21.15.144:6811/3162869174', 'heartbeat_back_addr': '172.21.15.144:6815/3162869174', 'heartbeat_front_addr': '172.21.15.144:6813/3162869174', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '970b128d-cd56-4dff-8345-3196b46c1257', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 229, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6816', 'nonce': 3439446288}, {'type': 'v1', 'addr': '172.21.15.144:6817', 'nonce': 3439446288}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6818', 'nonce': 3439446288}, {'type': 'v1', 'addr': '172.21.15.144:6819', 'nonce': 3439446288}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6822', 'nonce': 3439446288}, {'type': 'v1', 'addr': '172.21.15.144:6823', 'nonce': 3439446288}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6820', 'nonce': 3439446288}, {'type': 'v1', 'addr': '172.21.15.144:6821', 'nonce': 3439446288}]}, 'public_addr': '172.21.15.144:6817/3439446288', 'cluster_addr': '172.21.15.144:6819/3439446288', 'heartbeat_back_addr': '172.21.15.144:6823/3439446288', 'heartbeat_front_addr': '172.21.15.144:6821/3439446288', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '40100aaf-830d-4137-b037-5cf167cba308', 'up': 1, 'in': 1, 'weight': 0.3585662841796875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6824', 'nonce': 2371064429}, {'type': 'v1', 'addr': '172.21.15.144:6825', 'nonce': 2371064429}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6826', 'nonce': 2371064429}, {'type': 'v1', 'addr': '172.21.15.144:6827', 'nonce': 2371064429}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6830', 'nonce': 2371064429}, {'type': 'v1', 'addr': '172.21.15.144:6831', 'nonce': 2371064429}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6828', 'nonce': 2371064429}, {'type': 'v1', 'addr': '172.21.15.144:6829', 'nonce': 2371064429}]}, 'public_addr': '172.21.15.144:6825/2371064429', 'cluster_addr': '172.21.15.144:6827/2371064429', 'heartbeat_back_addr': '172.21.15.144:6831/2371064429', 'heartbeat_front_addr': '172.21.15.144:6829/2371064429', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:17:43.462969+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:16:55.452542+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:16:53.784059+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:17:41.544368+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:17:09.444319+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:17:30.251395+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:17:04.394684+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:17:08.125127+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.11', 'osds': [1, 0, 2]}], 'pg_upmap_items': [{'pgid': '4.9', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.10', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '5.1', 'mappings': [{'from': 0, 'to': 2}]}, {'pgid': '5.14', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '5.19', 'mappings': [{'from': 1, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.c', 'osds': [0, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.45:0/1901139892': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/1110781721': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:6800/2299156110': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:0/2331445880': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:6801/4256719031': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:6801/2299156110': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:6800/3882489981': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/272906373': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/11898002': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/2777740355': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:0/1903379086': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/926341889': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:6800/4256719031': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:0/2513993410': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:0/2584267737': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:0/3509006862': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:6801/3882489981': '2024-03-12T22:09:14.021723+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T22:18:02.225 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 5.11 2024-03-11T22:18:02.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd rm-pg-upmap 5.11 2024-03-11T22:18:02.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:02.256 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:02.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:02.293 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:18:02.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:02.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:02.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:02.486 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:02 smithi045 ceph-mon[39715]: osdmap e247: 8 total, 8 up, 8 in 2024-03-11T22:18:02.487 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:02 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3276694952' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:18:02.487 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:02 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:18:02.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:02 smithi045 ceph-mon[30178]: osdmap e247: 8 total, 8 up, 8 in 2024-03-11T22:18:02.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:02 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3276694952' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:18:02.487 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:02 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:18:02.506 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:02.539 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:02.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:02.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:02 smithi144 ceph-mon[33984]: osdmap e247: 8 total, 8 up, 8 in 2024-03-11T22:18:02.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:02 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3276694952' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:18:02.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:02 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:18:02.641 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:02.668 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:02.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:02.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:02.761 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:18:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:18:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:18:02.770 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:02.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:02.815 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:02.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:02.916 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:02.961 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:02.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:03.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:03.103 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:03.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:03.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:03.241 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:03.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:03.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:03.380 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:03.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:03.482 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:03.491 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:03 smithi045 ceph-mon[39715]: pgmap v428: 174 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 171 active+clean; 1.1 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 1546/54123 objects degraded (2.856%); 0 B/s, 82 objects/s recovering 2024-03-11T22:18:03.491 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:03 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:18:03.491 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:03 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:18:03.491 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:03 smithi045 ceph-mon[39715]: osdmap e248: 8 total, 8 up, 8 in 2024-03-11T22:18:03.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:03 smithi045 ceph-mon[30178]: pgmap v428: 174 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 171 active+clean; 1.1 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 1546/54123 objects degraded (2.856%); 0 B/s, 82 objects/s recovering 2024-03-11T22:18:03.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:03 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:18:03.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:03 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:18:03.492 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:03 smithi045 ceph-mon[30178]: osdmap e248: 8 total, 8 up, 8 in 2024-03-11T22:18:03.515 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:03.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:03.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:03.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:03 smithi144 ceph-mon[33984]: pgmap v428: 174 pgs: 1 active+recovering+undersized+degraded+remapped, 2 peering, 171 active+clean; 1.1 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 1546/54123 objects degraded (2.856%); 0 B/s, 82 objects/s recovering 2024-03-11T22:18:03.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:03 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:18:03.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:03 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:18:03.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:03 smithi144 ceph-mon[33984]: osdmap e248: 8 total, 8 up, 8 in 2024-03-11T22:18:03.617 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:03.645 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:03.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:03.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:03.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:03.785 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:03.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:03.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:03.928 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:03.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:04.030 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:04.060 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:04.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:04.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:04.200 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:04.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:04.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:04.302 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:04.336 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:04.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:04.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:04.438 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:04.468 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:04.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:04.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:04.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:04 smithi144 ceph-mon[33984]: osdmap e249: 8 total, 8 up, 8 in 2024-03-11T22:18:04.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:04 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T22:18:04.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:04.604 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:04.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:04.704 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:04.741 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:04.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:04.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:04 smithi045 ceph-mon[39715]: osdmap e249: 8 total, 8 up, 8 in 2024-03-11T22:18:04.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:04 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T22:18:04.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:04 smithi045 ceph-mon[30178]: osdmap e249: 8 total, 8 up, 8 in 2024-03-11T22:18:04.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:04 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T22:18:04.843 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:04.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:04.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:04.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:04.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:05.039 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:05.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:05.141 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:05.187 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:05.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:05.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:05.323 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:05.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:05.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:05.394 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:05 smithi045 ceph-mon[39715]: pgmap v431: 173 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 171 active+clean; 1.1 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 1546/54123 objects degraded (2.856%); 0 B/s, 82 objects/s recovering 2024-03-11T22:18:05.394 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:05 smithi045 ceph-mon[30178]: pgmap v431: 173 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 171 active+clean; 1.1 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 1546/54123 objects degraded (2.856%); 0 B/s, 82 objects/s recovering 2024-03-11T22:18:05.425 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:05.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:05.460 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:05.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:05.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:05.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:05 smithi144 ceph-mon[33984]: pgmap v431: 173 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 171 active+clean; 1.1 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 1546/54123 objects degraded (2.856%); 0 B/s, 82 objects/s recovering 2024-03-11T22:18:05.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:05 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]': finished 2024-03-11T22:18:05.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:05 smithi144 ceph-mon[33984]: osdmap e250: 8 total, 8 up, 8 in 2024-03-11T22:18:05.598 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:05.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:05.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:05.699 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:05.738 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:05.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:05.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:05.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:05 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]': finished 2024-03-11T22:18:05.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:05 smithi045 ceph-mon[39715]: osdmap e250: 8 total, 8 up, 8 in 2024-03-11T22:18:05.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:05 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "25"}]': finished 2024-03-11T22:18:05.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:05 smithi045 ceph-mon[30178]: osdmap e250: 8 total, 8 up, 8 in 2024-03-11T22:18:05.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:05.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:05.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:05.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:05.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:06.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:06.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:06.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:06.123 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:06.154 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:06.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:06.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:06.185 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:18:06.228 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:18:06.255 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:06.278 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:18:06.285 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:06.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:06.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:06.319 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:18:06.359 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:18:06.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:06.404 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:18:06.425 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:06.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:06.449 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:18:06.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:06 smithi045 ceph-mon[30178]: osdmap e251: 8 total, 8 up, 8 in 2024-03-11T22:18:06.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:06 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3160323559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:18:06.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:06 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 1546/54123 objects degraded (2.856%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:06.450 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:06 smithi045 ceph-mon[30178]: osdmap e252: 8 total, 8 up, 8 in 2024-03-11T22:18:06.450 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:06 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-11T22:18:06.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:06 smithi045 ceph-mon[39715]: osdmap e251: 8 total, 8 up, 8 in 2024-03-11T22:18:06.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:06 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3160323559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:18:06.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:06 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 1546/54123 objects degraded (2.856%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:06.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:06 smithi045 ceph-mon[39715]: osdmap e252: 8 total, 8 up, 8 in 2024-03-11T22:18:06.450 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:06 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-11T22:18:06.490 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:18:06.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:18:06.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:18:06.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:18:06.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:18:06.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:18:06.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:18:06.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:18:06.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:18:06.527 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:06.559 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:06.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:06.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:06.595 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:18:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:18:06.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:06 smithi144 ceph-mon[33984]: osdmap e251: 8 total, 8 up, 8 in 2024-03-11T22:18:06.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:06 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3160323559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:18:06.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:06 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 1546/54123 objects degraded (2.856%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:06.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:06 smithi144 ceph-mon[33984]: osdmap e252: 8 total, 8 up, 8 in 2024-03-11T22:18:06.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:06 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-03-11T22:18:06.661 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:06.696 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:06.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:18:06.770 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:18:06.797 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:06.830 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:06.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:06.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:06.932 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:06.962 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:06.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:06.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:07.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:07.101 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:07.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:07.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:07.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:07.252 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:07.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:07.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:07.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:07.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:07.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:07.532 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:07.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:07.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:07.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:07 smithi144 ceph-mon[33984]: pgmap v435: 172 pgs: 1 clean+premerge+peered, 1 active+recovering+undersized+degraded+remapped, 170 active+clean; 813 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.23k op/s; 1546/39585 objects degraded (3.906%) 2024-03-11T22:18:07.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:07 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:18:07.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:07 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:18:07.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:07 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]': finished 2024-03-11T22:18:07.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:07 smithi144 ceph-mon[33984]: osdmap e253: 8 total, 8 up, 8 in 2024-03-11T22:18:07.633 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:07.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:07.665 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:07.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:07.766 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:07.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:07.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:07 smithi045 ceph-mon[30178]: pgmap v435: 172 pgs: 1 clean+premerge+peered, 1 active+recovering+undersized+degraded+remapped, 170 active+clean; 813 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.23k op/s; 1546/39585 objects degraded (3.906%) 2024-03-11T22:18:07.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:07 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:18:07.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:07 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:18:07.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:07 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]': finished 2024-03-11T22:18:07.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:07 smithi045 ceph-mon[30178]: osdmap e253: 8 total, 8 up, 8 in 2024-03-11T22:18:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:07 smithi045 ceph-mon[39715]: pgmap v435: 172 pgs: 1 clean+premerge+peered, 1 active+recovering+undersized+degraded+remapped, 170 active+clean; 813 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.23k op/s; 1546/39585 objects degraded (3.906%) 2024-03-11T22:18:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:07 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:18:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:07 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:18:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:07 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "24"}]': finished 2024-03-11T22:18:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:07 smithi045 ceph-mon[39715]: osdmap e253: 8 total, 8 up, 8 in 2024-03-11T22:18:07.798 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:07.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:07.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:07.877 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:18:07.900 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:07.934 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:07.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:08.036 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:08.065 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:08.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:08.166 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:08.212 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:08.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:08.314 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:08.345 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:08.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:08.446 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:08.484 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:08.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:08.586 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:08.618 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:08.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:08.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:08.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:08.754 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:08.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:08.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:08.851 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:08.857 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:08.866 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T22:18:08.883 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:08.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:08.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:08.985 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:09.016 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:09.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:09.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:09.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:09.177 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:09.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:09.278 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:09.311 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:09.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:09.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:09.452 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:09.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:09.480 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:09 smithi045 ceph-mon[30178]: pgmap v438: 172 pgs: 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 170 active+clean; 711 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.65k op/s; 1233/34722 objects degraded (3.551%); 0 B/s, 79 objects/s recovering 2024-03-11T22:18:09.480 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:09 smithi045 ceph-mon[30178]: osdmap e254: 8 total, 8 up, 8 in 2024-03-11T22:18:09.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:09 smithi045 ceph-mon[39715]: pgmap v438: 172 pgs: 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 170 active+clean; 711 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.65k op/s; 1233/34722 objects degraded (3.551%); 0 B/s, 79 objects/s recovering 2024-03-11T22:18:09.480 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:09 smithi045 ceph-mon[39715]: osdmap e254: 8 total, 8 up, 8 in 2024-03-11T22:18:09.554 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:09.586 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:09.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:09.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:09 smithi144 ceph-mon[33984]: pgmap v438: 172 pgs: 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 170 active+clean; 711 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.65k op/s; 1233/34722 objects degraded (3.551%); 0 B/s, 79 objects/s recovering 2024-03-11T22:18:09.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:09 smithi144 ceph-mon[33984]: osdmap e254: 8 total, 8 up, 8 in 2024-03-11T22:18:09.688 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:09.706 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:09.718 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:09.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:09.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:09.867 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:09.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:09.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:09.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:10.009 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:10.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:10.110 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:10.144 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:10.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:10.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:10.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:10.292 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:10.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:10.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:18:10.394 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:10.428 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:10.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:10.449 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:10 smithi045 ceph-mon[39715]: osdmap e255: 8 total, 8 up, 8 in 2024-03-11T22:18:10.449 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:10 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-11T22:18:10.449 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:10 smithi045 ceph-mon[30178]: osdmap e255: 8 total, 8 up, 8 in 2024-03-11T22:18:10.450 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:10 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-11T22:18:10.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:10.573 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:10.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:10.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:10 smithi144 ceph-mon[33984]: osdmap e255: 8 total, 8 up, 8 in 2024-03-11T22:18:10.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:10 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-03-11T22:18:10.676 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:10.712 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:10.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:10.814 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:10.848 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:10.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:10.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:10.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:10.987 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:10.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:11.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:11.126 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:11.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:11.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:11.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:11.272 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:11.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:11.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:11.411 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:11.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:11.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:11.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:11.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:11 smithi045 ceph-mon[30178]: pgmap v440: 171 pgs: 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 169 active+clean; 711 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 403 op/s; 1233/34719 objects degraded (3.551%); 0 B/s, 77 objects/s recovering 2024-03-11T22:18:11.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:11 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]': finished 2024-03-11T22:18:11.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:11 smithi045 ceph-mon[30178]: osdmap e256: 8 total, 8 up, 8 in 2024-03-11T22:18:11.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:11 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 1233/34719 objects degraded (3.551%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:11.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:11 smithi045 ceph-mon[39715]: pgmap v440: 171 pgs: 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 169 active+clean; 711 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 403 op/s; 1233/34719 objects degraded (3.551%); 0 B/s, 77 objects/s recovering 2024-03-11T22:18:11.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:11 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]': finished 2024-03-11T22:18:11.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:11 smithi045 ceph-mon[39715]: osdmap e256: 8 total, 8 up, 8 in 2024-03-11T22:18:11.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:11 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 1233/34719 objects degraded (3.551%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:11.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:11.553 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:11.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:11.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:11 smithi144 ceph-mon[33984]: pgmap v440: 171 pgs: 1 active+recovering+undersized+degraded+remapped, 1 clean+premerge+peered, 169 active+clean; 711 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 403 op/s; 1233/34719 objects degraded (3.551%); 0 B/s, 77 objects/s recovering 2024-03-11T22:18:11.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:11 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "23"}]': finished 2024-03-11T22:18:11.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:11 smithi144 ceph-mon[33984]: osdmap e256: 8 total, 8 up, 8 in 2024-03-11T22:18:11.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:11 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 1233/34719 objects degraded (3.551%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:11.654 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:11.685 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:11.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:11.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:11.787 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:11.816 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:11.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:11.918 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:11.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:11.946 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:11.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:12.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:12.081 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:12.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:12.183 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:12.216 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:12.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:12.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:12.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:12.351 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:12.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:12.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:12.453 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:12.489 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:12.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:12.573 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:12 smithi045 ceph-mon[30178]: osdmap e257: 8 total, 8 up, 8 in 2024-03-11T22:18:12.573 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:12 smithi045 ceph-mon[39715]: osdmap e257: 8 total, 8 up, 8 in 2024-03-11T22:18:12.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:12.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:12 smithi144 ceph-mon[33984]: osdmap e257: 8 total, 8 up, 8 in 2024-03-11T22:18:12.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:12.630 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:12.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:12.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:12.782 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:12.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:12.886 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:12.919 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:12.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:13.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:13.031 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:18:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:18:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:18:13.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:13.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:13.146 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:13.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:13.187 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:13.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:13.288 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:13.322 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:13.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:13.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:13.424 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:13.456 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:13.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:13.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:13.558 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:13.589 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:13.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:13.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:13 smithi144 ceph-mon[33984]: pgmap v443: 171 pgs: 1 active+recovering+undersized+degraded+remapped, 170 active+clean; 306 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 856/15258 objects degraded (5.610%); 0 B/s, 94 objects/s recovering 2024-03-11T22:18:13.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:13 smithi144 ceph-mon[33984]: osdmap e258: 8 total, 8 up, 8 in 2024-03-11T22:18:13.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:13.729 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:13.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:13.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:13 smithi045 ceph-mon[39715]: pgmap v443: 171 pgs: 1 active+recovering+undersized+degraded+remapped, 170 active+clean; 306 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 856/15258 objects degraded (5.610%); 0 B/s, 94 objects/s recovering 2024-03-11T22:18:13.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:13 smithi045 ceph-mon[39715]: osdmap e258: 8 total, 8 up, 8 in 2024-03-11T22:18:13.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:13 smithi045 ceph-mon[30178]: pgmap v443: 171 pgs: 1 active+recovering+undersized+degraded+remapped, 170 active+clean; 306 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 856/15258 objects degraded (5.610%); 0 B/s, 94 objects/s recovering 2024-03-11T22:18:13.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:13 smithi045 ceph-mon[30178]: osdmap e258: 8 total, 8 up, 8 in 2024-03-11T22:18:13.831 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:13.864 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:13.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:13.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:13.966 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:14.001 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:14.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:14.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:14.104 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:14.131 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:14.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:14.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:14.232 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:14.264 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:14.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:14.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:14.366 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:14.389 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:14.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:14.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:14.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:14.520 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:14.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:14.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:14.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:14 smithi144 ceph-mon[33984]: osdmap e259: 8 total, 8 up, 8 in 2024-03-11T22:18:14.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:14 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-11T22:18:14.622 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:14.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:14.661 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:14.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:14.762 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:14.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:14 smithi045 ceph-mon[39715]: osdmap e259: 8 total, 8 up, 8 in 2024-03-11T22:18:14.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:14 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-11T22:18:14.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:14 smithi045 ceph-mon[30178]: osdmap e259: 8 total, 8 up, 8 in 2024-03-11T22:18:14.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:14 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-03-11T22:18:14.785 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:14.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:14.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:14.887 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:14.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:14.923 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:14.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:14.942 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Removed 124115 objects 2024-03-11T22:18:14.942 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Clean up completed and total clean up time :102.602 2024-03-11T22:18:15.024 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:15.048 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:15.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:15.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:15.149 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:15.173 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:15.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:15.275 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:15.297 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:15.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:15.399 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:15.422 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:15.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:15.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:15.524 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:15.547 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:15.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:15.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:15.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:15 smithi144 ceph-mon[33984]: pgmap v446: 170 pgs: 1 active+recovering+undersized+degraded+remapped, 169 active+clean; 306 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 856/15258 objects degraded (5.610%); 128 KiB/s, 94 objects/s recovering 2024-03-11T22:18:15.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]': finished 2024-03-11T22:18:15.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:15 smithi144 ceph-mon[33984]: osdmap e260: 8 total, 8 up, 8 in 2024-03-11T22:18:15.650 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:15.673 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:15.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:15.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:15.774 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:15.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:15 smithi045 ceph-mon[30178]: pgmap v446: 170 pgs: 1 active+recovering+undersized+degraded+remapped, 169 active+clean; 306 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 856/15258 objects degraded (5.610%); 128 KiB/s, 94 objects/s recovering 2024-03-11T22:18:15.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]': finished 2024-03-11T22:18:15.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:15 smithi045 ceph-mon[30178]: osdmap e260: 8 total, 8 up, 8 in 2024-03-11T22:18:15.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:15 smithi045 ceph-mon[39715]: pgmap v446: 170 pgs: 1 active+recovering+undersized+degraded+remapped, 169 active+clean; 306 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 856/15258 objects degraded (5.610%); 128 KiB/s, 94 objects/s recovering 2024-03-11T22:18:15.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "22"}]': finished 2024-03-11T22:18:15.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:15 smithi045 ceph-mon[39715]: osdmap e260: 8 total, 8 up, 8 in 2024-03-11T22:18:15.797 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:15.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:15.899 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:15.922 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:15.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:16.025 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:16.049 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:16.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:16.151 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:16.174 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:16.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:16.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:16.300 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:16.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:16.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:16.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:16.425 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:16.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:16.527 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:16.551 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:16.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:16.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:16.595 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:18:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:18:16.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:16 smithi144 ceph-mon[33984]: osdmap e261: 8 total, 8 up, 8 in 2024-03-11T22:18:16.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:18:16.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:16 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 856/15258 objects degraded (5.610%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:16.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:16 smithi144 ceph-mon[33984]: osdmap e262: 8 total, 8 up, 8 in 2024-03-11T22:18:16.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T22:18:16.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:16 smithi045 ceph-mon[30178]: osdmap e261: 8 total, 8 up, 8 in 2024-03-11T22:18:16.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:18:16.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:16 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 856/15258 objects degraded (5.610%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:16.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:16 smithi045 ceph-mon[30178]: osdmap e262: 8 total, 8 up, 8 in 2024-03-11T22:18:16.601 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T22:18:16.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:16 smithi045 ceph-mon[39715]: osdmap e261: 8 total, 8 up, 8 in 2024-03-11T22:18:16.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:18:16.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:16 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 856/15258 objects degraded (5.610%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:16.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:16 smithi045 ceph-mon[39715]: osdmap e262: 8 total, 8 up, 8 in 2024-03-11T22:18:16.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-11T22:18:16.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:16.678 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:16.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:16.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:16.780 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:16.804 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:16.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:16.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:16.906 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:16.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:16.931 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:16.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:17.032 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:17.056 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:17.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:17.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:17.158 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:17.182 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:17.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:17.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:17.238 INFO:teuthology.orchestra.run.smithi045.stderr:clear 5.11 pg_upmap mapping 2024-03-11T22:18:17.284 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:17.307 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:17.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:17.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:17.409 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:17.427 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:17.427 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[39715]: pgmap v450: 169 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 167 active+clean; 60 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 995 op/s; 856/3459 objects degraded (24.747%) 2024-03-11T22:18:17.427 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1080078271' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.11"}]: dispatch 2024-03-11T22:18:17.427 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.11"}]: dispatch 2024-03-11T22:18:17.428 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T22:18:17.428 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.11"}]': finished 2024-03-11T22:18:17.428 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[39715]: osdmap e263: 8 total, 8 up, 8 in 2024-03-11T22:18:17.428 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:17.428 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[30178]: pgmap v450: 169 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 167 active+clean; 60 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 995 op/s; 856/3459 objects degraded (24.747%) 2024-03-11T22:18:17.428 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1080078271' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.11"}]: dispatch 2024-03-11T22:18:17.428 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.11"}]: dispatch 2024-03-11T22:18:17.428 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T22:18:17.428 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.11"}]': finished 2024-03-11T22:18:17.428 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:17 smithi045 ceph-mon[30178]: osdmap e263: 8 total, 8 up, 8 in 2024-03-11T22:18:17.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:17.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:17.505 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-11T22:18:17.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-11T22:18:17.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:17.574 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:17.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:17.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:17.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:17 smithi144 ceph-mon[33984]: pgmap v450: 169 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 167 active+clean; 60 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 995 op/s; 856/3459 objects degraded (24.747%) 2024-03-11T22:18:17.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:17 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1080078271' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.11"}]: dispatch 2024-03-11T22:18:17.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:17 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.11"}]: dispatch 2024-03-11T22:18:17.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:17 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-11T22:18:17.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:17 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.11"}]': finished 2024-03-11T22:18:17.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:17 smithi144 ceph-mon[33984]: osdmap e263: 8 total, 8 up, 8 in 2024-03-11T22:18:17.630 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:17.632 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:17.675 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:17.699 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:17.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:17.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:17.800 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:17.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:17.825 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:17.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:17.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:17.950 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:17.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:18.052 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:18.077 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:18.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:18.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:18.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:18.202 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:18.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:18.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:18.236 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:18:18.304 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:18.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:18.331 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:18.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:18.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:18.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:18.455 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:18.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:18.556 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:18.584 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:18.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:18.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:18.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:18 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1164866458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:18:18.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:18 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:18:18.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:18 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1164866458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:18:18.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:18 smithi144 ceph-mon[33984]: osdmap e264: 8 total, 8 up, 8 in 2024-03-11T22:18:18.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:18.710 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:18.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:18.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:18.715 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:18 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1164866458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:18:18.715 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:18 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:18:18.715 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:18 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1164866458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:18:18.715 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:18 smithi045 ceph-mon[39715]: osdmap e264: 8 total, 8 up, 8 in 2024-03-11T22:18:18.715 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:18 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1164866458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:18:18.715 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:18 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:18:18.715 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:18 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1164866458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:18:18.716 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:18 smithi045 ceph-mon[30178]: osdmap e264: 8 total, 8 up, 8 in 2024-03-11T22:18:18.811 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:18.835 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:18.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:18.937 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:18.965 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:18.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:19.067 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:19.095 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:19.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:19.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:19.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:19.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:19.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:19.323 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:19.350 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:19.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:19.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:19.452 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:19.479 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:19.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:19.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:19.581 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:19.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:19 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:18:19.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:19 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:18:19.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:19 smithi144 ceph-mon[33984]: pgmap v453: 169 pgs: 1 active+undersized+remapped, 1 peering, 167 active+clean; 476 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s; 0 B/s, 217 objects/s recovering 2024-03-11T22:18:19.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:19 smithi144 ceph-mon[33984]: 5.a scrub starts 2024-03-11T22:18:19.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:19 smithi144 ceph-mon[33984]: 5.a scrub ok 2024-03-11T22:18:19.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:19 smithi144 ceph-mon[33984]: osdmap e265: 8 total, 8 up, 8 in 2024-03-11T22:18:19.605 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:19.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:19.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:19.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:19.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:19.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:19.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:19 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:18:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:19 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:18:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:19 smithi045 ceph-mon[39715]: pgmap v453: 169 pgs: 1 active+undersized+remapped, 1 peering, 167 active+clean; 476 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s; 0 B/s, 217 objects/s recovering 2024-03-11T22:18:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:19 smithi045 ceph-mon[39715]: 5.a scrub starts 2024-03-11T22:18:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:19 smithi045 ceph-mon[39715]: 5.a scrub ok 2024-03-11T22:18:19.777 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:19 smithi045 ceph-mon[39715]: osdmap e265: 8 total, 8 up, 8 in 2024-03-11T22:18:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:19 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:18:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:19 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:18:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:19 smithi045 ceph-mon[30178]: pgmap v453: 169 pgs: 1 active+undersized+remapped, 1 peering, 167 active+clean; 476 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s; 0 B/s, 217 objects/s recovering 2024-03-11T22:18:19.777 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:19 smithi045 ceph-mon[30178]: 5.a scrub starts 2024-03-11T22:18:19.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:19 smithi045 ceph-mon[30178]: 5.a scrub ok 2024-03-11T22:18:19.778 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:19 smithi045 ceph-mon[30178]: osdmap e265: 8 total, 8 up, 8 in 2024-03-11T22:18:19.832 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:19.860 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:19.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:19.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:19.985 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:19.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:19.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:20.087 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:20.115 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:20.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:20.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:20.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:20.242 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:20.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:20.344 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:20.372 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:20.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:20.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:20.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:20.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:20.500 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:20.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:20.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:20 smithi045 ceph-mon[30178]: 1.4 scrub starts 2024-03-11T22:18:20.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:20 smithi045 ceph-mon[30178]: 1.4 scrub ok 2024-03-11T22:18:20.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:20 smithi045 ceph-mon[30178]: 1.6 scrub starts 2024-03-11T22:18:20.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:20 smithi045 ceph-mon[30178]: 1.6 scrub ok 2024-03-11T22:18:20.508 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:20 smithi045 ceph-mon[30178]: osdmap e266: 8 total, 8 up, 8 in 2024-03-11T22:18:20.508 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:20 smithi045 ceph-mon[39715]: 1.4 scrub starts 2024-03-11T22:18:20.508 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:20 smithi045 ceph-mon[39715]: 1.4 scrub ok 2024-03-11T22:18:20.508 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:20 smithi045 ceph-mon[39715]: 1.6 scrub starts 2024-03-11T22:18:20.508 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:20 smithi045 ceph-mon[39715]: 1.6 scrub ok 2024-03-11T22:18:20.509 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:20 smithi045 ceph-mon[39715]: osdmap e266: 8 total, 8 up, 8 in 2024-03-11T22:18:20.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:20 smithi144 ceph-mon[33984]: 1.4 scrub starts 2024-03-11T22:18:20.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:20 smithi144 ceph-mon[33984]: 1.4 scrub ok 2024-03-11T22:18:20.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:20 smithi144 ceph-mon[33984]: 1.6 scrub starts 2024-03-11T22:18:20.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:20 smithi144 ceph-mon[33984]: 1.6 scrub ok 2024-03-11T22:18:20.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:20 smithi144 ceph-mon[33984]: osdmap e266: 8 total, 8 up, 8 in 2024-03-11T22:18:20.601 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:20.629 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:20.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:20.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:20.731 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:20.758 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:20.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:20.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:20.860 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:20.888 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:20.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:20.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:20.989 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:21.016 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:21.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:21.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:21.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:21.147 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:21.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:21.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:21.275 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:21.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:21.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:21.377 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:21.405 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:21.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:21.506 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:21.533 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:21.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:21.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:21.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:21 smithi144 ceph-mon[33984]: pgmap v456: 168 pgs: 1 active+undersized+remapped, 1 peering, 166 active+clean; 476 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 238 op/s; 0 B/s, 213 objects/s recovering 2024-03-11T22:18:21.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:21 smithi144 ceph-mon[33984]: 1.a scrub starts 2024-03-11T22:18:21.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:21 smithi144 ceph-mon[33984]: 1.a scrub ok 2024-03-11T22:18:21.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:21 smithi144 ceph-mon[33984]: 1.7 scrub starts 2024-03-11T22:18:21.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:21 smithi144 ceph-mon[33984]: 1.7 scrub ok 2024-03-11T22:18:21.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:21 smithi144 ceph-mon[33984]: 1.b scrub starts 2024-03-11T22:18:21.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:21 smithi144 ceph-mon[33984]: 1.b scrub ok 2024-03-11T22:18:21.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:21 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:21.635 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:21.665 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:21.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:21.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:21.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:21.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[39715]: pgmap v456: 168 pgs: 1 active+undersized+remapped, 1 peering, 166 active+clean; 476 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 238 op/s; 0 B/s, 213 objects/s recovering 2024-03-11T22:18:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[39715]: 1.a scrub starts 2024-03-11T22:18:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[39715]: 1.a scrub ok 2024-03-11T22:18:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[39715]: 1.7 scrub starts 2024-03-11T22:18:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[39715]: 1.7 scrub ok 2024-03-11T22:18:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[39715]: 1.b scrub starts 2024-03-11T22:18:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[39715]: 1.b scrub ok 2024-03-11T22:18:21.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:21.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[30178]: pgmap v456: 168 pgs: 1 active+undersized+remapped, 1 peering, 166 active+clean; 476 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 238 op/s; 0 B/s, 213 objects/s recovering 2024-03-11T22:18:21.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[30178]: 1.a scrub starts 2024-03-11T22:18:21.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[30178]: 1.a scrub ok 2024-03-11T22:18:21.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[30178]: 1.7 scrub starts 2024-03-11T22:18:21.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[30178]: 1.7 scrub ok 2024-03-11T22:18:21.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[30178]: 1.b scrub starts 2024-03-11T22:18:21.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[30178]: 1.b scrub ok 2024-03-11T22:18:21.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:21 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-11T22:18:21.794 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:21.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:21.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:21.896 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:21.923 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:21.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:22.024 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:22.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:22.049 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:22.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:22.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:22.174 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:22.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:22.276 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:22.299 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:22.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:22.400 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:22.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:22.425 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:22.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:22.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:22.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:22.551 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:22.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:22.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:22 smithi144 ceph-mon[33984]: 6.16 scrub starts 2024-03-11T22:18:22.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:22 smithi144 ceph-mon[33984]: 6.16 scrub ok 2024-03-11T22:18:22.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:22 smithi144 ceph-mon[33984]: 1.9 scrub starts 2024-03-11T22:18:22.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:22 smithi144 ceph-mon[33984]: 1.9 scrub ok 2024-03-11T22:18:22.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:22 smithi144 ceph-mon[33984]: 1.8 scrub starts 2024-03-11T22:18:22.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:22 smithi144 ceph-mon[33984]: 1.8 scrub ok 2024-03-11T22:18:22.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:22 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:22.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:22.676 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:22.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:22.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:22.781 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:18:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:18:22.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[39715]: 6.16 scrub starts 2024-03-11T22:18:22.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[39715]: 6.16 scrub ok 2024-03-11T22:18:22.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[39715]: 1.9 scrub starts 2024-03-11T22:18:22.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[39715]: 1.9 scrub ok 2024-03-11T22:18:22.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[39715]: 1.8 scrub starts 2024-03-11T22:18:22.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[39715]: 1.8 scrub ok 2024-03-11T22:18:22.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:22.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[30178]: 6.16 scrub starts 2024-03-11T22:18:22.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[30178]: 6.16 scrub ok 2024-03-11T22:18:22.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[30178]: 1.9 scrub starts 2024-03-11T22:18:22.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[30178]: 1.9 scrub ok 2024-03-11T22:18:22.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[30178]: 1.8 scrub starts 2024-03-11T22:18:22.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[30178]: 1.8 scrub ok 2024-03-11T22:18:22.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:22 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:18:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:22.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:22.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:22.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:22.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:23.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:23.053 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:23.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:23.154 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:23.180 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:23.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:23.282 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:23.308 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:23.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:23.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:23.410 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:23.435 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:23.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:23.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[30178]: 3.12 scrub starts 2024-03-11T22:18:23.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[30178]: 3.12 scrub ok 2024-03-11T22:18:23.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[30178]: 1.c scrub starts 2024-03-11T22:18:23.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[30178]: 1.c scrub ok 2024-03-11T22:18:23.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[30178]: pgmap v457: 168 pgs: 168 active+clean; 476 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-03-11T22:18:23.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[30178]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-11T22:18:23.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:18:23.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:18:23.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[30178]: 6.7 scrub starts 2024-03-11T22:18:23.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[30178]: 6.7 scrub ok 2024-03-11T22:18:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[39715]: 3.12 scrub starts 2024-03-11T22:18:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[39715]: 3.12 scrub ok 2024-03-11T22:18:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[39715]: 1.c scrub starts 2024-03-11T22:18:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[39715]: 1.c scrub ok 2024-03-11T22:18:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[39715]: pgmap v457: 168 pgs: 168 active+clean; 476 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-03-11T22:18:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[39715]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-11T22:18:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:18:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:18:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[39715]: 6.7 scrub starts 2024-03-11T22:18:23.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:23 smithi045 ceph-mon[39715]: 6.7 scrub ok 2024-03-11T22:18:23.537 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:23.564 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:23.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:23 smithi144 ceph-mon[33984]: 3.12 scrub starts 2024-03-11T22:18:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:23 smithi144 ceph-mon[33984]: 3.12 scrub ok 2024-03-11T22:18:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:23 smithi144 ceph-mon[33984]: 1.c scrub starts 2024-03-11T22:18:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:23 smithi144 ceph-mon[33984]: 1.c scrub ok 2024-03-11T22:18:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:23 smithi144 ceph-mon[33984]: pgmap v457: 168 pgs: 168 active+clean; 476 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-03-11T22:18:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:23 smithi144 ceph-mon[33984]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-11T22:18:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:23 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:18:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:23 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:18:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:23 smithi144 ceph-mon[33984]: 6.7 scrub starts 2024-03-11T22:18:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:23 smithi144 ceph-mon[33984]: 6.7 scrub ok 2024-03-11T22:18:23.665 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:23.691 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:23.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:23.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:23.741 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:18:23.741 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:18:23.742 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-11T22:18:23.742 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T22:18:23.742 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T22:18:23.742 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-11T22:18:23.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:23.817 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:23.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:23.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:23.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:23.944 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:23.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:23.986 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:18:24.045 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:24.070 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:24.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:24.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:24.197 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:24.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:24.298 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:24.323 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:24.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:24.424 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:24.449 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:24.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:24.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[30178]: 4.16 scrub starts 2024-03-11T22:18:24.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[30178]: 4.16 scrub ok 2024-03-11T22:18:24.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[30178]: 1.e scrub starts 2024-03-11T22:18:24.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[30178]: 1.e scrub ok 2024-03-11T22:18:24.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[30178]: osdmap e267: 8 total, 8 up, 8 in 2024-03-11T22:18:24.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:18:24.474 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:18:24.475 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[30178]: 6.1 scrub starts 2024-03-11T22:18:24.475 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:24.475 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:24.475 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:18:24.475 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[30178]: 6.1 scrub ok 2024-03-11T22:18:24.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[39715]: 4.16 scrub starts 2024-03-11T22:18:24.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[39715]: 4.16 scrub ok 2024-03-11T22:18:24.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[39715]: 1.e scrub starts 2024-03-11T22:18:24.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[39715]: 1.e scrub ok 2024-03-11T22:18:24.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[39715]: osdmap e267: 8 total, 8 up, 8 in 2024-03-11T22:18:24.475 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:18:24.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:18:24.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[39715]: 6.1 scrub starts 2024-03-11T22:18:24.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:24.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:24.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:18:24.476 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:24 smithi045 ceph-mon[39715]: 6.1 scrub ok 2024-03-11T22:18:24.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:24.575 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:24.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:24.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:24.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:24 smithi144 ceph-mon[33984]: 4.16 scrub starts 2024-03-11T22:18:24.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:24 smithi144 ceph-mon[33984]: 4.16 scrub ok 2024-03-11T22:18:24.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:24 smithi144 ceph-mon[33984]: 1.e scrub starts 2024-03-11T22:18:24.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:24 smithi144 ceph-mon[33984]: 1.e scrub ok 2024-03-11T22:18:24.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:24 smithi144 ceph-mon[33984]: osdmap e267: 8 total, 8 up, 8 in 2024-03-11T22:18:24.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:24 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:18:24.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:24 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:18:24.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:24 smithi144 ceph-mon[33984]: 6.1 scrub starts 2024-03-11T22:18:24.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:18:24.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:18:24.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:18:24.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:24 smithi144 ceph-mon[33984]: 6.1 scrub ok 2024-03-11T22:18:24.603 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:18:24.624 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:18:24.643 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:18:24.663 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:18:24.677 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:24.683 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:18:24.702 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:24.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:24.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:24.704 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:18:24.726 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:18:24.747 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:18:24.747 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:18:24.747 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:18:24.747 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:18:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:18:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:18:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:18:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:18:24.748 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:18:24.803 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:24.827 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:24.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:24.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:24.832 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:18:24.832 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:18:24.832 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-03-11T22:18:24.832 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-03-11T22:18:24.833 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:18:24.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:24.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:24.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:24.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:25.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:25.079 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:25.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:25.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:25.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:25.204 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:25.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:25.306 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:25.330 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:25.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:25.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:25.457 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:25.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:25.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[39715]: 6.1b scrub starts 2024-03-11T22:18:25.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[39715]: 6.1b scrub ok 2024-03-11T22:18:25.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[39715]: pgmap v459: 168 pgs: 168 active+clean; 476 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:18:25.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[39715]: osdmap e268: 8 total, 8 up, 8 in 2024-03-11T22:18:25.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2300067948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:18:25.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[39715]: 3.1f scrub starts 2024-03-11T22:18:25.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[39715]: 3.1f scrub ok 2024-03-11T22:18:25.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[30178]: 6.1b scrub starts 2024-03-11T22:18:25.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[30178]: 6.1b scrub ok 2024-03-11T22:18:25.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[30178]: pgmap v459: 168 pgs: 168 active+clean; 476 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:18:25.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[30178]: osdmap e268: 8 total, 8 up, 8 in 2024-03-11T22:18:25.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2300067948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:18:25.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[30178]: 3.1f scrub starts 2024-03-11T22:18:25.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:25 smithi045 ceph-mon[30178]: 3.1f scrub ok 2024-03-11T22:18:25.558 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:25.582 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:25.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:25.655 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:18:25.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:25.722 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:25.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:25.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:25.750 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:25 smithi144 ceph-mon[33984]: 6.1b scrub starts 2024-03-11T22:18:25.750 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:25 smithi144 ceph-mon[33984]: 6.1b scrub ok 2024-03-11T22:18:25.750 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:25 smithi144 ceph-mon[33984]: pgmap v459: 168 pgs: 168 active+clean; 476 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:18:25.750 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:25 smithi144 ceph-mon[33984]: osdmap e268: 8 total, 8 up, 8 in 2024-03-11T22:18:25.750 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:25 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2300067948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:18:25.751 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:25 smithi144 ceph-mon[33984]: 3.1f scrub starts 2024-03-11T22:18:25.751 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:25 smithi144 ceph-mon[33984]: 3.1f scrub ok 2024-03-11T22:18:25.794 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:25.824 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:25.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:25.848 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:25.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:25.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:25.973 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:25.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:26.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:26.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:26.099 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:26.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:26.200 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:26.224 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:26.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:26.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:26.314 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:18:26.315 INFO:teuthology.orchestra.run.smithi144.stdout: "success": "" 2024-03-11T22:18:26.315 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:18:26.326 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:26.340 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T22:18:26.345 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:18:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:18:26.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:26.352 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:26.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:26.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:26.476 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:26.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:26.578 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:26.602 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:26.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:26.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:26 smithi045 ceph-mon[39715]: 5.7 scrub starts 2024-03-11T22:18:26.617 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:26 smithi045 ceph-mon[39715]: 5.7 scrub ok 2024-03-11T22:18:26.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:26 smithi045 ceph-mon[30178]: 5.7 scrub starts 2024-03-11T22:18:26.617 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:26 smithi045 ceph-mon[30178]: 5.7 scrub ok 2024-03-11T22:18:26.624 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:26 smithi144 ceph-mon[33984]: 5.7 scrub starts 2024-03-11T22:18:26.624 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:26 smithi144 ceph-mon[33984]: 5.7 scrub ok 2024-03-11T22:18:26.705 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:26.728 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:26.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:26.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:26.855 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:26.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:26.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:26.980 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:26.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:26.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:27.081 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:27.106 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:27.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:27.108 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:27.208 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:27.232 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:27.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:27.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:27.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:27.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:27.360 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:27.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:27.460 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:27.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:27.485 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:27.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:27.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:27.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:27 smithi045 ceph-mon[39715]: pgmap v461: 168 pgs: 168 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:18:27.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:27.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:27 smithi045 ceph-mon[30178]: pgmap v461: 168 pgs: 168 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:18:27.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:18:27.587 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:27.611 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:27.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:27.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:27.713 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:27.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:27.738 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:27.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:27.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:27.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:27.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:27 smithi144 ceph-mon[33984]: pgmap v461: 168 pgs: 168 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:18:27.862 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:27.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:27.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:27.989 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:27.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:27.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:28.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:28.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:28.119 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:28.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:28.220 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:28.245 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:28.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:28.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:28.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:28.372 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:28.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:28.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:28.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:28.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:28.499 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:28.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:28.600 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:28.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:28.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:28.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:28.725 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:28.746 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:28 smithi045 ceph-mon[39715]: 6.1f scrub starts 2024-03-11T22:18:28.746 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:28 smithi045 ceph-mon[39715]: 2.0 scrub starts 2024-03-11T22:18:28.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:28 smithi045 ceph-mon[39715]: 2.0 scrub ok 2024-03-11T22:18:28.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:28 smithi045 ceph-mon[39715]: 6.1f scrub ok 2024-03-11T22:18:28.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:28 smithi045 ceph-mon[39715]: 4.1f scrub starts 2024-03-11T22:18:28.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:28 smithi045 ceph-mon[39715]: 4.1f scrub ok 2024-03-11T22:18:28.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:28 smithi045 ceph-mon[30178]: 6.1f scrub starts 2024-03-11T22:18:28.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:28 smithi045 ceph-mon[30178]: 2.0 scrub starts 2024-03-11T22:18:28.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:28 smithi045 ceph-mon[30178]: 2.0 scrub ok 2024-03-11T22:18:28.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:28 smithi045 ceph-mon[30178]: 6.1f scrub ok 2024-03-11T22:18:28.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:28 smithi045 ceph-mon[30178]: 4.1f scrub starts 2024-03-11T22:18:28.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:28 smithi045 ceph-mon[30178]: 4.1f scrub ok 2024-03-11T22:18:28.751 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:28.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:28.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:28.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:28 smithi144 ceph-mon[33984]: 6.1f scrub starts 2024-03-11T22:18:28.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:28 smithi144 ceph-mon[33984]: 2.0 scrub starts 2024-03-11T22:18:28.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:28 smithi144 ceph-mon[33984]: 2.0 scrub ok 2024-03-11T22:18:28.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:28 smithi144 ceph-mon[33984]: 6.1f scrub ok 2024-03-11T22:18:28.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:28 smithi144 ceph-mon[33984]: 4.1f scrub starts 2024-03-11T22:18:28.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:28 smithi144 ceph-mon[33984]: 4.1f scrub ok 2024-03-11T22:18:28.852 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:28.869 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:28.877 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:28.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:28.979 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:29.003 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:29.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:29.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:29.105 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:29.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:29.131 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:29.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:29.232 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:29.257 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:29.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:29.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:29.359 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:29.370 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'unique_pool_1' removed 2024-03-11T22:18:29.389 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:29.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:29.490 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:29.514 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:29.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:29.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:29.617 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:29.640 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:29.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:29.695 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[39715]: 6.9 scrub starts 2024-03-11T22:18:29.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[39715]: 6.9 scrub ok 2024-03-11T22:18:29.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[39715]: pgmap v462: 168 pgs: 168 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:18:29.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[39715]: 4.d scrub starts 2024-03-11T22:18:29.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[39715]: 4.d scrub ok 2024-03-11T22:18:29.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[39715]: 5.1b scrub starts 2024-03-11T22:18:29.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[39715]: 5.1b scrub ok 2024-03-11T22:18:29.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[39715]: 6.1e scrub starts 2024-03-11T22:18:29.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[39715]: 6.1e scrub ok 2024-03-11T22:18:29.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3414198461' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:18:29.696 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:18:29.696 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[30178]: 6.9 scrub starts 2024-03-11T22:18:29.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[30178]: 6.9 scrub ok 2024-03-11T22:18:29.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[30178]: pgmap v462: 168 pgs: 168 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:18:29.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[30178]: 4.d scrub starts 2024-03-11T22:18:29.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[30178]: 4.d scrub ok 2024-03-11T22:18:29.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[30178]: 5.1b scrub starts 2024-03-11T22:18:29.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[30178]: 5.1b scrub ok 2024-03-11T22:18:29.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[30178]: 6.1e scrub starts 2024-03-11T22:18:29.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[30178]: 6.1e scrub ok 2024-03-11T22:18:29.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3414198461' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:18:29.697 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:29 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:18:29.742 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:29.768 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:29.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:29.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:29 smithi144 ceph-mon[33984]: 6.9 scrub starts 2024-03-11T22:18:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:29 smithi144 ceph-mon[33984]: 6.9 scrub ok 2024-03-11T22:18:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:29 smithi144 ceph-mon[33984]: pgmap v462: 168 pgs: 168 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:18:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:29 smithi144 ceph-mon[33984]: 4.d scrub starts 2024-03-11T22:18:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:29 smithi144 ceph-mon[33984]: 4.d scrub ok 2024-03-11T22:18:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:29 smithi144 ceph-mon[33984]: 5.1b scrub starts 2024-03-11T22:18:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:29 smithi144 ceph-mon[33984]: 5.1b scrub ok 2024-03-11T22:18:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:29 smithi144 ceph-mon[33984]: 6.1e scrub starts 2024-03-11T22:18:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:29 smithi144 ceph-mon[33984]: 6.1e scrub ok 2024-03-11T22:18:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:29 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3414198461' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:18:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:29 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:18:29.871 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:29.894 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:29.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:29.996 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:30.020 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:30.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:30.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:30.122 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:30.145 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:30.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:30.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:30.221 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:18:30.221 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-03-11T22:18:30.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-03-11T22:18:30.221 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:18:30.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.221 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205592 7.e 7:78d8296a:::benchmark_data_smithi045_83007_object87121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:30.221 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.361866+0000", 2024-03-11T22:18:30.221 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.85661010500002, 2024-03-11T22:18:30.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23581138800000001, 2024-03-11T22:18:30.221 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.221 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:30.221 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205592 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.361866+0000", 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.361866+0000", 2024-03-11T22:18:30.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.361867+0000", 2024-03-11T22:18:30.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.299e-06 2024-03-11T22:18:30.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.361870+0000", 2024-03-11T22:18:30.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4290000000000001e-06 2024-03-11T22:18:30.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.361880+0000", 2024-03-11T22:18:30.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2939999999999994e-06 2024-03-11T22:18:30.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.361887+0000", 2024-03-11T22:18:30.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7389999999999999e-06 2024-03-11T22:18:30.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.362227+0000", 2024-03-11T22:18:30.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00033996299999999998 2024-03-11T22:18:30.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.362357+0000", 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000130001 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.362492+0000", 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013507600000000001 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.444224+0000", 2024-03-11T22:18:30.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.081731649000000003 2024-03-11T22:18:30.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:30.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.444244+0000", 2024-03-11T22:18:30.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9990999999999999e-05 2024-03-11T22:18:30.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.597523+0000", 2024-03-11T22:18:30.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15327923099999999 2024-03-11T22:18:30.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.597595+0000", 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1928999999999995e-05 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.597677+0000", 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1786999999999995e-05 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.227 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205756 7.e 7:76db1325:::benchmark_data_smithi045_83007_object87285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.650961+0000", 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.56751444499997, 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.33229374, 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205756 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.650961+0000", 2024-03-11T22:18:30.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.650961+0000", 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.650963+0000", 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2759999999999999e-06 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.650967+0000", 2024-03-11T22:18:30.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8389999999999997e-06 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.650979+0000", 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1386000000000001e-05 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.650987+0000", 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0150000000000003e-06 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.911846+0000", 2024-03-11T22:18:30.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26085960400000002 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.911983+0000", 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000136756 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.912124+0000", 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014117599999999999 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:30.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915845+0000", 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037210559999999999 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916955+0000", 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0011094259999999999 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.983174+0000", 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.066219263 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:30.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.983213+0000", 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8688e-05 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.983255+0000", 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2255000000000001e-05 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40360 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.665971+0000", 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.55250466500001, 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.249558946, 2024-03-11T22:18:30.233 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.234 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:18:30.234 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.665971+0000", 2024-03-11T22:18:30.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.665971+0000", 2024-03-11T22:18:30.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.235 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.665972+0000", 2024-03-11T22:18:30.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8000000000000004e-07 2024-03-11T22:18:30.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.665981+0000", 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8950000000000004e-06 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.665991+0000", 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0689999999999999e-05 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.665999+0000", 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6359999999999994e-06 2024-03-11T22:18:30.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915503+0000", 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24950422899999999 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915512+0000", 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2849999999999995e-06 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915530+0000", 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8331000000000001e-05 2024-03-11T22:18:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40360 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.668874+0000", 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.54960204899999, 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.246778778, 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668874+0000", 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668874+0000", 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668874+0000", 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0100000000000004e-07 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668879+0000", 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5709999999999999e-06 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668889+0000", 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7000000000000003e-06 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668895+0000", 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7819999999999996e-06 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915554+0000", 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24665883599999999 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915561+0000", 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3099999999999997e-06 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915652+0000", 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1878000000000004e-05 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.241 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40361 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.668998+0000", 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.54947813699999, 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.246704801, 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668998+0000", 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668998+0000", 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668998+0000", 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1899999999999998e-07 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.669001+0000", 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.683e-06 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.669008+0000", 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3409999999999997e-06 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.669013+0000", 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9579999999999998e-06 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915676+0000", 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24666300999999999 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915688+0000", 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1669e-05 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915702+0000", 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4820999999999999e-05 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40363 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:18:30.245 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.675731+0000", 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.54274519699999, 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24011618100000001, 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.675731+0000", 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.675731+0000", 2024-03-11T22:18:30.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.675732+0000", 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.082e-06 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.675738+0000", 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6610000000000003e-06 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.675752+0000", 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3913e-05 2024-03-11T22:18:30.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.675761+0000", 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3769999999999996e-06 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915807+0000", 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.240046381 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915815+0000", 2024-03-11T22:18:30.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5739999999999996e-06 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915847+0000", 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1192999999999998e-05 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205779 7.e 7:7733b75f:::benchmark_data_smithi045_83007_object87308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.678063+0000", 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.54041307799997, 2024-03-11T22:18:30.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30589906300000003, 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205779 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678063+0000", 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678063+0000", 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678065+0000", 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0590000000000001e-06 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678071+0000", 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9499999999999998e-06 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678085+0000", 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4847e-05 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678097+0000", 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1124000000000001e-05 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916985+0000", 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23888874500000001 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.917115+0000", 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012928500000000001 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.917246+0000", 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013107799999999999 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.924476+0000", 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072302360000000001 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.930799+0000", 2024-03-11T22:18:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0063235510000000002 2024-03-11T22:18:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.983827+0000", 2024-03-11T22:18:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.053027815999999998 2024-03-11T22:18:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.983895+0000", 2024-03-11T22:18:30.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7939000000000001e-05 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.983962+0000", 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6433000000000006e-05 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40364 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.679313+0000", 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.53916253699998, 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23669196200000001, 2024-03-11T22:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679313+0000", 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679313+0000", 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679314+0000", 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2800000000000005e-07 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679323+0000", 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3190000000000001e-06 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679336+0000", 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2721999999999999e-05 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679345+0000", 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6379999999999993e-06 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915980+0000", 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23663551199999999 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915988+0000", 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7069999999999992e-06 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916005+0000", 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7136e-05 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40361 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.679621+0000", 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.53885439099997, 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23650580700000001, 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.259 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679621+0000", 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679621+0000", 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679622+0000", 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4899999999999995e-07 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679626+0000", 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0280000000000002e-06 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679636+0000", 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7520000000000008e-06 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679642+0000", 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.319e-06 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916039+0000", 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23639743099999999 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916046+0000", 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9759999999999998e-06 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916127+0000", 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0752000000000002e-05 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.262 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40365 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.683465+0000", 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.53501097499998, 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23270696499999999, 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.683465+0000", 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.683465+0000", 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.683466+0000", 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7699999999999992e-07 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.683472+0000", 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8889999999999999e-06 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.683484+0000", 2024-03-11T22:18:30.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2340999999999999e-05 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.683492+0000", 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9109999999999993e-06 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916152+0000", 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23266065399999999 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916160+0000", 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.328e-06 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916172+0000", 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1865000000000001e-05 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205921 7.e 7:7bd432cf:::benchmark_data_smithi045_83007_object87450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.033967+0000", 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.184508827, 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.242674745, 2024-03-11T22:18:30.266 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.267 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:30.267 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:30.267 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:30.267 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:30.267 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205921 2024-03-11T22:18:30.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.267 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.033967+0000", 2024-03-11T22:18:30.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.033967+0000", 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.033968+0000", 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.093e-06 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.033972+0000", 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5559999999999999e-06 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.033982+0000", 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0745e-05 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.033990+0000", 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5449999999999998e-06 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.034047+0000", 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7414999999999998e-05 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.034196+0000", 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000148465 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.034443+0000", 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000247366 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.038503+0000", 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040595529999999996 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.046484+0000", 2024-03-11T22:18:30.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0079812549999999996 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.276516+0000", 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23003189099999999 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.276576+0000", 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0503e-05 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.276642+0000", 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5357999999999993e-05 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.271 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205927 7.e 7:72966589:::benchmark_data_smithi045_83007_object87456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.041175+0000", 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.17730044199999, 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23561992600000001, 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205927 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041175+0000", 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041175+0000", 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041177+0000", 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5960000000000001e-06 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041182+0000", 2024-03-11T22:18:30.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0499999999999999e-06 2024-03-11T22:18:30.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041203+0000", 2024-03-11T22:18:30.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1063000000000001e-05 2024-03-11T22:18:30.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041213+0000", 2024-03-11T22:18:30.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0322e-05 2024-03-11T22:18:30.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.275 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:30.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041628+0000", 2024-03-11T22:18:30.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000414666 2024-03-11T22:18:30.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041762+0000", 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013364599999999999 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041905+0000", 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014386600000000001 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.045920+0000", 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040150120000000001 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.046605+0000", 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00068451600000000003 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.276686+0000", 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23008144899999999 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.276741+0000", 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4286999999999997e-05 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.276795+0000", 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4453000000000002e-05 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205934 7.e 7:79ae0710:::benchmark_data_smithi045_83007_object87463:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.051197+0000", 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.16727897599998, 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26231636600000002, 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:30.279 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205934 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.051197+0000", 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.051197+0000", 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.051198+0000", 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.389e-06 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.051203+0000", 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8740000000000003e-06 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.051215+0000", 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2131e-05 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.051223+0000", 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1839999999999999e-06 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.272457+0000", 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22123373399999999 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.272609+0000", 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000152075 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.272782+0000", 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017291799999999999 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.280086+0000", 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0073040580000000004 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.306915+0000", 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.026828538999999998 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:30.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313321+0000", 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0064061220000000002 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313453+0000", 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013265299999999999 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313513+0000", 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9688999999999997e-05 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.284 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205936 7.e 7:7d99f5bd:::benchmark_data_smithi045_83007_object87465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.054435+0000", 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.16404044900003, 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.259277386, 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205936 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.054435+0000", 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.054435+0000", 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.054437+0000", 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6249999999999999e-06 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.054442+0000", 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2349999999999999e-06 2024-03-11T22:18:30.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.054455+0000", 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2476999999999999e-05 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.054463+0000", 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4819999999999994e-06 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.280116+0000", 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22565284499999999 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.280240+0000", 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000124079 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.280367+0000", 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000126608 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.293440+0000", 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013073342999999999 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.306951+0000", 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.01351135 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313567+0000", 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.006616116 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313648+0000", 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0630999999999994e-05 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313713+0000", 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4595000000000001e-05 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205940 7.e 7:7935b90f:::benchmark_data_smithi045_83007_object87469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.055176+0000", 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 284.16329999999999, 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25869365, 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:30.290 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205940 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.055176+0000", 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.055176+0000", 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.055176+0000", 2024-03-11T22:18:30.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3000000000000001e-07 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.055179+0000", 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2200000000000001e-06 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.055186+0000", 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8349999999999998e-06 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.055192+0000", 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4999999999999999e-06 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.288665+0000", 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.233473074 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.288787+0000", 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012224 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.288958+0000", 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000171206 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.293503+0000", 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0045447270000000001 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.306991+0000", 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013488348000000001 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313721+0000", 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0067295289999999997 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:30.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313799+0000", 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7831999999999996e-05 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313869+0000", 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0709000000000004e-05 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206080 7.e 7:70863e2d:::benchmark_data_smithi045_83007_object87609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.349294+0000", 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 283.86918132599999, 2024-03-11T22:18:30.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25670769999999998, 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206080 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.349294+0000", 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.349294+0000", 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.349295+0000", 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7299999999999995e-07 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.349298+0000", 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.298e-06 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.349306+0000", 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0439999999999996e-06 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.349311+0000", 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0359999999999997e-06 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.519694+0000", 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17038214800000001 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.519821+0000", 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012704799999999999 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.519969+0000", 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014822699999999999 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.524895+0000", 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0049257370000000003 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.533760+0000", 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0088654259999999992 2024-03-11T22:18:30.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.605877+0000", 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.072117052000000001 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.605939+0000", 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2008000000000007e-05 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.606002+0000", 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3003000000000006e-05 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.300 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206085 7.e 7:7809afd0:::benchmark_data_smithi045_83007_object87614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.361807+0000", 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 283.856669114, 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.246123166, 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206085 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.361807+0000", 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.361807+0000", 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.361808+0000", 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3689999999999999e-06 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.361812+0000", 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1710000000000004e-06 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.361822+0000", 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7969999999999995e-06 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.361830+0000", 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6199999999999999e-06 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.524930+0000", 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16310034100000001 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.525065+0000", 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013499200000000001 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.525212+0000", 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014669300000000001 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.533369+0000", 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0081575659999999998 2024-03-11T22:18:30.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.533765+0000", 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00039629299999999998 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.607806+0000", 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.074040410000000001 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.607875+0000", 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8998000000000006e-05 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.607930+0000", 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4916000000000001e-05 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206224 7.e 7:7359e978:::benchmark_data_smithi045_83007_object87753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.663615+0000", 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 283.55486043000002, 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25281374200000001, 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:30.306 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206224 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.663615+0000", 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.663615+0000", 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.663617+0000", 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2890000000000001e-06 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.663621+0000", 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.685e-06 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.663632+0000", 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0957e-05 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.663641+0000", 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7979999999999995e-06 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.760335+0000", 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.096694374999999999 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.760460+0000", 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012459299999999999 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.760587+0000", 2024-03-11T22:18:30.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012663900000000001 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.773226+0000", 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012639188000000001 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.773239+0000", 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3212e-05 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.916329+0000", 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.143089889 2024-03-11T22:18:30.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.916378+0000", 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9440000000000001e-05 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.916429+0000", 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0677000000000003e-05 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206230 7.e 7:7bd288ae:::benchmark_data_smithi045_83007_object87759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.670446+0000", 2024-03-11T22:18:30.311 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 283.54802981, 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25676780700000001, 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206230 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.670446+0000", 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.670446+0000", 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.670447+0000", 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.265e-06 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.670452+0000", 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0050000000000004e-06 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.670467+0000", 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4878000000000001e-05 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.670475+0000", 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0010000000000001e-06 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.766547+0000", 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.096072195999999999 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.766667+0000", 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119776 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.766791+0000", 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000123499 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.773354+0000", 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065630130000000004 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.773385+0000", 2024-03-11T22:18:30.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1952000000000002e-05 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.926976+0000", 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15359084000000001 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.927123+0000", 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014644199999999999 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.927214+0000", 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0939999999999993e-05 2024-03-11T22:18:30.316 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206249 7.e 7:79d284c3:::benchmark_data_smithi045_83007_object87778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.695778+0000", 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 283.52269788299998, 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23172493399999999, 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206249 2024-03-11T22:18:30.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.695778+0000", 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.695778+0000", 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.695780+0000", 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0619999999999999e-06 2024-03-11T22:18:30.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.695787+0000", 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8789999999999997e-06 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.695802+0000", 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5362999999999998e-05 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.695818+0000", 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6124999999999999e-05 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.769867+0000", 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.074049078000000004 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.770014+0000", 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014628500000000001 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.770137+0000", 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000122922 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.773419+0000", 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032821349999999998 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.783366+0000", 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0099476949999999995 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.927268+0000", 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14390134299999999 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.321 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:30.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.927431+0000", 2024-03-11T22:18:30.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016337400000000001 2024-03-11T22:18:30.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:30.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:30.322 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:30.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.927503+0000", 2024-03-11T22:18:30.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1673000000000003e-05 2024-03-11T22:18:30.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.322 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.322 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:30.322 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:30.322 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:18:30.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:30.323 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:30.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:30.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:30.448 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:30.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[30178]: 3.1 scrub starts 2024-03-11T22:18:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[30178]: 3.1 scrub ok 2024-03-11T22:18:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[30178]: 6.11 scrub starts 2024-03-11T22:18:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[30178]: 6.11 scrub ok 2024-03-11T22:18:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:18:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[30178]: osdmap e269: 8 total, 8 up, 8 in 2024-03-11T22:18:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[30178]: 3.1c scrub starts 2024-03-11T22:18:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[30178]: 3.1c scrub ok 2024-03-11T22:18:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:18:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:18:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[39715]: 3.1 scrub starts 2024-03-11T22:18:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[39715]: 3.1 scrub ok 2024-03-11T22:18:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[39715]: 6.11 scrub starts 2024-03-11T22:18:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[39715]: 6.11 scrub ok 2024-03-11T22:18:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:18:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[39715]: osdmap e269: 8 total, 8 up, 8 in 2024-03-11T22:18:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[39715]: 3.1c scrub starts 2024-03-11T22:18:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[39715]: 3.1c scrub ok 2024-03-11T22:18:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:18:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:18:30.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:30.573 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:30.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:30.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:30.699 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:30.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:30.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:30.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:30.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:30.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:30.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:30 smithi144 ceph-mon[33984]: 3.1 scrub starts 2024-03-11T22:18:30.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:30 smithi144 ceph-mon[33984]: 3.1 scrub ok 2024-03-11T22:18:30.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:30 smithi144 ceph-mon[33984]: 6.11 scrub starts 2024-03-11T22:18:30.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:30 smithi144 ceph-mon[33984]: 6.11 scrub ok 2024-03-11T22:18:30.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:30 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:18:30.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:30 smithi144 ceph-mon[33984]: osdmap e269: 8 total, 8 up, 8 in 2024-03-11T22:18:30.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:30 smithi144 ceph-mon[33984]: 3.1c scrub starts 2024-03-11T22:18:30.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:30 smithi144 ceph-mon[33984]: 3.1c scrub ok 2024-03-11T22:18:30.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:18:30.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:18:30.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:30.949 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:30.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:30.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:31.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:31.074 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:31.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:31.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:31.176 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:31.204 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:31.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:31.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:31.306 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:31.330 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:31.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:31.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:31.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:31.455 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:31.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:31.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:31.557 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:31.572 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T22:18:31.572 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T22:18:31.573 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-11T22:18:31.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-11T22:18:31.586 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:31.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:31.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:31.667 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:31 smithi045 ceph-mon[39715]: pgmap v464: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-03-11T22:18:31.667 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:31 smithi045 ceph-mon[39715]: 3.14 scrub starts 2024-03-11T22:18:31.667 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:31 smithi045 ceph-mon[39715]: 3.14 scrub ok 2024-03-11T22:18:31.667 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:31 smithi045 ceph-mon[39715]: osdmap e270: 8 total, 8 up, 8 in 2024-03-11T22:18:31.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:31 smithi045 ceph-mon[30178]: pgmap v464: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-03-11T22:18:31.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:31 smithi045 ceph-mon[30178]: 3.14 scrub starts 2024-03-11T22:18:31.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:31 smithi045 ceph-mon[30178]: 3.14 scrub ok 2024-03-11T22:18:31.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:31 smithi045 ceph-mon[30178]: osdmap e270: 8 total, 8 up, 8 in 2024-03-11T22:18:31.688 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:31.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:31.716 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:31.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:31.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:31.841 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:31.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:31.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:31.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:31 smithi144 ceph-mon[33984]: pgmap v464: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-03-11T22:18:31.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:31 smithi144 ceph-mon[33984]: 3.14 scrub starts 2024-03-11T22:18:31.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:31 smithi144 ceph-mon[33984]: 3.14 scrub ok 2024-03-11T22:18:31.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:31 smithi144 ceph-mon[33984]: osdmap e270: 8 total, 8 up, 8 in 2024-03-11T22:18:31.937 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:18:31.937 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:18:31.938 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-11T22:18:31.938 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T22:18:31.938 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T22:18:31.938 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-11T22:18:31.943 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:31.971 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:31.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:31.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:32.073 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:32.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:32.098 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:32.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:32.199 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:32.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:32.229 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:32.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:32.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:32.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:32.356 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:32.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:32.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T22:18:32.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:32.513 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:32.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:32.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:32.615 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:32.643 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:32.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:32.745 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:32.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:32.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:32.781 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:18:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:18:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:18:32.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:32 smithi045 ceph-mon[39715]: 5.1d scrub starts 2024-03-11T22:18:32.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:32 smithi045 ceph-mon[39715]: 5.1d scrub ok 2024-03-11T22:18:32.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:32 smithi045 ceph-mon[39715]: 2.1 scrub starts 2024-03-11T22:18:32.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:32 smithi045 ceph-mon[39715]: 2.1 scrub ok 2024-03-11T22:18:32.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:32 smithi045 ceph-mon[30178]: 5.1d scrub starts 2024-03-11T22:18:32.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:32 smithi045 ceph-mon[30178]: 5.1d scrub ok 2024-03-11T22:18:32.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:32 smithi045 ceph-mon[30178]: 2.1 scrub starts 2024-03-11T22:18:32.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:32 smithi045 ceph-mon[30178]: 2.1 scrub ok 2024-03-11T22:18:32.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:32 smithi144 ceph-mon[33984]: 5.1d scrub starts 2024-03-11T22:18:32.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:32 smithi144 ceph-mon[33984]: 5.1d scrub ok 2024-03-11T22:18:32.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:32 smithi144 ceph-mon[33984]: 2.1 scrub starts 2024-03-11T22:18:32.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:32 smithi144 ceph-mon[33984]: 2.1 scrub ok 2024-03-11T22:18:32.870 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:32.898 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:32.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:32.959 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:32.999 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:33.028 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:33.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:33.129 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:33.156 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:33.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:33.258 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:33.285 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:33.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:33.387 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:33.413 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:33.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:33.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:33.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:33 smithi045 ceph-mon[39715]: pgmap v466: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:18:33.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:33 smithi045 ceph-mon[39715]: 3.d scrub starts 2024-03-11T22:18:33.478 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:33 smithi045 ceph-mon[39715]: 3.d scrub ok 2024-03-11T22:18:33.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:33 smithi045 ceph-mon[30178]: pgmap v466: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:18:33.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:33 smithi045 ceph-mon[30178]: 3.d scrub starts 2024-03-11T22:18:33.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:33 smithi045 ceph-mon[30178]: 3.d scrub ok 2024-03-11T22:18:33.515 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:33.542 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:33.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:33.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:33.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:33.668 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:33.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:33.680 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:33.770 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:33.798 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:33.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:33.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:33 smithi144 ceph-mon[33984]: pgmap v466: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:18:33.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:33 smithi144 ceph-mon[33984]: 3.d scrub starts 2024-03-11T22:18:33.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:33 smithi144 ceph-mon[33984]: 3.d scrub ok 2024-03-11T22:18:33.900 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:33.928 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:33.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:33.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:34.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:34.054 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:34.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:34.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:34.148 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:34.156 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:34.183 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:34.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:34.285 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:34.312 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:34.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:34.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:34.381 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:18:34.414 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:34.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:34.443 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:34.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:34.496 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:34 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2787116736' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:18:34.496 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:34 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:18:34.497 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:34 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2787116736' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:18:34.497 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:34 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:18:34.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:34.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:34.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:34.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:34.670 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:34.698 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:34.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:34.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:34.800 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:34.827 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:34.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:34.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:34.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:34 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2787116736' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:18:34.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:34 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:18:34.929 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:34.955 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:34.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:34.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:35.057 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:35.083 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:35.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:35.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:35.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:35.208 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:35.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:35.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:35.334 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:35.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:35.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:35.435 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:35.463 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:35.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:35.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:35.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[39715]: pgmap v467: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T22:18:35.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:18:35.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:18:35.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:18:35.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[39715]: osdmap e271: 8 total, 8 up, 8 in 2024-03-11T22:18:35.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[39715]: 3.13 scrub starts 2024-03-11T22:18:35.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[39715]: 3.13 scrub ok 2024-03-11T22:18:35.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[30178]: pgmap v467: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T22:18:35.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:18:35.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:18:35.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:18:35.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[30178]: osdmap e271: 8 total, 8 up, 8 in 2024-03-11T22:18:35.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[30178]: 3.13 scrub starts 2024-03-11T22:18:35.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:35 smithi045 ceph-mon[30178]: 3.13 scrub ok 2024-03-11T22:18:35.564 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:35.592 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:35.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:35.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:35.693 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:35.719 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:35.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:35.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:35.747 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:35 smithi144 ceph-mon[33984]: pgmap v467: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T22:18:35.747 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:35 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:18:35.747 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:35 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:18:35.747 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:35 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:18:35.747 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:35 smithi144 ceph-mon[33984]: osdmap e271: 8 total, 8 up, 8 in 2024-03-11T22:18:35.747 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:35 smithi144 ceph-mon[33984]: 3.13 scrub starts 2024-03-11T22:18:35.747 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:35 smithi144 ceph-mon[33984]: 3.13 scrub ok 2024-03-11T22:18:35.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:35.845 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:35.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:35.946 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:35.975 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:35.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:35.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:36.076 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:36.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:36.102 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:36.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:36.203 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:36.228 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:36.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:36.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:36.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:36.352 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:36.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:36.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:36.454 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:36.477 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:36.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:36.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:36.579 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:36.596 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:18:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:18:36.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:36 smithi144 ceph-mon[33984]: 6.17 scrub starts 2024-03-11T22:18:36.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:36 smithi144 ceph-mon[33984]: 6.17 scrub ok 2024-03-11T22:18:36.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:36 smithi144 ceph-mon[33984]: 3.a scrub starts 2024-03-11T22:18:36.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:36 smithi144 ceph-mon[33984]: 3.a scrub ok 2024-03-11T22:18:36.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:36 smithi144 ceph-mon[33984]: osdmap e272: 8 total, 8 up, 8 in 2024-03-11T22:18:36.602 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:36.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:36.675 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:36 smithi045 ceph-mon[39715]: 6.17 scrub starts 2024-03-11T22:18:36.675 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:36 smithi045 ceph-mon[39715]: 6.17 scrub ok 2024-03-11T22:18:36.675 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:36 smithi045 ceph-mon[39715]: 3.a scrub starts 2024-03-11T22:18:36.676 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:36 smithi045 ceph-mon[39715]: 3.a scrub ok 2024-03-11T22:18:36.676 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:36 smithi045 ceph-mon[39715]: osdmap e272: 8 total, 8 up, 8 in 2024-03-11T22:18:36.676 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:36 smithi045 ceph-mon[30178]: 6.17 scrub starts 2024-03-11T22:18:36.676 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:36 smithi045 ceph-mon[30178]: 6.17 scrub ok 2024-03-11T22:18:36.676 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:36 smithi045 ceph-mon[30178]: 3.a scrub starts 2024-03-11T22:18:36.676 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:36 smithi045 ceph-mon[30178]: 3.a scrub ok 2024-03-11T22:18:36.676 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:36 smithi045 ceph-mon[30178]: osdmap e272: 8 total, 8 up, 8 in 2024-03-11T22:18:36.704 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:36.727 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:36.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:36.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:36.852 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:36.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:36.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:36.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:36.976 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:36.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:37.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:37.103 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:37.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:37.204 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:37.227 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:37.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:37.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:37.329 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:37.352 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:37.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:37.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:37.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[39715]: 5.3 scrub starts 2024-03-11T22:18:37.397 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[39715]: 5.3 scrub ok 2024-03-11T22:18:37.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:18:37.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:18:37.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[39715]: 6.4 scrub starts 2024-03-11T22:18:37.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[39715]: 6.4 scrub ok 2024-03-11T22:18:37.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:37.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[39715]: pgmap v470: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:18:37.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[39715]: 2.2 scrub starts 2024-03-11T22:18:37.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[39715]: 1.5 scrub starts 2024-03-11T22:18:37.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[39715]: 1.5 scrub ok 2024-03-11T22:18:37.398 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[39715]: 2.2 scrub ok 2024-03-11T22:18:37.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[30178]: 5.3 scrub starts 2024-03-11T22:18:37.398 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[30178]: 5.3 scrub ok 2024-03-11T22:18:37.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:18:37.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:18:37.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[30178]: 6.4 scrub starts 2024-03-11T22:18:37.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[30178]: 6.4 scrub ok 2024-03-11T22:18:37.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:37.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[30178]: pgmap v470: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:18:37.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[30178]: 2.2 scrub starts 2024-03-11T22:18:37.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[30178]: 1.5 scrub starts 2024-03-11T22:18:37.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[30178]: 1.5 scrub ok 2024-03-11T22:18:37.399 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:37 smithi045 ceph-mon[30178]: 2.2 scrub ok 2024-03-11T22:18:37.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:37.477 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:37.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:37.579 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:37.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:37 smithi144 ceph-mon[33984]: 5.3 scrub starts 2024-03-11T22:18:37.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:37 smithi144 ceph-mon[33984]: 5.3 scrub ok 2024-03-11T22:18:37.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:37 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:18:37.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:37 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:18:37.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:37 smithi144 ceph-mon[33984]: 6.4 scrub starts 2024-03-11T22:18:37.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:37 smithi144 ceph-mon[33984]: 6.4 scrub ok 2024-03-11T22:18:37.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:37.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:37 smithi144 ceph-mon[33984]: pgmap v470: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:18:37.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:37 smithi144 ceph-mon[33984]: 2.2 scrub starts 2024-03-11T22:18:37.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:37 smithi144 ceph-mon[33984]: 1.5 scrub starts 2024-03-11T22:18:37.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:37 smithi144 ceph-mon[33984]: 1.5 scrub ok 2024-03-11T22:18:37.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:37 smithi144 ceph-mon[33984]: 2.2 scrub ok 2024-03-11T22:18:37.602 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:37.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:37.704 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:37.728 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:37.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:37.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:37.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:18:37.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:37.854 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:37.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:37.955 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:37.982 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:37.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:38.083 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:38.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:38.108 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:38.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:38.209 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:38.232 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:38.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:38.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:38.358 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:38.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:38.459 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:38.482 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:38.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:38.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:38.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:38 smithi045 ceph-mon[39715]: 4.19 scrub starts 2024-03-11T22:18:38.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:38 smithi045 ceph-mon[39715]: 4.19 scrub ok 2024-03-11T22:18:38.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:38 smithi045 ceph-mon[39715]: 6.e scrub starts 2024-03-11T22:18:38.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:38 smithi045 ceph-mon[39715]: 6.e scrub ok 2024-03-11T22:18:38.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:38 smithi045 ceph-mon[39715]: 5.f scrub starts 2024-03-11T22:18:38.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:38 smithi045 ceph-mon[39715]: 5.f scrub ok 2024-03-11T22:18:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:38 smithi045 ceph-mon[30178]: 4.19 scrub starts 2024-03-11T22:18:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:38 smithi045 ceph-mon[30178]: 4.19 scrub ok 2024-03-11T22:18:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:38 smithi045 ceph-mon[30178]: 6.e scrub starts 2024-03-11T22:18:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:38 smithi045 ceph-mon[30178]: 6.e scrub ok 2024-03-11T22:18:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:38 smithi045 ceph-mon[30178]: 5.f scrub starts 2024-03-11T22:18:38.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:38 smithi045 ceph-mon[30178]: 5.f scrub ok 2024-03-11T22:18:38.584 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:38.613 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:38.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:38.715 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:38.742 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:38.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:38.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:38.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:38 smithi144 ceph-mon[33984]: 4.19 scrub starts 2024-03-11T22:18:38.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:38 smithi144 ceph-mon[33984]: 4.19 scrub ok 2024-03-11T22:18:38.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:38 smithi144 ceph-mon[33984]: 6.e scrub starts 2024-03-11T22:18:38.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:38 smithi144 ceph-mon[33984]: 6.e scrub ok 2024-03-11T22:18:38.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:38 smithi144 ceph-mon[33984]: 5.f scrub starts 2024-03-11T22:18:38.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:38 smithi144 ceph-mon[33984]: 5.f scrub ok 2024-03-11T22:18:38.847 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:18:38.847 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T22:18:38.847 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:18:38.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:38.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:38.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:38.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:38.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:38.997 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:38.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:39.098 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:39.121 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:39.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:39.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:39.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:39.246 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:39.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:39.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:39.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:39.371 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:39.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:39.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:39.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:39.496 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:39.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:39.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:39.622 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:39.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:39.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[30178]: 3.1e scrub starts 2024-03-11T22:18:39.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[30178]: 3.1e scrub ok 2024-03-11T22:18:39.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[30178]: 3.4 scrub starts 2024-03-11T22:18:39.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[30178]: 3.4 scrub ok 2024-03-11T22:18:39.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[30178]: pgmap v471: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-11T22:18:39.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[30178]: 5.8 scrub starts 2024-03-11T22:18:39.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[30178]: 5.8 scrub ok 2024-03-11T22:18:39.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[30178]: 6.19 scrub starts 2024-03-11T22:18:39.664 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[30178]: 6.19 scrub ok 2024-03-11T22:18:39.664 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[39715]: 3.1e scrub starts 2024-03-11T22:18:39.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[39715]: 3.1e scrub ok 2024-03-11T22:18:39.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[39715]: 3.4 scrub starts 2024-03-11T22:18:39.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[39715]: 3.4 scrub ok 2024-03-11T22:18:39.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[39715]: pgmap v471: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-11T22:18:39.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[39715]: 5.8 scrub starts 2024-03-11T22:18:39.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[39715]: 5.8 scrub ok 2024-03-11T22:18:39.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[39715]: 6.19 scrub starts 2024-03-11T22:18:39.665 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:39 smithi045 ceph-mon[39715]: 6.19 scrub ok 2024-03-11T22:18:39.694 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:18:39.694 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:18:39.694 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-03-11T22:18:39.694 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:18:39.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:39.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:39.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:39.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:39 smithi144 ceph-mon[33984]: 3.1e scrub starts 2024-03-11T22:18:39.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:39 smithi144 ceph-mon[33984]: 3.1e scrub ok 2024-03-11T22:18:39.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:39 smithi144 ceph-mon[33984]: 3.4 scrub starts 2024-03-11T22:18:39.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:39 smithi144 ceph-mon[33984]: 3.4 scrub ok 2024-03-11T22:18:39.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:39 smithi144 ceph-mon[33984]: pgmap v471: 152 pgs: 152 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 729 B/s rd, 0 op/s 2024-03-11T22:18:39.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:39 smithi144 ceph-mon[33984]: 5.8 scrub starts 2024-03-11T22:18:39.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:39 smithi144 ceph-mon[33984]: 5.8 scrub ok 2024-03-11T22:18:39.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:39 smithi144 ceph-mon[33984]: 6.19 scrub starts 2024-03-11T22:18:39.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:39 smithi144 ceph-mon[33984]: 6.19 scrub ok 2024-03-11T22:18:39.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:39.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:39.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:39.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:39.975 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:40.003 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:40.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:40.105 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:40.131 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:40.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:40.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:40.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:40.256 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:40.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:40.358 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:40.384 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:40.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:40.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:40.511 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:40.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:40.613 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:40.637 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:40.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:40.739 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:40.763 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:40.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:40.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:40 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/681570923' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-11T22:18:40.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:40 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/681570923' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-11T22:18:40.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:40 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/681570923' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-11T22:18:40.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:40.888 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:40.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:40.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:40.989 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:41.013 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:41.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:41.022 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'unique_pool_2' created 2024-03-11T22:18:41.115 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:41.138 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:41.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:41.240 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:41.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:41.264 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:41.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:41.365 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:41.389 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:41.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:41.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:41.517 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:41.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:41.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:41.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[39715]: 3.7 scrub starts 2024-03-11T22:18:41.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[39715]: 3.7 scrub ok 2024-03-11T22:18:41.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[39715]: pgmap v472: 152 pgs: 152 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:18:41.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/681570923' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-11T22:18:41.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[39715]: osdmap e273: 8 total, 8 up, 8 in 2024-03-11T22:18:41.619 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:18:41.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:18:41.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[39715]: 3.c scrub starts 2024-03-11T22:18:41.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[39715]: 3.c scrub ok 2024-03-11T22:18:41.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[39715]: 3.11 scrub starts 2024-03-11T22:18:41.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[39715]: 3.11 scrub ok 2024-03-11T22:18:41.620 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[39715]: osdmap e274: 8 total, 8 up, 8 in 2024-03-11T22:18:41.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[30178]: 3.7 scrub starts 2024-03-11T22:18:41.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[30178]: 3.7 scrub ok 2024-03-11T22:18:41.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[30178]: pgmap v472: 152 pgs: 152 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:18:41.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/681570923' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-11T22:18:41.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[30178]: osdmap e273: 8 total, 8 up, 8 in 2024-03-11T22:18:41.620 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:18:41.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:18:41.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[30178]: 3.c scrub starts 2024-03-11T22:18:41.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[30178]: 3.c scrub ok 2024-03-11T22:18:41.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[30178]: 3.11 scrub starts 2024-03-11T22:18:41.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[30178]: 3.11 scrub ok 2024-03-11T22:18:41.621 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:41 smithi045 ceph-mon[30178]: osdmap e274: 8 total, 8 up, 8 in 2024-03-11T22:18:41.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:41.646 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:41.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:41.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-11T22:18:41.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:41.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:41.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:41.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:41 smithi144 ceph-mon[33984]: 3.7 scrub starts 2024-03-11T22:18:41.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:41 smithi144 ceph-mon[33984]: 3.7 scrub ok 2024-03-11T22:18:41.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:41 smithi144 ceph-mon[33984]: pgmap v472: 152 pgs: 152 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:18:41.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:41 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/681570923' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-11T22:18:41.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:41 smithi144 ceph-mon[33984]: osdmap e273: 8 total, 8 up, 8 in 2024-03-11T22:18:41.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:41 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:18:41.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:41 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:18:41.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:41 smithi144 ceph-mon[33984]: 3.c scrub starts 2024-03-11T22:18:41.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:41 smithi144 ceph-mon[33984]: 3.c scrub ok 2024-03-11T22:18:41.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:41 smithi144 ceph-mon[33984]: 3.11 scrub starts 2024-03-11T22:18:41.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:41 smithi144 ceph-mon[33984]: 3.11 scrub ok 2024-03-11T22:18:41.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:41 smithi144 ceph-mon[33984]: osdmap e274: 8 total, 8 up, 8 in 2024-03-11T22:18:41.872 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:41.887 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:41.897 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:41.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:41.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:41.998 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:42.022 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:42.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:42.124 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:42.148 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:42.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:42.249 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:42.273 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T22:18:42.274 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:42.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:42.376 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:42.401 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:42.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:42.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:42.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:42.526 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:42.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:42.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:42.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[39715]: 1.d scrub starts 2024-03-11T22:18:42.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[39715]: 1.d scrub ok 2024-03-11T22:18:42.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[39715]: 5.b deep-scrub starts 2024-03-11T22:18:42.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[39715]: 5.b deep-scrub ok 2024-03-11T22:18:42.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[39715]: 4.13 scrub starts 2024-03-11T22:18:42.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[39715]: 4.13 scrub ok 2024-03-11T22:18:42.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[39715]: 1.1 scrub starts 2024-03-11T22:18:42.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[39715]: 1.1 scrub ok 2024-03-11T22:18:42.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[39715]: osdmap e275: 8 total, 8 up, 8 in 2024-03-11T22:18:42.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[30178]: 1.d scrub starts 2024-03-11T22:18:42.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[30178]: 1.d scrub ok 2024-03-11T22:18:42.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[30178]: 5.b deep-scrub starts 2024-03-11T22:18:42.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[30178]: 5.b deep-scrub ok 2024-03-11T22:18:42.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[30178]: 4.13 scrub starts 2024-03-11T22:18:42.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[30178]: 4.13 scrub ok 2024-03-11T22:18:42.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[30178]: 1.1 scrub starts 2024-03-11T22:18:42.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[30178]: 1.1 scrub ok 2024-03-11T22:18:42.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-mon[30178]: osdmap e275: 8 total, 8 up, 8 in 2024-03-11T22:18:42.628 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:42.654 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:42.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:42.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:42.781 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:42.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:42.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:42.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:42 smithi144 ceph-mon[33984]: 1.d scrub starts 2024-03-11T22:18:42.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:42 smithi144 ceph-mon[33984]: 1.d scrub ok 2024-03-11T22:18:42.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:42 smithi144 ceph-mon[33984]: 5.b deep-scrub starts 2024-03-11T22:18:42.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:42 smithi144 ceph-mon[33984]: 5.b deep-scrub ok 2024-03-11T22:18:42.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:42 smithi144 ceph-mon[33984]: 4.13 scrub starts 2024-03-11T22:18:42.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:42 smithi144 ceph-mon[33984]: 4.13 scrub ok 2024-03-11T22:18:42.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:42 smithi144 ceph-mon[33984]: 1.1 scrub starts 2024-03-11T22:18:42.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:42 smithi144 ceph-mon[33984]: 1.1 scrub ok 2024-03-11T22:18:42.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:42 smithi144 ceph-mon[33984]: osdmap e275: 8 total, 8 up, 8 in 2024-03-11T22:18:42.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:42.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:42.912 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:18:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:18:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:18:43.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:43.031 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:43.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:43.119 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:18:43.119 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:18:43.119 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:18:43.119 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:18:43.119 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:18:43.119 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:18:43.119 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:18:43.120 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:18:43.120 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:18:43.120 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:18:43.120 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:18:43.120 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:18:43.120 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:18:43.120 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:18:43.120 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:18:43.120 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:18:43.134 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:43.157 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:43.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:43.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:43.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:43.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:43.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:43.384 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:43.410 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:43.412 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:43.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:43.514 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:43.542 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:43.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:43.643 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:43.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:43.667 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:43.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:43.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:43.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[30178]: 4.11 deep-scrub starts 2024-03-11T22:18:43.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[30178]: 4.11 deep-scrub ok 2024-03-11T22:18:43.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[30178]: pgmap v476: 168 pgs: 1 creating+activating, 13 creating+peering, 154 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-03-11T22:18:43.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[30178]: 4.1c scrub starts 2024-03-11T22:18:43.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[30178]: 4.1c scrub ok 2024-03-11T22:18:43.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1866587339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:18:43.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[30178]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:18:43.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[39715]: 4.11 deep-scrub starts 2024-03-11T22:18:43.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[39715]: 4.11 deep-scrub ok 2024-03-11T22:18:43.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[39715]: pgmap v476: 168 pgs: 1 creating+activating, 13 creating+peering, 154 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-03-11T22:18:43.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[39715]: 4.1c scrub starts 2024-03-11T22:18:43.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[39715]: 4.1c scrub ok 2024-03-11T22:18:43.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1866587339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:18:43.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:43 smithi045 ceph-mon[39715]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:18:43.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:43.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:43.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:43.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:43 smithi144 ceph-mon[33984]: 4.11 deep-scrub starts 2024-03-11T22:18:43.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:43 smithi144 ceph-mon[33984]: 4.11 deep-scrub ok 2024-03-11T22:18:43.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:43 smithi144 ceph-mon[33984]: pgmap v476: 168 pgs: 1 creating+activating, 13 creating+peering, 154 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-03-11T22:18:43.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:43 smithi144 ceph-mon[33984]: 4.1c scrub starts 2024-03-11T22:18:43.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:43 smithi144 ceph-mon[33984]: 4.1c scrub ok 2024-03-11T22:18:43.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:43 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1866587339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:18:43.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:43 smithi144 ceph-mon[33984]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:18:43.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:43.917 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:43.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:44.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:44.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:44.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:44.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:44.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:44.170 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:44.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:44.272 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:44.299 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:44.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:44.400 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:44.427 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:44.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:44.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:44.503 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:44 smithi045 ceph-mon[30178]: 1.14 deep-scrub starts 2024-03-11T22:18:44.504 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:44 smithi045 ceph-mon[30178]: 1.14 deep-scrub ok 2024-03-11T22:18:44.504 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:44 smithi045 ceph-mon[39715]: 1.14 deep-scrub starts 2024-03-11T22:18:44.504 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:44 smithi045 ceph-mon[39715]: 1.14 deep-scrub ok 2024-03-11T22:18:44.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:44.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:44.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:44.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:44.654 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:44.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:44.682 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:44.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:44.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:44.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:44.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:44.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:44 smithi144 ceph-mon[33984]: 1.14 deep-scrub starts 2024-03-11T22:18:44.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:44 smithi144 ceph-mon[33984]: 1.14 deep-scrub ok 2024-03-11T22:18:44.908 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:44.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:44.937 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:44.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:45.038 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:45.065 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:45.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:45.167 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:45.191 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:45.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:45.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:45.293 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:45.321 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:45.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:45.422 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:45.442 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:18:45.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T22:18:45.450 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:45.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:45.551 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:45.579 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:45.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:45.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[39715]: 3.1b deep-scrub starts 2024-03-11T22:18:45.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[39715]: 3.1b deep-scrub ok 2024-03-11T22:18:45.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[39715]: 5.c scrub starts 2024-03-11T22:18:45.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[39715]: 5.c scrub ok 2024-03-11T22:18:45.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[39715]: pgmap v477: 168 pgs: 1 creating+activating, 13 creating+peering, 154 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-03-11T22:18:45.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[39715]: 6.8 scrub starts 2024-03-11T22:18:45.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[39715]: 6.8 scrub ok 2024-03-11T22:18:45.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:18:45.602 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:18:45.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[30178]: 3.1b deep-scrub starts 2024-03-11T22:18:45.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[30178]: 3.1b deep-scrub ok 2024-03-11T22:18:45.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[30178]: 5.c scrub starts 2024-03-11T22:18:45.602 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[30178]: 5.c scrub ok 2024-03-11T22:18:45.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[30178]: pgmap v477: 168 pgs: 1 creating+activating, 13 creating+peering, 154 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-03-11T22:18:45.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[30178]: 6.8 scrub starts 2024-03-11T22:18:45.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[30178]: 6.8 scrub ok 2024-03-11T22:18:45.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:18:45.603 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:18:45.642 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:45.681 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:45.708 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:45.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:45.810 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:45.837 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:45.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:45 smithi144 ceph-mon[33984]: 3.1b deep-scrub starts 2024-03-11T22:18:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:45 smithi144 ceph-mon[33984]: 3.1b deep-scrub ok 2024-03-11T22:18:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:45 smithi144 ceph-mon[33984]: 5.c scrub starts 2024-03-11T22:18:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:45 smithi144 ceph-mon[33984]: 5.c scrub ok 2024-03-11T22:18:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:45 smithi144 ceph-mon[33984]: pgmap v477: 168 pgs: 1 creating+activating, 13 creating+peering, 154 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-03-11T22:18:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:45 smithi144 ceph-mon[33984]: 6.8 scrub starts 2024-03-11T22:18:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:45 smithi144 ceph-mon[33984]: 6.8 scrub ok 2024-03-11T22:18:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:18:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:18:45.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:18:45.939 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:45.965 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:45.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:46.067 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:46.094 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:46.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:46.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:46.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:46.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:46.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:46.321 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:46.348 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:46.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:46.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:46.450 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:46.475 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:46.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:46.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:46.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[39715]: 1.0 scrub starts 2024-03-11T22:18:46.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[39715]: 1.0 scrub ok 2024-03-11T22:18:46.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[39715]: 3.9 scrub starts 2024-03-11T22:18:46.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[39715]: 3.9 scrub ok 2024-03-11T22:18:46.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[39715]: 1.12 scrub starts 2024-03-11T22:18:46.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[39715]: 1.12 scrub ok 2024-03-11T22:18:46.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[39715]: osdmap e276: 8 total, 8 up, 8 in 2024-03-11T22:18:46.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[30178]: 1.0 scrub starts 2024-03-11T22:18:46.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[30178]: 1.0 scrub ok 2024-03-11T22:18:46.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[30178]: 3.9 scrub starts 2024-03-11T22:18:46.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[30178]: 3.9 scrub ok 2024-03-11T22:18:46.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[30178]: 1.12 scrub starts 2024-03-11T22:18:46.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[30178]: 1.12 scrub ok 2024-03-11T22:18:46.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:46 smithi045 ceph-mon[30178]: osdmap e276: 8 total, 8 up, 8 in 2024-03-11T22:18:46.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:46.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:46 smithi144 ceph-mon[33984]: 1.0 scrub starts 2024-03-11T22:18:46.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:46 smithi144 ceph-mon[33984]: 1.0 scrub ok 2024-03-11T22:18:46.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:46 smithi144 ceph-mon[33984]: 3.9 scrub starts 2024-03-11T22:18:46.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:46 smithi144 ceph-mon[33984]: 3.9 scrub ok 2024-03-11T22:18:46.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:46 smithi144 ceph-mon[33984]: 1.12 scrub starts 2024-03-11T22:18:46.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:46 smithi144 ceph-mon[33984]: 1.12 scrub ok 2024-03-11T22:18:46.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:46 smithi144 ceph-mon[33984]: osdmap e276: 8 total, 8 up, 8 in 2024-03-11T22:18:46.596 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:18:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:18:46.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:46.602 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:46.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:46.703 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:46.727 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:46.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:46.829 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:46.857 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:46.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:46.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:46.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:46.982 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:46.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:46.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:47.084 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:47.112 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:47.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:47.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:47.238 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:47.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:47.340 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:47.343 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:47.349 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:47.367 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:47.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:47.403 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-11T22:18:47.468 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:47.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:47.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:47.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:47.497 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:47.497 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:47 smithi045 ceph-mon[39715]: pgmap v479: 168 pgs: 1 creating+activating, 167 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-03-11T22:18:47.497 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:47 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3070991506' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:18:47.497 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:47 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:18:47.497 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:47.497 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:47 smithi045 ceph-mon[30178]: pgmap v479: 168 pgs: 1 creating+activating, 167 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-03-11T22:18:47.497 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3070991506' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:18:47.497 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:47 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:18:47.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:47.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:47.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:47 smithi144 ceph-mon[33984]: pgmap v479: 168 pgs: 1 creating+activating, 167 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 121 op/s 2024-03-11T22:18:47.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:47 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3070991506' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:18:47.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:47 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:18:47.618 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:47.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:47.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:47.720 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:47.749 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:47.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:47.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:47.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:47.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:47.976 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:48.005 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:48.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:48.057 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:18:48.057 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:18:48.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:48.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:48.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:48.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:48.259 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:48.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:48.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:48.361 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:48.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:48.385 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:48.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:48.486 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:48.515 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:48.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:48.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:48.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:48 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:18:48.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:48 smithi045 ceph-mon[30178]: osdmap e277: 8 total, 8 up, 8 in 2024-03-11T22:18:48.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:48 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:18:48.518 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:48 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:18:48.518 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:48 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:18:48.518 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:48 smithi045 ceph-mon[39715]: osdmap e277: 8 total, 8 up, 8 in 2024-03-11T22:18:48.518 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:48 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:18:48.518 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:48 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:18:48.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:48.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:48.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:48.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:48.743 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:48.766 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:48.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:48.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:48 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:18:48.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:48 smithi144 ceph-mon[33984]: osdmap e277: 8 total, 8 up, 8 in 2024-03-11T22:18:48.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:48 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:18:48.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:48 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:18:48.868 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:48.895 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:48.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:48.997 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:49.025 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:49.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:49.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:49.127 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:49.154 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:49.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:49.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:49.281 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:49.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:49.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:49.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:49.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:49.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:49.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:49 smithi045 ceph-mon[39715]: 6.c scrub starts 2024-03-11T22:18:49.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:49 smithi045 ceph-mon[39715]: 6.c scrub ok 2024-03-11T22:18:49.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:49 smithi045 ceph-mon[39715]: pgmap v481: 168 pgs: 168 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-03-11T22:18:49.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:49 smithi045 ceph-mon[39715]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:18:49.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:49 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:18:49.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:49 smithi045 ceph-mon[39715]: osdmap e278: 8 total, 8 up, 8 in 2024-03-11T22:18:49.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:49 smithi045 ceph-mon[30178]: 6.c scrub starts 2024-03-11T22:18:49.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:49 smithi045 ceph-mon[30178]: 6.c scrub ok 2024-03-11T22:18:49.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:49 smithi045 ceph-mon[30178]: pgmap v481: 168 pgs: 168 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-03-11T22:18:49.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:49 smithi045 ceph-mon[30178]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:18:49.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:49 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:18:49.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:49 smithi045 ceph-mon[30178]: osdmap e278: 8 total, 8 up, 8 in 2024-03-11T22:18:49.510 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:49.538 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:49.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:49.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:49.664 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:49.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:49.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:49.789 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:49.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:49.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:49 smithi144 ceph-mon[33984]: 6.c scrub starts 2024-03-11T22:18:49.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:49 smithi144 ceph-mon[33984]: 6.c scrub ok 2024-03-11T22:18:49.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:49 smithi144 ceph-mon[33984]: pgmap v481: 168 pgs: 168 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-03-11T22:18:49.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:49 smithi144 ceph-mon[33984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:18:49.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:49 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:18:49.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:49 smithi144 ceph-mon[33984]: osdmap e278: 8 total, 8 up, 8 in 2024-03-11T22:18:49.891 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:49.919 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:49.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:50.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:50.046 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:50.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:50.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:50.172 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:50.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:50.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:50.274 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:50.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:50.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:50.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:50.423 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:50.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:50.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:50.524 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:50.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:50.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:50.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:50.650 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:50.678 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:50.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:50.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:50.779 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:50.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:50.807 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:50.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:50.908 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:50.934 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:50.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:50.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:51.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:51.061 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:51.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:51.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:51.163 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:51.188 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:51.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:51.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:51.312 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:51.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:51.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:51.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:51.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:51.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:51.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:51.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:51.562 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:51.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:51.662 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:51.691 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:51.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:51.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:51 smithi045 ceph-mon[39715]: pgmap v483: 168 pgs: 168 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-03-11T22:18:51.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:51 smithi045 ceph-mon[39715]: 5.1f scrub starts 2024-03-11T22:18:51.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:51 smithi045 ceph-mon[39715]: 5.1f scrub ok 2024-03-11T22:18:51.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:51 smithi045 ceph-mon[39715]: 6.d scrub starts 2024-03-11T22:18:51.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:51 smithi045 ceph-mon[39715]: 6.d scrub ok 2024-03-11T22:18:51.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:51 smithi045 ceph-mon[30178]: pgmap v483: 168 pgs: 168 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-03-11T22:18:51.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:51 smithi045 ceph-mon[30178]: 5.1f scrub starts 2024-03-11T22:18:51.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:51 smithi045 ceph-mon[30178]: 5.1f scrub ok 2024-03-11T22:18:51.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:51 smithi045 ceph-mon[30178]: 6.d scrub starts 2024-03-11T22:18:51.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:51 smithi045 ceph-mon[30178]: 6.d scrub ok 2024-03-11T22:18:51.792 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:51.819 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:51.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:51.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:51 smithi144 ceph-mon[33984]: pgmap v483: 168 pgs: 168 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-03-11T22:18:51.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:51 smithi144 ceph-mon[33984]: 5.1f scrub starts 2024-03-11T22:18:51.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:51 smithi144 ceph-mon[33984]: 5.1f scrub ok 2024-03-11T22:18:51.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:51 smithi144 ceph-mon[33984]: 6.d scrub starts 2024-03-11T22:18:51.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:51 smithi144 ceph-mon[33984]: 6.d scrub ok 2024-03-11T22:18:51.921 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:51.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T22:18:51.947 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:51.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:51.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:52.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:52.072 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:52.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:52.174 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:52.197 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:52.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:52.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:52.323 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:52.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:52.503 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:52.531 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:52.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:52.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:52 smithi045 ceph-mon[39715]: 4.1b scrub starts 2024-03-11T22:18:52.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:52 smithi045 ceph-mon[39715]: 4.1b scrub ok 2024-03-11T22:18:52.571 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:52 smithi045 ceph-mon[39715]: 6.f scrub starts 2024-03-11T22:18:52.572 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:52 smithi045 ceph-mon[39715]: 6.f scrub ok 2024-03-11T22:18:52.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:52 smithi045 ceph-mon[30178]: 4.1b scrub starts 2024-03-11T22:18:52.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:52 smithi045 ceph-mon[30178]: 4.1b scrub ok 2024-03-11T22:18:52.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:52 smithi045 ceph-mon[30178]: 6.f scrub starts 2024-03-11T22:18:52.572 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:52 smithi045 ceph-mon[30178]: 6.f scrub ok 2024-03-11T22:18:52.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:52.657 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:52.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:52.759 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:52.786 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:52.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:52.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:52.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:52 smithi144 ceph-mon[33984]: 4.1b scrub starts 2024-03-11T22:18:52.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:52 smithi144 ceph-mon[33984]: 4.1b scrub ok 2024-03-11T22:18:52.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:52 smithi144 ceph-mon[33984]: 6.f scrub starts 2024-03-11T22:18:52.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:52 smithi144 ceph-mon[33984]: 6.f scrub ok 2024-03-11T22:18:52.879 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:18:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:18:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:18:52.888 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:52.913 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:52.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:53.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:53.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:53.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:53.164 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:53.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:53.265 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:53.292 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:53.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:53.394 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:53.403 DEBUG:teuthology.orchestra.run.smithi045:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-11T22:18:53.416 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:53 smithi045 ceph-mon[30178]: pgmap v484: 168 pgs: 168 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-03-11T22:18:53.418 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T22:18:53.418 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T22:18:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:53.419 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:53.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:53.475 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:hints = 1 2024-03-11T22:18:53.475 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T22:18:53.476 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Object prefix: benchmark_data_smithi045_141553 2024-03-11T22:18:53.476 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:18:53.476 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 0 0 0 0 0 0 - 0 2024-03-11T22:18:53.520 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:53.553 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:53.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:53.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:53.654 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:53.668 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:53 smithi045 ceph-mon[39715]: pgmap v484: 168 pgs: 168 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-03-11T22:18:53.668 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:53 smithi045 ceph-mon[39715]: 5.9 scrub starts 2024-03-11T22:18:53.668 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:53 smithi045 ceph-mon[39715]: 5.9 scrub ok 2024-03-11T22:18:53.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:53 smithi045 ceph-mon[30178]: 5.9 scrub starts 2024-03-11T22:18:53.669 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:53 smithi045 ceph-mon[30178]: 5.9 scrub ok 2024-03-11T22:18:53.683 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:53.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:53.784 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:53.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:53.814 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:53.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:53.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:53 smithi144 ceph-mon[33984]: pgmap v484: 168 pgs: 168 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-03-11T22:18:53.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:53 smithi144 ceph-mon[33984]: 5.9 scrub starts 2024-03-11T22:18:53.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:53 smithi144 ceph-mon[33984]: 5.9 scrub ok 2024-03-11T22:18:53.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:53.957 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:53.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:53.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:54.059 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:54.088 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:54.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:54.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:54.190 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:54.220 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:54.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:54.322 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:54.330 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:18:54.330 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:18:54.352 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:54.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:54.369 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:18:54.404 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:54.413 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:18:54.446 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:18:54.450 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T22:18:54.453 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:54.454 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:18:54.478 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 1 16 2815 2799 174.468 174.938 0.00536952 0.00570116 2024-03-11T22:18:54.480 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:54.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:54.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:54.504 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:18:54.554 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:18:54.582 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:54.597 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:18:54.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[30178]: 4.10 scrub starts 2024-03-11T22:18:54.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[30178]: 4.10 scrub ok 2024-03-11T22:18:54.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/974282138' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:18:54.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[30178]: 1.10 scrub starts 2024-03-11T22:18:54.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[30178]: 1.10 scrub ok 2024-03-11T22:18:54.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/862344301' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:18:54.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:18:54.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3090877755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:18:54.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[39715]: 4.10 scrub starts 2024-03-11T22:18:54.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[39715]: 4.10 scrub ok 2024-03-11T22:18:54.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/974282138' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:18:54.599 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[39715]: 1.10 scrub starts 2024-03-11T22:18:54.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[39715]: 1.10 scrub ok 2024-03-11T22:18:54.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/862344301' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:18:54.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:18:54.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:54 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3090877755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:18:54.602 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:18:54.602 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:18:54.602 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:18:54.602 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:18:54.602 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:18:54.602 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:18:54.602 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:18:54.602 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:18:54.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:54.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:54.727 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:54.759 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:54.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:54.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:54 smithi144 ceph-mon[33984]: 4.10 scrub starts 2024-03-11T22:18:54.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:54 smithi144 ceph-mon[33984]: 4.10 scrub ok 2024-03-11T22:18:54.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:54 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/974282138' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:18:54.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:54 smithi144 ceph-mon[33984]: 1.10 scrub starts 2024-03-11T22:18:54.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:54 smithi144 ceph-mon[33984]: 1.10 scrub ok 2024-03-11T22:18:54.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:54 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/862344301' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:18:54.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:54 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:18:54.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:54 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3090877755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:18:54.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:54.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:54.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:54.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:55.003 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:55.047 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:55.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:55.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:55.188 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:55.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:55.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:55.289 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:55.320 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:55.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:55.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:55.461 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:55.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:55.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:55.478 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 2 16 5807 5791 180.717 187 0.00352767 0.00551588 2024-03-11T22:18:55.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:55.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T22:18:55.610 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:55.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:55.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:55.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:55.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:55.755 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:55.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:55.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:55 smithi045 ceph-mon[30178]: pgmap v485: 168 pgs: 168 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:18:55.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:55 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:18:55.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:55 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/974282138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:18:55.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:55 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:18:55.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:55 smithi045 ceph-mon[30178]: osdmap e279: 8 total, 8 up, 8 in 2024-03-11T22:18:55.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:55 smithi045 ceph-mon[39715]: pgmap v485: 168 pgs: 168 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:18:55.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:55 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:18:55.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:55 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/974282138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:18:55.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:55 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:18:55.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:55 smithi045 ceph-mon[39715]: osdmap e279: 8 total, 8 up, 8 in 2024-03-11T22:18:55.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:55 smithi144 ceph-mon[33984]: pgmap v485: 168 pgs: 168 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:18:55.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:55 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:18:55.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:55 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/974282138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:18:55.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:55 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:18:55.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:55 smithi144 ceph-mon[33984]: osdmap e279: 8 total, 8 up, 8 in 2024-03-11T22:18:55.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:55.897 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:55.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:55.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:55.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:56.040 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:56.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:56.142 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:56.173 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:56.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:56.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:56.274 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:56.305 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:56.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:56.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:56.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:56.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:56.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:56.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:56.478 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 3 16 8499 8483 176.56 168.25 0.00827069 0.005651 2024-03-11T22:18:56.549 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:56.580 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:56.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:56.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:56.596 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:18:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:18:56.682 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:56.721 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:56.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:56.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:56.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:56.863 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:56.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:56.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:57.004 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:57.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:57.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:57.149 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:57.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:57.251 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:57.280 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:57.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:57.370 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:57.370 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:57 smithi045 ceph-mon[39715]: osdmap e280: 8 total, 8 up, 8 in 2024-03-11T22:18:57.370 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:57 smithi045 ceph-mon[39715]: pgmap v488: 168 pgs: 168 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 42 MiB/s wr, 680 op/s 2024-03-11T22:18:57.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:57.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:57 smithi045 ceph-mon[30178]: osdmap e280: 8 total, 8 up, 8 in 2024-03-11T22:18:57.370 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:57 smithi045 ceph-mon[30178]: pgmap v488: 168 pgs: 168 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 42 MiB/s wr, 680 op/s 2024-03-11T22:18:57.382 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:18:57.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:57.414 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:57.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:18:57.479 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 4 15 11184 11169 174.387 167.875 0.00685923 0.0057269 2024-03-11T22:18:57.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:57.551 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:57.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:57.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:57.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:18:57.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:57 smithi144 ceph-mon[33984]: osdmap e280: 8 total, 8 up, 8 in 2024-03-11T22:18:57.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:57 smithi144 ceph-mon[33984]: pgmap v488: 168 pgs: 168 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 42 MiB/s wr, 680 op/s 2024-03-11T22:18:57.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:57.707 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:57.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:57.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:57.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:57.849 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:57.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:57.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:57.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:57.997 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:57.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:58.098 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:58.109 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:18:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:58.169 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:58.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:58.270 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:58.300 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:58.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:58.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:58.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:18:58.402 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:58.433 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:58.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:58.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:58.479 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 5 16 13673 13657 170.608 155.5 0.00410363 0.00585053 2024-03-11T22:18:58.534 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:18:58.563 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:58.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:18:58.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:58.709 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:58.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:58.810 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:58.840 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:58.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:58.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:58.856 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:18:58.941 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:18:58.970 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:58.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:58.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:18:59.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:18:59.116 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:59.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:18:59.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:59.269 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:59.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:59.371 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:18:59.411 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:59.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:18:59.479 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 6 16 15827 15811 164.611 134.625 0.00342945 0.00606611 2024-03-11T22:18:59.513 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:18:59.516 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:18:59 smithi045 ceph-mon[39715]: pgmap v489: 168 pgs: 168 active+clean; 437 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 873 op/s 2024-03-11T22:18:59.516 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:18:59 smithi045 ceph-mon[30178]: pgmap v489: 168 pgs: 168 active+clean; 437 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 873 op/s 2024-03-11T22:18:59.543 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:18:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:59.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:18:59.595 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:18:59 smithi144 ceph-mon[33984]: pgmap v489: 168 pgs: 168 active+clean; 437 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 873 op/s 2024-03-11T22:18:59.625 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:18:59.625 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-03-11T22:18:59.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-03-11T22:18:59.625 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:18:59.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.625 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:26450 7.2 7:4d05f3c4:::benchmark_data_smithi045_83007_object26449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:18:59.625 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.975397+0000", 2024-03-11T22:18:59.625 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 484.64700460500001, 2024-03-11T22:18:59.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.153187401, 2024-03-11T22:18:59.625 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.625 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 26450 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975397+0000", 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.626 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975397+0000", 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975398+0000", 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2810000000000001e-06 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975422+0000", 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4409999999999998e-05 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975433+0000", 2024-03-11T22:18:59.627 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.096e-05 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975442+0000", 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5709999999999998e-06 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975500+0000", 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8295999999999999e-05 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975629+0000", 2024-03-11T22:18:59.628 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012867099999999999 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975755+0000", 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012651700000000001 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126092+0000", 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.150336689 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126107+0000", 2024-03-11T22:18:59.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4783999999999999e-05 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128442+0000", 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023357540000000002 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128541+0000", 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8388999999999996e-05 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128584+0000", 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3078999999999999e-05 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:26453 7.a 7:53c80b4b:::benchmark_data_smithi045_83007_object26452:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.975879+0000", 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 484.64652244899997, 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15598842800000001, 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.631 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 26453 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975879+0000", 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975879+0000", 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975880+0000", 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6800000000000009e-07 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975902+0000", 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2144999999999999e-05 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975913+0000", 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1433e-05 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.975921+0000", 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7600000000000002e-06 2024-03-11T22:18:59.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.976199+0000", 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00027757599999999997 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.976413+0000", 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021448999999999999 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.976552+0000", 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013905699999999999 2024-03-11T22:18:59.634 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126114+0000", 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14956153999999999 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126151+0000", 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6961000000000001e-05 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.130760+0000", 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046094489999999998 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.131087+0000", 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00032702499999999999 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.131867+0000", 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00078002400000000004 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:26450 7.2 e106/99 ondisk, result = 0)", 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.977100+0000", 2024-03-11T22:18:59.636 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 484.64530078000001, 2024-03-11T22:18:59.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14901504199999999, 2024-03-11T22:18:59.637 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.637 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:18:59.637 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977100+0000", 2024-03-11T22:18:59.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977100+0000", 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977101+0000", 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4099999999999998e-07 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977106+0000", 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4730000000000002e-06 2024-03-11T22:18:59.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977116+0000", 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0492000000000001e-05 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977123+0000", 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9040000000000003e-06 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126047+0000", 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.148923889 2024-03-11T22:18:59.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126057+0000", 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7620000000000007e-06 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126115+0000", 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8780999999999998e-05 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.640 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:26450 7.2 e106/99 ondisk, result = 0)", 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.977360+0000", 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 484.64504078900001, 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15123065199999999, 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977360+0000", 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977360+0000", 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.641 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977361+0000", 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9800000000000003e-07 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977365+0000", 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3019999999999996e-06 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977383+0000", 2024-03-11T22:18:59.642 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7302000000000001e-05 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977392+0000", 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0990000000000005e-06 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.127691+0000", 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.150299025 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128436+0000", 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00074558000000000003 2024-03-11T22:18:59.643 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.128591+0000", 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015464600000000001 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.14790.0:26453 7.a e106/99 ondisk, result = 0)", 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:10:54.977682+0000", 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 484.64471910100002, 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.148475739, 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:18:59.644 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977682+0000", 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977682+0000", 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977683+0000", 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6899999999999997e-07 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.645 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:54.977685+0000", 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5600000000000001e-06 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.124416+0000", 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14673112999999999 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.124434+0000", 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7260000000000001e-05 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.646 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126138+0000", 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001704583 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126145+0000", 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.922e-06 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:10:55.126158+0000", 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2714999999999999e-05 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.647 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205787 7.a 7:5b6c94a5:::benchmark_data_smithi045_83007_object87316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.684916+0000", 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 313.93748480099998, 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14867815600000001, 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205787 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.684916+0000", 2024-03-11T22:18:59.648 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.684916+0000", 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.684917+0000", 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6799999999999999e-07 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.684920+0000", 2024-03-11T22:18:59.649 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.779e-06 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.684929+0000", 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7970000000000006e-06 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.684935+0000", 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0990000000000004e-06 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.686632+0000", 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016966870000000001 2024-03-11T22:18:59.650 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.686754+0000", 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012264 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.686888+0000", 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000133418 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.689112+0000", 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0022243200000000001 2024-03-11T22:18:59.651 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.691934+0000", 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028221349999999999 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.833089+0000", 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14115493000000001 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.833539+0000", 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00045021699999999999 2024-03-11T22:18:59.652 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.833594+0000", 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5266000000000002e-05 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205791 7.a 7:5f1b8033:::benchmark_data_smithi045_83007_object87320:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.686660+0000", 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 313.93574159100001, 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14750897499999999, 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.653 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205791 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.686660+0000", 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.686660+0000", 2024-03-11T22:18:59.654 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.686661+0000", 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.003e-06 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.686664+0000", 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.552e-06 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.686675+0000", 2024-03-11T22:18:59.655 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.131e-05 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.686683+0000", 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6930000000000007e-06 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.689187+0000", 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025035750000000001 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.689298+0000", 2024-03-11T22:18:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110952 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.689406+0000", 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010846 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.691923+0000", 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025166720000000002 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.691939+0000", 2024-03-11T22:18:59.657 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6028e-05 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.833749+0000", 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14180978 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.834001+0000", 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00025206300000000002 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.834169+0000", 2024-03-11T22:18:59.658 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000167887 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205944 7.a 7:57d1d1e8:::benchmark_data_smithi045_83007_object87473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.063609+0000", 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 313.55879220000003, 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14355891300000001, 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205944 2024-03-11T22:18:59.659 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.063609+0000", 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.063609+0000", 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.063610+0000", 2024-03-11T22:18:59.660 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.15e-06 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.063614+0000", 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1320000000000004e-06 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.063625+0000", 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0618000000000001e-05 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.063632+0000", 2024-03-11T22:18:59.661 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0480000000000001e-06 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.063712+0000", 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0490999999999999e-05 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.063869+0000", 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015707899999999999 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.063943+0000", 2024-03-11T22:18:59.662 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3485999999999994e-05 2024-03-11T22:18:59.663 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.663 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.663 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.663 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.065864+0000", 2024-03-11T22:18:59.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:18:59.664 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019207930000000001 2024-03-11T22:18:59.664 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.066075+0000", 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021159 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.207019+0000", 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.140943441 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.207119+0000", 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010072 2024-03-11T22:18:59.665 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.207168+0000", 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8365000000000002e-05 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:229200 7.2 7:4597acc3:::benchmark_data_smithi045_83007_object110729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:14:10.522172+0000", 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 289.10022873299999, 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15251725499999999, 2024-03-11T22:18:59.666 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 229200 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.522172+0000", 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.667 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.522172+0000", 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.522174+0000", 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3850000000000001e-06 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.522178+0000", 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4610000000000001e-06 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.522189+0000", 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1107000000000001e-05 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.522198+0000", 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3450000000000006e-06 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.528238+0000", 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060405479999999998 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.669 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.528354+0000", 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000115659 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.528465+0000", 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011124 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.534023+0000", 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.005557673 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.670 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.539813+0000", 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0057904749999999998 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.674601+0000", 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13478742499999999 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.674638+0000", 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7786e-05 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.674690+0000", 2024-03-11T22:18:59.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1150999999999999e-05 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:229207 7.2 7:412387a4:::benchmark_data_smithi045_83007_object110736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:14:10.529660+0000", 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 289.09274114800002, 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14801515500000001, 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.672 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 229207 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.529660+0000", 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.529660+0000", 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.529661+0000", 2024-03-11T22:18:59.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.173e-06 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.529665+0000", 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1880000000000004e-06 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.529676+0000", 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0994e-05 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.529685+0000", 2024-03-11T22:18:59.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4780000000000008e-06 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.531107+0000", 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014225290000000001 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.531210+0000", 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000102179 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.531317+0000", 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000107808 2024-03-11T22:18:59.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.534028+0000", 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027108200000000001 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.540548+0000", 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065193550000000001 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.677396+0000", 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13684828800000001 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.677583+0000", 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018717300000000001 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.677675+0000", 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2169999999999993e-05 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:229209 7.3 7:c6b6ecea:::benchmark_data_smithi045_83007_object110738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:18:59.677 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:14:10.530577+0000", 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 289.09182433799998, 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14816152899999999, 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 229209 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.530577+0000", 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.530577+0000", 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.530578+0000", 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2800000000000005e-07 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.530583+0000", 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8400000000000002e-06 2024-03-11T22:18:59.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.530598+0000", 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5174000000000001e-05 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.530606+0000", 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2840000000000006e-06 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.530678+0000", 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2261000000000005e-05 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.530810+0000", 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000132211 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,4", 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.532592+0000", 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017817320000000001 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.534843+0000", 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002251004 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.535617+0000", 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000774203 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.678644+0000", 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14302627200000001 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.678696+0000", 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2592000000000001e-05 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.678738+0000", 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2027999999999998e-05 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:229210 7.2 7:4d4b81f3:::benchmark_data_smithi045_83007_object110739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:14:10.531216+0000", 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 289.09118506700003, 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15048344699999999, 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.683 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 229210 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.531216+0000", 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.531216+0000", 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.531217+0000", 2024-03-11T22:18:59.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2799999999999995e-07 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.531221+0000", 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4429999999999996e-06 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.531232+0000", 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1070000000000001e-05 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.531240+0000", 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6690000000000006e-06 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.534065+0000", 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028250440000000001 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.534182+0000", 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011690500000000001 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.534298+0000", 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000115809 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.539380+0000", 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0050823600000000002 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.540597+0000", 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012166169999999999 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.681526+0000", 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.140929112 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.681624+0000", 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8343000000000004e-05 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.681700+0000", 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5247000000000006e-05 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:229215 7.2 7:48e7331e:::benchmark_data_smithi045_83007_object110744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:14:10.534911+0000", 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 289.08748991900001, 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14709581299999999, 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.688 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 229215 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.534911+0000", 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.534911+0000", 2024-03-11T22:18:59.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.534912+0000", 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1999999999999998e-07 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.534916+0000", 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1169999999999997e-06 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.534926+0000", 2024-03-11T22:18:59.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8870000000000004e-06 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.534934+0000", 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9950000000000005e-06 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.536638+0000", 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017041459999999999 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.536752+0000", 2024-03-11T22:18:59.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113674 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.536908+0000", 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015627699999999999 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.539422+0000", 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025133059999999999 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.540632+0000", 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012102650000000001 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.681739+0000", 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.141107555 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.681962+0000", 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000222974 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.682007+0000", 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4697000000000003e-05 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:229224 7.a 7:52a89015:::benchmark_data_smithi045_83007_object110753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:14:10.543579+0000", 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 289.07882231899998, 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.145273818, 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 229224 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.694 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.543579+0000", 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.543579+0000", 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.543580+0000", 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0130000000000001e-06 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.543584+0000", 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.481e-06 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.543597+0000", 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3039e-05 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.543605+0000", 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1310000000000006e-06 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.544520+0000", 2024-03-11T22:18:59.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00091450600000000004 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.544649+0000", 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012946 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.544786+0000", 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000136506 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.546393+0000", 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001607477 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.546878+0000", 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00048466999999999998 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.688693+0000", 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14181458199999999 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.688795+0000", 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000102453 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:10.688853+0000", 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7500000000000002e-05 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:229670 7.3 7:cde25fd4:::benchmark_data_smithi045_83007_object111199:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:14:11.131826+0000", 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 288.49057474599999, 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.159690046, 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.699 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 229670 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.131826+0000", 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.131826+0000", 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.131827+0000", 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.33e-07 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.131840+0000", 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2480000000000001e-05 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.131852+0000", 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2204999999999999e-05 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.131860+0000", 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7540000000000006e-06 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.131914+0000", 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3832999999999998e-05 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.132024+0000", 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000109924 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,4", 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.132131+0000", 2024-03-11T22:18:59.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010755 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.136461+0000", 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043295369999999996 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.139709+0000", 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003248187 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.291153+0000", 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15144460500000001 2024-03-11T22:18:59.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.291473+0000", 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00031962800000000002 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.291516+0000", 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3409999999999999e-05 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:229673 7.3 7:c39ebed5:::benchmark_data_smithi045_83007_object111202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:18:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:14:11.139797+0000", 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 288.48260367300003, 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15184858700000001, 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 229673 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.139797+0000", 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.139797+0000", 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.139799+0000", 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4339999999999999e-06 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.139805+0000", 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7400000000000001e-06 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.139819+0000", 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3995000000000001e-05 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.139827+0000", 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4919999999999993e-06 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.140276+0000", 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00044894299999999999 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.140405+0000", 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000128861 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,4", 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.140524+0000", 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119403 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.143771+0000", 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032471539999999999 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.270572+0000", 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.12680071600000001 2024-03-11T22:18:59.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.291557+0000", 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.020984962999999999 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.291608+0000", 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1291000000000001e-05 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.291646+0000", 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7595000000000002e-05 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.709 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:229680 7.a 7:5eacfd34:::benchmark_data_smithi045_83007_object111209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:14:11.142172+0000", 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 288.480228995, 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.149215295, 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 229680 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.710 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.142172+0000", 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.142172+0000", 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.142174+0000", 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.517e-06 2024-03-11T22:18:59.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.142179+0000", 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0250000000000002e-06 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.142191+0000", 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1843e-05 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.142198+0000", 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9160000000000001e-06 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.144828+0000", 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026296610000000002 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.144970+0000", 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000141483 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,5", 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.145100+0000", 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013044900000000001 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.713 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.271061+0000", 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.125961342 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.290792+0000", 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.019730457 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.291304+0000", 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00051187400000000005 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.291346+0000", 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2098999999999998e-05 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:14:11.291387+0000", 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1628999999999997e-05 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:177382 8.f 8:f90d6ab3:::benchmark_data_smithi045_115206_object53264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:17.282246+0000", 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 102.340155227, 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.152183813, 2024-03-11T22:18:59.715 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 177382 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.282246+0000", 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.282246+0000", 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.282248+0000", 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.84e-06 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.282252+0000", 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7230000000000004e-06 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.282265+0000", 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2500000000000001e-05 2024-03-11T22:18:59.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.282273+0000", 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7179999999999996e-06 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.282713+0000", 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00044070900000000001 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.282842+0000", 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012819200000000001 2024-03-11T22:18:59.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.282955+0000", 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000113819 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.285592+0000", 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026366470000000002 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.290977+0000", 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0053853240000000004 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.434334+0000", 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14335631700000001 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.434388+0000", 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3854999999999999e-05 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.434430+0000", 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2169000000000002e-05 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:177389 8.f 8:f4a4b2e2:::benchmark_data_smithi045_115206_object53271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:17.287809+0000", 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 102.33459181400001, 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14674206200000001, 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 177389 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.287809+0000", 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.287809+0000", 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.287811+0000", 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.223e-06 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.287815+0000", 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7530000000000001e-06 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.287827+0000", 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1975e-05 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.287844+0000", 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7020999999999998e-05 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.287908+0000", 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3796999999999999e-05 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288020+0000", 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000112308 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.288129+0000", 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010833299999999999 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.724 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.290763+0000", 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026342610000000002 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.291016+0000", 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00025271700000000001 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.434470+0000", 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14345446000000001 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.434516+0000", 2024-03-11T22:18:59.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5518e-05 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.434551+0000", 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5695999999999998e-05 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.24935.0:177396 8.b 8:d11732a0:::benchmark_data_smithi045_115206_object53278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:17:17.293211+0000", 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 102.329190251, 2024-03-11T22:18:59.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14663040699999999, 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.24935", 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 177396 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.293211+0000", 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:18:59.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.293211+0000", 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.293211+0000", 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7300000000000001e-07 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.293214+0000", 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6000000000000001e-06 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.293220+0000", 2024-03-11T22:18:59.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6690000000000003e-06 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.293224+0000", 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4000000000000002e-06 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.293267+0000", 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3371000000000002e-05 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.293342+0000", 2024-03-11T22:18:59.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4130999999999999e-05 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,4", 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.293409+0000", 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7269999999999998e-05 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.294972+0000", 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001562826 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.295282+0000", 2024-03-11T22:18:59.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00031046099999999999 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.439624+0000", 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14434195399999999 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.439788+0000", 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016435700000000001 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:17:17.439841+0000", 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2895000000000001e-05 2024-03-11T22:18:59.731 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.732 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.732 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.732 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:18:59.732 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:18:59.732 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:18:59.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:59.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:59.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:18:59.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:18:59.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:18:59.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:18:59.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:18:59.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:00.010 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:00.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:00.112 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:00.141 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:00.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:00.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:00.242 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:00.274 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:00.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:00.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:00.375 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:00.405 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:00.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:00.479 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 7 16 17991 17975 160.417 135.25 0.00492049 0.00622535 2024-03-11T22:19:00.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:00.548 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:00.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:00.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:00.651 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:00.680 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:00.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:00.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:00.782 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:00.812 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:00.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:00.914 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:00.943 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:00.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:01.044 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:01.076 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:01.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:01.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:01.215 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:01.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:01.317 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:01.347 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:01.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:01.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:01.449 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:01.476 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:01.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:01.482 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 8 16 20292 20276 158.328 143.812 0.0116811 0.00630653 2024-03-11T22:19:01.499 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T22:19:01.578 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:01.610 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:01.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:01.617 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:01.619 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:01.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:01.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:01 smithi045 ceph-mon[39715]: pgmap v490: 168 pgs: 168 active+clean; 437 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 872 op/s 2024-03-11T22:19:01.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:19:01.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:19:01.727 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:01 smithi045 ceph-mon[30178]: pgmap v490: 168 pgs: 168 active+clean; 437 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 872 op/s 2024-03-11T22:19:01.727 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:19:01.728 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:19:01.756 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:01.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:01.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:01 smithi144 ceph-mon[33984]: pgmap v490: 168 pgs: 168 active+clean; 437 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 872 op/s 2024-03-11T22:19:01.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:19:01.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:19:01.858 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:01.869 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:19:01.896 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:01.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:01.998 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:02.037 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:02.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:02.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:02.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:02.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:02.286 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:02.320 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:02.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:02.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:02.466 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:02.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:02.480 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 9 16 22453 22437 155.743 135.062 0.00830225 0.00641452 2024-03-11T22:19:02.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:02.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:02.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:02.704 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:19:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:19:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:19:02.718 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:02.757 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:02.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:02.858 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:02.890 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:02.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:02.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:02.992 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:03.019 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:03.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:03.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:03.121 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:03.149 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:03.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:03.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:03.251 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:03.282 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:03.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:03.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:03.384 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:03.427 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:03.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:03.480 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 10 16 24432 24416 152.537 123.688 0.00691492 0.0065494 2024-03-11T22:19:03.528 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:03.558 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:03.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:03.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:03.577 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:03 smithi144 ceph-mon[33984]: pgmap v491: 168 pgs: 168 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.38k op/s 2024-03-11T22:19:03.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:03 smithi045 ceph-mon[39715]: pgmap v491: 168 pgs: 168 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.38k op/s 2024-03-11T22:19:03.638 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:03 smithi045 ceph-mon[30178]: pgmap v491: 168 pgs: 168 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.38k op/s 2024-03-11T22:19:03.659 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:03.690 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:03.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:03.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:03.791 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:03.824 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:03.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:03.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:03.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:03.926 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:19:03.926 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:19:03.926 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.927 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.25142.0:25364 9.e e281/273 9:7da26fbd:::benchmark_data_smithi045_141553_object25363:head v 281'1620, mlcod=281'1618)", 2024-03-11T22:19:03.927 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:19:03.923164+0000", 2024-03-11T22:19:03.927 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.001080316, 2024-03-11T22:19:03.927 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011117329999999999, 2024-03-11T22:19:03.927 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:03.927 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:19:03.927 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:03.927 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.927 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:03.927 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923164+0000", 2024-03-11T22:19:03.927 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:03.927 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.927 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923164+0000", 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923165+0000", 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7599999999999997e-07 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923234+0000", 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9461000000000005e-05 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.928 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923256+0000", 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1358000000000001e-05 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923264+0000", 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9009999999999994e-06 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923334+0000", 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9987000000000006e-05 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.929 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923353+0000", 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9165999999999999e-05 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.25142.0:25362 9.4 e281/273 9:2ae90ed4:::benchmark_data_smithi045_141553_object25361:head v 281'1592, mlcod=281'1589)", 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:19:03.923283+0000", 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.00096152800000000001, 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001066762, 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:19:03.930 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923283+0000", 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923283+0000", 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923283+0000", 2024-03-11T22:19:03.931 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1100000000000001e-07 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923315+0000", 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1415000000000003e-05 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923327+0000", 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1841999999999999e-05 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923333+0000", 2024-03-11T22:19:03.932 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7680000000000003e-06 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923373+0000", 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9444999999999999e-05 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:03.923391+0000", 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7858000000000001e-05 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-03-11T22:19:03.933 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 2 2024-03-11T22:19:03.934 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:19:03.975 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:03.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:04.077 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:04.106 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:04.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:04.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:04.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:04.254 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:04.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:04.356 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:04.381 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:04.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:04.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:04.389 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T22:19:04.480 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 11 16 26265 26249 149.085 114.562 0.00385347 0.00670159 2024-03-11T22:19:04.482 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:04.514 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:04.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:04.616 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:04.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:04.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:04.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:04.743 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:04.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:04 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:19:04.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:04 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:19:04.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:04 smithi045 ceph-mon[30178]: osdmap e281: 8 total, 8 up, 8 in 2024-03-11T22:19:04.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:04 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1012158672' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:19:04.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:04 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:19:04.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:04 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:19:04.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:04 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:19:04.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:04 smithi045 ceph-mon[39715]: osdmap e281: 8 total, 8 up, 8 in 2024-03-11T22:19:04.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:04 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1012158672' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:19:04.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:04 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:19:04.783 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:04.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:04.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:04 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:19:04.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:04 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:19:04.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:04 smithi144 ceph-mon[33984]: osdmap e281: 8 total, 8 up, 8 in 2024-03-11T22:19:04.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:04 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1012158672' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:19:04.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:04 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:19:04.885 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:04.914 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:04.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:04.946 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T22:19:05.015 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:05.067 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:05.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:05.168 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:05.199 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:05.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:05.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:05.309 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:19:05.323 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:05.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:05.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:05.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:05.441 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:05 smithi045 ceph-mon[39715]: pgmap v493: 168 pgs: 168 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 638 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-03-11T22:19:05.441 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:05 smithi045 ceph-mon[30178]: pgmap v493: 168 pgs: 168 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 638 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-03-11T22:19:05.453 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:05.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:05.480 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 12 16 27715 27699 144.214 90.625 0.00541497 0.00692829 2024-03-11T22:19:05.555 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:05.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:05.592 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:05.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:05.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:05 smithi144 ceph-mon[33984]: pgmap v493: 168 pgs: 168 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 638 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-03-11T22:19:05.599 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:05 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:19:05.599 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:05 smithi144 ceph-mon[33984]: osdmap e282: 8 total, 8 up, 8 in 2024-03-11T22:19:05.692 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:05.721 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:05.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:05.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:05 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:19:05.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:05 smithi045 ceph-mon[30178]: osdmap e282: 8 total, 8 up, 8 in 2024-03-11T22:19:05.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:05 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:19:05.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:05 smithi045 ceph-mon[39715]: osdmap e282: 8 total, 8 up, 8 in 2024-03-11T22:19:05.823 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:05.858 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:05.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:05.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:05.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:05.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:05.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:06.099 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:06.131 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:06.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:06.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:06.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:06.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:06.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:06.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:06.404 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:06.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:06.480 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 13 16 29220 29204 140.356 94.0625 0.00311877 0.00710725 2024-03-11T22:19:06.505 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:06.531 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:06.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:06.596 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:19:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:19:06.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:06.660 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:06.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:06.761 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:06.799 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:06.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:06.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:06.901 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:06.924 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:06.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:07.026 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:07.058 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:07.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:07.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:07.197 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:07.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:07.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:07.337 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:07.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:07.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:07.439 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:07.462 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:07.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:07.480 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 14 16 30564 30548 136.331 84 0.00461468 0.00732968 2024-03-11T22:19:07.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:07.586 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:07.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:07.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:07.688 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:07.717 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:07.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:07.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:07.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:07.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:07 smithi045 ceph-mon[30178]: pgmap v495: 168 pgs: 168 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 154 MiB/s wr, 2.47k op/s 2024-03-11T22:19:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:07 smithi045 ceph-mon[39715]: pgmap v495: 168 pgs: 168 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 154 MiB/s wr, 2.47k op/s 2024-03-11T22:19:07.819 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:07.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:07.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:07 smithi144 ceph-mon[33984]: pgmap v495: 168 pgs: 168 active+clean; 1.6 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 154 MiB/s wr, 2.47k op/s 2024-03-11T22:19:07.853 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:07.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:07.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:07.955 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:07.985 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:07.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:08.087 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:08.117 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:08.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:08.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:08.256 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:08.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:08.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:08.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:08.397 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:08.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:08.480 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 15 15 32034 32019 133.372 91.9375 0.0096623 0.00749174 2024-03-11T22:19:08.499 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:08.530 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:08.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:08.631 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:08.664 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:08.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:08.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:08.737 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:08 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:19:08.737 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:08 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:19:08.766 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:08.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:08 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:19:08.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:08 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:19:08.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:08 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:19:08.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:08 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:19:08.803 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:08.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:08.904 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:08.936 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:08.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:09.036 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:19:09.037 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-03-11T22:19:09.037 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-03-11T22:19:09.037 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-03-11T22:19:09.037 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:19:09.038 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:09.067 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:09.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:09.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:09.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:09.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:09.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:09.306 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:09.343 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:09.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:09.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:09.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:09.481 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 16 16 33807 33791 131.957 110.75 0.0105647 0.00757056 2024-03-11T22:19:09.485 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:09.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:09.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:09.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:09.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:09.726 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:09.739 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:09 smithi144 ceph-mon[33984]: pgmap v496: 168 pgs: 168 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 167 MiB/s wr, 2.68k op/s 2024-03-11T22:19:09.739 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:09 smithi144 ceph-mon[33984]: osdmap e283: 8 total, 8 up, 8 in 2024-03-11T22:19:09.760 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:09.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:09.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:09 smithi045 ceph-mon[39715]: pgmap v496: 168 pgs: 168 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 167 MiB/s wr, 2.68k op/s 2024-03-11T22:19:09.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:09 smithi045 ceph-mon[39715]: osdmap e283: 8 total, 8 up, 8 in 2024-03-11T22:19:09.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:09 smithi045 ceph-mon[30178]: pgmap v496: 168 pgs: 168 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 167 MiB/s wr, 2.68k op/s 2024-03-11T22:19:09.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:09 smithi045 ceph-mon[30178]: osdmap e283: 8 total, 8 up, 8 in 2024-03-11T22:19:09.785 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T22:19:09.862 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:09.904 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:09.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:09.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T22:19:10.006 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:10.037 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:10.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:10.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:10.138 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:10.165 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:19:10.171 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:10.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:10.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:10.273 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:10.305 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:10.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:10.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:10.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:10.451 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:10.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:10.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:10.481 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 17 16 35597 35581 130.776 111.875 0.00924544 0.00764064 2024-03-11T22:19:10.552 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:10.583 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:10.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:10.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:10.684 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:10.714 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:10.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:10.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:10.816 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:10.847 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:10.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:10.949 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:10.978 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:10.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:10.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:11.080 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:11.108 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:11.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:11.210 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:11.239 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:11.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:11.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:11.340 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:11.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:11.383 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:11.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:11.481 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 18 15 37252 37237 129.26 103.5 0.011587 0.0077303 2024-03-11T22:19:11.484 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:11.525 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:11.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:11.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:11.627 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:11.667 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:11.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:11.757 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:11 smithi144 ceph-mon[33984]: pgmap v498: 168 pgs: 168 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 744 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T22:19:11.757 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:11 smithi144 ceph-mon[33984]: osdmap e284: 8 total, 8 up, 8 in 2024-03-11T22:19:11.757 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:11 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2716389332' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:19:11.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:11.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:11 smithi045 ceph-mon[39715]: pgmap v498: 168 pgs: 168 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 744 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T22:19:11.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:11 smithi045 ceph-mon[39715]: osdmap e284: 8 total, 8 up, 8 in 2024-03-11T22:19:11.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:11 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2716389332' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:19:11.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:11 smithi045 ceph-mon[30178]: pgmap v498: 168 pgs: 168 active+clean; 1.7 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 744 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T22:19:11.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:11 smithi045 ceph-mon[30178]: osdmap e284: 8 total, 8 up, 8 in 2024-03-11T22:19:11.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:11 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2716389332' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:19:11.797 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:11.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:11.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:11.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:11.940 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:11.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:12.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:12.070 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:19:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-03-11T22:19:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-03-11T22:19:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:19:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:28074 7.1 7:868323d2:::benchmark_data_smithi045_83007_object28073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:19:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:56.076197+0000", 2024-03-11T22:19:12.070 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 495.98975327400001, 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13456032600000001, 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 28074 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.076197+0000", 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.071 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.076197+0000", 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.076198+0000", 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.279e-06 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.076257+0000", 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8640000000000001e-05 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.072 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.076270+0000", 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3172e-05 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.076278+0000", 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3469999999999999e-06 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.076346+0000", 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7345e-05 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.073 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.076503+0000", 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015714099999999999 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.076654+0000", 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015118299999999999 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.078064+0000", 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0014101859999999999 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.074 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079687+0000", 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016227920000000001 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210671+0000", 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13098432400000001 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210714+0000", 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3061e-05 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210757+0000", 2024-03-11T22:19:12.075 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2855999999999998e-05 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:41645 7.1 7:889a6bdd:::benchmark_data_smithi045_83007_object41644:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.883007+0000", 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 488.18294293000002, 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101232269, 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 41645 2024-03-11T22:19:12.076 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.883007+0000", 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.883007+0000", 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.883008+0000", 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2759999999999999e-06 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.077 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.883053+0000", 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4968e-05 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.883064+0000", 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.077e-05 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.883072+0000", 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3640000000000001e-06 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.078 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.883120+0000", 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8078000000000003e-05 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.883202+0000", 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1936999999999998e-05 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.883281+0000", 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9680999999999998e-05 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.079 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.884613+0000", 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0013314959999999999 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.889629+0000", 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0050157939999999996 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.984076+0000", 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.094447059 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.984173+0000", 2024-03-11T22:19:12.080 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7449999999999997e-05 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.984239+0000", 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6396000000000006e-05 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:41662 7.8 e109/99 7:1c1bd511:::benchmark_data_smithi045_83007_object41661:head v 109'2622, mlcod=109'2620)", 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.897986+0000", 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 488.16796460400002, 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.074807589999999993, 2024-03-11T22:19:12.081 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.897986+0000", 2024-03-11T22:19:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.897986+0000", 2024-03-11T22:19:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.082 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.897986+0000", 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.92e-07 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.898040+0000", 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3332999999999999e-05 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.898048+0000", 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9300000000000003e-06 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.083 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.898053+0000", 2024-03-11T22:19:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9359999999999998e-06 2024-03-11T22:19:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.898093+0000", 2024-03-11T22:19:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9987999999999997e-05 2024-03-11T22:19:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.898106+0000", 2024-03-11T22:19:12.084 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3833e-05 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.899138+0000", 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010315109999999999 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.972793+0000", 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.073655266999999996 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.085 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.14790.0:41663 7.4 e109/99 7:2b138a04:::benchmark_data_smithi045_83007_object41662:head v 109'2625, mlcod=109'2623)", 2024-03-11T22:19:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.900470+0000", 2024-03-11T22:19:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 488.16548003000003, 2024-03-11T22:19:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.074036282999999994, 2024-03-11T22:19:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.900470+0000", 2024-03-11T22:19:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.086 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.900470+0000", 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.900472+0000", 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5600000000000001e-06 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.900537+0000", 2024-03-11T22:19:12.087 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5335999999999999e-05 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.972815+0000", 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.072277583000000006 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.972823+0000", 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2560000000000002e-06 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.972877+0000", 2024-03-11T22:19:12.088 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4194000000000003e-05 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.972897+0000", 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9947999999999999e-05 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.974469+0000", 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001572026 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.974506+0000", 2024-03-11T22:19:12.089 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7379999999999998e-05 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:82787 7.1 7:86c18b82:::benchmark_data_smithi045_83007_object82786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:36.335499+0000", 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 455.73045080899999, 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.083959864999999995, 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.090 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 82787 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.335499+0000", 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.335499+0000", 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.091 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.335501+0000", 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.451e-06 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.335643+0000", 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000141822 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.335660+0000", 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7555e-05 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.092 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.335668+0000", 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0099999999999995e-06 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.335733+0000", 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4759999999999997e-05 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.335858+0000", 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000125105 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.093 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.336188+0000", 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000330014 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.338741+0000", 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0025528270000000001 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.339511+0000", 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00077001100000000002 2024-03-11T22:19:12.094 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.419338+0000", 2024-03-11T22:19:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.079827224000000002 2024-03-11T22:19:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.419401+0000", 2024-03-11T22:19:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3040000000000006e-05 2024-03-11T22:19:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.095 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:36.419459+0000", 2024-03-11T22:19:12.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:12.096 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8046e-05 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:141141 7.1 7:846e6614:::benchmark_data_smithi045_83007_object22670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:37.001627+0000", 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 395.06432312800001, 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.070072997999999997, 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.097 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 141141 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.001627+0000", 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.001627+0000", 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.098 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.001629+0000", 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.548e-06 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.001634+0000", 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2580000000000003e-06 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.001646+0000", 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1891e-05 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.099 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.001654+0000", 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5439999999999999e-06 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.001719+0000", 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4326999999999996e-05 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.001825+0000", 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010635 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.001897+0000", 2024-03-11T22:19:12.100 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1645999999999998e-05 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.003926+0000", 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002029028 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.003971+0000", 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5391e-05 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.071595+0000", 2024-03-11T22:19:12.101 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.067623612 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.071657+0000", 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2184e-05 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.071700+0000", 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3219000000000001e-05 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.102 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:141143 7.1 7:86741f22:::benchmark_data_smithi045_83007_object22672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:37.004384+0000", 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 395.06156648400002, 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.067450716999999993, 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 141143 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.004384+0000", 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.103 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.004384+0000", 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.004385+0000", 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0410000000000001e-06 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.004390+0000", 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8380000000000001e-06 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.104 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.004402+0000", 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2865e-05 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.004411+0000", 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3450000000000006e-06 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.004462+0000", 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0794000000000002e-05 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.105 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.004583+0000", 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000121334 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.004708+0000", 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012523100000000001 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.006510+0000", 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018015469999999999 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.006628+0000", 2024-03-11T22:19:12.106 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011791200000000001 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.071741+0000", 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.065113684000000005 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.071795+0000", 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3557999999999998e-05 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.071834+0000", 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9567999999999998e-05 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.107 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:141375 7.1 7:81e3d253:::benchmark_data_smithi045_83007_object22904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:37.295315+0000", 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 394.77063469699999, 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.069488184999999994, 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 141375 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.108 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.295315+0000", 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.295315+0000", 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.295316+0000", 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5700000000000002e-07 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.109 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.295319+0000", 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.971e-06 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.295328+0000", 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4940000000000003e-06 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.295334+0000", 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5980000000000002e-06 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.297560+0000", 2024-03-11T22:19:12.110 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002225346 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.297690+0000", 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013082600000000001 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.297817+0000", 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012694900000000001 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.299921+0000", 2024-03-11T22:19:12.111 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0021038929999999999 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.300709+0000", 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00078798099999999999 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.364733+0000", 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.064023423999999995 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.364769+0000", 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6023999999999999e-05 2024-03-11T22:19:12.112 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.364804+0000", 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5021999999999998e-05 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:142726 7.1 7:86e1bcfa:::benchmark_data_smithi045_83007_object24255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:38.771392+0000", 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 393.29455791700002, 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.078343636999999994, 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.113 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 142726 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771392+0000", 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771392+0000", 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.114 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771393+0000", 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.155e-06 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771398+0000", 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.228e-06 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771409+0000", 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0978000000000001e-05 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.115 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771417+0000", 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0209999999999999e-06 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771467+0000", 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0868999999999997e-05 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771569+0000", 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000101777 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.771644+0000", 2024-03-11T22:19:12.116 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5050999999999997e-05 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.776095+0000", 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0044505769999999998 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.776147+0000", 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2210999999999999e-05 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.849669+0000", 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.073522044999999994 2024-03-11T22:19:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.849705+0000", 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5757999999999999e-05 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.849736+0000", 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0966999999999997e-05 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:142730 7.1 7:845bca03:::benchmark_data_smithi045_83007_object24259:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:38.772511+0000", 2024-03-11T22:19:12.118 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 393.29343919899998, 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.077327228999999997, 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 142730 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.772511+0000", 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.119 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.772511+0000", 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.772512+0000", 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0869999999999999e-06 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.772519+0000", 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7970000000000003e-06 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.772532+0000", 2024-03-11T22:19:12.120 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3147e-05 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.772539+0000", 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5229999999999998e-06 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.774014+0000", 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001474597 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.774126+0000", 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000111854 2024-03-11T22:19:12.121 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.774235+0000", 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000108698 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.777144+0000", 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0029089900000000002 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.778382+0000", 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001238495 2024-03-11T22:19:12.122 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.849767+0000", 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.071384699999999995 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.849801+0000", 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4567999999999998e-05 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:38.849838+0000", 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6773000000000002e-05 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.123 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:206119 7.1 7:84ba3b37:::benchmark_data_smithi045_83007_object87648:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:46.405016+0000", 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 325.66093369499998, 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.100269341, 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 206119 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.124 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.405016+0000", 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.405016+0000", 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.405018+0000", 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5650000000000001e-06 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.405028+0000", 2024-03-11T22:19:12.125 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.9539999999999999e-06 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.405040+0000", 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2233e-05 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.405049+0000", 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7199999999999995e-06 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.126 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.405109+0000", 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0226999999999998e-05 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.405195+0000", 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6033999999999997e-05 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.405270+0000", 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4532999999999999e-05 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.407339+0000", 2024-03-11T22:19:12.127 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0020691049999999999 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.407707+0000", 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00036827700000000002 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.505168+0000", 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.097460529000000004 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.505232+0000", 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.4012999999999999e-05 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.128 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:46.505286+0000", 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4150999999999997e-05 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:207223 7.1 7:8c0c391a:::benchmark_data_smithi045_83007_object88752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:48.160954+0000", 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 323.90499601599998, 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.096515078000000004, 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.129 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 207223 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160954+0000", 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160954+0000", 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.130 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160955+0000", 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2529999999999999e-06 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160960+0000", 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6290000000000001e-06 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160973+0000", 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2746e-05 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.131 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.160981+0000", 2024-03-11T22:19:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7919999999999993e-06 2024-03-11T22:19:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.161036+0000", 2024-03-11T22:19:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5531000000000001e-05 2024-03-11T22:19:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.132 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.161150+0000", 2024-03-11T22:19:12.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:12.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000114391 2024-03-11T22:19:12.133 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.161256+0000", 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010576700000000001 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.163459+0000", 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0022030320000000002 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.163646+0000", 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018716499999999999 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.134 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.257398+0000", 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.093751127000000004 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.257438+0000", 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0151000000000002e-05 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:48.257469+0000", 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1494e-05 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.135 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:227150 7.1 7:88e4136d:::benchmark_data_smithi045_83007_object108679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:08.354926+0000", 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 303.71102394899998, 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.078393189000000002, 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 227150 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.136 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.354926+0000", 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.354926+0000", 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.354927+0000", 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2699999999999999e-06 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.137 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.354931+0000", 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0350000000000003e-06 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.354941+0000", 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3910000000000004e-06 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.354947+0000", 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8309999999999997e-06 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.138 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.355002+0000", 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4984000000000001e-05 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.355159+0000", 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015741600000000001 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.355281+0000", 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012234400000000001 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.360744+0000", 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0054626800000000001 2024-03-11T22:19:12.139 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.360830+0000", 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6067000000000001e-05 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.433236+0000", 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.072406149000000003 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.433274+0000", 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7450000000000002e-05 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.140 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.433319+0000", 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5571999999999997e-05 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:227156 7.1 7:8a230474:::benchmark_data_smithi045_83007_object108685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:08.357965+0000", 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 303.707984737, 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.075695344999999997, 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.141 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 227156 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.357965+0000", 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.357965+0000", 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.142 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.357967+0000", 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3820000000000001e-06 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.357973+0000", 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9190000000000004e-06 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.357984+0000", 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1085e-05 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.143 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.357993+0000", 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7290000000000006e-06 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.358058+0000", 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5166999999999995e-05 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.358182+0000", 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000124778 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.144 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.358301+0000", 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011892699999999999 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.360870+0000", 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0025691080000000001 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.361568+0000", 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00069712099999999998 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.145 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.433598+0000", 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.072030283000000001 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.433627+0000", 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9002000000000001e-05 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:08.433661+0000", 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3844000000000003e-05 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.146 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229387 7.1 7:8823dab5:::benchmark_data_smithi045_83007_object110916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:10.797925+0000", 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 301.26802475, 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.075513844999999996, 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229387 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.147 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.797925+0000", 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.797925+0000", 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.797927+0000", 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.564e-06 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.797932+0000", 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8069999999999999e-06 2024-03-11T22:19:12.148 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.797945+0000", 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3395e-05 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.797953+0000", 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2500000000000006e-06 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.798017+0000", 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3553000000000006e-05 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.149 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.798174+0000", 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015694 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.798290+0000", 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000116008 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.800758+0000", 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024676329999999999 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.150 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.801151+0000", 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00039380300000000002 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.873352+0000", 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.072200390000000003 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.873392+0000", 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0738000000000002e-05 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.151 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:10.873439+0000", 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6764000000000002e-05 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229679 7.1 7:81a4a561:::benchmark_data_smithi045_83007_object111208:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:11.141233+0000", 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 300.924717493, 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.12941819800000001, 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.152 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229679 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.141233+0000", 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.141233+0000", 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.141234+0000", 2024-03-11T22:19:12.153 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.376e-06 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.141239+0000", 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2070000000000003e-06 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.141251+0000", 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2133999999999999e-05 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.141259+0000", 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6529999999999994e-06 2024-03-11T22:19:12.154 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.141317+0000", 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7773999999999999e-05 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.141407+0000", 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9740000000000005e-05 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.141479+0000", 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2256999999999996e-05 2024-03-11T22:19:12.155 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.143683+0000", 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002203834 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.143802+0000", 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011980399999999999 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.270537+0000", 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.126735031 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.156 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.270601+0000", 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3762000000000004e-05 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.270651+0000", 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9626000000000002e-05 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:230402 7.1 7:8474a197:::benchmark_data_smithi045_83007_object111931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:11.993778+0000", 2024-03-11T22:19:12.157 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 300.07217183400002, 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.082643357000000001, 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 230402 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.993778+0000", 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.158 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.993778+0000", 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.993780+0000", 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6789999999999999e-06 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.993785+0000", 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4430000000000002e-06 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.159 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.993798+0000", 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2486999999999999e-05 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.993806+0000", 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8299999999999996e-06 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.993869+0000", 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2978999999999995e-05 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.160 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.993997+0000", 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012806499999999999 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.994128+0000", 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000130782 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.996670+0000", 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0025419779999999999 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.161 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.996901+0000", 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00023117799999999999 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:12.076349+0000", 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.079448314000000006 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:12.076384+0000", 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5120999999999998e-05 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.162 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:12.076422+0000", 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7500999999999999e-05 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:231439 7.1 7:82f90b18:::benchmark_data_smithi045_83007_object112968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:13.106696+0000", 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 298.95925429300001, 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.071186097000000004, 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:12.163 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 231439 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.106696+0000", 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.106696+0000", 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.106697+0000", 2024-03-11T22:19:12.164 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1599999999999999e-06 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.106700+0000", 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3280000000000002e-06 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.106711+0000", 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0774999999999999e-05 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.106718+0000", 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5769999999999999e-06 2024-03-11T22:19:12.165 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.106765+0000", 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7729999999999999e-05 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.106855+0000", 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9287000000000001e-05 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,6", 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.106938+0000", 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3372000000000001e-05 2024-03-11T22:19:12.166 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.109258+0000", 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0023199399999999999 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.110329+0000", 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001071362 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.177704+0000", 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.067374978000000002 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.167 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.177831+0000", 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012676300000000001 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:13.177882+0000", 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0825000000000002e-05 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:93598 8.2 e200/175 8:45e1a51c:::benchmark_data_smithi045_115206_object93597:head v 200'5810, mlcod=200'5806)", 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:03.430107+0000", 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 188.635842674, 2024-03-11T22:19:12.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.081047534000000004, 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.430107+0000", 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.430107+0000", 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.430111+0000", 2024-03-11T22:19:12.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.72e-06 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.430150+0000", 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9195e-05 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.430166+0000", 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5823000000000001e-05 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.430174+0000", 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2090000000000005e-06 2024-03-11T22:19:12.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.430235+0000", 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0652999999999997e-05 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.430256+0000", 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0550000000000001e-05 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.511112+0000", 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.080856270999999993 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.511155+0000", 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3112999999999999e-05 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.24935.0:93605 8.4 e200/175 8:25182649:::benchmark_data_smithi045_115206_object93604:head v 200'5940, mlcod=200'5938)", 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:03.431161+0000", 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 188.63478891099999, 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.085336334999999999, 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:12.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.431161+0000", 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.431161+0000", 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.431162+0000", 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4499999999999996e-07 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.431200+0000", 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8405999999999998e-05 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.511098+0000", 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.079897093000000002 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.511112+0000", 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4549000000000001e-05 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.511164+0000", 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2358000000000003e-05 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.511186+0000", 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1868000000000001e-05 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.516448+0000", 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.005261735 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:03.516498+0000", 2024-03-11T22:19:12.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9481000000000003e-05 2024-03-11T22:19:12.176 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.176 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.176 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.176 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:12.176 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:12.176 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:19:12.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:12.254 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:19:12.274 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:12.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:12.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:12.377 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:12.408 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:12.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:12.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:12.481 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 19 16 38689 38673 127.181 89.75 0.00926106 0.00785679 2024-03-11T22:19:12.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:12.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:12 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:19:12.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:12 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2716389332' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:19:12.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:12 smithi045 ceph-mon[39715]: osdmap e285: 8 total, 8 up, 8 in 2024-03-11T22:19:12.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:12 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:19:12.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:12 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2716389332' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:19:12.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:12 smithi045 ceph-mon[30178]: osdmap e285: 8 total, 8 up, 8 in 2024-03-11T22:19:12.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:12.548 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:12.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:12.649 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:12.672 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:12.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:12.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:12.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:12.816 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:12.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:12.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:12.829 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:19:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:19:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:19:12.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:12 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:19:12.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:12 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2716389332' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:19:12.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:12 smithi144 ceph-mon[33984]: osdmap e285: 8 total, 8 up, 8 in 2024-03-11T22:19:12.918 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:12.949 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:12.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:12.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:13.003 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T22:19:13.051 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:13.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:13.090 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:13.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:13.191 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:13.231 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:13.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:13.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:13.323 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:19:13.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:13.374 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:13.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:13.476 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:13.481 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:19:13.481053+0000 min lat: 0.00145181 max lat: 0.218324 avg lat: 0.00795679 2024-03-11T22:19:13.481 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:19:13.481 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 20 16 40186 40170 125.498 93.5625 0.0496398 0.00795679 2024-03-11T22:19:13.515 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:13.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:13.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:13.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:13.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:13.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:13.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:13.748 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:13 smithi144 ceph-mon[33984]: pgmap v501: 168 pgs: 168 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.61k op/s 2024-03-11T22:19:13.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:13.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:13 smithi045 ceph-mon[39715]: pgmap v501: 168 pgs: 168 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.61k op/s 2024-03-11T22:19:13.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:13 smithi045 ceph-mon[30178]: pgmap v501: 168 pgs: 168 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 100 MiB/s wr, 1.61k op/s 2024-03-11T22:19:13.796 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:13.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:13.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:13.897 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:13.928 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:13.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:13.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:14.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:14.073 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:14.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:14.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:14.080 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:19:14.125 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:19:14.168 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:19:14.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:14.215 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:19:14.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:14.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:14.258 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:19:14.299 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:19:14.320 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:14.338 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:19:14.352 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:14.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:14.379 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:19:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:19:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:19:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:19:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:19:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:19:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:19:14.385 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:19:14.386 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:19:14.454 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:14.481 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 21 16 41667 41651 123.93 92.5625 0.00870033 0.00806348 2024-03-11T22:19:14.485 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:14.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:14.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:14.554 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:14 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1142618271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:19:14.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:14.629 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:14.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:14.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:14.774 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:14.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:14.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:14.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:14 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1142618271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:19:14.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:14 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1142618271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:19:14.877 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:14.908 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:14.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:14.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:15.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:15.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:15.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:15.060 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:15.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:15.194 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:15.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:15.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:15.296 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:15.327 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:15.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:15.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:15.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:15.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:15.481 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 22 16 43345 43329 123.063 104.875 0.00725828 0.00812066 2024-03-11T22:19:15.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:15 smithi045 ceph-mon[30178]: pgmap v502: 168 pgs: 168 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-11T22:19:15.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:19:15.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:15 smithi045 ceph-mon[39715]: pgmap v502: 168 pgs: 168 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-11T22:19:15.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:19:15.573 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:15.605 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:15.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:15.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:15.644 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:15 smithi144 ceph-mon[33984]: pgmap v502: 168 pgs: 168 active+clean; 2.2 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-11T22:19:15.644 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:19:15.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:15.711 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:19:15.711 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:19:15.746 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:15.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:15.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:15.888 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:15.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:15.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:15.990 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:16.021 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:16.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:16.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:16.122 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:16.151 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:16.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:16.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:16.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:16.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:16.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:16.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:16.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:16.444 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:16.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:16.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:16.481 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 23 15 44873 44858 121.867 95.5625 0.00826271 0.00820031 2024-03-11T22:19:16.546 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:16.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:16.570 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:16.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:16.596 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:19:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:19:16.671 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:16.702 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:16.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:16.804 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:16.835 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:16.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:19:16.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:16.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:16.938 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:16.975 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:16.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:16.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:16.993 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T22:19:17.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:17.128 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:17.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:17.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:17.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:17.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:17.298 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:17.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:17.353 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:19:17.359 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:17.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:17.449 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:17.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:17.453 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:17.454 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:17 smithi045 ceph-mon[39715]: osdmap e286: 8 total, 8 up, 8 in 2024-03-11T22:19:17.454 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:17 smithi045 ceph-mon[39715]: pgmap v504: 168 pgs: 168 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-11T22:19:17.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:17.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:17 smithi045 ceph-mon[30178]: osdmap e286: 8 total, 8 up, 8 in 2024-03-11T22:19:17.454 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:17 smithi045 ceph-mon[30178]: pgmap v504: 168 pgs: 168 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-11T22:19:17.481 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 24 16 46264 46248 120.409 86.875 0.0141526 0.00830002 2024-03-11T22:19:17.516 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:17.516 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:17 smithi144 ceph-mon[33984]: osdmap e286: 8 total, 8 up, 8 in 2024-03-11T22:19:17.516 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:17 smithi144 ceph-mon[33984]: pgmap v504: 168 pgs: 168 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-11T22:19:17.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:17.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:17.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:17.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:17.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:19:17.695 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:17.728 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:17.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:17.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:17.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:17.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:17.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:17.976 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:18.006 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:18.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:18.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:18.150 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:18.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:18.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:18.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:18.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:18.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:18.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:18.397 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:18.424 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:18.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:18.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:18.482 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 25 16 47595 47579 118.92 83.1875 0.00888985 0.00840243 2024-03-11T22:19:18.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:18.570 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:18.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:18.672 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:18.701 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:18.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:18.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:18.854 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:18.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:18.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:18.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:18.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:18.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:19.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:19.140 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:19.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:19.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:19.198 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:19:19.198 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:19:19.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:19.271 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:19.288 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T22:19:19.300 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:19.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:19.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:19.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:19.451 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:19.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:19.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:19.482 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 26 16 48816 48800 117.281 76.3125 0.00501073 0.00852126 2024-03-11T22:19:19.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:19 smithi045 ceph-mon[30178]: pgmap v505: 168 pgs: 168 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 730 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-11T22:19:19.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:19 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/553202895' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:19:19.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:19 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:19:19.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:19 smithi045 ceph-mon[39715]: pgmap v505: 168 pgs: 168 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 730 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-11T22:19:19.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:19 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/553202895' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:19:19.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:19 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:19:19.544 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:19 smithi144 ceph-mon[33984]: pgmap v505: 168 pgs: 168 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 730 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-11T22:19:19.544 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:19 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/553202895' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:19:19.544 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:19 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:19:19.553 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:19.584 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:19.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:19.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:19.685 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:19.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:19.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:19.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:19.834 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:19.864 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:19.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:19.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:19.893 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T22:19:19.965 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:20.013 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:20.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:20.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T22:19:20.115 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:20.147 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:20.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:20.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:20.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:20.262 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:19:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:20.318 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:20.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:20.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:20.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:20.480 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:20.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:20.484 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 27 16 50170 50154 116.063 84.625 0.0159344 0.00861054 2024-03-11T22:19:20.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:20 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:19:20.541 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:20 smithi045 ceph-mon[30178]: osdmap e287: 8 total, 8 up, 8 in 2024-03-11T22:19:20.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:20 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:19:20.542 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:20 smithi045 ceph-mon[39715]: osdmap e287: 8 total, 8 up, 8 in 2024-03-11T22:19:20.548 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:20 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:19:20.548 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:20 smithi144 ceph-mon[33984]: osdmap e287: 8 total, 8 up, 8 in 2024-03-11T22:19:20.581 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:20.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:20.614 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:20.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:20.715 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:20.750 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:20.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:20.852 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:20.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:20.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:20.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:20.991 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:21.020 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:21.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:21.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:21.162 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:21.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:21.263 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:21.291 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:21.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:21.392 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:21.423 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:21.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:21.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:21.462 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:21 smithi045 ceph-mon[39715]: pgmap v507: 168 pgs: 168 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 60 MiB/s wr, 957 op/s 2024-03-11T22:19:21.462 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:21 smithi045 ceph-mon[39715]: osdmap e288: 8 total, 8 up, 8 in 2024-03-11T22:19:21.462 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:21 smithi045 ceph-mon[30178]: pgmap v507: 168 pgs: 168 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 60 MiB/s wr, 957 op/s 2024-03-11T22:19:21.463 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:21 smithi045 ceph-mon[30178]: osdmap e288: 8 total, 8 up, 8 in 2024-03-11T22:19:21.479 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:21 smithi144 ceph-mon[33984]: pgmap v507: 168 pgs: 168 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 60 MiB/s wr, 957 op/s 2024-03-11T22:19:21.480 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:21 smithi144 ceph-mon[33984]: osdmap e288: 8 total, 8 up, 8 in 2024-03-11T22:19:21.484 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 28 16 51661 51645 115.246 93.1875 0.00429274 0.00867048 2024-03-11T22:19:21.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:21.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:21.571 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:21.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:21.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:21.723 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:21.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:21.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:21.825 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:21.862 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:21.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:21.963 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:22.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:22.002 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:22.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:22.103 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:22.138 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:22.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:22.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:22.164 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:19:22.164 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:19:22.240 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:22.283 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:22.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:22.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:22.311 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:19:22.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:22 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/91369268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:19:22.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:22 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:19:22.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:22.430 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:22.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:22.487 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 29 16 52980 52964 114.114 82.4375 0.00479846 0.00875785 2024-03-11T22:19:22.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:22.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:22 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/91369268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:19:22.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:22 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:19:22.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:22 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/91369268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:19:22.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:22 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:19:22.597 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:22.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:22.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:22.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:22.737 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:22.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:22.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:22.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:22.877 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:19:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:19:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:19:22.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:22.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:22.898 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T22:19:22.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:23.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:23.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:23.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:23.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:23.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:23.269 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:19:23.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:23.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:23.302 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:23.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:23.404 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:23.413 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:23 smithi045 ceph-mon[30178]: pgmap v509: 168 pgs: 168 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 851 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-11T22:19:23.413 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:23 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:19:23.413 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:23 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:19:23.413 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:23 smithi045 ceph-mon[30178]: osdmap e289: 8 total, 8 up, 8 in 2024-03-11T22:19:23.413 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:23 smithi045 ceph-mon[39715]: pgmap v509: 168 pgs: 168 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 851 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-11T22:19:23.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:23 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:19:23.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:23 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:19:23.414 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:23 smithi045 ceph-mon[39715]: osdmap e289: 8 total, 8 up, 8 in 2024-03-11T22:19:23.433 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:23.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:23.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:23.484 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 30 16 54316 54300 113.094 83.5 0.0101207 0.00883638 2024-03-11T22:19:23.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:23.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:23.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:23 smithi144 ceph-mon[33984]: pgmap v509: 168 pgs: 168 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 851 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-11T22:19:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:23 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:19:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:23 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:19:23.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:23 smithi144 ceph-mon[33984]: osdmap e289: 8 total, 8 up, 8 in 2024-03-11T22:19:23.681 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:23.711 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:23.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:23.813 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:23.841 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:23.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:23.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:23.943 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:23.971 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:23.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:24.073 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:24.097 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:24.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:24.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:24.199 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:24.238 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:24.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:24.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:24.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:24.387 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:24.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:24.484 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 31 16 55534 55518 111.901 76.125 0.00723434 0.0089285 2024-03-11T22:19:24.488 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:24.511 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:24.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:24.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:24.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:24 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2659798013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:19:24.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:24.613 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:24.654 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:24.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:24.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:24.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:24 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2659798013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:19:24.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:24.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:24 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2659798013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:19:24.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:19:24.812 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:24.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:24.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:24.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:24.957 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:24.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:24.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:25.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:25.101 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:25.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:25.203 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:25.233 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:25.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:25.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:25.335 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:25.373 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:25.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:25.475 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:25.484 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 32 16 56677 56661 110.637 71.4375 0.0173806 0.00903381 2024-03-11T22:19:25.503 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:25.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:25.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:25.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:25 smithi144 ceph-mon[33984]: pgmap v511: 168 pgs: 168 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T22:19:25.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:25 smithi144 ceph-mon[33984]: 3.2 scrub starts 2024-03-11T22:19:25.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:25 smithi144 ceph-mon[33984]: 3.2 scrub ok 2024-03-11T22:19:25.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:25 smithi144 ceph-mon[33984]: 5.4 scrub starts 2024-03-11T22:19:25.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:25 smithi144 ceph-mon[33984]: 5.4 scrub ok 2024-03-11T22:19:25.605 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:25.629 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:25.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:25.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:25.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:25.766 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:25.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:25.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:25.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:25 smithi045 ceph-mon[39715]: pgmap v511: 168 pgs: 168 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T22:19:25.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:25 smithi045 ceph-mon[39715]: 3.2 scrub starts 2024-03-11T22:19:25.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:25 smithi045 ceph-mon[39715]: 3.2 scrub ok 2024-03-11T22:19:25.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:25 smithi045 ceph-mon[39715]: 5.4 scrub starts 2024-03-11T22:19:25.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:25 smithi045 ceph-mon[39715]: 5.4 scrub ok 2024-03-11T22:19:25.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:25 smithi045 ceph-mon[30178]: pgmap v511: 168 pgs: 168 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T22:19:25.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:25 smithi045 ceph-mon[30178]: 3.2 scrub starts 2024-03-11T22:19:25.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:25 smithi045 ceph-mon[30178]: 3.2 scrub ok 2024-03-11T22:19:25.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:25 smithi045 ceph-mon[30178]: 5.4 scrub starts 2024-03-11T22:19:25.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:25 smithi045 ceph-mon[30178]: 5.4 scrub ok 2024-03-11T22:19:25.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:25.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:25.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:25.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:25.997 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:26.026 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:26.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:26.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:26.127 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:26.156 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:26.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:26.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:26.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:26.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:26.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:26.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:26.424 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:26.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:26.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:26.484 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:26 smithi144 ceph-mon[33984]: 1.2 scrub starts 2024-03-11T22:19:26.485 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:26 smithi144 ceph-mon[33984]: 1.2 scrub ok 2024-03-11T22:19:26.485 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:26 smithi144 ceph-mon[33984]: 3.18 scrub starts 2024-03-11T22:19:26.485 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:26 smithi144 ceph-mon[33984]: 3.18 scrub ok 2024-03-11T22:19:26.485 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:26 smithi144 ceph-mon[33984]: osdmap e290: 8 total, 8 up, 8 in 2024-03-11T22:19:26.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:19:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:19:26.485 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 33 16 57748 57732 109.313 66.9375 0.0083767 0.00914226 2024-03-11T22:19:26.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:26.561 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:26.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:26.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:26.687 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:26.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:26.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:26 smithi045 ceph-mon[30178]: 1.2 scrub starts 2024-03-11T22:19:26.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:26 smithi045 ceph-mon[30178]: 1.2 scrub ok 2024-03-11T22:19:26.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:26 smithi045 ceph-mon[30178]: 3.18 scrub starts 2024-03-11T22:19:26.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:26 smithi045 ceph-mon[30178]: 3.18 scrub ok 2024-03-11T22:19:26.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:26 smithi045 ceph-mon[30178]: osdmap e290: 8 total, 8 up, 8 in 2024-03-11T22:19:26.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:26 smithi045 ceph-mon[39715]: 1.2 scrub starts 2024-03-11T22:19:26.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:26 smithi045 ceph-mon[39715]: 1.2 scrub ok 2024-03-11T22:19:26.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:26 smithi045 ceph-mon[39715]: 3.18 scrub starts 2024-03-11T22:19:26.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:26 smithi045 ceph-mon[39715]: 3.18 scrub ok 2024-03-11T22:19:26.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:26 smithi045 ceph-mon[39715]: osdmap e290: 8 total, 8 up, 8 in 2024-03-11T22:19:26.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:26.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:26.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:26.929 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:26.959 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:26.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:26.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:27.061 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:27.086 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:27.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:27.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:27.120 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:19:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-03-11T22:19:27.120 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-03-11T22:19:27.121 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:19:27.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:27.231 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:27.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:27.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:27.265 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:19:27.266 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:19:27.266 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:19:27.266 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:19:27.266 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:19:27.266 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:19:27.266 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:19:27.266 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:19:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:19:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:19:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:19:27.266 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:19:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:19:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:19:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:19:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:19:27.332 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:27.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:27.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:27 smithi144 ceph-mon[33984]: pgmap v513: 168 pgs: 168 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 125 MiB/s wr, 1.99k op/s 2024-03-11T22:19:27.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:27 smithi144 ceph-mon[33984]: 6.10 scrub starts 2024-03-11T22:19:27.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:27 smithi144 ceph-mon[33984]: 6.10 scrub ok 2024-03-11T22:19:27.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:27 smithi144 ceph-mon[33984]: 6.1a scrub starts 2024-03-11T22:19:27.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:27 smithi144 ceph-mon[33984]: 6.1a scrub ok 2024-03-11T22:19:27.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:27 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:27.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:27 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:27.346 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:27 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:27.347 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:27 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:27.347 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:27 smithi144 ceph-mon[33984]: Adjusting osd_memory_target on smithi144 to 3248M 2024-03-11T22:19:27.347 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:27 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:19:27.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:27.360 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:27.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:27.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:27.485 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 34 16 58808 58792 108.046 66.25 0.00843173 0.00925002 2024-03-11T22:19:27.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:27.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:27.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:27.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[39715]: pgmap v513: 168 pgs: 168 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 125 MiB/s wr, 1.99k op/s 2024-03-11T22:19:27.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[39715]: 6.10 scrub starts 2024-03-11T22:19:27.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[39715]: 6.10 scrub ok 2024-03-11T22:19:27.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[39715]: 6.1a scrub starts 2024-03-11T22:19:27.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[39715]: 6.1a scrub ok 2024-03-11T22:19:27.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:27.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:27.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:27.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:27.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[39715]: Adjusting osd_memory_target on smithi144 to 3248M 2024-03-11T22:19:27.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:19:27.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:27.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[30178]: pgmap v513: 168 pgs: 168 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 125 MiB/s wr, 1.99k op/s 2024-03-11T22:19:27.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[30178]: 6.10 scrub starts 2024-03-11T22:19:27.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[30178]: 6.10 scrub ok 2024-03-11T22:19:27.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[30178]: 6.1a scrub starts 2024-03-11T22:19:27.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[30178]: 6.1a scrub ok 2024-03-11T22:19:27.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:27.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:27.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:27.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:19:27.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[30178]: Adjusting osd_memory_target on smithi144 to 3248M 2024-03-11T22:19:27.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:27 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:19:27.601 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:27.626 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:27.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:27.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:27.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:27.771 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:27.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:27.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:27.785 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T22:19:27.873 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:27.914 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:27.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:27.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:28.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:28.053 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:28.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:28.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:28.099 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:19:28.155 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:28.186 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:28.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:28.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:28.328 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:28.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:28.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:28.472 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:28.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:28.485 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 35 16 59962 59946 107.02 72.125 0.0085298 0.00933867 2024-03-11T22:19:28.541 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:28 smithi144 ceph-mon[33984]: 1.13 scrub starts 2024-03-11T22:19:28.541 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:28 smithi144 ceph-mon[33984]: 1.13 scrub ok 2024-03-11T22:19:28.541 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:28 smithi144 ceph-mon[33984]: 6.1c scrub starts 2024-03-11T22:19:28.541 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:28 smithi144 ceph-mon[33984]: 6.1c scrub ok 2024-03-11T22:19:28.575 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:28.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:28.604 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:28.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:28.705 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:28.735 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:28.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:28 smithi045 ceph-mon[30178]: 1.13 scrub starts 2024-03-11T22:19:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:28 smithi045 ceph-mon[30178]: 1.13 scrub ok 2024-03-11T22:19:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:28 smithi045 ceph-mon[30178]: 6.1c scrub starts 2024-03-11T22:19:28.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:28 smithi045 ceph-mon[30178]: 6.1c scrub ok 2024-03-11T22:19:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:28 smithi045 ceph-mon[39715]: 1.13 scrub starts 2024-03-11T22:19:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:28 smithi045 ceph-mon[39715]: 1.13 scrub ok 2024-03-11T22:19:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:28 smithi045 ceph-mon[39715]: 6.1c scrub starts 2024-03-11T22:19:28.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:28 smithi045 ceph-mon[39715]: 6.1c scrub ok 2024-03-11T22:19:28.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:28.879 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:28.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:28.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:28.974 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:28.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:29.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:29.055 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:29.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:29.156 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:29.183 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:29.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:29.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:29.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:29.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:29.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:29.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:29.427 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:29.465 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:29.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:29.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:29.485 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 36 15 61247 61232 106.28 80.375 0.0068903 0.00940341 2024-03-11T22:19:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[39715]: pgmap v514: 168 pgs: 168 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 730 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-11T22:19:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[39715]: 3.6 scrub starts 2024-03-11T22:19:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[39715]: 3.6 scrub ok 2024-03-11T22:19:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[39715]: 3.b scrub starts 2024-03-11T22:19:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[39715]: 3.b scrub ok 2024-03-11T22:19:29.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:29.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:29.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:19:29.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[30178]: pgmap v514: 168 pgs: 168 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 730 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-11T22:19:29.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[30178]: 3.6 scrub starts 2024-03-11T22:19:29.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[30178]: 3.6 scrub ok 2024-03-11T22:19:29.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[30178]: 3.b scrub starts 2024-03-11T22:19:29.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[30178]: 3.b scrub ok 2024-03-11T22:19:29.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:29.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:29.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:29 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:19:29.568 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:29.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:29 smithi144 ceph-mon[33984]: pgmap v514: 168 pgs: 168 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 730 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-11T22:19:29.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:29 smithi144 ceph-mon[33984]: 3.6 scrub starts 2024-03-11T22:19:29.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:29 smithi144 ceph-mon[33984]: 3.6 scrub ok 2024-03-11T22:19:29.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:29 smithi144 ceph-mon[33984]: 3.b scrub starts 2024-03-11T22:19:29.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:29 smithi144 ceph-mon[33984]: 3.b scrub ok 2024-03-11T22:19:29.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:29 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:19:29.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:29 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:19:29.591 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:29 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:19:29.603 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:29.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:29.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:29.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:29.743 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:29.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:29.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:29.872 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:29.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:29.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:29.974 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:30.006 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:30.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:30.033 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:19:30.033 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-03-11T22:19:30.033 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-03-11T22:19:30.033 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-03-11T22:19:30.033 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:19:30.108 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:30.139 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:30.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:30.240 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:30.270 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:30.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:30.372 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:30.404 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:30.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:30.485 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 37 16 62463 62447 105.459 75.9375 0.00817371 0.0094771 2024-03-11T22:19:30.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:30 smithi045 ceph-mon[30178]: 1.11 scrub starts 2024-03-11T22:19:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:30 smithi045 ceph-mon[30178]: 1.3 scrub starts 2024-03-11T22:19:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:30 smithi045 ceph-mon[30178]: 1.11 scrub ok 2024-03-11T22:19:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:30 smithi045 ceph-mon[30178]: 1.3 scrub ok 2024-03-11T22:19:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:30 smithi045 ceph-mon[30178]: 4.15 scrub starts 2024-03-11T22:19:30.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:30 smithi045 ceph-mon[30178]: 4.15 scrub ok 2024-03-11T22:19:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:30 smithi045 ceph-mon[39715]: 1.11 scrub starts 2024-03-11T22:19:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:30 smithi045 ceph-mon[39715]: 1.3 scrub starts 2024-03-11T22:19:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:30 smithi045 ceph-mon[39715]: 1.11 scrub ok 2024-03-11T22:19:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:30 smithi045 ceph-mon[39715]: 1.3 scrub ok 2024-03-11T22:19:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:30 smithi045 ceph-mon[39715]: 4.15 scrub starts 2024-03-11T22:19:30.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:30 smithi045 ceph-mon[39715]: 4.15 scrub ok 2024-03-11T22:19:30.545 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:30.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:30.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:30.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:30 smithi144 ceph-mon[33984]: 1.11 scrub starts 2024-03-11T22:19:30.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:30 smithi144 ceph-mon[33984]: 1.3 scrub starts 2024-03-11T22:19:30.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:30 smithi144 ceph-mon[33984]: 1.11 scrub ok 2024-03-11T22:19:30.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:30 smithi144 ceph-mon[33984]: 1.3 scrub ok 2024-03-11T22:19:30.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:30 smithi144 ceph-mon[33984]: 4.15 scrub starts 2024-03-11T22:19:30.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:30 smithi144 ceph-mon[33984]: 4.15 scrub ok 2024-03-11T22:19:30.648 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:30.682 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:30.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:30.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:30.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:30.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:30.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:30.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:30.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:30.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:31.019 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T22:19:31.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:19:31.073 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:31.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:31.104 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:31.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:31.205 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:31.237 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:31.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:31.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:31.352 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:19:31.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:31.388 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:31.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:31.487 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 38 16 63735 63719 104.773 79.5 0.00886272 0.00953785 2024-03-11T22:19:31.489 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:31.517 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:31.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:31.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:31 smithi144 ceph-mon[33984]: pgmap v515: 168 pgs: 168 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 784 op/s 2024-03-11T22:19:31.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:19:31.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:31 smithi144 ceph-mon[33984]: 4.18 scrub starts 2024-03-11T22:19:31.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:31 smithi144 ceph-mon[33984]: 4.18 scrub ok 2024-03-11T22:19:31.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:31.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:19:31.665 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:31.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:31.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:31 smithi045 ceph-mon[30178]: pgmap v515: 168 pgs: 168 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 784 op/s 2024-03-11T22:19:31.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:19:31.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:31 smithi045 ceph-mon[30178]: 4.18 scrub starts 2024-03-11T22:19:31.671 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:31 smithi045 ceph-mon[30178]: 4.18 scrub ok 2024-03-11T22:19:31.671 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:31 smithi045 ceph-mon[39715]: pgmap v515: 168 pgs: 168 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 784 op/s 2024-03-11T22:19:31.671 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:19:31.671 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:31 smithi045 ceph-mon[39715]: 4.18 scrub starts 2024-03-11T22:19:31.671 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:31 smithi045 ceph-mon[39715]: 4.18 scrub ok 2024-03-11T22:19:31.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:31.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:31.822 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:31.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:31.922 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:31.952 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:31.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:32.054 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:32.082 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:32.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:32.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:32.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:32.240 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:32.251 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:32.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:32.352 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:32.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:32.487 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 39 15 64862 64847 103.894 70.5 0.0114447 0.00961998 2024-03-11T22:19:32.506 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:32.535 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:32.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:32.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:32.637 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:32.666 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:32.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:32.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:32.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:32.782 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:19:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:19:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:19:32.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:32 smithi045 ceph-mon[39715]: 4.2 scrub starts 2024-03-11T22:19:32.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:32 smithi045 ceph-mon[39715]: 4.2 scrub ok 2024-03-11T22:19:32.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:32 smithi045 ceph-mon[30178]: 4.2 scrub starts 2024-03-11T22:19:32.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:32 smithi045 ceph-mon[30178]: 4.2 scrub ok 2024-03-11T22:19:32.802 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:32 smithi144 ceph-mon[33984]: 4.2 scrub starts 2024-03-11T22:19:32.802 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:32 smithi144 ceph-mon[33984]: 4.2 scrub ok 2024-03-11T22:19:32.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:32.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:32.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:32.911 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:32.947 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:32.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:32.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:33.048 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:33.082 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:33.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:33.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:33.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:33.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:33.242 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:33.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:33.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:33.381 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T22:19:33.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:33.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:33.456 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:33 smithi144 ceph-mon[33984]: pgmap v516: 168 pgs: 168 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T22:19:33.456 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:33 smithi144 ceph-mon[33984]: 5.14 scrub starts 2024-03-11T22:19:33.456 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:33 smithi144 ceph-mon[33984]: 5.14 scrub ok 2024-03-11T22:19:33.456 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:33 smithi144 ceph-mon[33984]: 4.b scrub starts 2024-03-11T22:19:33.456 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:33 smithi144 ceph-mon[33984]: 4.b scrub ok 2024-03-11T22:19:33.456 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:33 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2034536677' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:19:33.456 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:33 smithi144 ceph-mon[33984]: 6.12 scrub starts 2024-03-11T22:19:33.456 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:33 smithi144 ceph-mon[33984]: 6.12 scrub ok 2024-03-11T22:19:33.458 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:19:33.458 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-03-11T22:19:33.458 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-03-11T22:19:33.458 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:19:33.458 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.458 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24335.0:1130 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94036442452352 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-03-11T22:19:33.458 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:34.398788+0000", 2024-03-11T22:19:33.458 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 539.05377614099996, 2024-03-11T22:19:33.458 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.019798728, 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24335", 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/3033943457", 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 1130 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398788+0000", 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.459 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398788+0000", 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398791+0000", 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2590000000000001e-06 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398805+0000", 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3721e-05 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.460 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398835+0000", 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0238000000000001e-05 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398851+0000", 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5838999999999999e-05 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398945+0000", 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3986000000000004e-05 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.461 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op must wait for map", 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.398953+0000", 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9119999999999998e-06 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.406225+0000", 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0072720850000000002 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:34.406238+0000", 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2743e-05 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.462 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:35.414245+0000", 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.008007466 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:35.414414+0000", 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016910300000000001 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:35.418587+0000", 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0041723760000000002 2024-03-11T22:19:33.463 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:26438 7.7 7:e97d8559:::benchmark_data_smithi045_83007_object26437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:54.971431+0000", 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 518.48113367300004, 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.15983009400000001, 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 26438 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.464 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971431+0000", 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971431+0000", 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971432+0000", 2024-03-11T22:19:33.465 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.265e-06 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971483+0000", 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1215999999999999e-05 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971499+0000", 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5633000000000001e-05 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971507+0000", 2024-03-11T22:19:33.466 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8539999999999997e-06 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971568+0000", 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1066000000000001e-05 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971680+0000", 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000112273 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.971792+0000", 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011223200000000001 2024-03-11T22:19:33.467 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.972903+0000", 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001111293 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:54.975356+0000", 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024528470000000002 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.131147+0000", 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.155790505 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.468 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.131211+0000", 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3792000000000002e-05 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.131261+0000", 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0117999999999997e-05 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:27627 7.c 7:3bf1c813:::benchmark_data_smithi045_83007_object27626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:55.718204+0000", 2024-03-11T22:19:33.469 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 517.73436067499995, 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13720726899999999, 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 27627 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718204+0000", 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.470 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718204+0000", 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718205+0000", 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6050000000000001e-06 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718251+0000", 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6177999999999997e-05 2024-03-11T22:19:33.471 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718265+0000", 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3200999999999999e-05 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718272+0000", 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0539999999999997e-06 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718328+0000", 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6705e-05 2024-03-11T22:19:33.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718465+0000", 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013699599999999999 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.718557+0000", 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1218999999999995e-05 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.719504+0000", 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00094716799999999997 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.719651+0000", 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000147496 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.855305+0000", 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13565408200000001 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.855364+0000", 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8455999999999997e-05 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.474 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:55.855411+0000", 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7108999999999999e-05 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:28077 7.c 7:32c5189f:::benchmark_data_smithi045_83007_object28076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:56.079615+0000", 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 517.37294900899997, 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13117891700000001, 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.475 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 28077 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079615+0000", 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079615+0000", 2024-03-11T22:19:33.476 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079617+0000", 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4440000000000001e-06 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079648+0000", 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1707e-05 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.477 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079660+0000", 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1491999999999999e-05 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079668+0000", 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1869999999999997e-06 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079717+0000", 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9237999999999999e-05 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.478 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079871+0000", 2024-03-11T22:19:33.479 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015396299999999999 2024-03-11T22:19:33.479 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.479 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.479 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:19:33.479 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.080033+0000", 2024-03-11T22:19:33.479 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016142199999999999 2024-03-11T22:19:33.479 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.479 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.479 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.479 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.081317+0000", 2024-03-11T22:19:33.479 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001284232 2024-03-11T22:19:33.479 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.479 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.081507+0000", 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018976400000000001 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210699+0000", 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.12919265399999999 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210746+0000", 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6906000000000001e-05 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.480 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.481 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.481 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210794+0000", 2024-03-11T22:19:33.481 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7908000000000003e-05 2024-03-11T22:19:33.481 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.481 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.481 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.481 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.481 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.481 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:28080 7.7 7:eb74b751:::benchmark_data_smithi045_83007_object28079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-11T22:19:33.481 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:10:56.079682+0000", 2024-03-11T22:19:33.481 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 517.37288247200001, 2024-03-11T22:19:33.481 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.131107114, 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 28080 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079682+0000", 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.482 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079682+0000", 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079682+0000", 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0200000000000002e-07 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079704+0000", 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1733e-05 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.483 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.484 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.484 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079711+0000", 2024-03-11T22:19:33.484 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0609999999999998e-06 2024-03-11T22:19:33.484 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.484 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.484 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.079718+0000", 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3080000000000002e-06 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.080007+0000", 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00028894499999999998 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.080112+0000", 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000104268 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:19:33.485 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.080196+0000", 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4815999999999996e-05 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.081217+0000", 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010204439999999999 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.208091+0000", 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.12687417200000001 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.486 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210683+0000", 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0025918550000000001 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210746+0000", 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2639999999999997e-05 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:10:56.210789+0000", 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.337e-05 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.487 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:41624 7.d 7:bc5fdf06:::benchmark_data_smithi045_83007_object41623:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.868679+0000", 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 509.58388484400001, 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.112665528, 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 41624 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.488 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868679+0000", 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868679+0000", 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868680+0000", 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6499999999999999e-07 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.489 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868724+0000", 2024-03-11T22:19:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4027999999999999e-05 2024-03-11T22:19:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868732+0000", 2024-03-11T22:19:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6240000000000002e-06 2024-03-11T22:19:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868741+0000", 2024-03-11T22:19:33.490 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8060000000000001e-06 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868771+0000", 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0066e-05 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868871+0000", 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010005600000000001 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868991+0000", 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000120525 2024-03-11T22:19:33.491 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981178+0000", 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11218715 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981193+0000", 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5003e-05 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981229+0000", 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5941e-05 2024-03-11T22:19:33.492 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981306+0000", 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6853000000000004e-05 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981345+0000", 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8810999999999998e-05 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.493 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:41628 7.d 7:b3acd120:::benchmark_data_smithi045_83007_object41627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.868789+0000", 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 509.58377520499999, 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11711809199999999, 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 41628 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868789+0000", 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.494 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868789+0000", 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868790+0000", 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9800000000000004e-07 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868858+0000", 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8860000000000001e-05 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.495 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868864+0000", 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2249999999999999e-06 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868867+0000", 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.613e-06 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.870148+0000", 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0012809970000000001 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.496 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.870273+0000", 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012430699999999999 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.979501+0000", 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.109228742 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.985659+0000", 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0061581730000000003 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.497 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.985719+0000", 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9308999999999997e-05 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.985807+0000", 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7916000000000003e-05 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.985864+0000", 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7082e-05 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.498 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.985907+0000", 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.337e-05 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:41629 7.c 7:386f5e1e:::benchmark_data_smithi045_83007_object41628:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:11:03.868874+0000", 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 509.58369044900002, 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.112468074, 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.499 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 41629 2024-03-11T22:19:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868874+0000", 2024-03-11T22:19:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868874+0000", 2024-03-11T22:19:33.500 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868874+0000", 2024-03-11T22:19:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4200000000000002e-07 2024-03-11T22:19:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868904+0000", 2024-03-11T22:19:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9839999999999999e-05 2024-03-11T22:19:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.501 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868908+0000", 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4379999999999997e-06 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868915+0000", 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5169999999999996e-06 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.868960+0000", 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4909e-05 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.502 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.869049+0000", 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8858999999999997e-05 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.869144+0000", 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.5629000000000002e-05 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.979159+0000", 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.110014764 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.503 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.979730+0000", 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00057078200000000004 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981217+0000", 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001486996 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981290+0000", 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3380999999999994e-05 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.504 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:11:03.981342+0000", 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1616999999999997e-05 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:141386 7.c 7:3f6866e0:::benchmark_data_smithi045_83007_object22915:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:12:37.304655+0000", 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 416.14790933400002, 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.112149399, 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.505 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 141386 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304655+0000", 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304655+0000", 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.506 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304656+0000", 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3200000000000001e-06 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304660+0000", 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2080000000000002e-06 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304672+0000", 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1901e-05 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.507 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304679+0000", 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1069999999999999e-06 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304726+0000", 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6161000000000001e-05 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304813+0000", 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.7842000000000004e-05 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.508 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,7", 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.304887+0000", 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3063000000000002e-05 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.308138+0000", 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003251463 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.308322+0000", 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000184146 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.509 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.510 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.416735+0000", 2024-03-11T22:19:33.510 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.108413038 2024-03-11T22:19:33.510 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.510 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.510 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.510 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.416771+0000", 2024-03-11T22:19:33.510 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6125999999999999e-05 2024-03-11T22:19:33.510 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.510 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.510 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.510 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:12:37.416804+0000", 2024-03-11T22:19:33.510 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3024000000000001e-05 2024-03-11T22:19:33.510 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:185996 7.7 7:ef067263:::benchmark_data_smithi045_83007_object67525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:22.317287+0000", 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 371.13527718099999, 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.11014377, 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 185996 2024-03-11T22:19:33.511 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317287+0000", 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317287+0000", 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.512 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317288+0000", 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6400000000000001e-07 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317292+0000", 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.566e-06 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317307+0000", 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4748e-05 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.513 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317315+0000", 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3830000000000003e-06 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317372+0000", 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7550000000000003e-05 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.317605+0000", 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00023298599999999999 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:19:33.514 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.320901+0000", 2024-03-11T22:19:33.515 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003296284 2024-03-11T22:19:33.515 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.515 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.515 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.515 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.323882+0000", 2024-03-11T22:19:33.515 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0029803780000000001 2024-03-11T22:19:33.515 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.515 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.515 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.515 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.374715+0000", 2024-03-11T22:19:33.515 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.050833138 2024-03-11T22:19:33.515 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.427331+0000", 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.052615833000000001 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.427383+0000", 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2349999999999999e-05 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:22.427431+0000", 2024-03-11T22:19:33.516 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7790000000000002e-05 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:200301 7.7 7:ef47b068:::benchmark_data_smithi045_83007_object81830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:38.577251+0000", 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 354.875313683, 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.111378923, 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 200301 2024-03-11T22:19:33.517 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.577251+0000", 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.577251+0000", 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.518 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.577251+0000", 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3099999999999999e-07 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.577253+0000", 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7889999999999999e-06 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.577258+0000", 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.169e-06 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.577263+0000", 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7910000000000004e-06 2024-03-11T22:19:33.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.579849+0000", 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0025862680000000001 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.579941+0000", 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.2008000000000004e-05 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.580011+0000", 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0154999999999995e-05 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.520 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.581929+0000", 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0019174629999999999 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.661648+0000", 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.079719124000000002 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.688539+0000", 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.026891770999999998 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.521 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.688588+0000", 2024-03-11T22:19:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8210000000000001e-05 2024-03-11T22:19:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:38.688630+0000", 2024-03-11T22:19:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1844000000000001e-05 2024-03-11T22:19:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.522 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:33.524 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:19:33.486195+0000 min lat: 0.00145181 max lat: 0.218324 avg lat: 0.00968238 2024-03-11T22:19:33.524 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:19:33.524 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 40 16 66097 66081 103.225 77.125 0.00935154 0.00968238 2024-03-11T22:19:33.524 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:204820 7.7 7:ec16411b:::benchmark_data_smithi045_83007_object86349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:19:33.524 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:44.296015+0000", 2024-03-11T22:19:33.524 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 349.15654896199999, 2024-03-11T22:19:33.524 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.129960253, 2024-03-11T22:19:33.524 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.525 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.525 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.525 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.525 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.525 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 204820 2024-03-11T22:19:33.525 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.525 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.525 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.525 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.525 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296015+0000", 2024-03-11T22:19:33.525 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.525 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.525 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296015+0000", 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296017+0000", 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5030000000000001e-06 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296029+0000", 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2198000000000001e-05 2024-03-11T22:19:33.526 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296040+0000", 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1009000000000001e-05 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296046+0000", 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1859999999999997e-06 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296097+0000", 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0995999999999998e-05 2024-03-11T22:19:33.527 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296183+0000", 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5535e-05 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296254+0000", 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1216000000000004e-05 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.298666+0000", 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002411641 2024-03-11T22:19:33.528 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.418974+0000", 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.120308251 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.425855+0000", 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0068807549999999997 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.425924+0000", 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8912999999999996e-05 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.425976+0000", 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2049999999999998e-05 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:205790 7.d 7:bd5a76ed:::benchmark_data_smithi045_83007_object87319:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:45.686540+0000", 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 347.76602384400002, 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.157224592, 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.530 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 205790 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686540+0000", 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686540+0000", 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686542+0000", 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4649999999999999e-06 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686546+0000", 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5700000000000003e-06 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686560+0000", 2024-03-11T22:19:33.532 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3190000000000001e-05 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686567+0000", 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7360000000000001e-06 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686609+0000", 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1242000000000003e-05 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686695+0000", 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6576000000000006e-05 2024-03-11T22:19:33.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.534 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:19:33.534 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686764+0000", 2024-03-11T22:19:33.534 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8880999999999993e-05 2024-03-11T22:19:33.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.534 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.534 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.689199+0000", 2024-03-11T22:19:33.534 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024353690000000002 2024-03-11T22:19:33.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.534 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.691011+0000", 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018111799999999999 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.843698+0000", 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.15268752499999999 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.843732+0000", 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3476000000000003e-05 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.535 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.843765+0000", 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3382e-05 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:228199 7.7 7:ee665074:::benchmark_data_smithi045_83007_object109728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:09.462446+0000", 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 323.99011860000002, 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.14675967300000001, 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.536 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 228199 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462446+0000", 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462446+0000", 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462447+0000", 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9100000000000002e-07 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462451+0000", 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2880000000000003e-06 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462466+0000", 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.467e-05 2024-03-11T22:19:33.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462473+0000", 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7500000000000003e-06 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462524+0000", 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1223000000000001e-05 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462633+0000", 2024-03-11T22:19:33.539 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000108226 2024-03-11T22:19:33.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.540 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.540 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:19:33.540 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462729+0000", 2024-03-11T22:19:33.540 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.5805999999999997e-05 2024-03-11T22:19:33.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.540 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.540 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.540 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.465857+0000", 2024-03-11T22:19:33.540 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0031280800000000001 2024-03-11T22:19:33.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.540 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.540 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.590331+0000", 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.12447485599999999 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.609081+0000", 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.018749670999999999 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.609142+0000", 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0516000000000001e-05 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.542 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.542 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.609205+0000", 2024-03-11T22:19:33.542 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3695999999999995e-05 2024-03-11T22:19:33.542 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.542 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.542 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.542 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.542 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.542 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:228205 7.d 7:bc358dc8:::benchmark_data_smithi045_83007_object109734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:19:33.542 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:09.466216+0000", 2024-03-11T22:19:33.542 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 323.98634871399997, 2024-03-11T22:19:33.542 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13463407299999999, 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 228205 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466216+0000", 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.543 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466216+0000", 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466217+0000", 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7199999999999997e-07 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466221+0000", 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5539999999999999e-06 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.544 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466230+0000", 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1810000000000007e-06 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466236+0000", 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4970000000000001e-06 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466283+0000", 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7160999999999998e-05 2024-03-11T22:19:33.545 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466380+0000", 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7510000000000007e-05 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466508+0000", 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012758100000000001 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.469514+0000", 2024-03-11T22:19:33.546 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003005495 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.469937+0000", 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00042370599999999998 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.600748+0000", 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13081040799999999 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.600800+0000", 2024-03-11T22:19:33.547 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2818000000000002e-05 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.600850+0000", 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9190000000000002e-05 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229663 7.d 7:bdf7b07c:::benchmark_data_smithi045_83007_object111192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:11.125093+0000", 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 322.32747132100002, 2024-03-11T22:19:33.548 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.17061443900000001, 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229663 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.125093+0000", 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.549 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.125093+0000", 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.125093+0000", 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.84e-07 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.125095+0000", 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.984e-06 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.551 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.551 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.125102+0000", 2024-03-11T22:19:33.551 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.075e-06 2024-03-11T22:19:33.551 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.551 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.551 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.125108+0000", 2024-03-11T22:19:33.551 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1710000000000001e-06 2024-03-11T22:19:33.551 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.551 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.551 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129918+0000", 2024-03-11T22:19:33.551 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0048100460000000001 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.130040+0000", 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000122862 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.130170+0000", 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000130001 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.552 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.139457+0000", 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0092864469999999998 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.142545+0000", 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0030881459999999999 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.295632+0000", 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.15308732 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.553 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.295669+0000", 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6362000000000002e-05 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.295707+0000", 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8640999999999998e-05 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229666 7.d 7:b2f2096b:::benchmark_data_smithi045_83007_object111195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:11.129450+0000", 2024-03-11T22:19:33.554 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 322.32311436100002, 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.16635360900000001, 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229666 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129450+0000", 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129450+0000", 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129451+0000", 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.19e-06 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129456+0000", 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7119999999999999e-06 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129467+0000", 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1005999999999999e-05 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129474+0000", 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9550000000000003e-06 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.133139+0000", 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0036649059999999999 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.558 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.558 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.133269+0000", 2024-03-11T22:19:33.558 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012985899999999999 2024-03-11T22:19:33.558 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.558 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.558 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:19:33.558 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.133398+0000", 2024-03-11T22:19:33.558 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012982900000000001 2024-03-11T22:19:33.558 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.558 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.558 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.558 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.139462+0000", 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0060638630000000001 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.150243+0000", 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010780919999999999 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.295743+0000", 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.14549944300000001 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.559 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.295776+0000", 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3683999999999998e-05 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.295804+0000", 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7242000000000001e-05 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229669 7.d 7:bd6d76c8:::benchmark_data_smithi045_83007_object111198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:19:33.560 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:11.131546+0000", 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 322.32101777000003, 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.164917068, 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229669 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.131546+0000", 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.561 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.562 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.562 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.131546+0000", 2024-03-11T22:19:33.562 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.562 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.562 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.131548+0000", 2024-03-11T22:19:33.562 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.124e-06 2024-03-11T22:19:33.562 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.562 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.562 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.562 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.131553+0000", 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0159999999999999e-06 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.131567+0000", 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4267000000000001e-05 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.131575+0000", 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9060000000000002e-06 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.563 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.564 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.135734+0000", 2024-03-11T22:19:33.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.004159177 2024-03-11T22:19:33.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.564 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.564 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.135856+0000", 2024-03-11T22:19:33.564 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012204100000000001 2024-03-11T22:19:33.564 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.564 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.565 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:19:33.565 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.135975+0000", 2024-03-11T22:19:33.565 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000118817 2024-03-11T22:19:33.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.566 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.566 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.139791+0000", 2024-03-11T22:19:33.566 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0038162249999999999 2024-03-11T22:19:33.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.566 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.566 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.150272+0000", 2024-03-11T22:19:33.566 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010480487 2024-03-11T22:19:33.566 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.566 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.566 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296386+0000", 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.14611407200000001 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296428+0000", 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2633999999999999e-05 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296464+0000", 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5302000000000002e-05 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.567 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229681 7.d 7:b04b550f:::benchmark_data_smithi045_83007_object111210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:11.144736+0000", 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 322.30782826500001, 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.151893167, 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229681 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.568 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144736+0000", 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144736+0000", 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144737+0000", 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4360000000000001e-06 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.569 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144743+0000", 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6849999999999998e-06 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144756+0000", 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3251000000000001e-05 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144765+0000", 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2709999999999992e-06 2024-03-11T22:19:33.570 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144814+0000", 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9109e-05 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144902+0000", 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8319000000000006e-05 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144975+0000", 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2630000000000001e-05 2024-03-11T22:19:33.571 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.150217+0000", 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0052418409999999997 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.150302+0000", 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5792999999999997e-05 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296526+0000", 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.14622318300000001 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.572 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296586+0000", 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0689000000000001e-05 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296629+0000", 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2960000000000002e-05 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229683 7.d 7:b3eb5a94:::benchmark_data_smithi045_83007_object111212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:11.145873+0000", 2024-03-11T22:19:33.573 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 322.30669144199999, 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.15093811200000001, 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229683 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.145873+0000", 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.574 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.145873+0000", 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.145874+0000", 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2099999999999996e-07 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.145877+0000", 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9480000000000001e-06 2024-03-11T22:19:33.575 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.145888+0000", 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0115000000000001e-05 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.145896+0000", 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3189999999999995e-06 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.147846+0000", 2024-03-11T22:19:33.576 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0019499140000000001 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.147952+0000", 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000106032 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.148030+0000", 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8530999999999997e-05 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.150280+0000", 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002249858 2024-03-11T22:19:33.577 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.578 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.578 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.292276+0000", 2024-03-11T22:19:33.578 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.14199621200000001 2024-03-11T22:19:33.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.578 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:33.578 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296691+0000", 2024-03-11T22:19:33.578 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0044147809999999996 2024-03-11T22:19:33.578 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.578 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.578 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:19:33.578 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296768+0000", 2024-03-11T22:19:33.579 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6314000000000002e-05 2024-03-11T22:19:33.579 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:33.579 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:33.579 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:19:33.579 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296811+0000", 2024-03-11T22:19:33.579 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3367e-05 2024-03-11T22:19:33.579 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.579 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.579 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.579 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:33.579 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:33.579 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:19:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:33.580 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:33.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:33.681 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:33.712 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:33.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:33.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[30178]: pgmap v516: 168 pgs: 168 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T22:19:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[30178]: 5.14 scrub starts 2024-03-11T22:19:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[30178]: 5.14 scrub ok 2024-03-11T22:19:33.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[30178]: 4.b scrub starts 2024-03-11T22:19:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[30178]: 4.b scrub ok 2024-03-11T22:19:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2034536677' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:19:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[30178]: 6.12 scrub starts 2024-03-11T22:19:33.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[30178]: 6.12 scrub ok 2024-03-11T22:19:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[39715]: pgmap v516: 168 pgs: 168 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-11T22:19:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[39715]: 5.14 scrub starts 2024-03-11T22:19:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[39715]: 5.14 scrub ok 2024-03-11T22:19:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[39715]: 4.b scrub starts 2024-03-11T22:19:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[39715]: 4.b scrub ok 2024-03-11T22:19:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2034536677' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:19:33.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[39715]: 6.12 scrub starts 2024-03-11T22:19:33.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:33 smithi045 ceph-mon[39715]: 6.12 scrub ok 2024-03-11T22:19:33.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:33.851 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:33.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:33.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:33.953 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:33.983 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:33.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:33.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:34.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:34.129 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:34.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:34.230 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:34.260 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:34.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:34.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:34.362 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:34.374 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T22:19:34.395 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:34.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:34.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:34.486 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 41 16 67380 67364 102.663 80.1875 0.0112803 0.009735 2024-03-11T22:19:34.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:34.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:34 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2034536677' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:19:34.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:34 smithi045 ceph-mon[39715]: osdmap e291: 8 total, 8 up, 8 in 2024-03-11T22:19:34.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:34 smithi045 ceph-mon[39715]: 6.a scrub starts 2024-03-11T22:19:34.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:34 smithi045 ceph-mon[39715]: 6.a scrub ok 2024-03-11T22:19:34.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:34 smithi045 ceph-mon[39715]: 6.14 scrub starts 2024-03-11T22:19:34.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:34 smithi045 ceph-mon[39715]: 6.14 scrub ok 2024-03-11T22:19:34.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:34 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2034536677' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:19:34.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:34 smithi045 ceph-mon[30178]: osdmap e291: 8 total, 8 up, 8 in 2024-03-11T22:19:34.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:34 smithi045 ceph-mon[30178]: 6.a scrub starts 2024-03-11T22:19:34.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:34 smithi045 ceph-mon[30178]: 6.a scrub ok 2024-03-11T22:19:34.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:34 smithi045 ceph-mon[30178]: 6.14 scrub starts 2024-03-11T22:19:34.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:34 smithi045 ceph-mon[30178]: 6.14 scrub ok 2024-03-11T22:19:34.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:34.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:34.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:34 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2034536677' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:19:34.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:34 smithi144 ceph-mon[33984]: osdmap e291: 8 total, 8 up, 8 in 2024-03-11T22:19:34.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:34 smithi144 ceph-mon[33984]: 6.a scrub starts 2024-03-11T22:19:34.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:34 smithi144 ceph-mon[33984]: 6.a scrub ok 2024-03-11T22:19:34.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:34 smithi144 ceph-mon[33984]: 6.14 scrub starts 2024-03-11T22:19:34.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:34 smithi144 ceph-mon[33984]: 6.14 scrub ok 2024-03-11T22:19:34.638 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:34.673 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:34.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:34.734 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:19:34.775 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:34.803 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:34.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:34.905 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:34.937 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:34.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:34.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:35.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:35.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:35.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:35.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:35.181 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:35.219 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:35.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:35.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:35.321 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:35.354 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:35.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:35.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:35.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:35.486 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:35.487 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 42 16 68699 68683 102.181 82.4375 0.00568261 0.00978065 2024-03-11T22:19:35.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:35.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:35.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[39715]: pgmap v518: 168 pgs: 168 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 638 B/s rd, 55 MiB/s wr, 886 op/s 2024-03-11T22:19:35.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[39715]: 6.0 scrub starts 2024-03-11T22:19:35.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[39715]: 6.0 scrub ok 2024-03-11T22:19:35.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[39715]: 4.7 scrub starts 2024-03-11T22:19:35.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[39715]: 4.7 scrub ok 2024-03-11T22:19:35.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[39715]: 4.5 scrub starts 2024-03-11T22:19:35.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[39715]: 4.5 scrub ok 2024-03-11T22:19:35.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[39715]: 3.5 scrub starts 2024-03-11T22:19:35.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[39715]: 3.5 scrub ok 2024-03-11T22:19:35.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:19:35.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:19:35.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[30178]: pgmap v518: 168 pgs: 168 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 638 B/s rd, 55 MiB/s wr, 886 op/s 2024-03-11T22:19:35.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[30178]: 6.0 scrub starts 2024-03-11T22:19:35.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[30178]: 6.0 scrub ok 2024-03-11T22:19:35.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[30178]: 4.7 scrub starts 2024-03-11T22:19:35.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[30178]: 4.7 scrub ok 2024-03-11T22:19:35.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[30178]: 4.5 scrub starts 2024-03-11T22:19:35.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[30178]: 4.5 scrub ok 2024-03-11T22:19:35.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[30178]: 3.5 scrub starts 2024-03-11T22:19:35.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[30178]: 3.5 scrub ok 2024-03-11T22:19:35.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:19:35.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:35 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:19:35.588 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:35.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:35 smithi144 ceph-mon[33984]: pgmap v518: 168 pgs: 168 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 638 B/s rd, 55 MiB/s wr, 886 op/s 2024-03-11T22:19:35.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:35 smithi144 ceph-mon[33984]: 6.0 scrub starts 2024-03-11T22:19:35.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:35 smithi144 ceph-mon[33984]: 6.0 scrub ok 2024-03-11T22:19:35.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:35 smithi144 ceph-mon[33984]: 4.7 scrub starts 2024-03-11T22:19:35.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:35 smithi144 ceph-mon[33984]: 4.7 scrub ok 2024-03-11T22:19:35.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:35 smithi144 ceph-mon[33984]: 4.5 scrub starts 2024-03-11T22:19:35.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:35 smithi144 ceph-mon[33984]: 4.5 scrub ok 2024-03-11T22:19:35.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:35 smithi144 ceph-mon[33984]: 3.5 scrub starts 2024-03-11T22:19:35.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:35 smithi144 ceph-mon[33984]: 3.5 scrub ok 2024-03-11T22:19:35.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:35 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:19:35.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:35 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:19:35.618 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:35.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:35.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:35.761 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:35.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:35.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:35.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:35.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:36.001 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:36.039 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:36.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:36.141 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:36.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:36.174 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:36.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:36.275 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:36.305 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:36.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:36.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:36.319 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T22:19:36.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:36.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:36.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:36.487 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 43 16 69838 69822 101.46 71.1875 0.0141187 0.00985114 2024-03-11T22:19:36.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:36 smithi045 ceph-mon[30178]: osdmap e292: 8 total, 8 up, 8 in 2024-03-11T22:19:36.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:36 smithi045 ceph-mon[30178]: 6.3 scrub starts 2024-03-11T22:19:36.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:36 smithi045 ceph-mon[30178]: 6.3 scrub ok 2024-03-11T22:19:36.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:36 smithi045 ceph-mon[39715]: osdmap e292: 8 total, 8 up, 8 in 2024-03-11T22:19:36.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:36 smithi045 ceph-mon[39715]: 6.3 scrub starts 2024-03-11T22:19:36.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:36 smithi045 ceph-mon[39715]: 6.3 scrub ok 2024-03-11T22:19:36.564 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:36.595 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:36.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:36.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:36 smithi144 ceph-mon[33984]: osdmap e292: 8 total, 8 up, 8 in 2024-03-11T22:19:36.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:36 smithi144 ceph-mon[33984]: 6.3 scrub starts 2024-03-11T22:19:36.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:36 smithi144 ceph-mon[33984]: 6.3 scrub ok 2024-03-11T22:19:36.596 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:19:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:19:36.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:36.738 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:36.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:36.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:36.840 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:36.870 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:36.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:36.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:36.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:37.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:37.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:37.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:37.059 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:19:37.059 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:19:37.059 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.059 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.25142.0:70497 9.a e292/273 9:5acf0bdf:::benchmark_data_smithi045_141553_object70496:head v 292'4467, mlcod=292'4465)", 2024-03-11T22:19:37.059 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:19:37.055061+0000", 2024-03-11T22:19:37.059 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.001688567, 2024-03-11T22:19:37.059 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0017179630000000001, 2024-03-11T22:19:37.059 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:37.059 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055061+0000", 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055061+0000", 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:37.060 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055062+0000", 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3699999999999999e-07 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055088+0000", 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6173000000000002e-05 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055121+0000", 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2654999999999998e-05 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055136+0000", 2024-03-11T22:19:37.061 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5690000000000001e-05 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055216+0000", 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9451999999999998e-05 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055237+0000", 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1250000000000002e-05 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.062 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.25142.0:70489 9.c e292/273 9:3a5bb9dd:::benchmark_data_smithi045_141553_object70488:head v 292'4325, mlcod=292'4322)", 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:19:37.054990+0000", 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.0017594799999999999, 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001854496, 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.054990+0000", 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:37.063 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.054990+0000", 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.054991+0000", 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0809999999999999e-06 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055095+0000", 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000104119 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055141+0000", 2024-03-11T22:19:37.064 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6032999999999998e-05 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055151+0000", 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7089999999999997e-06 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055225+0000", 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4133000000000003e-05 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055251+0000", 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5673e-05 2024-03-11T22:19:37.065 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.25142.0:70491 9.c e292/273)", 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:19:37.055797+0000", 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.00095272100000000001, 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011049149999999999, 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "queued for pg", 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055797+0000", 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:37.066 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055797+0000", 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055797+0000", 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1200000000000003e-07 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055878+0000", 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0357000000000003e-05 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.067 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055887+0000", 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1239999999999994e-06 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:37.055893+0000", 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3829999999999997e-06 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 3 2024-03-11T22:19:37.068 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:19:37.129 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:37.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:37.183 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:37.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:37.267 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:37.284 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:37.319 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:37.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:37.419 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:37.466 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:37.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:37.487 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 44 16 71027 71011 100.843 74.3125 0.0131978 0.00991105 2024-03-11T22:19:37.567 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:37.601 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:37.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:37.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[39715]: pgmap v520: 168 pgs: 168 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.37k op/s 2024-03-11T22:19:37.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:37.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[39715]: 4.1e scrub starts 2024-03-11T22:19:37.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[39715]: 3.1a scrub starts 2024-03-11T22:19:37.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[39715]: 4.1e scrub ok 2024-03-11T22:19:37.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[39715]: 3.1a scrub ok 2024-03-11T22:19:37.650 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[39715]: 5.6 scrub starts 2024-03-11T22:19:37.651 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[39715]: 5.6 scrub ok 2024-03-11T22:19:37.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[30178]: pgmap v520: 168 pgs: 168 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.37k op/s 2024-03-11T22:19:37.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:37.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[30178]: 4.1e scrub starts 2024-03-11T22:19:37.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[30178]: 3.1a scrub starts 2024-03-11T22:19:37.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[30178]: 4.1e scrub ok 2024-03-11T22:19:37.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[30178]: 3.1a scrub ok 2024-03-11T22:19:37.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[30178]: 5.6 scrub starts 2024-03-11T22:19:37.651 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:37 smithi045 ceph-mon[30178]: 5.6 scrub ok 2024-03-11T22:19:37.686 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:37 smithi144 ceph-mon[33984]: pgmap v520: 168 pgs: 168 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.37k op/s 2024-03-11T22:19:37.686 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:37.686 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:37 smithi144 ceph-mon[33984]: 4.1e scrub starts 2024-03-11T22:19:37.686 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:37 smithi144 ceph-mon[33984]: 3.1a scrub starts 2024-03-11T22:19:37.686 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:37 smithi144 ceph-mon[33984]: 4.1e scrub ok 2024-03-11T22:19:37.686 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:37 smithi144 ceph-mon[33984]: 3.1a scrub ok 2024-03-11T22:19:37.686 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:37 smithi144 ceph-mon[33984]: 5.6 scrub starts 2024-03-11T22:19:37.686 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:37 smithi144 ceph-mon[33984]: 5.6 scrub ok 2024-03-11T22:19:37.703 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:37.736 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:37.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:37.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:37.801 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T22:19:37.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:37.888 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:37.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:37.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:37.990 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:38.019 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:19:38.019 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:19:38.019 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:19:38.031 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:38.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:38.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:38.048 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:19:38.098 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:19:38.129 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:19:38.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:38.141 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:19:38.172 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:38.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:38.194 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:19:38.240 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:19:38.257 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:19:38.257 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:19:38.257 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:19:38.257 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:19:38.257 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:19:38.257 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:19:38.257 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:19:38.257 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:19:38.274 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:38.307 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:38.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:38.409 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:38.434 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:19:38.445 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:38.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:38.487 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 45 16 72082 72066 100.068 65.9375 0.0175003 0.00998768 2024-03-11T22:19:38.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:38.586 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:38.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:38.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:38 smithi144 ceph-mon[33984]: 3.e scrub starts 2024-03-11T22:19:38.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:38 smithi144 ceph-mon[33984]: 3.e scrub ok 2024-03-11T22:19:38.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:38 smithi144 ceph-mon[33984]: 6.5 scrub starts 2024-03-11T22:19:38.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:38 smithi144 ceph-mon[33984]: 6.5 scrub ok 2024-03-11T22:19:38.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:38 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2856227410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:19:38.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:38 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2665636021' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:19:38.688 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:38.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:38.728 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:38.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:38.744 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:38 smithi045 ceph-mon[30178]: 3.e scrub starts 2024-03-11T22:19:38.744 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:38 smithi045 ceph-mon[30178]: 3.e scrub ok 2024-03-11T22:19:38.745 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:38 smithi045 ceph-mon[30178]: 6.5 scrub starts 2024-03-11T22:19:38.745 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:38 smithi045 ceph-mon[30178]: 6.5 scrub ok 2024-03-11T22:19:38.745 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:38 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2856227410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:19:38.745 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:38 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2665636021' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:19:38.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:38 smithi045 ceph-mon[39715]: 3.e scrub starts 2024-03-11T22:19:38.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:38 smithi045 ceph-mon[39715]: 3.e scrub ok 2024-03-11T22:19:38.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:38 smithi045 ceph-mon[39715]: 6.5 scrub starts 2024-03-11T22:19:38.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:38 smithi045 ceph-mon[39715]: 6.5 scrub ok 2024-03-11T22:19:38.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:38 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2856227410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:19:38.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:38 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2665636021' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:19:38.830 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:38.863 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:38.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:38.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:38.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:39.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:39.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:39.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:39.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:39.152 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:39.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:39.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:39.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:39.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:39.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:39.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:39.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:39.441 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:39.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:39.487 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 46 16 73376 73360 99.6502 80.875 0.00436572 0.01003 2024-03-11T22:19:39.543 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:39.583 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:39.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:39.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:39.685 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:39.728 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:39.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:39.769 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:39 smithi144 ceph-mon[33984]: pgmap v521: 168 pgs: 168 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-11T22:19:39.769 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:39 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:19:39.769 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:39 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:19:39.769 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:39 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2665636021' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:19:39.769 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:39 smithi144 ceph-mon[33984]: osdmap e293: 8 total, 8 up, 8 in 2024-03-11T22:19:39.769 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:39 smithi144 ceph-mon[33984]: 4.3 scrub starts 2024-03-11T22:19:39.770 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:39 smithi144 ceph-mon[33984]: 4.3 scrub ok 2024-03-11T22:19:39.770 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:39 smithi144 ceph-mon[33984]: 4.9 scrub starts 2024-03-11T22:19:39.770 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:39 smithi144 ceph-mon[33984]: 4.9 scrub ok 2024-03-11T22:19:39.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[39715]: pgmap v521: 168 pgs: 168 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-11T22:19:39.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:19:39.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:19:39.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2665636021' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:19:39.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[39715]: osdmap e293: 8 total, 8 up, 8 in 2024-03-11T22:19:39.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[39715]: 4.3 scrub starts 2024-03-11T22:19:39.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[39715]: 4.3 scrub ok 2024-03-11T22:19:39.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[39715]: 4.9 scrub starts 2024-03-11T22:19:39.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[39715]: 4.9 scrub ok 2024-03-11T22:19:39.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[30178]: pgmap v521: 168 pgs: 168 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-11T22:19:39.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:19:39.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:19:39.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2665636021' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:19:39.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[30178]: osdmap e293: 8 total, 8 up, 8 in 2024-03-11T22:19:39.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[30178]: 4.3 scrub starts 2024-03-11T22:19:39.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[30178]: 4.3 scrub ok 2024-03-11T22:19:39.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[30178]: 4.9 scrub starts 2024-03-11T22:19:39.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:39 smithi045 ceph-mon[30178]: 4.9 scrub ok 2024-03-11T22:19:39.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:39.868 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:39.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:39.970 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:40.009 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:40.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:40.109 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:40.132 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:19:40.132 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:19:40.143 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:40.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:40.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:40.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:40.287 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:40.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:40.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:40.444 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:40.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:40.487 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 47 16 74674 74658 99.256 81.125 0.0137183 0.0100698 2024-03-11T22:19:40.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:40.585 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:40.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:40.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:40.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:40.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:40.774 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:40 smithi144 ceph-mon[33984]: 4.4 scrub starts 2024-03-11T22:19:40.774 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:40 smithi144 ceph-mon[33984]: 4.4 scrub ok 2024-03-11T22:19:40.774 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:40 smithi144 ceph-mon[33984]: 5.13 scrub starts 2024-03-11T22:19:40.774 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:40 smithi144 ceph-mon[33984]: 5.13 scrub ok 2024-03-11T22:19:40.775 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:40 smithi144 ceph-mon[33984]: 5.d deep-scrub starts 2024-03-11T22:19:40.775 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:40 smithi144 ceph-mon[33984]: 5.d deep-scrub ok 2024-03-11T22:19:40.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:40 smithi045 ceph-mon[30178]: 4.4 scrub starts 2024-03-11T22:19:40.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:40 smithi045 ceph-mon[30178]: 4.4 scrub ok 2024-03-11T22:19:40.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:40 smithi045 ceph-mon[30178]: 5.13 scrub starts 2024-03-11T22:19:40.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:40 smithi045 ceph-mon[30178]: 5.13 scrub ok 2024-03-11T22:19:40.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:40 smithi045 ceph-mon[30178]: 5.d deep-scrub starts 2024-03-11T22:19:40.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:40 smithi045 ceph-mon[30178]: 5.d deep-scrub ok 2024-03-11T22:19:40.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:40 smithi045 ceph-mon[39715]: 4.4 scrub starts 2024-03-11T22:19:40.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:40 smithi045 ceph-mon[39715]: 4.4 scrub ok 2024-03-11T22:19:40.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:40 smithi045 ceph-mon[39715]: 5.13 scrub starts 2024-03-11T22:19:40.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:40 smithi045 ceph-mon[39715]: 5.13 scrub ok 2024-03-11T22:19:40.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:40 smithi045 ceph-mon[39715]: 5.d deep-scrub starts 2024-03-11T22:19:40.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:40 smithi045 ceph-mon[39715]: 5.d deep-scrub ok 2024-03-11T22:19:40.819 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T22:19:40.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:40.875 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:40.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:40.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:40.977 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:41.010 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:41.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:41.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:41.150 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:41.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:41.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:41.162 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:19:41.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:41.277 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:41.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:41.379 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:41.410 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:41.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:41.487 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 48 16 75932 75916 98.8261 78.625 0.0165122 0.0101134 2024-03-11T22:19:41.511 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:41.540 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:41.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:41.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:41.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:41 smithi144 ceph-mon[33984]: pgmap v523: 168 pgs: 168 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 742 B/s rd, 56 MiB/s wr, 890 op/s 2024-03-11T22:19:41.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:41 smithi144 ceph-mon[33984]: 3.1d scrub starts 2024-03-11T22:19:41.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:41 smithi144 ceph-mon[33984]: 3.1d scrub ok 2024-03-11T22:19:41.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:41 smithi144 ceph-mon[33984]: osdmap e294: 8 total, 8 up, 8 in 2024-03-11T22:19:41.642 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:41.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:41 smithi045 ceph-mon[30178]: pgmap v523: 168 pgs: 168 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 742 B/s rd, 56 MiB/s wr, 890 op/s 2024-03-11T22:19:41.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:41 smithi045 ceph-mon[30178]: 3.1d scrub starts 2024-03-11T22:19:41.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:41 smithi045 ceph-mon[30178]: 3.1d scrub ok 2024-03-11T22:19:41.652 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:41 smithi045 ceph-mon[30178]: osdmap e294: 8 total, 8 up, 8 in 2024-03-11T22:19:41.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:41 smithi045 ceph-mon[39715]: pgmap v523: 168 pgs: 168 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 742 B/s rd, 56 MiB/s wr, 890 op/s 2024-03-11T22:19:41.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:41 smithi045 ceph-mon[39715]: 3.1d scrub starts 2024-03-11T22:19:41.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:41 smithi045 ceph-mon[39715]: 3.1d scrub ok 2024-03-11T22:19:41.652 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:41 smithi045 ceph-mon[39715]: osdmap e294: 8 total, 8 up, 8 in 2024-03-11T22:19:41.671 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:41.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:41.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:41.816 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:41.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:41.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:41.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:19:41.918 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:41.947 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:41.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:42.049 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:42.077 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:42.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:42.178 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:42.205 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:42.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:42.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:42.307 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:42.335 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:42.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:42.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:42.368 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:42.426 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:19:42.438 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:42.469 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:42.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:42.488 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 49 16 76982 76966 98.1474 65.625 0.0102074 0.0101819 2024-03-11T22:19:42.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:42.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:42.618 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:42.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:42.698 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:42 smithi045 ceph-mon[30178]: 4.c deep-scrub starts 2024-03-11T22:19:42.698 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:42 smithi045 ceph-mon[30178]: 4.c deep-scrub ok 2024-03-11T22:19:42.699 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:19:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:19:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:19:42.699 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:42 smithi045 ceph-mon[39715]: 4.c deep-scrub starts 2024-03-11T22:19:42.699 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:42 smithi045 ceph-mon[39715]: 4.c deep-scrub ok 2024-03-11T22:19:42.719 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:42.747 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:42.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:42.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:42.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:42 smithi144 ceph-mon[33984]: 4.c deep-scrub starts 2024-03-11T22:19:42.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:42 smithi144 ceph-mon[33984]: 4.c deep-scrub ok 2024-03-11T22:19:42.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:42.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:42.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:42.998 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:43.034 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:43.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:43.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:43.179 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:43.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:43.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:43.326 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:43.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:43.428 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:43.457 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T22:19:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:43.463 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:43.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:43.488 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 50 16 78201 78185 97.7081 76.1875 0.0111987 0.0102287 2024-03-11T22:19:43.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:43.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:43 smithi144 ceph-mon[33984]: pgmap v525: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-11T22:19:43.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:43 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3749379878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:19:43.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:43 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:19:43.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:43 smithi144 ceph-mon[33984]: 5.15 scrub starts 2024-03-11T22:19:43.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:43 smithi144 ceph-mon[33984]: 5.15 scrub ok 2024-03-11T22:19:43.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:43 smithi144 ceph-mon[33984]: 5.a scrub starts 2024-03-11T22:19:43.590 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:43 smithi144 ceph-mon[33984]: 5.a scrub ok 2024-03-11T22:19:43.591 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:19:43.591 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:19:43.603 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:43.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:43.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:43.767 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:43.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:43.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[30178]: pgmap v525: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-11T22:19:43.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3749379878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:19:43.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:19:43.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[30178]: 5.15 scrub starts 2024-03-11T22:19:43.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[30178]: 5.15 scrub ok 2024-03-11T22:19:43.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[30178]: 5.a scrub starts 2024-03-11T22:19:43.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[30178]: 5.a scrub ok 2024-03-11T22:19:43.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[39715]: pgmap v525: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-11T22:19:43.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3749379878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:19:43.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:19:43.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[39715]: 5.15 scrub starts 2024-03-11T22:19:43.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[39715]: 5.15 scrub ok 2024-03-11T22:19:43.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[39715]: 5.a scrub starts 2024-03-11T22:19:43.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:43 smithi045 ceph-mon[39715]: 5.a scrub ok 2024-03-11T22:19:43.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:43.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:43.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:43.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:44.013 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:44.042 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:44.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:44.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:44.144 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:44.184 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:44.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:44.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:44.286 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:44.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T22:19:44.314 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:44.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:44.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:44.416 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:44.443 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:44.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:44.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:44.488 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 51 15 79384 79369 97.2428 74 0.0172029 0.0102771 2024-03-11T22:19:44.545 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:44.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:44.595 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:44.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:44.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:44 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:19:44.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:44 smithi144 ceph-mon[33984]: osdmap e295: 8 total, 8 up, 8 in 2024-03-11T22:19:44.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:44 smithi144 ceph-mon[33984]: 5.17 scrub starts 2024-03-11T22:19:44.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:44 smithi144 ceph-mon[33984]: 5.17 scrub ok 2024-03-11T22:19:44.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:44 smithi144 ceph-mon[33984]: 5.2 scrub starts 2024-03-11T22:19:44.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:44 smithi144 ceph-mon[33984]: 5.2 scrub ok 2024-03-11T22:19:44.696 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:44.708 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:44 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:19:44.708 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:44 smithi045 ceph-mon[39715]: osdmap e295: 8 total, 8 up, 8 in 2024-03-11T22:19:44.708 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:44 smithi045 ceph-mon[39715]: 5.17 scrub starts 2024-03-11T22:19:44.708 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:44 smithi045 ceph-mon[39715]: 5.17 scrub ok 2024-03-11T22:19:44.709 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:44 smithi045 ceph-mon[39715]: 5.2 scrub starts 2024-03-11T22:19:44.709 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:44 smithi045 ceph-mon[39715]: 5.2 scrub ok 2024-03-11T22:19:44.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:44 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:19:44.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:44 smithi045 ceph-mon[30178]: osdmap e295: 8 total, 8 up, 8 in 2024-03-11T22:19:44.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:44 smithi045 ceph-mon[30178]: 5.17 scrub starts 2024-03-11T22:19:44.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:44 smithi045 ceph-mon[30178]: 5.17 scrub ok 2024-03-11T22:19:44.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:44 smithi045 ceph-mon[30178]: 5.2 scrub starts 2024-03-11T22:19:44.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:44 smithi045 ceph-mon[30178]: 5.2 scrub ok 2024-03-11T22:19:44.725 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:44.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:44.827 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:44.859 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:44.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:44.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:45.001 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:45.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:45.103 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:45.132 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:45.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:45.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:45.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T22:19:45.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:45.309 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:45.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:45.352 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:45.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:45.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:45.465 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:45.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:45.488 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 52 16 80564 80548 96.7898 73.6875 0.0110338 0.0103256 2024-03-11T22:19:45.566 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:45.598 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:45.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:45.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:45.700 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:45.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:45 smithi045 ceph-mon[30178]: pgmap v527: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-11T22:19:45.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:45 smithi045 ceph-mon[30178]: 3.15 scrub starts 2024-03-11T22:19:45.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:45 smithi045 ceph-mon[30178]: 3.15 scrub ok 2024-03-11T22:19:45.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:19:45.732 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:19:45.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:45 smithi045 ceph-mon[39715]: pgmap v527: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-11T22:19:45.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:45 smithi045 ceph-mon[39715]: 3.15 scrub starts 2024-03-11T22:19:45.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:45 smithi045 ceph-mon[39715]: 3.15 scrub ok 2024-03-11T22:19:45.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:19:45.734 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:19:45.743 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:45.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:45.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:45.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:45 smithi144 ceph-mon[33984]: pgmap v527: 168 pgs: 168 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-11T22:19:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:45 smithi144 ceph-mon[33984]: 3.15 scrub starts 2024-03-11T22:19:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:45 smithi144 ceph-mon[33984]: 3.15 scrub ok 2024-03-11T22:19:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:19:45.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:19:45.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:45.885 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:45.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:45.986 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:46.015 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:46.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:46.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:46.117 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:46.145 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:46.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:46.246 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:46.275 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:46.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:46.377 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:46.405 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:46.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:46.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:46.488 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 53 16 81709 81693 96.3138 71.5625 0.0111062 0.0103768 2024-03-11T22:19:46.507 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:46.536 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:46.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:46.580 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:46.596 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:19:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:19:46.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:46 smithi144 ceph-mon[33984]: 3.17 scrub starts 2024-03-11T22:19:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:46 smithi144 ceph-mon[33984]: 3.17 scrub ok 2024-03-11T22:19:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:46 smithi144 ceph-mon[33984]: 6.1d scrub starts 2024-03-11T22:19:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:46 smithi144 ceph-mon[33984]: 5.19 scrub starts 2024-03-11T22:19:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:46 smithi144 ceph-mon[33984]: 5.19 scrub ok 2024-03-11T22:19:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:46 smithi144 ceph-mon[33984]: 6.1d scrub ok 2024-03-11T22:19:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:46 smithi144 ceph-mon[33984]: osdmap e296: 8 total, 8 up, 8 in 2024-03-11T22:19:46.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:46.693 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:46.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:46.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[30178]: 3.17 scrub starts 2024-03-11T22:19:46.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[30178]: 3.17 scrub ok 2024-03-11T22:19:46.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[30178]: 6.1d scrub starts 2024-03-11T22:19:46.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[30178]: 5.19 scrub starts 2024-03-11T22:19:46.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[30178]: 5.19 scrub ok 2024-03-11T22:19:46.757 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[30178]: 6.1d scrub ok 2024-03-11T22:19:46.758 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[30178]: osdmap e296: 8 total, 8 up, 8 in 2024-03-11T22:19:46.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[39715]: 3.17 scrub starts 2024-03-11T22:19:46.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[39715]: 3.17 scrub ok 2024-03-11T22:19:46.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[39715]: 6.1d scrub starts 2024-03-11T22:19:46.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[39715]: 5.19 scrub starts 2024-03-11T22:19:46.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[39715]: 5.19 scrub ok 2024-03-11T22:19:46.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[39715]: 6.1d scrub ok 2024-03-11T22:19:46.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:46 smithi045 ceph-mon[39715]: osdmap e296: 8 total, 8 up, 8 in 2024-03-11T22:19:46.794 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:46.824 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:46.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:46.926 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:46.958 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:46.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:47.060 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:47.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:47.091 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:47.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:47.192 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:47.221 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:47.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:47.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:47.367 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:47.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:47.468 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:47.488 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 54 16 82913 82897 95.9236 75.25 0.0198296 0.0104186 2024-03-11T22:19:47.495 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:47.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:47.502 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:47.504 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:19:47.597 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:47.628 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:47.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:47.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:47.729 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:47.757 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:47.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[39715]: pgmap v529: 168 pgs: 168 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-11T22:19:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[39715]: 3.0 deep-scrub starts 2024-03-11T22:19:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[39715]: 3.0 deep-scrub ok 2024-03-11T22:19:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[39715]: 4.a scrub starts 2024-03-11T22:19:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[39715]: 4.a scrub ok 2024-03-11T22:19:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3445985489' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:19:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:19:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[39715]: 4.8 scrub starts 2024-03-11T22:19:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[39715]: 4.8 scrub ok 2024-03-11T22:19:47.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[30178]: pgmap v529: 168 pgs: 168 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-11T22:19:47.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:47.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[30178]: 3.0 deep-scrub starts 2024-03-11T22:19:47.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[30178]: 3.0 deep-scrub ok 2024-03-11T22:19:47.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[30178]: 4.a scrub starts 2024-03-11T22:19:47.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[30178]: 4.a scrub ok 2024-03-11T22:19:47.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3445985489' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:19:47.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:19:47.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[30178]: 4.8 scrub starts 2024-03-11T22:19:47.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:47 smithi045 ceph-mon[30178]: 4.8 scrub ok 2024-03-11T22:19:47.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:47 smithi144 ceph-mon[33984]: pgmap v529: 168 pgs: 168 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-11T22:19:47.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:47.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:47 smithi144 ceph-mon[33984]: 3.0 deep-scrub starts 2024-03-11T22:19:47.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:47 smithi144 ceph-mon[33984]: 3.0 deep-scrub ok 2024-03-11T22:19:47.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:47 smithi144 ceph-mon[33984]: 4.a scrub starts 2024-03-11T22:19:47.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:47 smithi144 ceph-mon[33984]: 4.a scrub ok 2024-03-11T22:19:47.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:47 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3445985489' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:19:47.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:47 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:19:47.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:47 smithi144 ceph-mon[33984]: 4.8 scrub starts 2024-03-11T22:19:47.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:47 smithi144 ceph-mon[33984]: 4.8 scrub ok 2024-03-11T22:19:47.859 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:47.887 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:47.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:47.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:48.029 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:48.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:48.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:48.174 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:48.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:48.276 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:48.305 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:48.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:48.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:48.408 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:48.438 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:48.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:48.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:48.488 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 55 16 83930 83914 95.3352 63.5625 0.00764594 0.0104839 2024-03-11T22:19:48.540 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:48.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:48.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:48.669 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:48.702 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:48.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:48.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:48 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:19:48.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:48 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:19:48.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:48 smithi045 ceph-mon[30178]: osdmap e297: 8 total, 8 up, 8 in 2024-03-11T22:19:48.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:48 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:19:48.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:48 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:19:48.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:48 smithi045 ceph-mon[39715]: osdmap e297: 8 total, 8 up, 8 in 2024-03-11T22:19:48.803 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:48.836 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:48.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:48.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:48.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:48 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:19:48.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:48 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:19:48.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:48 smithi144 ceph-mon[33984]: osdmap e297: 8 total, 8 up, 8 in 2024-03-11T22:19:48.938 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:48.967 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:48.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:48.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:49.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:49.106 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:49.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:49.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:49.208 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:49.236 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:49.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:49.338 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:49.369 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:49.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:49.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:49.471 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:49.489 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 56 16 85035 85019 94.866 69.0625 0.00566003 0.0105361 2024-03-11T22:19:49.502 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:49.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:49.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:49 smithi045 ceph-mon[30178]: pgmap v531: 168 pgs: 168 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 982 op/s 2024-03-11T22:19:49.526 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:49 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:19:49.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:49.651 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:49.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:49.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:49.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:49 smithi045 ceph-mon[39715]: pgmap v531: 168 pgs: 168 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 982 op/s 2024-03-11T22:19:49.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:49 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:19:49.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:49 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:19:49.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:49 smithi045 ceph-mon[39715]: osdmap e298: 8 total, 8 up, 8 in 2024-03-11T22:19:49.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:49 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:19:49.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:49 smithi045 ceph-mon[30178]: osdmap e298: 8 total, 8 up, 8 in 2024-03-11T22:19:49.790 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:49.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:49.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:49 smithi144 ceph-mon[33984]: pgmap v531: 168 pgs: 168 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 982 op/s 2024-03-11T22:19:49.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:49 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:19:49.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:49 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:19:49.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:49 smithi144 ceph-mon[33984]: osdmap e298: 8 total, 8 up, 8 in 2024-03-11T22:19:49.889 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:19:49.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:49.937 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:49.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:49.946 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:19:49.985 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:19:50.031 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:19:50.038 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:50.058 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:19:50.063 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:50.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:50.079 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:19:50.099 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:19:50.133 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:19:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:19:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:19:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:19:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:19:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:19:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:19:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:19:50.138 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:19:50.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:50.206 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:50.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:50.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:50.308 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:50.336 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:50.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:50.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:50.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:50.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:50.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:50.489 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 57 16 86104 86088 94.3738 66.8125 0.00940734 0.0105908 2024-03-11T22:19:50.578 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:50.608 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:50.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:50.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:50.710 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:50.742 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:50.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:50.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:50.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:50.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:50 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2795187104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:19:50.874 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:50 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2795187104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:19:50.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:50 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2795187104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:19:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:50.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:50.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:50.988 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:50.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:19:51.017 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:51.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:51.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:51.118 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:51.143 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:51.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:51.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:51.244 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:51.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:51.275 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:51.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:51.376 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:51.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:51.401 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:51.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:51.489 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 58 16 87215 87199 93.9438 69.4375 0.00715595 0.0106399 2024-03-11T22:19:51.502 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:51.532 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:51.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:51.633 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:51.663 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:51.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:51.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:51.797 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:51.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:51.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:51 smithi144 ceph-mon[33984]: pgmap v533: 168 pgs: 168 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 982 op/s 2024-03-11T22:19:51.899 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:51.926 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:51.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:51.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:51 smithi045 ceph-mon[30178]: pgmap v533: 168 pgs: 168 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 982 op/s 2024-03-11T22:19:51.987 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:51 smithi045 ceph-mon[39715]: pgmap v533: 168 pgs: 168 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 982 op/s 2024-03-11T22:19:52.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:52.076 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:52.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:52.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:52.177 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:52.207 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:52.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:52.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:52.331 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:52.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:52.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:52.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:52.458 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:52.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:52.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:52.489 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 59 16 88209 88193 93.4045 62.125 0.0177468 0.0107009 2024-03-11T22:19:52.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:52.598 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:52.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:52.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:52.702 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:52.730 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:52.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:52.831 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:52.860 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:52.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:52.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:52.961 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:52.990 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:52.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:52.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:53.032 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:19:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:19:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:19:53.092 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:53.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:53.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:53.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:53.233 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:53.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:53.268 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:53.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:53.369 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:53.399 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:53.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:53.489 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:19:53.488709+0000 min lat: 0.00145181 max lat: 0.218324 avg lat: 0.0107315 2024-03-11T22:19:53.489 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:19:53.489 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 60 15 89449 89434 93.1404 77.5625 0.0210572 0.0107315 2024-03-11T22:19:53.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:53.546 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:53.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:53.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:53.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:53.697 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:53.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:53.742 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:53 smithi045 ceph-mon[39715]: pgmap v534: 168 pgs: 168 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 850 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-11T22:19:53.743 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:53 smithi045 ceph-mon[30178]: pgmap v534: 168 pgs: 168 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 850 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-11T22:19:53.798 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:53.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:53 smithi144 ceph-mon[33984]: pgmap v534: 168 pgs: 168 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 850 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-11T22:19:53.858 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:53.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:53.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:53.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:53.985 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:53.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:53.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:54.087 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:54.115 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:54.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:54.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:54.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:54.257 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:54.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:54.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:54.400 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:54.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:54.489 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 61 16 90671 90655 92.8644 76.3125 0.0110845 0.0107639 2024-03-11T22:19:54.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:54.540 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:54.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:54.643 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:54.672 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:54.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:54.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:54.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:54.818 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:54.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:54.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:54 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:19:54.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:54 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:19:54.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:54 smithi144 ceph-mon[33984]: osdmap e299: 8 total, 8 up, 8 in 2024-03-11T22:19:54.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:54.946 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:54.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:54.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:55.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:54 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:19:55.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:54 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:19:55.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:54 smithi045 ceph-mon[39715]: osdmap e299: 8 total, 8 up, 8 in 2024-03-11T22:19:55.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:54 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:19:55.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:54 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:19:55.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:54 smithi045 ceph-mon[30178]: osdmap e299: 8 total, 8 up, 8 in 2024-03-11T22:19:55.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:55.089 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:55.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:55.191 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:55.233 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:55.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:55.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:55.335 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:55.363 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:55.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:55.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:55.489 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 62 16 91797 91781 92.5016 70.375 0.0137002 0.0108055 2024-03-11T22:19:55.505 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:55.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:55.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:55.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:55.647 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:55.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:55.748 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:55.778 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:55.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:55.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:55.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:55 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:19:55.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:55 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:19:55.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:55 smithi045 ceph-mon[30178]: pgmap v536: 168 pgs: 168 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 753 B/s rd, 51 MiB/s wr, 820 op/s 2024-03-11T22:19:55.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:55 smithi045 ceph-mon[30178]: osdmap e300: 8 total, 8 up, 8 in 2024-03-11T22:19:55.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:55 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:19:55.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:55 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:19:55.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:55 smithi045 ceph-mon[39715]: pgmap v536: 168 pgs: 168 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 753 B/s rd, 51 MiB/s wr, 820 op/s 2024-03-11T22:19:55.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:55 smithi045 ceph-mon[39715]: osdmap e300: 8 total, 8 up, 8 in 2024-03-11T22:19:55.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:55 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:19:55.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:55 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:19:55.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:55 smithi144 ceph-mon[33984]: pgmap v536: 168 pgs: 168 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 753 B/s rd, 51 MiB/s wr, 820 op/s 2024-03-11T22:19:55.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:55 smithi144 ceph-mon[33984]: osdmap e300: 8 total, 8 up, 8 in 2024-03-11T22:19:55.879 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:55.909 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:55.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:55.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:56.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:56.049 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:56.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:56.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:56.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:56.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:56.297 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:56.334 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:56.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:56.436 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:56.486 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:56.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:56.490 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 63 16 92980 92964 92.2056 73.9375 0.0135205 0.0108403 2024-03-11T22:19:56.587 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:56.596 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:19:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:19:56.621 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:56.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:56.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:56.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:56.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:56.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:56.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:56.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:56.908 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:56.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:57.009 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:57.059 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:57.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:57.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:57.161 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:57.207 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:57.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:57.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:57.350 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:57.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:57.452 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:57.483 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:57.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:57.490 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 64 16 94178 94162 91.9348 74.875 0.00669764 0.0108722 2024-03-11T22:19:57.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:57 smithi045 ceph-mon[30178]: pgmap v538: 168 pgs: 168 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:19:57.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:57.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:57 smithi045 ceph-mon[39715]: pgmap v538: 168 pgs: 168 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:19:57.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:57.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:57.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:57 smithi144 ceph-mon[33984]: pgmap v538: 168 pgs: 168 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:19:57.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:19:57.626 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:57.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:57.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:57.728 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:57.750 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:19:57.750 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:19:57.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.750 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:93844 9.9 9:9bbb2c08:::benchmark_data_smithi045_141553_object93843:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-11T22:19:57.750 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:19:57.218540+0000", 2024-03-11T22:19:57.750 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0096956079999999997, 2024-03-11T22:19:57.750 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0097297879999999996, 2024-03-11T22:19:57.750 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:19:57.750 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:19:57.750 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:19:57.750 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 93844 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.218540+0000", 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.218540+0000", 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.218541+0000", 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4139999999999999e-06 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.218595+0000", 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4073000000000002e-05 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.218615+0000", 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9527000000000001e-05 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.218625+0000", 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0148e-05 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.218695+0000", 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9441000000000001e-05 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.218862+0000", 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016697899999999999 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,5", 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.219053+0000", 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019157899999999999 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.222686+0000", 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003632611 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.222729+0000", 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2778000000000003e-05 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:19:57.754 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:93848 9.b 9:d5271279:::benchmark_data_smithi045_141553_object93847:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:19:57.221961+0000", 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0062740790000000001, 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0064075130000000001, 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 93848 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.755 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.221961+0000", 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.221961+0000", 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.221963+0000", 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1629999999999999e-06 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:19:57.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.222025+0000", 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2366999999999994e-05 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.222036+0000", 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1199000000000001e-05 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.222045+0000", 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6389999999999998e-06 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:19:57.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.222123+0000", 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8002000000000003e-05 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.222461+0000", 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00033810699999999998 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 5,6", 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.222970+0000", 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00050873499999999996 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:19:57.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.25142.0:93848 9.b e300/273)", 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:19:57.226986+0000", 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.001249453, 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014547710000000001, 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.226986+0000", 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.226986+0000", 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.226987+0000", 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0999999999999998e-07 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.226995+0000", 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3089999999999996e-06 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:19:57.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.227005+0000", 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7899999999999994e-06 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.227012+0000", 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6869999999999997e-06 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.25142.0:93848 9.b e300/273)", 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:19:57.227038+0000", 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.001197872, 2024-03-11T22:19:57.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014484369999999999, 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.227038+0000", 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.227038+0000", 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:19:57.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.227038+0000", 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.75e-07 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.227041+0000", 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1489999999999998e-06 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.227052+0000", 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0268999999999999e-05 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:57.227057+0000", 2024-03-11T22:19:57.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5729999999999998e-06 2024-03-11T22:19:57.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:19:57.764 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:19:57.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:19:57.764 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:19:57.764 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:19:57.764 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 4 2024-03-11T22:19:57.764 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:19:57.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:57.765 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:57.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:57.865 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:57.892 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:57.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:57.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:57.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:58.031 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:58.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:58.133 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:58.163 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:58.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:58.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:58.264 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:19:58.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:58.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:58.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:19:58.392 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:19:58.422 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:58.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:19:58.490 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 65 16 95363 95347 91.6598 74.0625 0.0174648 0.0109047 2024-03-11T22:19:58.523 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:58.546 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:58.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:58.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:58.647 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:19:58.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:58.686 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:58.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:19:58.787 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:58.818 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:58.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:58.919 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:58.946 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:58.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:58.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:59.048 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:19:59.091 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:59.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:59.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:19:59.094 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T22:19:59.196 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:19:59.224 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:59.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:19:59.326 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:19:59.353 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:59.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:19:59.455 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:59.482 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:59.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:59.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:59.491 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 66 16 96436 96420 91.2866 67.0625 0.0248856 0.010949 2024-03-11T22:19:59.547 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:59.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:59.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:19:59 smithi144 ceph-mon[33984]: pgmap v539: 168 pgs: 168 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-11T22:19:59.608 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:19:59 smithi045 ceph-mon[39715]: pgmap v539: 168 pgs: 168 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-11T22:19:59.610 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:19:59 smithi045 ceph-mon[30178]: pgmap v539: 168 pgs: 168 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-11T22:19:59.614 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:19:59.640 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:59.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:59.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:19:59.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:19:59.742 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:19:59.770 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:19:59.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:59.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:19:59.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:19:59.922 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:19:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:19:59.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:00.023 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:00.053 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:00.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:00.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:00.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:00.297 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:00.337 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:00.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:00.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:00.355 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T22:20:00.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:00.460 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:00 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1171129447' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:20:00.460 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:00 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:20:00.460 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:00 smithi045 ceph-mon[39715]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-03-11T22:20:00.460 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:00 smithi045 ceph-mon[39715]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-11T22:20:00.460 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:00 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1171129447' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:20:00.460 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:00 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:20:00.461 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:00 smithi045 ceph-mon[30178]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-03-11T22:20:00.461 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:00 smithi045 ceph-mon[30178]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-11T22:20:00.481 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:00.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:00.491 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 67 16 97488 97472 90.9054 65.75 0.0130091 0.0109956 2024-03-11T22:20:00.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:00.622 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:00.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:00.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:00.767 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:00.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:00.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:00.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:00 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1171129447' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:20:00.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:00 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:20:00.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:00 smithi144 ceph-mon[33984]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-03-11T22:20:00.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:00 smithi144 ceph-mon[33984]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-11T22:20:00.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:00.909 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:00.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:01.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:01.047 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:01.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:01.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:01.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:01.193 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:01.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:01.294 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:01.324 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:01.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:01.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:01.426 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:01.456 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:01.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:01.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:01.491 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 68 16 98774 98758 90.7505 80.375 0.00819896 0.0110143 2024-03-11T22:20:01.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:01 smithi045 ceph-mon[30178]: pgmap v540: 168 pgs: 168 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 711 op/s 2024-03-11T22:20:01.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:01 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:20:01.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:01 smithi045 ceph-mon[30178]: osdmap e301: 8 total, 8 up, 8 in 2024-03-11T22:20:01.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:20:01.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:01.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:01 smithi045 ceph-mon[30178]: osdmap e302: 8 total, 8 up, 8 in 2024-03-11T22:20:01.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:01 smithi045 ceph-mon[39715]: pgmap v540: 168 pgs: 168 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 711 op/s 2024-03-11T22:20:01.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:01 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:20:01.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:01 smithi045 ceph-mon[39715]: osdmap e301: 8 total, 8 up, 8 in 2024-03-11T22:20:01.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:20:01.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:01.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:01 smithi045 ceph-mon[39715]: osdmap e302: 8 total, 8 up, 8 in 2024-03-11T22:20:01.558 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:01.585 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:01.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:01.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:01.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:01 smithi144 ceph-mon[33984]: pgmap v540: 168 pgs: 168 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 711 op/s 2024-03-11T22:20:01.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:01 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:20:01.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:01 smithi144 ceph-mon[33984]: osdmap e301: 8 total, 8 up, 8 in 2024-03-11T22:20:01.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:20:01.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:01.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:01 smithi144 ceph-mon[33984]: osdmap e302: 8 total, 8 up, 8 in 2024-03-11T22:20:01.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:01.742 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:01.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:01.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:01.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:01.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:01.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T22:20:01.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:02.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:02.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:02.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:02.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:02.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:02.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:02.312 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:02.372 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:02.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:02.473 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:02.491 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 69 16 99913 99897 90.4669 71.1875 0.0094281 0.0110488 2024-03-11T22:20:02.502 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:02.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:02.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:02.604 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:02.633 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:02.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:02.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:02.735 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:02.764 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:02.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:02.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:02.769 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:20:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:20:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:20:02.866 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:02.909 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:02.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:03.011 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:03.041 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:03.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:03.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:03.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:03.191 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:03.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:03.292 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:03.320 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:03.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:03.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:03.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:03 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:20:03.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:03 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:20:03.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:03 smithi045 ceph-mon[30178]: pgmap v543: 168 pgs: 168 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-11T22:20:03.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:03 smithi045 ceph-mon[30178]: osdmap e303: 8 total, 8 up, 8 in 2024-03-11T22:20:03.423 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:03 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:20:03.423 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:03 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:20:03.423 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:03 smithi045 ceph-mon[39715]: pgmap v543: 168 pgs: 168 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-11T22:20:03.423 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:03 smithi045 ceph-mon[39715]: osdmap e303: 8 total, 8 up, 8 in 2024-03-11T22:20:03.432 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:03.451 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:03.491 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 70 16 101109 101093 90.2423 74.75 0.0340346 0.0110757 2024-03-11T22:20:03.508 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:03.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:03.610 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:03.657 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:03.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:03.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:03.758 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:03.786 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:03.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:03.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:03 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:20:03.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:03 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:20:03.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:03 smithi144 ceph-mon[33984]: pgmap v543: 168 pgs: 168 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-11T22:20:03.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:03 smithi144 ceph-mon[33984]: osdmap e303: 8 total, 8 up, 8 in 2024-03-11T22:20:03.888 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:03.932 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:03.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:03.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:04.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:04.085 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:04.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:04.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:04.187 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:04.214 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:04.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:04.316 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:04.345 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:04.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:04.411 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:20:04.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:04.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:04.491 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 71 16 102241 102225 89.9677 70.75 0.00515609 0.0111104 2024-03-11T22:20:04.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:04.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:04.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:04.638 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:04.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:04.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:04.661 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:04 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/230177224' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:20:04.661 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:04 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:20:04.662 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:04 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/230177224' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:20:04.662 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:04 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:20:04.739 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:04.768 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:04.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:04.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:04.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:04 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/230177224' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:20:04.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:04 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:20:04.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:04.913 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:04.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:04.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:05.015 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:05.046 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:05.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:05.147 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:05.176 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:05.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:05.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:05.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:05.323 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:05.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:05.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:05.425 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:05.459 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:05.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:05.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:05.491 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 72 16 103438 103422 89.7572 74.8125 0.0178244 0.0111364 2024-03-11T22:20:05.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:05.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:05 smithi045 ceph-mon[39715]: pgmap v545: 168 pgs: 168 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 958 op/s 2024-03-11T22:20:05.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:05 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:20:05.566 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:05 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:20:05.567 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:05 smithi045 ceph-mon[39715]: osdmap e304: 8 total, 8 up, 8 in 2024-03-11T22:20:05.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:05 smithi045 ceph-mon[30178]: pgmap v545: 168 pgs: 168 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 958 op/s 2024-03-11T22:20:05.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:05 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:20:05.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:05 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:20:05.568 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:05 smithi045 ceph-mon[30178]: osdmap e304: 8 total, 8 up, 8 in 2024-03-11T22:20:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:05.607 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:05.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:05.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:05.747 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:05 smithi144 ceph-mon[33984]: pgmap v545: 168 pgs: 168 active+clean; 6.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 958 op/s 2024-03-11T22:20:05.747 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:05 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:20:05.747 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:05 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:20:05.747 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:05 smithi144 ceph-mon[33984]: osdmap e304: 8 total, 8 up, 8 in 2024-03-11T22:20:05.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:05.753 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:05.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:05.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:05.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:05.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:05.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:05.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:06.033 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:06.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:06.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:06.135 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:06.170 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:06.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:06.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:06.271 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:06.394 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:06.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:06.491 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:20:06.491 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:20:06.491 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-03-11T22:20:06.491 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-03-11T22:20:06.491 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:20:06.492 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 73 16 104712 104696 89.6183 79.625 0.00938164 0.0111539 2024-03-11T22:20:06.496 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:06.512 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:20:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:20:06.528 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:06.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:06.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:06.630 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:06.665 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:06.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:06.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:06.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:06.804 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:06.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:06.905 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:06.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:06.950 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:06.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:07.051 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:07.084 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:07.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:07.185 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:07.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:07.215 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:07.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:07.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:07.360 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:07.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:07.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:07.461 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:07.491 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:07.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:07.493 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 74 16 105997 105981 89.4925 80.3125 0.0060769 0.0111692 2024-03-11T22:20:07.593 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:07.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:07 smithi144 ceph-mon[33984]: pgmap v547: 168 pgs: 168 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-11T22:20:07.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:07.625 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:07.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:07.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:07.728 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:07.759 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:07.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:07.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:07 smithi045 ceph-mon[39715]: pgmap v547: 168 pgs: 168 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-11T22:20:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:07.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:07 smithi045 ceph-mon[30178]: pgmap v547: 168 pgs: 168 active+clean; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-11T22:20:07.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:07.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:07.904 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:07.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:07.923 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T22:20:08.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:08.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:08.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:08.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:08.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:08.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:08.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:08.296 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:08.328 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:08.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:08.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:08.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:08.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:20:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:08.473 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:08.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:08.493 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 75 16 107212 107196 89.31 75.9375 0.00872453 0.011192 2024-03-11T22:20:08.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:08.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:08.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:08.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:08.765 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:08.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:08.866 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:08.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:08.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:09.029 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:09.057 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:09.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:09.159 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:09.190 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:09.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:09.292 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:09.318 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:09.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:09.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:09.354 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:09 smithi045 ceph-mon[39715]: pgmap v548: 168 pgs: 168 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 729 B/s rd, 62 MiB/s wr, 998 op/s 2024-03-11T22:20:09.355 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:09 smithi045 ceph-mon[30178]: pgmap v548: 168 pgs: 168 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 729 B/s rd, 62 MiB/s wr, 998 op/s 2024-03-11T22:20:09.420 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:09.452 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:09.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:09.495 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 76 16 108366 108350 89.0839 72.125 0.0165972 0.0112207 2024-03-11T22:20:09.554 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:09.583 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:09.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:09.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:09 smithi144 ceph-mon[33984]: pgmap v548: 168 pgs: 168 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 729 B/s rd, 62 MiB/s wr, 998 op/s 2024-03-11T22:20:09.684 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:09.695 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:20:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:09.714 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:09.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:09.753 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:20:09.801 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:20:09.814 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:09.841 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:09.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:09.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:09.857 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:20:09.890 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:09.906 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:20:09.943 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:09.946 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:20:09.970 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:09.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:09.997 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:20:10.048 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:20:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:20:10.061 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:20:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:20:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:20:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:20:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:20:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:20:10.062 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:20:10.071 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:10.159 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:10.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:10.261 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:10.289 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:10.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:10.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:10.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:10.433 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:10.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:10.493 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 77 16 109339 109323 88.7167 60.8125 0.00356623 0.0112648 2024-03-11T22:20:10.534 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:10.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:10.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:10.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:10.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:10 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/921398309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:20:10.681 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:10.693 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:10 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/921398309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:20:10.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:10 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/921398309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:20:10.712 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:10.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:10.814 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:10.845 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:10.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:10.946 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:10.980 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:10.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:11.081 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:11.113 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:11.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:11.214 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:11.263 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:11.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:11.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:11.327 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T22:20:11.366 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:11.391 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:11.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:11.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:11.493 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:11.494 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 78 16 110536 110520 88.5384 74.8125 0.00747832 0.0112898 2024-03-11T22:20:11.525 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:11.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:11.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:11 smithi045 ceph-mon[30178]: pgmap v549: 168 pgs: 168 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 744 op/s 2024-03-11T22:20:11.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:11 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1414340403' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:20:11.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:11 smithi045 ceph-mon[39715]: pgmap v549: 168 pgs: 168 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 744 op/s 2024-03-11T22:20:11.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:11 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1414340403' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:20:11.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:11 smithi144 ceph-mon[33984]: pgmap v549: 168 pgs: 168 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 744 op/s 2024-03-11T22:20:11.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:11 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1414340403' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:20:11.627 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:11.658 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:11.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:11.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:11.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:11.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:11.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:11.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:11.944 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:11.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:11.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:12.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:12.088 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:12.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:12.189 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:12.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:12.223 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:12.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:12.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:12.359 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:12.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:12.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:12.493 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 79 16 111822 111806 88.4349 80.375 0.0194447 0.011303 2024-03-11T22:20:12.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:12.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:12.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:12.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:12 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1414340403' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:20:12.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:12 smithi045 ceph-mon[30178]: osdmap e305: 8 total, 8 up, 8 in 2024-03-11T22:20:12.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:12 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1414340403' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:20:12.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:12 smithi045 ceph-mon[39715]: osdmap e305: 8 total, 8 up, 8 in 2024-03-11T22:20:12.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:12 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1414340403' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:20:12.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:12 smithi144 ceph-mon[33984]: osdmap e305: 8 total, 8 up, 8 in 2024-03-11T22:20:12.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:12.647 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:12.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:12.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T22:20:12.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:12.792 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:12.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:12.848 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:20:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:20:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:20:12.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:12.938 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:12.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:13.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:13.087 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:13.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:13.097 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:13.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:13.188 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:20:13.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:13.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:13.283 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:13.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:13.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:13.432 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:13.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:13.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:13.494 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:20:13.493049+0000 min lat: 0.00145181 max lat: 0.218324 avg lat: 0.0113282 2024-03-11T22:20:13.494 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:20:13.494 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 80 16 112980 112964 88.2341 72.375 0.0148811 0.0113282 2024-03-11T22:20:13.521 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:13 smithi045 ceph-mon[39715]: pgmap v551: 168 pgs: 168 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-11T22:20:13.521 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:13 smithi045 ceph-mon[30178]: pgmap v551: 168 pgs: 168 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-11T22:20:13.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:13.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:13 smithi144 ceph-mon[33984]: pgmap v551: 168 pgs: 168 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-11T22:20:13.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:13.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:13.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:13.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:13.767 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:13.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:13.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:13.869 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:13.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:13.892 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:13.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:13.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:14.034 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:14.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:14.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:14.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:14.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:14.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:14.305 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:14.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:14.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:14.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:14.431 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:14.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:14.494 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 81 16 114042 114026 87.9642 66.375 0.0178426 0.0113636 2024-03-11T22:20:14.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:14.573 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:14.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:14.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:14.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:14.717 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:14.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:14.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:14.819 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:14.853 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:14.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:14.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:14.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:15.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:15.003 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:15.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:15.104 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:15.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:15.151 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:15.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:15.251 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:15.281 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:15.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:15.383 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:15.410 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:15.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:15.494 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 82 16 115188 115172 87.7649 71.625 0.0169958 0.0113892 2024-03-11T22:20:15.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:15.532 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:15 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:20:15.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:15 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:20:15.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:15 smithi045 ceph-mon[39715]: pgmap v552: 168 pgs: 168 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T22:20:15.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:15 smithi045 ceph-mon[39715]: osdmap e306: 8 total, 8 up, 8 in 2024-03-11T22:20:15.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:15 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:20:15.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:15 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:20:15.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:15 smithi045 ceph-mon[30178]: pgmap v552: 168 pgs: 168 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T22:20:15.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:15 smithi045 ceph-mon[30178]: osdmap e306: 8 total, 8 up, 8 in 2024-03-11T22:20:15.539 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:15.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:15.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:15.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:15 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:20:15.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:15 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:20:15.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:15 smithi144 ceph-mon[33984]: pgmap v552: 168 pgs: 168 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T22:20:15.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:15 smithi144 ceph-mon[33984]: osdmap e306: 8 total, 8 up, 8 in 2024-03-11T22:20:15.640 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:15.666 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:15.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:15.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:15.807 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:15.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:15.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:15.909 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:15.939 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:15.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:16.040 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:16.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:16.200 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:16.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:16.301 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:16.330 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:16.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:16.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:16.432 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:16.459 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:16.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:16.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:16.494 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 83 16 116347 116331 87.5802 72.4375 0.0104702 0.0114135 2024-03-11T22:20:16.532 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:16.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:16.562 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:16.593 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:16.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:16.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:16.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:16.596 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:20:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:20:16.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:16.736 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:16.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:16.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:16.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:16.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:16.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:16.989 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:17.020 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:17.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:17.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:17.121 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:17.152 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:17.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:17.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:17.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:17.283 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:17.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:17.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:17.385 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:17.414 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:17.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:17.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:17.494 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 84 16 117581 117565 87.4556 77.125 0.00693214 0.0114278 2024-03-11T22:20:17.515 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:17.547 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:17.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:17.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:17.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:17 smithi144 ceph-mon[33984]: pgmap v554: 168 pgs: 168 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-11T22:20:17.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:17.650 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:17.679 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:17.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:17.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:17.781 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:17.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:17 smithi045 ceph-mon[30178]: pgmap v554: 168 pgs: 168 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-11T22:20:17.782 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:17.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:17 smithi045 ceph-mon[39715]: pgmap v554: 168 pgs: 168 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-11T22:20:17.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:17.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:17.810 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:17.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:17.911 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:17.947 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:17.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:17.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:18.048 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:18.076 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:18.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:18.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:18.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:18.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:18.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:18.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:18.321 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:18.346 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:20:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=cleanup t=2024-03-11T22:20:18.067090565Z level=info msg="Completed cleanup jobs" duration=2.343058ms 2024-03-11T22:20:18.349 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:18.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:18.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:18.475 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:18.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:18.494 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 85 16 118611 118595 87.184 64.375 0.0606511 0.0114629 2024-03-11T22:20:18.576 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:18.604 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:18.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:18.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:18.745 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:18.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:18.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:18.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:18.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:18.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:18.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:19.025 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:19.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:19.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:19.127 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:19.153 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:19.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:19.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:19.255 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:19.294 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:19.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:19.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:19.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:19.427 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:19.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:19.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:19.494 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 86 16 119661 119645 86.9333 65.625 0.0138191 0.0114982 2024-03-11T22:20:19.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:19.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:19.570 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:19.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:19.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:19 smithi144 ceph-mon[33984]: pgmap v555: 168 pgs: 168 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T22:20:19.671 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:19.700 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:19.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:19.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:19.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:19 smithi045 ceph-mon[39715]: pgmap v555: 168 pgs: 168 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T22:20:19.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:19 smithi045 ceph-mon[30178]: pgmap v555: 168 pgs: 168 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T22:20:19.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:19.843 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:19.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:19.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:19.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:20.001 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:20.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:20.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:20.144 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:20.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:20.246 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:20.279 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:20.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:20.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:20.422 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:20.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:20.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:20.494 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 87 16 120790 120774 86.7451 70.5625 0.00360219 0.0115233 2024-03-11T22:20:20.524 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:20.562 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:20.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:20.626 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:20:20.626 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:20:20.664 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:20.694 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:20.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:20.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:20.795 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:20.823 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:20.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:20.925 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:20.955 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:20.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:21.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:21.494 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 88 16 121874 121858 86.5292 67.75 0.0119944 0.0115519 2024-03-11T22:20:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:21.685 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:21.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:21.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:21 smithi045 ceph-mon[39715]: pgmap v556: 168 pgs: 168 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 571 B/s rd, 48 MiB/s wr, 764 op/s 2024-03-11T22:20:21.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:21 smithi045 ceph-mon[30178]: pgmap v556: 168 pgs: 168 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 571 B/s rd, 48 MiB/s wr, 764 op/s 2024-03-11T22:20:21.786 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:21.820 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:21.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:21.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:21 smithi144 ceph-mon[33984]: pgmap v556: 168 pgs: 168 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 571 B/s rd, 48 MiB/s wr, 764 op/s 2024-03-11T22:20:21.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:21.962 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:21.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:22.063 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:22.094 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:22.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:22.195 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:22.222 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:22.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:22.259 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:22.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:22.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:22.372 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:22.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:22.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:22.495 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 89 16 122934 122918 86.3004 66.25 0.0124813 0.0115827 2024-03-11T22:20:22.518 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:22.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:22.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:22.620 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:22.666 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:22.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:22.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:22.672 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:20:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:20:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:20:22.770 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:22.801 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:22.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:22.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:22.950 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:22.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:22.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:23.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:23.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:23.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:23.198 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:23.227 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:23.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:23.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:23.329 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:23.356 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:23.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:23.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T22:20:23.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:23.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:23.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:23.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:23.496 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Total time run: 90.0094 2024-03-11T22:20:23.496 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Total writes made: 124117 2024-03-11T22:20:23.496 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Write size: 65536 2024-03-11T22:20:23.496 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Object size: 65536 2024-03-11T22:20:23.496 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Bandwidth (MB/sec): 86.1834 2024-03-11T22:20:23.496 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev Bandwidth: 26.9232 2024-03-11T22:20:23.496 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max bandwidth (MB/sec): 187 2024-03-11T22:20:23.496 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min bandwidth (MB/sec): 60.8125 2024-03-11T22:20:23.497 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Average IOPS: 1378 2024-03-11T22:20:23.497 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev IOPS: 430.771 2024-03-11T22:20:23.497 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max IOPS: 2992 2024-03-11T22:20:23.497 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min IOPS: 973 2024-03-11T22:20:23.497 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Average Latency(s): 0.0115988 2024-03-11T22:20:23.497 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev Latency(s): 0.00914215 2024-03-11T22:20:23.497 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max latency(s): 0.218324 2024-03-11T22:20:23.497 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min latency(s): 0.00145181 2024-03-11T22:20:23.499 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T22:20:23.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:23.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:23 smithi045 ceph-mon[39715]: pgmap v557: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T22:20:23.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:23 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:20:23.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:23 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:20:23.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:23 smithi045 ceph-mon[39715]: osdmap e307: 8 total, 8 up, 8 in 2024-03-11T22:20:23.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:23 smithi045 ceph-mon[30178]: pgmap v557: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T22:20:23.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:23 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:20:23.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:23 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:20:23.607 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:23 smithi045 ceph-mon[30178]: osdmap e307: 8 total, 8 up, 8 in 2024-03-11T22:20:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:23.644 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:23.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:23.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:23.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:23.793 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:23.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:23.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:23 smithi144 ceph-mon[33984]: pgmap v557: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T22:20:23.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:23 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:20:23.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:23 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:20:23.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:23 smithi144 ceph-mon[33984]: osdmap e307: 8 total, 8 up, 8 in 2024-03-11T22:20:23.894 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:23.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:23.926 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:23.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:24.027 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:24.053 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:24.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:24.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:24.156 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:24.184 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:24.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:24.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:24.286 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:24.314 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:24.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:24.416 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:24.445 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:24.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:24.547 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:24.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:24.577 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:24.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:24.678 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:24.706 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:24.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:24.746 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:24.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:24.853 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:24.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:24.954 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:24.981 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:24.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:25.083 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:25.111 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:25.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:25.212 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:25.240 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:25.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:25.263 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:25.341 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:25.369 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:25.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:25.416 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:20:25.470 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:25.521 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:25.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:25.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:25 smithi045 ceph-mon[39715]: pgmap v559: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T22:20:25.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:25 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4269920512' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:20:25.529 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:25 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:20:25.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:25 smithi045 ceph-mon[30178]: pgmap v559: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T22:20:25.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:25 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4269920512' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:20:25.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:25 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:20:25.621 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:25.647 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:25.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:25.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:25.786 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:25.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:25.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:25.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:25 smithi144 ceph-mon[33984]: pgmap v559: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T22:20:25.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:25 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4269920512' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:20:25.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:25 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:20:25.888 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:25.915 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:25.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:26.017 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:26.047 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:26.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:26.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:26.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:26.187 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:26.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:26.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:26.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:26.324 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:26.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:26.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:26.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:26.466 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:26.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:26.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:26.568 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:26.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:26 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:20:26.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:26 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:20:26.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:26 smithi144 ceph-mon[33984]: osdmap e308: 8 total, 8 up, 8 in 2024-03-11T22:20:26.597 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:20:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:20:26.603 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:26.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:26.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:26.705 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:26.734 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:26.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:26.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:26.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:26 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:20:26.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:26 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:20:26.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:26 smithi045 ceph-mon[39715]: osdmap e308: 8 total, 8 up, 8 in 2024-03-11T22:20:26.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:26 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:20:26.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:26 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:20:26.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:26 smithi045 ceph-mon[30178]: osdmap e308: 8 total, 8 up, 8 in 2024-03-11T22:20:26.836 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:26.865 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:26.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:26.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:26.967 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:27.013 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:27.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:27.114 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:27.144 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:27.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:27.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:27.245 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:27.272 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:27.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:27.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:27.374 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:27.401 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:27.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:27.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:27.503 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:27.530 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:27.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:27.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[30178]: 5.18 scrub starts 2024-03-11T22:20:27.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[30178]: 5.18 scrub ok 2024-03-11T22:20:27.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[30178]: pgmap v561: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 62 MiB/s wr, 1.29k op/s 2024-03-11T22:20:27.599 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:27.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[30178]: 3.f scrub starts 2024-03-11T22:20:27.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[30178]: 3.f scrub ok 2024-03-11T22:20:27.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[30178]: 4.f scrub starts 2024-03-11T22:20:27.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[30178]: 4.f scrub ok 2024-03-11T22:20:27.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[30178]: 4.e scrub starts 2024-03-11T22:20:27.600 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[30178]: 4.e scrub ok 2024-03-11T22:20:27.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[39715]: 5.18 scrub starts 2024-03-11T22:20:27.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[39715]: 5.18 scrub ok 2024-03-11T22:20:27.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[39715]: pgmap v561: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 62 MiB/s wr, 1.29k op/s 2024-03-11T22:20:27.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:27.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[39715]: 3.f scrub starts 2024-03-11T22:20:27.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[39715]: 3.f scrub ok 2024-03-11T22:20:27.600 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[39715]: 4.f scrub starts 2024-03-11T22:20:27.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[39715]: 4.f scrub ok 2024-03-11T22:20:27.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[39715]: 4.e scrub starts 2024-03-11T22:20:27.601 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:27 smithi045 ceph-mon[39715]: 4.e scrub ok 2024-03-11T22:20:27.631 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:27.657 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:27.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:27.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:27.807 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:27.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:27.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:27 smithi144 ceph-mon[33984]: 5.18 scrub starts 2024-03-11T22:20:27.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:27 smithi144 ceph-mon[33984]: 5.18 scrub ok 2024-03-11T22:20:27.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:27 smithi144 ceph-mon[33984]: pgmap v561: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 62 MiB/s wr, 1.29k op/s 2024-03-11T22:20:27.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:27.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:27 smithi144 ceph-mon[33984]: 3.f scrub starts 2024-03-11T22:20:27.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:27 smithi144 ceph-mon[33984]: 3.f scrub ok 2024-03-11T22:20:27.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:27 smithi144 ceph-mon[33984]: 4.f scrub starts 2024-03-11T22:20:27.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:27 smithi144 ceph-mon[33984]: 4.f scrub ok 2024-03-11T22:20:27.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:27 smithi144 ceph-mon[33984]: 4.e scrub starts 2024-03-11T22:20:27.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:27 smithi144 ceph-mon[33984]: 4.e scrub ok 2024-03-11T22:20:27.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:27.945 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:27.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:27.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:28.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:28.088 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:28.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:28.190 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:28.228 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:28.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:28.329 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:28.360 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:28.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:28.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:28.486 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:28.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:28.586 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:28.610 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:28.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:28.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:28.724 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T22:20:28.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:28 smithi045 ceph-mon[30178]: 5.1e scrub starts 2024-03-11T22:20:28.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:28 smithi045 ceph-mon[30178]: 5.1e scrub ok 2024-03-11T22:20:28.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:28 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:20:28.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:28 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:20:28.725 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:28 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4212159590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:20:28.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:28 smithi045 ceph-mon[39715]: 5.1e scrub starts 2024-03-11T22:20:28.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:28 smithi045 ceph-mon[39715]: 5.1e scrub ok 2024-03-11T22:20:28.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:28 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:20:28.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:28 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:20:28.726 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:28 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4212159590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:20:28.735 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:28.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:28.837 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:28.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:28 smithi144 ceph-mon[33984]: 5.1e scrub starts 2024-03-11T22:20:28.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:28 smithi144 ceph-mon[33984]: 5.1e scrub ok 2024-03-11T22:20:28.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:28 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:20:28.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:28 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:20:28.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:28 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4212159590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:20:28.860 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:28.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:28.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:28.985 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:28.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:28.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:29.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:29.110 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:29.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:29.212 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:29.235 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:29.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:29.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:29.336 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:29.361 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:29.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:29.462 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:29.470 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T22:20:29.470 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:20:29.487 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:29.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:29.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:29.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:29.614 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:29.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:29.714 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:29.739 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:29.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:29 smithi045 ceph-mon[39715]: pgmap v562: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 65 MiB/s wr, 1.43k op/s 2024-03-11T22:20:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:29 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4212159590' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:20:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:29 smithi045 ceph-mon[39715]: osdmap e309: 8 total, 8 up, 8 in 2024-03-11T22:20:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:29 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:29.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:29 smithi045 ceph-mon[39715]: osdmap e310: 8 total, 8 up, 8 in 2024-03-11T22:20:29.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:29 smithi045 ceph-mon[30178]: pgmap v562: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 65 MiB/s wr, 1.43k op/s 2024-03-11T22:20:29.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:29 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4212159590' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:20:29.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:29 smithi045 ceph-mon[30178]: osdmap e309: 8 total, 8 up, 8 in 2024-03-11T22:20:29.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:29 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:29.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:29 smithi045 ceph-mon[30178]: osdmap e310: 8 total, 8 up, 8 in 2024-03-11T22:20:29.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:29 smithi144 ceph-mon[33984]: pgmap v562: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 65 MiB/s wr, 1.43k op/s 2024-03-11T22:20:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:29 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4212159590' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:20:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:29 smithi144 ceph-mon[33984]: osdmap e309: 8 total, 8 up, 8 in 2024-03-11T22:20:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:29 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:20:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:29 smithi144 ceph-mon[33984]: osdmap e310: 8 total, 8 up, 8 in 2024-03-11T22:20:29.863 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:29.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:29.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:29.990 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:29.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:29.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:30.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:30.115 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:30.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:30.216 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:30.240 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:30.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:30.342 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:30.366 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:30.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:30.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:30.468 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:30.494 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:30.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:30.596 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:30.602 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:20:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:30.622 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:30.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:30.723 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:30.744 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[39715]: 4.6 scrub starts 2024-03-11T22:20:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[39715]: 4.6 scrub ok 2024-03-11T22:20:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[39715]: 5.11 scrub starts 2024-03-11T22:20:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[39715]: 5.11 scrub ok 2024-03-11T22:20:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:20:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:20:30.745 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[39715]: osdmap e311: 8 total, 8 up, 8 in 2024-03-11T22:20:30.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[30178]: 4.6 scrub starts 2024-03-11T22:20:30.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[30178]: 4.6 scrub ok 2024-03-11T22:20:30.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[30178]: 5.11 scrub starts 2024-03-11T22:20:30.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[30178]: 5.11 scrub ok 2024-03-11T22:20:30.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:30.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:30.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:30.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:30.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:20:30.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:30.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:30.746 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:20:30.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:30 smithi045 ceph-mon[30178]: osdmap e311: 8 total, 8 up, 8 in 2024-03-11T22:20:30.754 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:30.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:30.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:30.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:30 smithi144 ceph-mon[33984]: 4.6 scrub starts 2024-03-11T22:20:30.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:30 smithi144 ceph-mon[33984]: 4.6 scrub ok 2024-03-11T22:20:30.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:30 smithi144 ceph-mon[33984]: 5.11 scrub starts 2024-03-11T22:20:30.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:30 smithi144 ceph-mon[33984]: 5.11 scrub ok 2024-03-11T22:20:30.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:30.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:30.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:30.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:20:30.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:20:30.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:20:30.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:20:30.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:20:30.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:30 smithi144 ceph-mon[33984]: osdmap e311: 8 total, 8 up, 8 in 2024-03-11T22:20:30.856 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:30.886 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:30.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:30.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:30.987 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:31.037 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:31.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:31.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:31.103 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:20:31.103 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:20:31.109 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:20:31.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:31.153 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:20:31.195 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:20:31.205 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:31.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:31.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:31.242 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:20:31.284 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:20:31.307 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:31.331 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:20:31.336 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:20:31.336 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:20:31.336 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:20:31.336 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:20:31.336 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:20:31.336 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:20:31.336 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:20:31.336 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:20:31.338 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:31.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:31.440 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:31.474 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:31.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:31.575 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:31.606 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:31.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:31.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: Adjusting osd_memory_target on smithi045 to 3249M 2024-03-11T22:20:31.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: pgmap v565: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 28 MiB/s wr, 974 op/s 2024-03-11T22:20:31.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: 1.9 starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:20:31.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: 9.9 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8125 2024-03-11T22:20:31.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: 1.e starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:20:31.633 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: 9.7 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8180 2024-03-11T22:20:31.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:20:31.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: 1.d starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:20:31.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: 9.d starting backfill to osd.7 from (0'0,0'0] MAX to 308'8144 2024-03-11T22:20:31.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: 9.e starting backfill to osd.7 from (0'0,0'0] MAX to 308'8077 2024-03-11T22:20:31.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: 9.5 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8324 2024-03-11T22:20:31.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: 9.3 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8225 2024-03-11T22:20:31.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: 9.1 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8211 2024-03-11T22:20:31.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: 6.18 scrub starts 2024-03-11T22:20:31.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: 6.18 scrub ok 2024-03-11T22:20:31.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:31.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2102969542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:20:31.634 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[30178]: osdmap e312: 8 total, 8 up, 8 in 2024-03-11T22:20:31.636 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: Adjusting osd_memory_target on smithi045 to 3249M 2024-03-11T22:20:31.636 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: pgmap v565: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 28 MiB/s wr, 974 op/s 2024-03-11T22:20:31.636 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: 1.9 starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: 9.9 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8125 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: 1.e starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: 9.7 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8180 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: 1.d starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: 9.d starting backfill to osd.7 from (0'0,0'0] MAX to 308'8144 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: 9.e starting backfill to osd.7 from (0'0,0'0] MAX to 308'8077 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: 9.5 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8324 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: 9.3 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8225 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: 9.1 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8211 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: 6.18 scrub starts 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: 6.18 scrub ok 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2102969542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:20:31.637 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:31 smithi045 ceph-mon[39715]: osdmap e312: 8 total, 8 up, 8 in 2024-03-11T22:20:31.708 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:31.774 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:31.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:31.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:31.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: Adjusting osd_memory_target on smithi045 to 3249M 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: pgmap v565: 168 pgs: 168 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 28 MiB/s wr, 974 op/s 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: 1.9 starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: 9.9 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8125 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: 1.e starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: 9.7 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8180 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: 1.d starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: 9.d starting backfill to osd.7 from (0'0,0'0] MAX to 308'8144 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: 9.e starting backfill to osd.7 from (0'0,0'0] MAX to 308'8077 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: 9.5 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8324 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: 9.3 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8225 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: 9.1 starting backfill to osd.7 from (0'0,0'0] MAX to 308'8211 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: 6.18 scrub starts 2024-03-11T22:20:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: 6.18 scrub ok 2024-03-11T22:20:31.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:31.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2102969542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:20:31.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:31 smithi144 ceph-mon[33984]: osdmap e312: 8 total, 8 up, 8 in 2024-03-11T22:20:31.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:31.916 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:31.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:32.018 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:32.050 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:32.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:32.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:32.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:32.196 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:32.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:32.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:32.297 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:32.328 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:32.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:32.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:32.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:32.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:32.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:32.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:32.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:32 smithi045 ceph-mon[30178]: 3.13 scrub starts 2024-03-11T22:20:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:32 smithi045 ceph-mon[30178]: 3.13 scrub ok 2024-03-11T22:20:32.576 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:32 smithi045 ceph-mon[30178]: osdmap e313: 8 total, 8 up, 8 in 2024-03-11T22:20:32.576 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:20:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:20:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:20:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:32 smithi045 ceph-mon[39715]: 3.13 scrub starts 2024-03-11T22:20:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:32 smithi045 ceph-mon[39715]: 3.13 scrub ok 2024-03-11T22:20:32.577 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:32 smithi045 ceph-mon[39715]: osdmap e313: 8 total, 8 up, 8 in 2024-03-11T22:20:32.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:32.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:32.725 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:32.760 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:32.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:32.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:32 smithi144 ceph-mon[33984]: 3.13 scrub starts 2024-03-11T22:20:32.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:32 smithi144 ceph-mon[33984]: 3.13 scrub ok 2024-03-11T22:20:32.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:32 smithi144 ceph-mon[33984]: osdmap e313: 8 total, 8 up, 8 in 2024-03-11T22:20:32.862 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:32.900 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:32.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:32.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:33.001 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:33.035 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:33.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:33.135 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:33.167 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:33.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:33.268 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:33.299 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:33.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:33.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:33.398 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:33.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:33.429 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:20:33.429 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:20:33.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.429 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.25142.0:133385 9.0 e313/273)", 2024-03-11T22:20:33.429 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:33.414670+0000", 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0085714889999999998, 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0085984289999999994, 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.414670+0000", 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.414670+0000", 2024-03-11T22:20:33.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.414671+0000", 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1399999999999995e-07 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.414680+0000", 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3659999999999992e-06 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.414695+0000", 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.49e-05 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.414703+0000", 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1939999999999998e-06 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25142.0:133393 9.5 e313/310 9:a57081d2:::benchmark_data_smithi045_141553_object9273:head v 313'8521, mlcod=313'8519)", 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:33.419297+0000", 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0039447140000000002, 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040284689999999998, 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:20:33.432 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.419297+0000", 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.419297+0000", 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.419297+0000", 2024-03-11T22:20:33.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8700000000000001e-07 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.419302+0000", 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.792e-06 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.419309+0000", 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9949999999999999e-06 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:20:33.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.419314+0000", 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1580000000000004e-06 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.419396+0000", 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1305999999999997e-05 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.419420+0000", 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4307999999999998e-05 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.25142.0:133392 9.0 e313/273)", 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:33.421221+0000", 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0020205219999999999, 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002166154, 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.421221+0000", 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:20:33.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.421221+0000", 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.421222+0000", 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6299999999999993e-07 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.421232+0000", 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8870000000000004e-06 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:20:33.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.421244+0000", 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1902000000000001e-05 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.421252+0000", 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.807e-06 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:133395 9.4 9.a95a2984 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:33.421274+0000", 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0019683130000000002, 2024-03-11T22:20:33.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021687049999999999, 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 133395 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.421274+0000", 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:20:33.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.421274+0000", 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.421275+0000", 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3149999999999999e-06 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.421278+0000", 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6210000000000001e-06 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:20:33.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.421287+0000", 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9670000000000007e-06 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.421294+0000", 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7970000000000003e-06 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:133381 9.0 9:00c489b8:::benchmark_data_smithi045_141553_object9261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:33.393491+0000", 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.029750670999999999, 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.030013614000000001, 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:20:33.441 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 133381 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.393491+0000", 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:20:33.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.393491+0000", 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.393493+0000", 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.705e-06 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.393499+0000", 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1990000000000003e-06 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.393511+0000", 2024-03-11T22:20:33.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1929999999999999e-05 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.393519+0000", 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3380000000000005e-06 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.394437+0000", 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00091810000000000004 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:20:33.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.394601+0000", 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000163764 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.394721+0000", 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119529 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.397437+0000", 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002716483 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:20:33.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.404337+0000", 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068999730000000002 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:133385 9.0 9:0ca775fc:::benchmark_data_smithi045_141553_object9265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:33.411294+0000", 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.011948145, 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012303400000000001, 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:20:33.446 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 133385 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411294+0000", 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411294+0000", 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:20:33.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411296+0000", 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2989999999999999e-06 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411306+0000", 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.027e-05 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411333+0000", 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6628e-05 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411347+0000", 2024-03-11T22:20:33.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4481000000000001e-05 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411494+0000", 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000146348 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411644+0000", 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000150505 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411780+0000", 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013582500000000001 2024-03-11T22:20:33.449 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:133392 9.0 9:06358c41:::benchmark_data_smithi045_141553_object9272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:33.411439+0000", 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.011802897999999999, 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012237582, 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 133392 2024-03-11T22:20:33.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411439+0000", 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411439+0000", 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411440+0000", 2024-03-11T22:20:33.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2099999999999996e-07 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411444+0000", 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0450000000000002e-06 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411453+0000", 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9870000000000005e-06 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.411460+0000", 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9090000000000003e-06 2024-03-11T22:20:33.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.415021+0000", 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035618360000000001 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.415142+0000", 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000120493 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.415251+0000", 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000108925 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:20:33.453 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25142.0:133389 9.a e313/273 9:5b1b5aab:::benchmark_data_smithi045_141553_object9269:head v 313'8440, mlcod=313'8438)", 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:33.413104+0000", 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.010137811, 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010647278, 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.413104+0000", 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.413104+0000", 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.413105+0000", 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1999999999999999e-07 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.413112+0000", 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7729999999999999e-06 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.455 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.413123+0000", 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.062e-05 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.413130+0000", 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9439999999999999e-06 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.418863+0000", 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0057333749999999998 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:20:33.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.418883+0000", 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0052e-05 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.25142.0:133381 9.0 e313/273)", 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:33.413164+0000", 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.01007765, 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010645385, 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.413164+0000", 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.413164+0000", 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.413165+0000", 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4099999999999999e-07 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.413168+0000", 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2169999999999999e-06 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.413175+0000", 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.323e-06 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:33.413180+0000", 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2249999999999999e-06 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.459 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:20:33.460 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:20:33.460 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 9 2024-03-11T22:20:33.460 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:20:33.460 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:33.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:33.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:33.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:33.619 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:33.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:33.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:33.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:33 smithi045 ceph-mon[30178]: pgmap v569: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 1 activating+remapped, 2 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 154 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 912 op/s; 51442/352908 objects misplaced (14.577%); 2.2 MiB/s, 48 objects/s recovering 2024-03-11T22:20:33.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:33 smithi045 ceph-mon[30178]: 6.15 scrub starts 2024-03-11T22:20:33.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:33 smithi045 ceph-mon[30178]: 6.15 scrub ok 2024-03-11T22:20:33.692 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:33 smithi045 ceph-mon[30178]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:20:33.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:33 smithi045 ceph-mon[39715]: pgmap v569: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 1 activating+remapped, 2 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 154 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 912 op/s; 51442/352908 objects misplaced (14.577%); 2.2 MiB/s, 48 objects/s recovering 2024-03-11T22:20:33.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:33 smithi045 ceph-mon[39715]: 6.15 scrub starts 2024-03-11T22:20:33.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:33 smithi045 ceph-mon[39715]: 6.15 scrub ok 2024-03-11T22:20:33.693 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:33 smithi045 ceph-mon[39715]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:20:33.721 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:33.758 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:33.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:33.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:33.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:33 smithi144 ceph-mon[33984]: pgmap v569: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 1 activating+remapped, 2 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 154 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 912 op/s; 51442/352908 objects misplaced (14.577%); 2.2 MiB/s, 48 objects/s recovering 2024-03-11T22:20:33.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:33 smithi144 ceph-mon[33984]: 6.15 scrub starts 2024-03-11T22:20:33.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:33 smithi144 ceph-mon[33984]: 6.15 scrub ok 2024-03-11T22:20:33.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:33 smithi144 ceph-mon[33984]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:20:33.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:33.904 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:33.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:34.005 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:34.042 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:34.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:34.144 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:34.176 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:34.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:34.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:34.278 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:34.311 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:34.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:34.412 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:34.448 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:34.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:34.533 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T22:20:34.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:34.633 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:34.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:34.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:34.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:34.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:34.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:34.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:34.987 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:35.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:20:35.035 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:35.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:35.137 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:35.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:35.170 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:35.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:35.271 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:35.301 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:35.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:35.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:35.403 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:35.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:35.481 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:35 smithi045 ceph-mon[39715]: pgmap v570: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 1 activating+remapped, 2 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 154 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 725 op/s; 51442/352908 objects misplaced (14.577%); 1.8 MiB/s, 38 objects/s recovering 2024-03-11T22:20:35.481 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:35 smithi045 ceph-mon[30178]: pgmap v570: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 1 activating+remapped, 2 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 154 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 725 op/s; 51442/352908 objects misplaced (14.577%); 1.8 MiB/s, 38 objects/s recovering 2024-03-11T22:20:35.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:35.589 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:35.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:35.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:35.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:35.730 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:35.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:35.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:35.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:35 smithi144 ceph-mon[33984]: pgmap v570: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 1 activating+remapped, 2 active+remapped+backfilling, 3 active+clean+remapped, 2 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 154 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 725 op/s; 51442/352908 objects misplaced (14.577%); 1.8 MiB/s, 38 objects/s recovering 2024-03-11T22:20:35.877 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:35.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:35.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:35.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:36.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:36.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:36.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:36.124 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:36.165 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:36.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:36.266 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:36.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:36.302 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:36.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:36.403 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:36.438 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:36.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:36.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:36.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:36 smithi045 ceph-mon[39715]: 1.f scrub starts 2024-03-11T22:20:36.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:36 smithi045 ceph-mon[39715]: 1.f scrub ok 2024-03-11T22:20:36.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:36 smithi045 ceph-mon[30178]: 1.f scrub starts 2024-03-11T22:20:36.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:36 smithi045 ceph-mon[30178]: 1.f scrub ok 2024-03-11T22:20:36.539 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:36.576 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:36.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:36.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:36.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:36 smithi144 ceph-mon[33984]: 1.f scrub starts 2024-03-11T22:20:36.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:36 smithi144 ceph-mon[33984]: 1.f scrub ok 2024-03-11T22:20:36.597 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:20:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:20:36.679 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:36.715 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:36.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:36.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:36.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:36.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:36.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:36.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:36.985 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:36.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:37.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:37.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:37.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:37.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:37.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:37.260 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:37.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:37.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:37.404 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:37.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:37.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:37.533 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:37.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:37.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:37.671 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:37.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:37.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:37.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:37.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:37 smithi045 ceph-mon[39715]: pgmap v571: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 159 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.34k op/s; 47123/339396 objects misplaced (13.884%); 29 MiB/s, 468 objects/s recovering 2024-03-11T22:20:37.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:37.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:37 smithi045 ceph-mon[30178]: pgmap v571: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 159 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.34k op/s; 47123/339396 objects misplaced (13.884%); 29 MiB/s, 468 objects/s recovering 2024-03-11T22:20:37.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:37.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:37.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:37.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:37.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:37 smithi144 ceph-mon[33984]: pgmap v571: 168 pgs: 2 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 159 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.34k op/s; 47123/339396 objects misplaced (13.884%); 29 MiB/s, 468 objects/s recovering 2024-03-11T22:20:37.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:37.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:37.947 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:37.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:37.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:38.049 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:38.084 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:38.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:38.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:38.186 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:38.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:38.223 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:38.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:38.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:38.366 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:38.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:38.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:38.467 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:38.503 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:38.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:38.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:38.604 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:38.637 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:38.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:38.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:38.739 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:38.961 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:38.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:39.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:39.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:39.100 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:39.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:39.200 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:39.242 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:39.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:39.344 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:39.377 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:39.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:39.478 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:39.512 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:39.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:39.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:39.559 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:39.614 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:39.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:39.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:39.757 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:39.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:39 smithi045 ceph-mon[30178]: pgmap v572: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.16k op/s; 46409/336207 objects misplaced (13.804%); 22 MiB/s, 354 objects/s recovering 2024-03-11T22:20:39.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:39 smithi045 ceph-mon[39715]: pgmap v572: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.16k op/s; 46409/336207 objects misplaced (13.804%); 22 MiB/s, 354 objects/s recovering 2024-03-11T22:20:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:39.799 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:39.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:39.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:39 smithi144 ceph-mon[33984]: pgmap v572: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.16k op/s; 46409/336207 objects misplaced (13.804%); 22 MiB/s, 354 objects/s recovering 2024-03-11T22:20:39.901 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:39.928 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:39.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:40.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:40.066 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:40.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:40.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:40.168 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:40.200 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:40.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:40.301 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:40.333 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:40.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:40.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:40.476 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:40.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:40.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:40.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:40.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:40.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:40.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:40.717 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:40.741 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:40.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:40.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:40.782 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:40 smithi045 ceph-mon[39715]: 5.b scrub starts 2024-03-11T22:20:40.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:40 smithi045 ceph-mon[39715]: 5.b scrub ok 2024-03-11T22:20:40.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:40 smithi045 ceph-mon[30178]: 5.b scrub starts 2024-03-11T22:20:40.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:40 smithi045 ceph-mon[30178]: 5.b scrub ok 2024-03-11T22:20:40.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:40.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:40 smithi144 ceph-mon[33984]: 5.b scrub starts 2024-03-11T22:20:40.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:40 smithi144 ceph-mon[33984]: 5.b scrub ok 2024-03-11T22:20:40.879 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:40.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:40.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:40.980 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:41.019 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:41.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:41.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:41.157 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:41.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:41.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:41.259 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:41.293 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:41.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:41.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:41.395 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:41.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:41.427 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:41.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:41.528 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:41.566 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:41.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:41.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:41 smithi045 ceph-mon[30178]: pgmap v573: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 568 B/s rd, 0 B/s wr, 682 op/s; 46409/336207 objects misplaced (13.804%); 18 MiB/s, 292 objects/s recovering 2024-03-11T22:20:41.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:41 smithi045 ceph-mon[30178]: 4.d scrub starts 2024-03-11T22:20:41.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:41 smithi045 ceph-mon[30178]: 4.d scrub ok 2024-03-11T22:20:41.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:41 smithi045 ceph-mon[30178]: 5.10 scrub starts 2024-03-11T22:20:41.604 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:41 smithi045 ceph-mon[30178]: 5.10 scrub ok 2024-03-11T22:20:41.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:41 smithi045 ceph-mon[39715]: pgmap v573: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 568 B/s rd, 0 B/s wr, 682 op/s; 46409/336207 objects misplaced (13.804%); 18 MiB/s, 292 objects/s recovering 2024-03-11T22:20:41.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:41 smithi045 ceph-mon[39715]: 4.d scrub starts 2024-03-11T22:20:41.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:41 smithi045 ceph-mon[39715]: 4.d scrub ok 2024-03-11T22:20:41.606 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:41 smithi045 ceph-mon[39715]: 5.10 scrub starts 2024-03-11T22:20:41.607 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:41 smithi045 ceph-mon[39715]: 5.10 scrub ok 2024-03-11T22:20:41.667 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:41.700 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:41.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:41.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:41.846 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:41.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:41.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:41 smithi144 ceph-mon[33984]: pgmap v573: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 568 B/s rd, 0 B/s wr, 682 op/s; 46409/336207 objects misplaced (13.804%); 18 MiB/s, 292 objects/s recovering 2024-03-11T22:20:41.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:41 smithi144 ceph-mon[33984]: 4.d scrub starts 2024-03-11T22:20:41.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:41 smithi144 ceph-mon[33984]: 4.d scrub ok 2024-03-11T22:20:41.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:41 smithi144 ceph-mon[33984]: 5.10 scrub starts 2024-03-11T22:20:41.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:41 smithi144 ceph-mon[33984]: 5.10 scrub ok 2024-03-11T22:20:41.948 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:41.980 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:41.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:41.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:42.007 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:42.082 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:42.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:42.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:42.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:42.239 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:42.271 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:42.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:42.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:42.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:42.422 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:42.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:42.505 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:20:42.513 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:20:42.523 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:42.555 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:42.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:42.657 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:42.690 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:42.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:42.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:42.782 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:20:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:20:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:20:42.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:42.828 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:42.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:42.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:42.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:42.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:43.067 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:43.100 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:43.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:43.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:43.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:43.245 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:43.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:43.347 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:43.380 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:43.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:43.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:43.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:43.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:43.525 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:43.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:43.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:43.649 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:43.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:43.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:43.750 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:43.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:43 smithi045 ceph-mon[30178]: pgmap v574: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.07k op/s; 41836/320841 objects misplaced (13.039%); 32 MiB/s, 513 objects/s recovering 2024-03-11T22:20:43.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:43 smithi045 ceph-mon[30178]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:20:43.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:43 smithi045 ceph-mon[30178]: 3.10 scrub starts 2024-03-11T22:20:43.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:43 smithi045 ceph-mon[30178]: 3.10 scrub ok 2024-03-11T22:20:43.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:43 smithi045 ceph-mon[39715]: pgmap v574: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.07k op/s; 41836/320841 objects misplaced (13.039%); 32 MiB/s, 513 objects/s recovering 2024-03-11T22:20:43.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:43 smithi045 ceph-mon[39715]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:20:43.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:43 smithi045 ceph-mon[39715]: 3.10 scrub starts 2024-03-11T22:20:43.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:43 smithi045 ceph-mon[39715]: 3.10 scrub ok 2024-03-11T22:20:43.787 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:43.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:43.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:43 smithi144 ceph-mon[33984]: pgmap v574: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.07k op/s; 41836/320841 objects misplaced (13.039%); 32 MiB/s, 513 objects/s recovering 2024-03-11T22:20:43.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:43 smithi144 ceph-mon[33984]: from='client.15390 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:20:43.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:43 smithi144 ceph-mon[33984]: 3.10 scrub starts 2024-03-11T22:20:43.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:43 smithi144 ceph-mon[33984]: 3.10 scrub ok 2024-03-11T22:20:43.889 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:43.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:43.925 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:43.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:44.026 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:44.059 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:44.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:44.160 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:44.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:44.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:44.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:44.303 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:44.338 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:44.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:44.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:44.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:44.481 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:44.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:44.582 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:44.604 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:44.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:44.706 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:44.738 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:44.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:44.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:44.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:44 smithi045 ceph-mon[39715]: 1.7 scrub starts 2024-03-11T22:20:44.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:44 smithi045 ceph-mon[39715]: 1.7 scrub ok 2024-03-11T22:20:44.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:44 smithi045 ceph-mon[30178]: 1.7 scrub starts 2024-03-11T22:20:44.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:44 smithi045 ceph-mon[30178]: 1.7 scrub ok 2024-03-11T22:20:44.839 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:44.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:44 smithi144 ceph-mon[33984]: 1.7 scrub starts 2024-03-11T22:20:44.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:44 smithi144 ceph-mon[33984]: 1.7 scrub ok 2024-03-11T22:20:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:44.881 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:44.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:44.982 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:45.016 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:45.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:45.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:45.117 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:45.151 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:45.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:45.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:45.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:45.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:45.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:45.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:45.439 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:45.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:45.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:45.541 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:45.578 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:45.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:45.679 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:45.724 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:45.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:45.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:45.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:45 smithi045 ceph-mon[39715]: pgmap v575: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 891 op/s; 41836/320841 objects misplaced (13.039%); 27 MiB/s, 428 objects/s recovering 2024-03-11T22:20:45.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:45 smithi045 ceph-mon[39715]: 1.6 scrub starts 2024-03-11T22:20:45.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:45 smithi045 ceph-mon[39715]: 1.6 scrub ok 2024-03-11T22:20:45.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:45 smithi045 ceph-mon[30178]: pgmap v575: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 891 op/s; 41836/320841 objects misplaced (13.039%); 27 MiB/s, 428 objects/s recovering 2024-03-11T22:20:45.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:45 smithi045 ceph-mon[30178]: 1.6 scrub starts 2024-03-11T22:20:45.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:45 smithi045 ceph-mon[30178]: 1.6 scrub ok 2024-03-11T22:20:45.825 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:45 smithi144 ceph-mon[33984]: pgmap v575: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 891 op/s; 41836/320841 objects misplaced (13.039%); 27 MiB/s, 428 objects/s recovering 2024-03-11T22:20:45.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:45 smithi144 ceph-mon[33984]: 1.6 scrub starts 2024-03-11T22:20:45.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:45 smithi144 ceph-mon[33984]: 1.6 scrub ok 2024-03-11T22:20:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:45.860 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:45.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:45.961 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:45.987 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:45.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:46.089 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:46.124 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:46.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:46.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:46.253 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:46.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:46.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:46.355 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:46.382 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:46.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:46.420 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":574,"stamp":"2024-03-11T22:20:42.292766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6996433268,"num_objects":106947,"num_object_clones":0,"num_object_copies":320841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":41836,"num_objects_unfound":0,"num_objects_dirty":106947,"num_whiteouts":0,"num_read":1818,"num_read_kb":1570,"num_write":142209,"num_write_kb":7943534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5392,"num_bytes_recovered":346098052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74327,"ondisk_log_size":74327,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":512,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":24050280,"kb_used_data":21000680,"kb_used_omap":12,"kb_used_meta":3049523,"kb_avail":725910936,"statfs":{"total":767960285184,"available":743332798464,"internally_reserved":0,"allocated":21504696320,"data_stored":21502066340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":3122712115},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,4,6,0,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-700514304,"num_objects":-10689,"num_object_clones":0,"num_object_copies":-32067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-9606,"num_objects_unfound":0,"num_objects_dirty":-10689,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10689,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5144,"num_bytes_recovered":337117184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.014832"},"pg_stats":[{"pgid":"9.1","version":"313'8980","reported_seq":11643,"reported_epoch":313,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T22:20:41.120005+0000","last_change":"2024-03-11T22:20:30.592256+0000","last_active":"2024-03-11T22:20:41.120005+0000","last_peered":"2024-03-11T22:20:41.120005+0000","last_clean":"2024-03-11T22:20:28.399763+0000","last_became_active":"2024-03-11T22:20:30.566997+0000","last_became_peered":"2024-03-11T22:20:30.566997+0000","last_unstale":"2024-03-11T22:20:41.120005+0000","last_undegraded":"2024-03-11T22:20:41.120005+0000","last_fullsized":"2024-03-11T22:20:41.120005+0000","mapping_epoch":310,"log_start":"294'4900","ondisk_log_start":"294'4900","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4080,"log_dups_size":0,"ondisk_log_size":4080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434372608,"num_objects":6628,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5417,"num_objects_unfound":0,"num_objects_dirty":6628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1282,"num_bytes_recovered":83951616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":6628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"294'25","reported_seq":350,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.287157+0000","last_change":"2024-03-11T22:19:41.508884+0000","last_active":"2024-03-11T22:20:32.287157+0000","last_peered":"2024-03-11T22:20:32.287157+0000","last_clean":"2024-03-11T22:20:32.287157+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:20:32.287157+0000","last_undegraded":"2024-03-11T22:20:32.287157+0000","last_fullsized":"2024-03-11T22:20:32.287157+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:19:41.508161+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:45.768331+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":538,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284468+0000","last_change":"2024-03-11T22:19:28.586829+0000","last_active":"2024-03-11T22:20:32.284468+0000","last_peered":"2024-03-11T22:20:32.284468+0000","last_clean":"2024-03-11T22:20:32.284468+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:20:32.284468+0000","last_undegraded":"2024-03-11T22:20:32.284468+0000","last_fullsized":"2024-03-11T22:20:32.284468+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:28.586614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:28.586614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":709,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:33.493982+0000","last_change":"2024-03-11T22:19:40.008490+0000","last_active":"2024-03-11T22:20:33.493982+0000","last_peered":"2024-03-11T22:20:33.493982+0000","last_clean":"2024-03-11T22:20:33.493982+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:20:33.493982+0000","last_undegraded":"2024-03-11T22:20:33.493982+0000","last_fullsized":"2024-03-11T22:20:33.493982+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:19:40.008325+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:19:40.008325+0000","last_clean_scrub_stamp":"2024-03-11T22:19:40.008325+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:45.464443+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":304,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.290879+0000","last_change":"2024-03-11T22:18:37.116172+0000","last_active":"2024-03-11T22:20:32.290879+0000","last_peered":"2024-03-11T22:20:32.290879+0000","last_clean":"2024-03-11T22:20:32.290879+0000","last_became_active":"2024-03-11T22:14:52.243690+0000","last_became_peered":"2024-03-11T22:14:52.243690+0000","last_unstale":"2024-03-11T22:20:32.290879+0000","last_undegraded":"2024-03-11T22:20:32.290879+0000","last_fullsized":"2024-03-11T22:20:32.290879+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:37.116093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.116093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":861,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.294412+0000","last_change":"2024-03-11T22:20:32.294412+0000","last_active":"2024-03-11T22:20:32.294412+0000","last_peered":"2024-03-11T22:20:32.294412+0000","last_clean":"2024-03-11T22:20:32.294412+0000","last_became_active":"2024-03-11T22:20:32.293356+0000","last_became_peered":"2024-03-11T22:20:32.293356+0000","last_unstale":"2024-03-11T22:20:32.294412+0000","last_undegraded":"2024-03-11T22:20:32.294412+0000","last_fullsized":"2024-03-11T22:20:32.294412+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:20.907743+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.907743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"313'8895","reported_seq":8986,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:41.088437+0000","last_change":"2024-03-11T22:18:41.783594+0000","last_active":"2024-03-11T22:20:41.088437+0000","last_peered":"2024-03-11T22:20:41.088437+0000","last_clean":"2024-03-11T22:20:41.088437+0000","last_became_active":"2024-03-11T22:18:41.783248+0000","last_became_peered":"2024-03-11T22:18:41.783248+0000","last_unstale":"2024-03-11T22:20:41.088437+0000","last_undegraded":"2024-03-11T22:20:41.088437+0000","last_fullsized":"2024-03-11T22:20:41.088437+0000","mapping_epoch":273,"log_start":"292'4400","ondisk_log_start":"292'4400","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436797440,"num_objects":6665,"num_object_clones":0,"num_object_copies":19995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8895,"num_write_kb":497920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"273'31","reported_seq":388,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:40.790439+0000","last_change":"2024-03-11T22:20:40.790500+0000","last_active":"2024-03-11T22:20:40.790439+0000","last_peered":"2024-03-11T22:20:40.790439+0000","last_clean":"2024-03-11T22:20:40.790439+0000","last_became_active":"2024-03-11T22:14:52.239649+0000","last_became_peered":"2024-03-11T22:14:52.239649+0000","last_unstale":"2024-03-11T22:20:40.790439+0000","last_undegraded":"2024-03-11T22:20:40.790439+0000","last_fullsized":"2024-03-11T22:20:40.790439+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'31","last_scrub_stamp":"2024-03-11T22:20:40.790339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:40.790339+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:48.818452+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":554,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284357+0000","last_change":"2024-03-11T22:18:35.820337+0000","last_active":"2024-03-11T22:20:32.284357+0000","last_peered":"2024-03-11T22:20:32.284357+0000","last_clean":"2024-03-11T22:20:32.284357+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:20:32.284357+0000","last_undegraded":"2024-03-11T22:20:32.284357+0000","last_fullsized":"2024-03-11T22:20:32.284357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:35.820233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:18:35.820233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":538,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.285360+0000","last_change":"2024-03-11T22:18:44.130366+0000","last_active":"2024-03-11T22:20:32.285360+0000","last_peered":"2024-03-11T22:20:32.285360+0000","last_clean":"2024-03-11T22:20:32.285360+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:20:32.285360+0000","last_undegraded":"2024-03-11T22:20:32.285360+0000","last_fullsized":"2024-03-11T22:20:32.285360+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:44.130252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:18:44.130252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":542,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.280898+0000","last_change":"2024-03-11T22:18:51.794420+0000","last_active":"2024-03-11T22:20:32.280898+0000","last_peered":"2024-03-11T22:20:32.280898+0000","last_clean":"2024-03-11T22:20:32.280898+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:20:32.280898+0000","last_undegraded":"2024-03-11T22:20:32.280898+0000","last_fullsized":"2024-03-11T22:20:32.280898+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:51.794313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:51.794313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":837,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.289618+0000","last_change":"2024-03-11T22:18:21.646464+0000","last_active":"2024-03-11T22:20:32.289618+0000","last_peered":"2024-03-11T22:20:32.289618+0000","last_clean":"2024-03-11T22:20:32.289618+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:20:32.289618+0000","last_undegraded":"2024-03-11T22:20:32.289618+0000","last_fullsized":"2024-03-11T22:20:32.289618+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:21.646305+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.646305+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"313'8880","reported_seq":9019,"reported_epoch":313,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-11T22:20:41.091319+0000","last_change":"2024-03-11T22:20:30.647673+0000","last_active":"2024-03-11T22:20:41.091319+0000","last_peered":"2024-03-11T22:20:41.091319+0000","last_clean":"2024-03-11T22:20:28.402388+0000","last_became_active":"2024-03-11T22:20:30.515089+0000","last_became_peered":"2024-03-11T22:20:30.515089+0000","last_unstale":"2024-03-11T22:20:41.091319+0000","last_undegraded":"2024-03-11T22:20:41.091319+0000","last_fullsized":"2024-03-11T22:20:41.091319+0000","mapping_epoch":310,"log_start":"294'4800","ondisk_log_start":"294'4800","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4080,"log_dups_size":0,"ondisk_log_size":4080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446693376,"num_objects":6816,"num_object_clones":0,"num_object_copies":20448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6802,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8880,"num_write_kb":502272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,1,5],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":6816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"273'19","reported_seq":362,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.286498+0000","last_change":"2024-03-11T22:20:27.089007+0000","last_active":"2024-03-11T22:20:32.286498+0000","last_peered":"2024-03-11T22:20:32.286498+0000","last_clean":"2024-03-11T22:20:32.286498+0000","last_became_active":"2024-03-11T22:14:52.240437+0000","last_became_peered":"2024-03-11T22:14:52.240437+0000","last_unstale":"2024-03-11T22:20:32.286498+0000","last_undegraded":"2024-03-11T22:20:32.286498+0000","last_fullsized":"2024-03-11T22:20:32.286498+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:20:27.088856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:27.088856+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:30.994147+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":313,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.279732+0000","last_change":"2024-03-11T22:18:45.711951+0000","last_active":"2024-03-11T22:20:32.279732+0000","last_peered":"2024-03-11T22:20:32.279732+0000","last_clean":"2024-03-11T22:20:32.279732+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:20:32.279732+0000","last_undegraded":"2024-03-11T22:20:32.279732+0000","last_fullsized":"2024-03-11T22:20:32.279732+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:45.711862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:45.711862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":530,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.293570+0000","last_change":"2024-03-11T22:18:37.705396+0000","last_active":"2024-03-11T22:20:32.293570+0000","last_peered":"2024-03-11T22:20:32.293570+0000","last_clean":"2024-03-11T22:20:32.293570+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:20:32.293570+0000","last_undegraded":"2024-03-11T22:20:32.293570+0000","last_fullsized":"2024-03-11T22:20:32.293570+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:37.705305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.705305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":518,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.286601+0000","last_change":"2024-03-11T22:18:48.086934+0000","last_active":"2024-03-11T22:20:32.286601+0000","last_peered":"2024-03-11T22:20:32.286601+0000","last_clean":"2024-03-11T22:20:32.286601+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:20:32.286601+0000","last_undegraded":"2024-03-11T22:20:32.286601+0000","last_fullsized":"2024-03-11T22:20:32.286601+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:48.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:48.086856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"93'130","reported_seq":865,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284724+0000","last_change":"2024-03-11T22:18:20.968124+0000","last_active":"2024-03-11T22:20:32.284724+0000","last_peered":"2024-03-11T22:20:32.284724+0000","last_clean":"2024-03-11T22:20:32.284724+0000","last_became_active":"2024-03-11T22:18:00.271764+0000","last_became_peered":"2024-03-11T22:18:00.271764+0000","last_unstale":"2024-03-11T22:20:32.284724+0000","last_undegraded":"2024-03-11T22:20:32.284724+0000","last_fullsized":"2024-03-11T22:20:32.284724+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:20.968038+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.968038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"313'8758","reported_seq":8848,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:41.090940+0000","last_change":"2024-03-11T22:18:41.238608+0000","last_active":"2024-03-11T22:20:41.090940+0000","last_peered":"2024-03-11T22:20:41.090940+0000","last_clean":"2024-03-11T22:20:41.090940+0000","last_became_active":"2024-03-11T22:18:41.238269+0000","last_became_peered":"2024-03-11T22:18:41.238269+0000","last_unstale":"2024-03-11T22:20:41.090940+0000","last_undegraded":"2024-03-11T22:20:41.090940+0000","last_fullsized":"2024-03-11T22:20:41.090940+0000","mapping_epoch":273,"log_start":"294'4700","ondisk_log_start":"294'4700","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4058,"log_dups_size":0,"ondisk_log_size":4058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429785088,"num_objects":6558,"num_object_clones":0,"num_object_copies":19674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8758,"num_write_kb":490112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"273'34","reported_seq":600,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.288686+0000","last_change":"2024-03-11T22:20:26.599126+0000","last_active":"2024-03-11T22:20:32.288686+0000","last_peered":"2024-03-11T22:20:32.288686+0000","last_clean":"2024-03-11T22:20:32.288686+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:20:32.288686+0000","last_undegraded":"2024-03-11T22:20:32.288686+0000","last_fullsized":"2024-03-11T22:20:32.288686+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:20:26.598991+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:20:26.598991+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:48.861807+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":542,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283303+0000","last_change":"2024-03-11T22:20:29.432709+0000","last_active":"2024-03-11T22:20:32.283303+0000","last_peered":"2024-03-11T22:20:32.283303+0000","last_clean":"2024-03-11T22:20:32.283303+0000","last_became_active":"2024-03-11T22:20:29.426224+0000","last_became_peered":"2024-03-11T22:20:29.426224+0000","last_unstale":"2024-03-11T22:20:32.283303+0000","last_undegraded":"2024-03-11T22:20:32.283303+0000","last_fullsized":"2024-03-11T22:20:32.283303+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:38.013058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:38.013058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":392,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:33.493897+0000","last_change":"2024-03-11T22:15:39.663447+0000","last_active":"2024-03-11T22:20:33.493897+0000","last_peered":"2024-03-11T22:20:33.493897+0000","last_clean":"2024-03-11T22:20:33.493897+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:20:33.493897+0000","last_undegraded":"2024-03-11T22:20:33.493897+0000","last_fullsized":"2024-03-11T22:20:33.493897+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:15:39.663324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:39.663324+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":526,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.294252+0000","last_change":"2024-03-11T22:18:50.805816+0000","last_active":"2024-03-11T22:20:32.294252+0000","last_peered":"2024-03-11T22:20:32.294252+0000","last_clean":"2024-03-11T22:20:32.294252+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:20:32.294252+0000","last_undegraded":"2024-03-11T22:20:32.294252+0000","last_fullsized":"2024-03-11T22:20:32.294252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:50.805698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:18:50.805698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":838,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.289609+0000","last_change":"2024-03-11T22:18:20.663763+0000","last_active":"2024-03-11T22:20:32.289609+0000","last_peered":"2024-03-11T22:20:32.289609+0000","last_clean":"2024-03-11T22:20:32.289609+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:20:32.289609+0000","last_undegraded":"2024-03-11T22:20:32.289609+0000","last_fullsized":"2024-03-11T22:20:32.289609+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:20.663661+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.663661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"313'8843","reported_seq":8939,"reported_epoch":313,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-11T22:20:41.097172+0000","last_change":"2024-03-11T22:20:30.592868+0000","last_active":"2024-03-11T22:20:41.097172+0000","last_peered":"2024-03-11T22:20:41.097172+0000","last_clean":"2024-03-11T22:20:28.382555+0000","last_became_active":"2024-03-11T22:20:30.580794+0000","last_became_peered":"2024-03-11T22:20:30.580794+0000","last_unstale":"2024-03-11T22:20:41.097172+0000","last_undegraded":"2024-03-11T22:20:41.097172+0000","last_fullsized":"2024-03-11T22:20:41.097172+0000","mapping_epoch":310,"log_start":"291'4300","ondisk_log_start":"291'4300","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434372632,"num_objects":6629,"num_object_clones":0,"num_object_copies":19887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6629,"num_objects_unfound":0,"num_objects_dirty":6629,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8843,"num_write_kb":495041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,5,4],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":6629}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":314,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284820+0000","last_change":"2024-03-11T22:20:29.441577+0000","last_active":"2024-03-11T22:20:32.284820+0000","last_peered":"2024-03-11T22:20:32.284820+0000","last_clean":"2024-03-11T22:20:32.284820+0000","last_became_active":"2024-03-11T22:20:29.438896+0000","last_became_peered":"2024-03-11T22:20:29.438896+0000","last_unstale":"2024-03-11T22:20:32.284820+0000","last_undegraded":"2024-03-11T22:20:32.284820+0000","last_fullsized":"2024-03-11T22:20:32.284820+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.293977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.293977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"273'30","reported_seq":628,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283444+0000","last_change":"2024-03-11T22:20:29.473203+0000","last_active":"2024-03-11T22:20:32.283444+0000","last_peered":"2024-03-11T22:20:32.283444+0000","last_clean":"2024-03-11T22:20:32.283444+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:20:32.283444+0000","last_undegraded":"2024-03-11T22:20:32.283444+0000","last_fullsized":"2024-03-11T22:20:32.283444+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:15:33.497732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:33.497732+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":550,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.286211+0000","last_change":"2024-03-11T22:20:29.441068+0000","last_active":"2024-03-11T22:20:32.286211+0000","last_peered":"2024-03-11T22:20:32.286211+0000","last_clean":"2024-03-11T22:20:32.286211+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:20:32.286211+0000","last_undegraded":"2024-03-11T22:20:32.286211+0000","last_fullsized":"2024-03-11T22:20:32.286211+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.133631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.133631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":296,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283301+0000","last_change":"2024-03-11T22:15:12.339167+0000","last_active":"2024-03-11T22:20:32.283301+0000","last_peered":"2024-03-11T22:20:32.283301+0000","last_clean":"2024-03-11T22:20:32.283301+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:20:32.283301+0000","last_undegraded":"2024-03-11T22:20:32.283301+0000","last_fullsized":"2024-03-11T22:20:32.283301+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:12.339050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:12.339050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":790,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.280177+0000","last_change":"2024-03-11T22:18:36.683358+0000","last_active":"2024-03-11T22:20:32.280177+0000","last_peered":"2024-03-11T22:20:32.280177+0000","last_clean":"2024-03-11T22:20:32.280177+0000","last_became_active":"2024-03-11T22:18:20.240728+0000","last_became_peered":"2024-03-11T22:18:20.240728+0000","last_unstale":"2024-03-11T22:20:32.280177+0000","last_undegraded":"2024-03-11T22:20:32.280177+0000","last_fullsized":"2024-03-11T22:20:32.280177+0000","mapping_epoch":175,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:36.683219+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:36.683219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.c","version":"313'8775","reported_seq":8865,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:41.104337+0000","last_change":"2024-03-11T22:18:41.260878+0000","last_active":"2024-03-11T22:20:41.104337+0000","last_peered":"2024-03-11T22:20:41.104337+0000","last_clean":"2024-03-11T22:20:41.104337+0000","last_became_active":"2024-03-11T22:18:41.260611+0000","last_became_peered":"2024-03-11T22:18:41.260611+0000","last_unstale":"2024-03-11T22:20:41.104337+0000","last_undegraded":"2024-03-11T22:20:41.104337+0000","last_fullsized":"2024-03-11T22:20:41.104337+0000","mapping_epoch":273,"log_start":"294'4700","ondisk_log_start":"294'4700","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436273152,"num_objects":6657,"num_object_clones":0,"num_object_copies":19971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8775,"num_write_kb":493824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":677,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:38.498033+0000","last_change":"2024-03-11T22:15:31.519229+0000","last_active":"2024-03-11T22:20:38.498033+0000","last_peered":"2024-03-11T22:20:38.498033+0000","last_clean":"2024-03-11T22:20:38.498033+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:20:38.498033+0000","last_undegraded":"2024-03-11T22:20:38.498033+0000","last_fullsized":"2024-03-11T22:20:38.498033+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:15:31.513336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:31.513336+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"273'28","reported_seq":357,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.290286+0000","last_change":"2024-03-11T22:15:36.626072+0000","last_active":"2024-03-11T22:20:32.290286+0000","last_peered":"2024-03-11T22:20:32.290286+0000","last_clean":"2024-03-11T22:20:32.290286+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:20:32.290286+0000","last_undegraded":"2024-03-11T22:20:32.290286+0000","last_fullsized":"2024-03-11T22:20:32.290286+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:15:36.625916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:36.625916+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":539,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.288844+0000","last_change":"2024-03-11T22:19:28.428409+0000","last_active":"2024-03-11T22:20:32.288844+0000","last_peered":"2024-03-11T22:20:32.288844+0000","last_clean":"2024-03-11T22:20:32.288844+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:20:32.288844+0000","last_undegraded":"2024-03-11T22:20:32.288844+0000","last_fullsized":"2024-03-11T22:20:32.288844+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:19:28.428283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:28.428283+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:36.521430+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":484,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284550+0000","last_change":"2024-03-11T22:19:35.746810+0000","last_active":"2024-03-11T22:20:32.284550+0000","last_peered":"2024-03-11T22:20:32.284550+0000","last_clean":"2024-03-11T22:20:32.284550+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:20:32.284550+0000","last_undegraded":"2024-03-11T22:20:32.284550+0000","last_fullsized":"2024-03-11T22:20:32.284550+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:35.746693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:19:35.746693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:39.475020+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":854,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.293956+0000","last_change":"2024-03-11T22:20:32.293956+0000","last_active":"2024-03-11T22:20:32.293956+0000","last_peered":"2024-03-11T22:20:32.293956+0000","last_clean":"2024-03-11T22:20:32.293956+0000","last_became_active":"2024-03-11T22:20:32.293085+0000","last_became_peered":"2024-03-11T22:20:32.293085+0000","last_unstale":"2024-03-11T22:20:32.293956+0000","last_undegraded":"2024-03-11T22:20:32.293956+0000","last_fullsized":"2024-03-11T22:20:32.293956+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:18.905939+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:18.905939+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"313'8948","reported_seq":9038,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:40.445160+0000","last_change":"2024-03-11T22:18:41.236691+0000","last_active":"2024-03-11T22:20:40.445160+0000","last_peered":"2024-03-11T22:20:40.445160+0000","last_clean":"2024-03-11T22:20:40.445160+0000","last_became_active":"2024-03-11T22:18:41.236462+0000","last_became_peered":"2024-03-11T22:18:41.236462+0000","last_unstale":"2024-03-11T22:20:40.445160+0000","last_undegraded":"2024-03-11T22:20:40.445160+0000","last_fullsized":"2024-03-11T22:20:40.445160+0000","mapping_epoch":273,"log_start":"292'4400","ondisk_log_start":"292'4400","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448004096,"num_objects":6836,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8948,"num_write_kb":505088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":684,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:33.493484+0000","last_change":"2024-03-11T22:18:35.815130+0000","last_active":"2024-03-11T22:20:33.493484+0000","last_peered":"2024-03-11T22:20:33.493484+0000","last_clean":"2024-03-11T22:20:33.493484+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:20:33.493484+0000","last_undegraded":"2024-03-11T22:20:33.493484+0000","last_fullsized":"2024-03-11T22:20:33.493484+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:18:35.815031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:35.815031+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"273'20","reported_seq":580,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.290633+0000","last_change":"2024-03-11T22:19:31.503395+0000","last_active":"2024-03-11T22:20:32.290633+0000","last_peered":"2024-03-11T22:20:32.290633+0000","last_clean":"2024-03-11T22:20:32.290633+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:20:32.290633+0000","last_undegraded":"2024-03-11T22:20:32.290633+0000","last_fullsized":"2024-03-11T22:20:32.290633+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:19:31.502993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:31.502993+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:53.451821+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":546,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.294897+0000","last_change":"2024-03-11T22:19:34.706343+0000","last_active":"2024-03-11T22:20:32.294897+0000","last_peered":"2024-03-11T22:20:32.294897+0000","last_clean":"2024-03-11T22:20:32.294897+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:20:32.294897+0000","last_undegraded":"2024-03-11T22:20:32.294897+0000","last_fullsized":"2024-03-11T22:20:32.294897+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.706222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.706222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:49.315002+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":526,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.287261+0000","last_change":"2024-03-11T22:19:34.477472+0000","last_active":"2024-03-11T22:20:32.287261+0000","last_peered":"2024-03-11T22:20:32.287261+0000","last_clean":"2024-03-11T22:20:32.287261+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:20:32.287261+0000","last_undegraded":"2024-03-11T22:20:32.287261+0000","last_fullsized":"2024-03-11T22:20:32.287261+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.477349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.477349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:36.425362+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":771,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.290885+0000","last_change":"2024-03-11T22:20:32.290885+0000","last_active":"2024-03-11T22:20:32.290885+0000","last_peered":"2024-03-11T22:20:32.290885+0000","last_clean":"2024-03-11T22:20:32.290885+0000","last_became_active":"2024-03-11T22:20:32.290512+0000","last_became_peered":"2024-03-11T22:20:32.290512+0000","last_unstale":"2024-03-11T22:20:32.290885+0000","last_undegraded":"2024-03-11T22:20:32.290885+0000","last_fullsized":"2024-03-11T22:20:32.290885+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:20.906663+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.906663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"313'8787","reported_seq":11493,"reported_epoch":313,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T22:20:41.107999+0000","last_change":"2024-03-11T22:20:30.553695+0000","last_active":"2024-03-11T22:20:41.107999+0000","last_peered":"2024-03-11T22:20:41.107999+0000","last_clean":"2024-03-11T22:20:28.378192+0000","last_became_active":"2024-03-11T22:20:30.544603+0000","last_became_peered":"2024-03-11T22:20:30.544603+0000","last_unstale":"2024-03-11T22:20:41.107999+0000","last_undegraded":"2024-03-11T22:20:41.107999+0000","last_fullsized":"2024-03-11T22:20:41.107999+0000","mapping_epoch":310,"log_start":"292'4300","ondisk_log_start":"292'4300","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4487,"log_dups_size":0,"ondisk_log_size":4487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432996352,"num_objects":6607,"num_object_clones":0,"num_object_copies":19821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5368,"num_objects_unfound":0,"num_objects_dirty":6607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8787,"num_write_kb":492608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1305,"num_bytes_recovered":85524480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,4,0],"avail_no_missing":["1","0","4"],"object_location_counts":[{"shards":"0,1,4","objects":6607}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":546,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284265+0000","last_change":"2024-03-11T22:19:43.625744+0000","last_active":"2024-03-11T22:20:32.284265+0000","last_peered":"2024-03-11T22:20:32.284265+0000","last_clean":"2024-03-11T22:20:32.284265+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:20:32.284265+0000","last_undegraded":"2024-03-11T22:20:32.284265+0000","last_fullsized":"2024-03-11T22:20:32.284265+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:43.625634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:19:43.625634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:45.406841+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"273'29","reported_seq":585,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.287497+0000","last_change":"2024-03-11T22:19:38.450843+0000","last_active":"2024-03-11T22:20:32.287497+0000","last_peered":"2024-03-11T22:20:32.287497+0000","last_clean":"2024-03-11T22:20:32.287497+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:20:32.287497+0000","last_undegraded":"2024-03-11T22:20:32.287497+0000","last_fullsized":"2024-03-11T22:20:32.287497+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'29","last_scrub_stamp":"2024-03-11T22:19:38.450581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.450581+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:05.577079+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":546,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.285148+0000","last_change":"2024-03-11T22:18:38.153695+0000","last_active":"2024-03-11T22:20:32.285148+0000","last_peered":"2024-03-11T22:20:32.285148+0000","last_clean":"2024-03-11T22:20:32.285148+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:20:32.285148+0000","last_undegraded":"2024-03-11T22:20:32.285148+0000","last_fullsized":"2024-03-11T22:20:32.285148+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:38.153614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.153614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":534,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283975+0000","last_change":"2024-03-11T22:18:23.998360+0000","last_active":"2024-03-11T22:20:32.283975+0000","last_peered":"2024-03-11T22:20:32.283975+0000","last_clean":"2024-03-11T22:20:32.283975+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:20:32.283975+0000","last_undegraded":"2024-03-11T22:20:32.283975+0000","last_fullsized":"2024-03-11T22:20:32.283975+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:23.998251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:23.998251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":746,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.286800+0000","last_change":"2024-03-11T22:18:19.936959+0000","last_active":"2024-03-11T22:20:32.286800+0000","last_peered":"2024-03-11T22:20:32.286800+0000","last_clean":"2024-03-11T22:20:32.286800+0000","last_became_active":"2024-03-11T22:18:17.238247+0000","last_became_peered":"2024-03-11T22:18:17.238247+0000","last_unstale":"2024-03-11T22:20:32.286800+0000","last_undegraded":"2024-03-11T22:20:32.286800+0000","last_fullsized":"2024-03-11T22:20:32.286800+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:19.936857+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:19.936857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"313'8705","reported_seq":8795,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:41.751572+0000","last_change":"2024-03-11T22:18:41.783592+0000","last_active":"2024-03-11T22:20:41.751572+0000","last_peered":"2024-03-11T22:20:41.751572+0000","last_clean":"2024-03-11T22:20:41.751572+0000","last_became_active":"2024-03-11T22:18:41.783277+0000","last_became_peered":"2024-03-11T22:18:41.783277+0000","last_unstale":"2024-03-11T22:20:41.751572+0000","last_undegraded":"2024-03-11T22:20:41.751572+0000","last_fullsized":"2024-03-11T22:20:41.751572+0000","mapping_epoch":273,"log_start":"289'3500","ondisk_log_start":"289'3500","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5205,"log_dups_size":0,"ondisk_log_size":5205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431161344,"num_objects":6579,"num_object_clones":0,"num_object_copies":19737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8705,"num_write_kb":489088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":530,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.293743+0000","last_change":"2024-03-11T22:18:25.668662+0000","last_active":"2024-03-11T22:20:32.293743+0000","last_peered":"2024-03-11T22:20:32.293743+0000","last_clean":"2024-03-11T22:20:32.293743+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:20:32.293743+0000","last_undegraded":"2024-03-11T22:20:32.293743+0000","last_fullsized":"2024-03-11T22:20:32.293743+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:25.668581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:25.668581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"273'24","reported_seq":578,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.288742+0000","last_change":"2024-03-11T22:20:29.380624+0000","last_active":"2024-03-11T22:20:32.288742+0000","last_peered":"2024-03-11T22:20:32.288742+0000","last_clean":"2024-03-11T22:20:32.288742+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:20:32.288742+0000","last_undegraded":"2024-03-11T22:20:32.288742+0000","last_fullsized":"2024-03-11T22:20:32.288742+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'24","last_scrub_stamp":"2024-03-11T22:20:29.380527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:29.380527+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:49.652759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":1255,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:39.368801+0000","last_change":"2024-03-11T22:18:27.077438+0000","last_active":"2024-03-11T22:20:39.368801+0000","last_peered":"2024-03-11T22:20:39.368801+0000","last_clean":"2024-03-11T22:20:39.368801+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:20:39.368801+0000","last_undegraded":"2024-03-11T22:20:39.368801+0000","last_fullsized":"2024-03-11T22:20:39.368801+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:18:27.077275+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.077275+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":693,"num_read_kb":688,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":538,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.288450+0000","last_change":"2024-03-11T22:18:28.722689+0000","last_active":"2024-03-11T22:20:32.288450+0000","last_peered":"2024-03-11T22:20:32.288450+0000","last_clean":"2024-03-11T22:20:32.288450+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:20:32.288450+0000","last_undegraded":"2024-03-11T22:20:32.288450+0000","last_fullsized":"2024-03-11T22:20:32.288450+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:28.722615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:28.722615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":542,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284953+0000","last_change":"2024-03-11T22:18:36.127513+0000","last_active":"2024-03-11T22:20:32.284953+0000","last_peered":"2024-03-11T22:20:32.284953+0000","last_clean":"2024-03-11T22:20:32.284953+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:20:32.284953+0000","last_undegraded":"2024-03-11T22:20:32.284953+0000","last_fullsized":"2024-03-11T22:20:32.284953+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:36.127435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:36.127435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":844,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284959+0000","last_change":"2024-03-11T22:19:29.435468+0000","last_active":"2024-03-11T22:20:32.284959+0000","last_peered":"2024-03-11T22:20:32.284959+0000","last_clean":"2024-03-11T22:20:32.284959+0000","last_became_active":"2024-03-11T22:14:38.164101+0000","last_became_peered":"2024-03-11T22:14:38.164101+0000","last_unstale":"2024-03-11T22:20:32.284959+0000","last_undegraded":"2024-03-11T22:20:32.284959+0000","last_fullsized":"2024-03-11T22:20:32.284959+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:29.435334+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.435334+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:50.439118+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"313'8899","reported_seq":8989,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:41.099227+0000","last_change":"2024-03-11T22:18:41.240089+0000","last_active":"2024-03-11T22:20:41.099227+0000","last_peered":"2024-03-11T22:20:41.099227+0000","last_clean":"2024-03-11T22:20:41.099227+0000","last_became_active":"2024-03-11T22:18:41.239829+0000","last_became_peered":"2024-03-11T22:18:41.239829+0000","last_unstale":"2024-03-11T22:20:41.099227+0000","last_undegraded":"2024-03-11T22:20:41.099227+0000","last_fullsized":"2024-03-11T22:20:41.099227+0000","mapping_epoch":273,"log_start":"294'4800","ondisk_log_start":"294'4800","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438108160,"num_objects":6685,"num_object_clones":0,"num_object_copies":20055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8899,"num_write_kb":498688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":504,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.280220+0000","last_change":"2024-03-11T22:20:29.420829+0000","last_active":"2024-03-11T22:20:32.280220+0000","last_peered":"2024-03-11T22:20:32.280220+0000","last_clean":"2024-03-11T22:20:32.280220+0000","last_became_active":"2024-03-11T22:20:29.420497+0000","last_became_peered":"2024-03-11T22:20:29.420497+0000","last_unstale":"2024-03-11T22:20:32.280220+0000","last_undegraded":"2024-03-11T22:20:32.280220+0000","last_fullsized":"2024-03-11T22:20:32.280220+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:24.875284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:19:24.875284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:44.177047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"273'28","reported_seq":736,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:33.494426+0000","last_change":"2024-03-11T22:19:34.671571+0000","last_active":"2024-03-11T22:20:33.494426+0000","last_peered":"2024-03-11T22:20:33.494426+0000","last_clean":"2024-03-11T22:20:33.494426+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:20:33.494426+0000","last_undegraded":"2024-03-11T22:20:33.494426+0000","last_fullsized":"2024-03-11T22:20:33.494426+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:19:34.671335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.671335+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:45.435800+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":538,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283030+0000","last_change":"2024-03-11T22:19:24.533661+0000","last_active":"2024-03-11T22:20:32.283030+0000","last_peered":"2024-03-11T22:20:32.283030+0000","last_clean":"2024-03-11T22:20:32.283030+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:20:32.283030+0000","last_undegraded":"2024-03-11T22:20:32.283030+0000","last_fullsized":"2024-03-11T22:20:32.283030+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:24.533532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:19:24.533532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:46.126118+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":854,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.295320+0000","last_change":"2024-03-11T22:20:32.294325+0000","last_active":"2024-03-11T22:20:32.295320+0000","last_peered":"2024-03-11T22:20:32.295320+0000","last_clean":"2024-03-11T22:20:32.295320+0000","last_became_active":"2024-03-11T22:20:32.293740+0000","last_became_peered":"2024-03-11T22:20:32.293740+0000","last_unstale":"2024-03-11T22:20:32.295320+0000","last_undegraded":"2024-03-11T22:20:32.295320+0000","last_fullsized":"2024-03-11T22:20:32.295320+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:44.952136+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:18:44.952136+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":526,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.293379+0000","last_change":"2024-03-11T22:18:22.688031+0000","last_active":"2024-03-11T22:20:32.293379+0000","last_peered":"2024-03-11T22:20:32.293379+0000","last_clean":"2024-03-11T22:20:32.293379+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:20:32.293379+0000","last_undegraded":"2024-03-11T22:20:32.293379+0000","last_fullsized":"2024-03-11T22:20:32.293379+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:22.687922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.687922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"313'8879","reported_seq":8967,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:40.429366+0000","last_change":"2024-03-11T22:18:41.260551+0000","last_active":"2024-03-11T22:20:40.429366+0000","last_peered":"2024-03-11T22:20:40.429366+0000","last_clean":"2024-03-11T22:20:40.429366+0000","last_became_active":"2024-03-11T22:18:41.260185+0000","last_became_peered":"2024-03-11T22:18:41.260185+0000","last_unstale":"2024-03-11T22:20:40.429366+0000","last_undegraded":"2024-03-11T22:20:40.429366+0000","last_fullsized":"2024-03-11T22:20:40.429366+0000","mapping_epoch":273,"log_start":"292'4400","ondisk_log_start":"292'4400","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4479,"log_dups_size":0,"ondisk_log_size":4479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448724992,"num_objects":6847,"num_object_clones":0,"num_object_copies":20541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8879,"num_write_kb":503232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":554,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.280493+0000","last_change":"2024-03-11T22:19:37.032204+0000","last_active":"2024-03-11T22:20:32.280493+0000","last_peered":"2024-03-11T22:20:32.280493+0000","last_clean":"2024-03-11T22:20:32.280493+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:20:32.280493+0000","last_undegraded":"2024-03-11T22:20:32.280493+0000","last_fullsized":"2024-03-11T22:20:32.280493+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.032058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.032058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:54.420326+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"273'17","reported_seq":583,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.288847+0000","last_change":"2024-03-11T22:19:34.486422+0000","last_active":"2024-03-11T22:20:32.288847+0000","last_peered":"2024-03-11T22:20:32.288847+0000","last_clean":"2024-03-11T22:20:32.288847+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:20:32.288847+0000","last_undegraded":"2024-03-11T22:20:32.288847+0000","last_fullsized":"2024-03-11T22:20:32.288847+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'17","last_scrub_stamp":"2024-03-11T22:19:34.486239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.486239+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:49.516370+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":571,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.279829+0000","last_change":"2024-03-11T22:18:31.559422+0000","last_active":"2024-03-11T22:20:32.279829+0000","last_peered":"2024-03-11T22:20:32.279829+0000","last_clean":"2024-03-11T22:20:32.279829+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:20:32.279829+0000","last_undegraded":"2024-03-11T22:20:32.279829+0000","last_fullsized":"2024-03-11T22:20:32.279829+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:31.559330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:18:31.559330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":555,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284630+0000","last_change":"2024-03-11T22:19:46.587019+0000","last_active":"2024-03-11T22:20:32.284630+0000","last_peered":"2024-03-11T22:20:32.284630+0000","last_clean":"2024-03-11T22:20:32.284630+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:20:32.284630+0000","last_undegraded":"2024-03-11T22:20:32.284630+0000","last_fullsized":"2024-03-11T22:20:32.284630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:19:46.586446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:48.778337+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":484,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284889+0000","last_change":"2024-03-11T22:19:37.776868+0000","last_active":"2024-03-11T22:20:32.284889+0000","last_peered":"2024-03-11T22:20:32.284889+0000","last_clean":"2024-03-11T22:20:32.284889+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:20:32.284889+0000","last_undegraded":"2024-03-11T22:20:32.284889+0000","last_fullsized":"2024-03-11T22:20:32.284889+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.776600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.776600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:58.256015+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":861,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.295561+0000","last_change":"2024-03-11T22:20:32.295561+0000","last_active":"2024-03-11T22:20:32.295561+0000","last_peered":"2024-03-11T22:20:32.295561+0000","last_clean":"2024-03-11T22:20:32.295561+0000","last_became_active":"2024-03-11T22:20:32.295175+0000","last_became_peered":"2024-03-11T22:20:32.295175+0000","last_unstale":"2024-03-11T22:20:32.295561+0000","last_undegraded":"2024-03-11T22:20:32.295561+0000","last_fullsized":"2024-03-11T22:20:32.295561+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:25.392244+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:25.392244+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:48.078541+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"313'8880","reported_seq":8965,"reported_epoch":313,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-11T22:20:41.773490+0000","last_change":"2024-03-11T22:20:30.589165+0000","last_active":"2024-03-11T22:20:41.773490+0000","last_peered":"2024-03-11T22:20:41.773490+0000","last_clean":"2024-03-11T22:20:28.362243+0000","last_became_active":"2024-03-11T22:20:30.567189+0000","last_became_peered":"2024-03-11T22:20:30.567189+0000","last_unstale":"2024-03-11T22:20:41.773490+0000","last_undegraded":"2024-03-11T22:20:41.773490+0000","last_fullsized":"2024-03-11T22:20:41.773490+0000","mapping_epoch":310,"log_start":"290'3700","ondisk_log_start":"290'3700","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432668672,"num_objects":6602,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6602,"num_objects_unfound":0,"num_objects_dirty":6602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8880,"num_write_kb":495424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,5,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":6602}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":530,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.289805+0000","last_change":"2024-03-11T22:15:30.922162+0000","last_active":"2024-03-11T22:20:32.289805+0000","last_peered":"2024-03-11T22:20:32.289805+0000","last_clean":"2024-03-11T22:20:32.289805+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:20:32.289805+0000","last_undegraded":"2024-03-11T22:20:32.289805+0000","last_fullsized":"2024-03-11T22:20:32.289805+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:30.922001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:30.922001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"273'40","reported_seq":762,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:33.494571+0000","last_change":"2024-03-11T22:19:39.417690+0000","last_active":"2024-03-11T22:20:33.494571+0000","last_peered":"2024-03-11T22:20:33.494571+0000","last_clean":"2024-03-11T22:20:33.494571+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:20:33.494571+0000","last_undegraded":"2024-03-11T22:20:33.494571+0000","last_fullsized":"2024-03-11T22:20:33.494571+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'40","last_scrub_stamp":"2024-03-11T22:19:39.417411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:39.417411+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:44.098715+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":297,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.287411+0000","last_change":"2024-03-11T22:15:24.899285+0000","last_active":"2024-03-11T22:20:32.287411+0000","last_peered":"2024-03-11T22:20:32.287411+0000","last_clean":"2024-03-11T22:20:32.287411+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:20:32.287411+0000","last_undegraded":"2024-03-11T22:20:32.287411+0000","last_fullsized":"2024-03-11T22:20:32.287411+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:24.899168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:24.899168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":550,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.289836+0000","last_change":"2024-03-11T22:18:36.684061+0000","last_active":"2024-03-11T22:20:32.289836+0000","last_peered":"2024-03-11T22:20:32.289836+0000","last_clean":"2024-03-11T22:20:32.289836+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:20:32.289836+0000","last_undegraded":"2024-03-11T22:20:32.289836+0000","last_fullsized":"2024-03-11T22:20:32.289836+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:18:36.683930+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:18:36.683930+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":530,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284104+0000","last_change":"2024-03-11T22:20:29.434967+0000","last_active":"2024-03-11T22:20:32.284104+0000","last_peered":"2024-03-11T22:20:32.284104+0000","last_clean":"2024-03-11T22:20:32.284104+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:20:32.284104+0000","last_undegraded":"2024-03-11T22:20:32.284104+0000","last_fullsized":"2024-03-11T22:20:32.284104+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:32.518054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:15:32.518054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":835,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.290193+0000","last_change":"2024-03-11T22:18:41.790977+0000","last_active":"2024-03-11T22:20:32.290193+0000","last_peered":"2024-03-11T22:20:32.290193+0000","last_clean":"2024-03-11T22:20:32.290193+0000","last_became_active":"2024-03-11T22:14:38.164859+0000","last_became_peered":"2024-03-11T22:14:38.164859+0000","last_unstale":"2024-03-11T22:20:32.290193+0000","last_undegraded":"2024-03-11T22:20:32.290193+0000","last_fullsized":"2024-03-11T22:20:32.290193+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:41.790896+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:41.790896+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"313'8532","reported_seq":8622,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:40.434017+0000","last_change":"2024-03-11T22:18:41.233217+0000","last_active":"2024-03-11T22:20:40.434017+0000","last_peered":"2024-03-11T22:20:40.434017+0000","last_clean":"2024-03-11T22:20:40.434017+0000","last_became_active":"2024-03-11T22:18:41.232989+0000","last_became_peered":"2024-03-11T22:18:41.232989+0000","last_unstale":"2024-03-11T22:20:40.434017+0000","last_undegraded":"2024-03-11T22:20:40.434017+0000","last_fullsized":"2024-03-11T22:20:40.434017+0000","mapping_epoch":273,"log_start":"290'4000","ondisk_log_start":"290'4000","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422838272,"num_objects":6452,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8532,"num_write_kb":479488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"273'15","reported_seq":575,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.287720+0000","last_change":"2024-03-11T22:19:32.486126+0000","last_active":"2024-03-11T22:20:32.287720+0000","last_peered":"2024-03-11T22:20:32.287720+0000","last_clean":"2024-03-11T22:20:32.287720+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:20:32.287720+0000","last_undegraded":"2024-03-11T22:20:32.287720+0000","last_fullsized":"2024-03-11T22:20:32.287720+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:19:32.485571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.485571+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:57.322796+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":538,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.293222+0000","last_change":"2024-03-11T22:18:40.745227+0000","last_active":"2024-03-11T22:20:32.293222+0000","last_peered":"2024-03-11T22:20:32.293222+0000","last_clean":"2024-03-11T22:20:32.293222+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:20:32.293222+0000","last_undegraded":"2024-03-11T22:20:32.293222+0000","last_fullsized":"2024-03-11T22:20:32.293222+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.745151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.745151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":554,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.282006+0000","last_change":"2024-03-11T22:19:42.927329+0000","last_active":"2024-03-11T22:20:32.282006+0000","last_peered":"2024-03-11T22:20:32.282006+0000","last_clean":"2024-03-11T22:20:32.282006+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:20:32.282006+0000","last_undegraded":"2024-03-11T22:20:32.282006+0000","last_fullsized":"2024-03-11T22:20:32.282006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:42.926178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:19:42.926178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:49.199630+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":538,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.288660+0000","last_change":"2024-03-11T22:20:29.418105+0000","last_active":"2024-03-11T22:20:32.288660+0000","last_peered":"2024-03-11T22:20:32.288660+0000","last_clean":"2024-03-11T22:20:32.288660+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:20:32.288660+0000","last_undegraded":"2024-03-11T22:20:32.288660+0000","last_fullsized":"2024-03-11T22:20:32.288660+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:27.721245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.721245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"93'130","reported_seq":688,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.293722+0000","last_change":"2024-03-11T22:20:32.293722+0000","last_active":"2024-03-11T22:20:32.293722+0000","last_peered":"2024-03-11T22:20:32.293722+0000","last_clean":"2024-03-11T22:20:32.293722+0000","last_became_active":"2024-03-11T22:20:32.292676+0000","last_became_peered":"2024-03-11T22:20:32.292676+0000","last_unstale":"2024-03-11T22:20:32.293722+0000","last_undegraded":"2024-03-11T22:20:32.293722+0000","last_fullsized":"2024-03-11T22:20:32.293722+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:22.939363+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.939363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.7","version":"313'8827","reported_seq":14281,"reported_epoch":313,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T22:20:40.456988+0000","last_change":"2024-03-11T22:20:30.536596+0000","last_active":"2024-03-11T22:20:40.456988+0000","last_peered":"2024-03-11T22:20:40.456988+0000","last_clean":"2024-03-11T22:20:28.395206+0000","last_became_active":"2024-03-11T22:20:30.524339+0000","last_became_peered":"2024-03-11T22:20:30.524339+0000","last_unstale":"2024-03-11T22:20:40.456988+0000","last_undegraded":"2024-03-11T22:20:40.456988+0000","last_fullsized":"2024-03-11T22:20:40.456988+0000","mapping_epoch":310,"log_start":"292'4300","ondisk_log_start":"292'4300","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444399616,"num_objects":6781,"num_object_clones":0,"num_object_copies":20343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4227,"num_objects_unfound":0,"num_objects_dirty":6781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2679,"num_bytes_recovered":175570944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,3,2],"avail_no_missing":["5","2","3"],"object_location_counts":[{"shards":"2,3,5","objects":6781}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"273'35","reported_seq":657,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284152+0000","last_change":"2024-03-11T22:20:29.437134+0000","last_active":"2024-03-11T22:20:32.284152+0000","last_peered":"2024-03-11T22:20:32.284152+0000","last_clean":"2024-03-11T22:20:32.284152+0000","last_became_active":"2024-03-11T22:20:29.418041+0000","last_became_peered":"2024-03-11T22:20:29.418041+0000","last_unstale":"2024-03-11T22:20:32.284152+0000","last_undegraded":"2024-03-11T22:20:32.284152+0000","last_fullsized":"2024-03-11T22:20:32.284152+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'35","last_scrub_stamp":"2024-03-11T22:19:46.640720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:46.640720+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:52.029441+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":489,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.294678+0000","last_change":"2024-03-11T22:20:29.414029+0000","last_active":"2024-03-11T22:20:32.294678+0000","last_peered":"2024-03-11T22:20:32.294678+0000","last_clean":"2024-03-11T22:20:32.294678+0000","last_became_active":"2024-03-11T22:20:29.413664+0000","last_became_peered":"2024-03-11T22:20:29.413664+0000","last_unstale":"2024-03-11T22:20:32.294678+0000","last_undegraded":"2024-03-11T22:20:32.294678+0000","last_fullsized":"2024-03-11T22:20:32.294678+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:32.682157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:32.682157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":554,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:39.805734+0000","last_change":"2024-03-11T22:20:39.805771+0000","last_active":"2024-03-11T22:20:39.805734+0000","last_peered":"2024-03-11T22:20:39.805734+0000","last_clean":"2024-03-11T22:20:39.805734+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:20:39.805734+0000","last_undegraded":"2024-03-11T22:20:39.805734+0000","last_fullsized":"2024-03-11T22:20:39.805734+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:39.805647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:20:39.805647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:55.656366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":504,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.280036+0000","last_change":"2024-03-11T22:20:29.416481+0000","last_active":"2024-03-11T22:20:32.280036+0000","last_peered":"2024-03-11T22:20:32.280036+0000","last_clean":"2024-03-11T22:20:32.280036+0000","last_became_active":"2024-03-11T22:20:29.415565+0000","last_became_peered":"2024-03-11T22:20:29.415565+0000","last_unstale":"2024-03-11T22:20:32.280036+0000","last_undegraded":"2024-03-11T22:20:32.280036+0000","last_fullsized":"2024-03-11T22:20:32.280036+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:44.717740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:20.931122+0000","last_clean_scrub_stamp":"2024-03-11T22:18:44.717740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"93'130","reported_seq":739,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:36.022115+0000","last_change":"2024-03-11T22:20:36.022151+0000","last_active":"2024-03-11T22:20:36.022115+0000","last_peered":"2024-03-11T22:20:36.022115+0000","last_clean":"2024-03-11T22:20:36.022115+0000","last_became_active":"2024-03-11T22:14:43.195433+0000","last_became_peered":"2024-03-11T22:14:43.195433+0000","last_unstale":"2024-03-11T22:20:36.022115+0000","last_undegraded":"2024-03-11T22:20:36.022115+0000","last_fullsized":"2024-03-11T22:20:36.022115+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:36.022039+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:20:36.022039+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:45.766952+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"313'8810","reported_seq":8900,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:41.104086+0000","last_change":"2024-03-11T22:18:41.237962+0000","last_active":"2024-03-11T22:20:41.104086+0000","last_peered":"2024-03-11T22:20:41.104086+0000","last_clean":"2024-03-11T22:20:41.104086+0000","last_became_active":"2024-03-11T22:18:41.237565+0000","last_became_peered":"2024-03-11T22:18:41.237565+0000","last_unstale":"2024-03-11T22:20:41.104086+0000","last_undegraded":"2024-03-11T22:20:41.104086+0000","last_fullsized":"2024-03-11T22:20:41.104086+0000","mapping_epoch":273,"log_start":"292'4300","ondisk_log_start":"292'4300","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4510,"log_dups_size":0,"ondisk_log_size":4510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433586176,"num_objects":6616,"num_object_clones":0,"num_object_copies":19848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8810,"num_write_kb":493632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":546,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.279594+0000","last_change":"2024-03-11T22:18:38.699012+0000","last_active":"2024-03-11T22:20:32.279594+0000","last_peered":"2024-03-11T22:20:32.279594+0000","last_clean":"2024-03-11T22:20:32.279594+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:20:32.279594+0000","last_undegraded":"2024-03-11T22:20:32.279594+0000","last_fullsized":"2024-03-11T22:20:32.279594+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:38.698891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.698891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"273'18","reported_seq":362,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.290650+0000","last_change":"2024-03-11T22:19:38.795869+0000","last_active":"2024-03-11T22:20:32.290650+0000","last_peered":"2024-03-11T22:20:32.290650+0000","last_clean":"2024-03-11T22:20:32.290650+0000","last_became_active":"2024-03-11T22:14:52.247400+0000","last_became_peered":"2024-03-11T22:14:52.247400+0000","last_unstale":"2024-03-11T22:20:32.290650+0000","last_undegraded":"2024-03-11T22:20:32.290650+0000","last_fullsized":"2024-03-11T22:20:32.290650+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:19:38.795650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.795650+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:52.893327+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":328,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.291106+0000","last_change":"2024-03-11T22:20:29.417391+0000","last_active":"2024-03-11T22:20:32.291106+0000","last_peered":"2024-03-11T22:20:32.291106+0000","last_clean":"2024-03-11T22:20:32.291106+0000","last_became_active":"2024-03-11T22:20:29.416729+0000","last_became_peered":"2024-03-11T22:20:29.416729+0000","last_unstale":"2024-03-11T22:20:32.291106+0000","last_undegraded":"2024-03-11T22:20:32.291106+0000","last_fullsized":"2024-03-11T22:20:32.291106+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.439756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.439756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:56.331649+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":540,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284733+0000","last_change":"2024-03-11T22:20:29.470787+0000","last_active":"2024-03-11T22:20:32.284733+0000","last_peered":"2024-03-11T22:20:32.284733+0000","last_clean":"2024-03-11T22:20:32.284733+0000","last_became_active":"2024-03-11T22:20:29.438413+0000","last_became_peered":"2024-03-11T22:20:29.438413+0000","last_unstale":"2024-03-11T22:20:32.284733+0000","last_undegraded":"2024-03-11T22:20:32.284733+0000","last_fullsized":"2024-03-11T22:20:32.284733+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:15:40.678629+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:15:40.678629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"93'130","reported_seq":864,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284397+0000","last_change":"2024-03-11T22:18:21.956432+0000","last_active":"2024-03-11T22:20:32.284397+0000","last_peered":"2024-03-11T22:20:32.284397+0000","last_clean":"2024-03-11T22:20:32.284397+0000","last_became_active":"2024-03-11T22:17:57.246241+0000","last_became_peered":"2024-03-11T22:17:57.246241+0000","last_unstale":"2024-03-11T22:20:32.284397+0000","last_undegraded":"2024-03-11T22:20:32.284397+0000","last_fullsized":"2024-03-11T22:20:32.284397+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:21.956334+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.956334+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"313'9089","reported_seq":9184,"reported_epoch":313,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-11T22:20:41.740904+0000","last_change":"2024-03-11T22:20:30.593378+0000","last_active":"2024-03-11T22:20:41.740904+0000","last_peered":"2024-03-11T22:20:41.740904+0000","last_clean":"2024-03-11T22:20:28.376799+0000","last_became_active":"2024-03-11T22:20:30.582612+0000","last_became_peered":"2024-03-11T22:20:30.582612+0000","last_unstale":"2024-03-11T22:20:41.740904+0000","last_undegraded":"2024-03-11T22:20:41.740904+0000","last_fullsized":"2024-03-11T22:20:41.740904+0000","mapping_epoch":310,"log_start":"290'3900","ondisk_log_start":"290'3900","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5189,"log_dups_size":0,"ondisk_log_size":5189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445054976,"num_objects":6791,"num_object_clones":0,"num_object_copies":20373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6791,"num_objects_unfound":0,"num_objects_dirty":6791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":508160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":6791}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"310'11","reported_seq":652,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:33.494691+0000","last_change":"2024-03-11T22:20:29.432306+0000","last_active":"2024-03-11T22:20:33.494691+0000","last_peered":"2024-03-11T22:20:33.494691+0000","last_clean":"2024-03-11T22:20:33.494691+0000","last_became_active":"2024-03-11T22:20:29.418355+0000","last_became_peered":"2024-03-11T22:20:29.418355+0000","last_unstale":"2024-03-11T22:20:33.494691+0000","last_undegraded":"2024-03-11T22:20:33.494691+0000","last_fullsized":"2024-03-11T22:20:33.494691+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'10","last_scrub_stamp":"2024-03-11T22:18:52.719695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:52.719695+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"273'34","reported_seq":636,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.289389+0000","last_change":"2024-03-11T22:20:29.431072+0000","last_active":"2024-03-11T22:20:32.289389+0000","last_peered":"2024-03-11T22:20:32.289389+0000","last_clean":"2024-03-11T22:20:32.289389+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:20:32.289389+0000","last_undegraded":"2024-03-11T22:20:32.289389+0000","last_fullsized":"2024-03-11T22:20:32.289389+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:19:46.888741+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:19:46.888741+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:54.523213+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":338,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.286246+0000","last_change":"2024-03-11T22:20:29.440395+0000","last_active":"2024-03-11T22:20:32.286246+0000","last_peered":"2024-03-11T22:20:32.286246+0000","last_clean":"2024-03-11T22:20:32.286246+0000","last_became_active":"2024-03-11T22:20:29.415226+0000","last_became_peered":"2024-03-11T22:20:29.415226+0000","last_unstale":"2024-03-11T22:20:32.286246+0000","last_undegraded":"2024-03-11T22:20:32.286246+0000","last_fullsized":"2024-03-11T22:20:32.286246+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:20:26.335219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:26.335219+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:47.255101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":526,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.290100+0000","last_change":"2024-03-11T22:19:33.760079+0000","last_active":"2024-03-11T22:20:32.290100+0000","last_peered":"2024-03-11T22:20:32.290100+0000","last_clean":"2024-03-11T22:20:32.290100+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:20:32.290100+0000","last_undegraded":"2024-03-11T22:20:32.290100+0000","last_fullsized":"2024-03-11T22:20:32.290100+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:33.759955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:33.759955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:50.321376+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"93'130","reported_seq":881,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.300771+0000","last_change":"2024-03-11T22:20:32.300771+0000","last_active":"2024-03-11T22:20:32.300771+0000","last_peered":"2024-03-11T22:20:32.300771+0000","last_clean":"2024-03-11T22:20:32.300771+0000","last_became_active":"2024-03-11T22:20:32.295606+0000","last_became_peered":"2024-03-11T22:20:32.295606+0000","last_unstale":"2024-03-11T22:20:32.300771+0000","last_undegraded":"2024-03-11T22:20:32.300771+0000","last_fullsized":"2024-03-11T22:20:32.300771+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:41.158328+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:41.158328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"273'12","reported_seq":571,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.286983+0000","last_change":"2024-03-11T22:20:29.483060+0000","last_active":"2024-03-11T22:20:32.286983+0000","last_peered":"2024-03-11T22:20:32.286983+0000","last_clean":"2024-03-11T22:20:32.286983+0000","last_became_active":"2024-03-11T22:20:29.441599+0000","last_became_peered":"2024-03-11T22:20:29.441599+0000","last_unstale":"2024-03-11T22:20:32.286983+0000","last_undegraded":"2024-03-11T22:20:32.286983+0000","last_fullsized":"2024-03-11T22:20:32.286983+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:15:27.448002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:27.448002+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":546,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283611+0000","last_change":"2024-03-11T22:17:33.927480+0000","last_active":"2024-03-11T22:20:32.283611+0000","last_peered":"2024-03-11T22:20:32.283611+0000","last_clean":"2024-03-11T22:20:32.283611+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:20:32.283611+0000","last_undegraded":"2024-03-11T22:20:32.283611+0000","last_fullsized":"2024-03-11T22:20:32.283611+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:33.927367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:17:33.927367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":530,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.293051+0000","last_change":"2024-03-11T22:15:41.760176+0000","last_active":"2024-03-11T22:20:32.293051+0000","last_peered":"2024-03-11T22:20:32.293051+0000","last_clean":"2024-03-11T22:20:32.293051+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:20:32.293051+0000","last_undegraded":"2024-03-11T22:20:32.293051+0000","last_fullsized":"2024-03-11T22:20:32.293051+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:41.760044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:07.195289+0000","last_clean_scrub_stamp":"2024-03-11T22:15:41.760044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":484,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.993405+0000","last_change":"2024-03-11T22:20:32.993441+0000","last_active":"2024-03-11T22:20:32.993405+0000","last_peered":"2024-03-11T22:20:32.993405+0000","last_clean":"2024-03-11T22:20:32.993405+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:20:32.993405+0000","last_undegraded":"2024-03-11T22:20:32.993405+0000","last_fullsized":"2024-03-11T22:20:32.993405+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:32.993301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:20:32.993301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:01.295107+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"93'130","reported_seq":844,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.289504+0000","last_change":"2024-03-11T22:18:45.781426+0000","last_active":"2024-03-11T22:20:32.289504+0000","last_peered":"2024-03-11T22:20:32.289504+0000","last_clean":"2024-03-11T22:20:32.289504+0000","last_became_active":"2024-03-11T22:14:43.194413+0000","last_became_peered":"2024-03-11T22:14:43.194413+0000","last_unstale":"2024-03-11T22:20:32.289504+0000","last_undegraded":"2024-03-11T22:20:32.289504+0000","last_fullsized":"2024-03-11T22:20:32.289504+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:45.781296+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:45.781296+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"93'130","reported_seq":851,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284541+0000","last_change":"2024-03-11T22:19:27.413677+0000","last_active":"2024-03-11T22:20:32.284541+0000","last_peered":"2024-03-11T22:20:32.284541+0000","last_clean":"2024-03-11T22:20:32.284541+0000","last_became_active":"2024-03-11T22:14:43.198216+0000","last_became_peered":"2024-03-11T22:14:43.198216+0000","last_unstale":"2024-03-11T22:20:32.284541+0000","last_undegraded":"2024-03-11T22:20:32.284541+0000","last_fullsized":"2024-03-11T22:20:32.284541+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:27.413560+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:27.413560+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:42.992752+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"273'19","reported_seq":595,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.289094+0000","last_change":"2024-03-11T22:20:29.451236+0000","last_active":"2024-03-11T22:20:32.289094+0000","last_peered":"2024-03-11T22:20:32.289094+0000","last_clean":"2024-03-11T22:20:32.289094+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:20:32.289094+0000","last_undegraded":"2024-03-11T22:20:32.289094+0000","last_fullsized":"2024-03-11T22:20:32.289094+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:18:22.657089+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.657089+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":308,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284565+0000","last_change":"2024-03-11T22:18:40.801794+0000","last_active":"2024-03-11T22:20:32.284565+0000","last_peered":"2024-03-11T22:20:32.284565+0000","last_clean":"2024-03-11T22:20:32.284565+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:20:32.284565+0000","last_undegraded":"2024-03-11T22:20:32.284565+0000","last_fullsized":"2024-03-11T22:20:32.284565+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.801713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.801713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":541,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.286441+0000","last_change":"2024-03-11T22:20:29.433304+0000","last_active":"2024-03-11T22:20:32.286441+0000","last_peered":"2024-03-11T22:20:32.286441+0000","last_clean":"2024-03-11T22:20:32.286441+0000","last_became_active":"2024-03-11T22:20:29.433005+0000","last_became_peered":"2024-03-11T22:20:29.433005+0000","last_unstale":"2024-03-11T22:20:32.286441+0000","last_undegraded":"2024-03-11T22:20:32.286441+0000","last_fullsized":"2024-03-11T22:20:32.286441+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:43.544959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:43.544959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:58.636623+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":550,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.280120+0000","last_change":"2024-03-11T22:19:33.966233+0000","last_active":"2024-03-11T22:20:32.280120+0000","last_peered":"2024-03-11T22:20:32.280120+0000","last_clean":"2024-03-11T22:20:32.280120+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:20:32.280120+0000","last_undegraded":"2024-03-11T22:20:32.280120+0000","last_fullsized":"2024-03-11T22:20:32.280120+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:33.966085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:19:33.966085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:35.247434+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"273'19","reported_seq":571,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.294814+0000","last_change":"2024-03-11T22:19:29.865779+0000","last_active":"2024-03-11T22:20:32.294814+0000","last_peered":"2024-03-11T22:20:32.294814+0000","last_clean":"2024-03-11T22:20:32.294814+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:20:32.294814+0000","last_undegraded":"2024-03-11T22:20:32.294814+0000","last_fullsized":"2024-03-11T22:20:32.294814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:19:29.865528+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.865528+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:43.762836+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":550,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.287110+0000","last_change":"2024-03-11T22:20:29.417404+0000","last_active":"2024-03-11T22:20:32.287110+0000","last_peered":"2024-03-11T22:20:32.287110+0000","last_clean":"2024-03-11T22:20:32.287110+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:20:32.287110+0000","last_undegraded":"2024-03-11T22:20:32.287110+0000","last_fullsized":"2024-03-11T22:20:32.287110+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:21.693554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.693554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'11","reported_seq":719,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:33.494411+0000","last_change":"2024-03-11T22:20:29.445996+0000","last_active":"2024-03-11T22:20:33.494411+0000","last_peered":"2024-03-11T22:20:33.494411+0000","last_clean":"2024-03-11T22:20:33.494411+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:20:33.494411+0000","last_undegraded":"2024-03-11T22:20:33.494411+0000","last_fullsized":"2024-03-11T22:20:33.494411+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'10","last_scrub_stamp":"2024-03-11T22:19:32.461683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.461683+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:01.555630+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":312,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.280642+0000","last_change":"2024-03-11T22:18:35.713146+0000","last_active":"2024-03-11T22:20:32.280642+0000","last_peered":"2024-03-11T22:20:32.280642+0000","last_clean":"2024-03-11T22:20:32.280642+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:20:32.280642+0000","last_undegraded":"2024-03-11T22:20:32.280642+0000","last_fullsized":"2024-03-11T22:20:32.280642+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:35.713009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:35.713009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"93'130","reported_seq":837,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.289723+0000","last_change":"2024-03-11T22:18:53.739248+0000","last_active":"2024-03-11T22:20:32.289723+0000","last_peered":"2024-03-11T22:20:32.289723+0000","last_clean":"2024-03-11T22:20:32.289723+0000","last_became_active":"2024-03-11T22:14:43.194350+0000","last_became_peered":"2024-03-11T22:14:43.194350+0000","last_unstale":"2024-03-11T22:20:32.289723+0000","last_undegraded":"2024-03-11T22:20:32.289723+0000","last_fullsized":"2024-03-11T22:20:32.289723+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:53.738316+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:18:53.738316+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"273'20","reported_seq":574,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.286816+0000","last_change":"2024-03-11T22:16:16.055622+0000","last_active":"2024-03-11T22:20:32.286816+0000","last_peered":"2024-03-11T22:20:32.286816+0000","last_clean":"2024-03-11T22:20:32.286816+0000","last_became_active":"2024-03-11T22:16:15.815814+0000","last_became_peered":"2024-03-11T22:16:15.815814+0000","last_unstale":"2024-03-11T22:20:32.286816+0000","last_undegraded":"2024-03-11T22:20:32.286816+0000","last_fullsized":"2024-03-11T22:20:32.286816+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:15:29.460297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.460297+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":336,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.279505+0000","last_change":"2024-03-11T22:20:31.797304+0000","last_active":"2024-03-11T22:20:32.279505+0000","last_peered":"2024-03-11T22:20:32.279505+0000","last_clean":"2024-03-11T22:20:32.279505+0000","last_became_active":"2024-03-11T22:20:29.415019+0000","last_became_peered":"2024-03-11T22:20:29.415019+0000","last_unstale":"2024-03-11T22:20:32.279505+0000","last_undegraded":"2024-03-11T22:20:32.279505+0000","last_fullsized":"2024-03-11T22:20:32.279505+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:31.796712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:31.796712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:47.886151+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":692,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:38.494970+0000","last_change":"2024-03-11T22:19:42.854028+0000","last_active":"2024-03-11T22:20:38.494970+0000","last_peered":"2024-03-11T22:20:38.494970+0000","last_clean":"2024-03-11T22:20:38.494970+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:20:38.494970+0000","last_undegraded":"2024-03-11T22:20:38.494970+0000","last_fullsized":"2024-03-11T22:20:38.494970+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:19:42.853428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:42.853428+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:57.865194+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":529,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.288644+0000","last_change":"2024-03-11T22:20:29.431434+0000","last_active":"2024-03-11T22:20:32.288644+0000","last_peered":"2024-03-11T22:20:32.288644+0000","last_clean":"2024-03-11T22:20:32.288644+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:20:32.288644+0000","last_undegraded":"2024-03-11T22:20:32.288644+0000","last_fullsized":"2024-03-11T22:20:32.288644+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:20.763181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.763181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"93'130","reported_seq":793,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.287982+0000","last_change":"2024-03-11T22:19:29.435235+0000","last_active":"2024-03-11T22:20:32.287982+0000","last_peered":"2024-03-11T22:20:32.287982+0000","last_clean":"2024-03-11T22:20:32.287982+0000","last_became_active":"2024-03-11T22:14:43.191297+0000","last_became_peered":"2024-03-11T22:14:43.191297+0000","last_unstale":"2024-03-11T22:20:32.287982+0000","last_undegraded":"2024-03-11T22:20:32.287982+0000","last_fullsized":"2024-03-11T22:20:32.287982+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:29.434647+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.434647+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:46.425189+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"273'19","reported_seq":354,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284399+0000","last_change":"2024-03-11T22:18:41.785037+0000","last_active":"2024-03-11T22:20:32.284399+0000","last_peered":"2024-03-11T22:20:32.284399+0000","last_clean":"2024-03-11T22:20:32.284399+0000","last_became_active":"2024-03-11T22:14:52.239810+0000","last_became_peered":"2024-03-11T22:14:52.239810+0000","last_unstale":"2024-03-11T22:20:32.284399+0000","last_undegraded":"2024-03-11T22:20:32.284399+0000","last_fullsized":"2024-03-11T22:20:32.284399+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:18:41.784905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:41.784905+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":333,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284185+0000","last_change":"2024-03-11T22:20:29.434792+0000","last_active":"2024-03-11T22:20:32.284185+0000","last_peered":"2024-03-11T22:20:32.284185+0000","last_clean":"2024-03-11T22:20:32.284185+0000","last_became_active":"2024-03-11T22:20:29.420867+0000","last_became_peered":"2024-03-11T22:20:29.420867+0000","last_unstale":"2024-03-11T22:20:32.284185+0000","last_undegraded":"2024-03-11T22:20:32.284185+0000","last_fullsized":"2024-03-11T22:20:32.284185+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:18:30.847205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:30.847205+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":538,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.290548+0000","last_change":"2024-03-11T22:17:24.726548+0000","last_active":"2024-03-11T22:20:32.290548+0000","last_peered":"2024-03-11T22:20:32.290548+0000","last_clean":"2024-03-11T22:20:32.290548+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:20:32.290548+0000","last_undegraded":"2024-03-11T22:20:32.290548+0000","last_fullsized":"2024-03-11T22:20:32.290548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:24.726421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:17:24.726421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":518,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284557+0000","last_change":"2024-03-11T22:18:29.014246+0000","last_active":"2024-03-11T22:20:32.284557+0000","last_peered":"2024-03-11T22:20:32.284557+0000","last_clean":"2024-03-11T22:20:32.284557+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:20:32.284557+0000","last_undegraded":"2024-03-11T22:20:32.284557+0000","last_fullsized":"2024-03-11T22:20:32.284557+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:29.014149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:29.014149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":571,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.290640+0000","last_change":"2024-03-11T22:17:35.738522+0000","last_active":"2024-03-11T22:20:32.290640+0000","last_peered":"2024-03-11T22:20:32.290640+0000","last_clean":"2024-03-11T22:20:32.290640+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:20:32.290640+0000","last_undegraded":"2024-03-11T22:20:32.290640+0000","last_fullsized":"2024-03-11T22:20:32.290640+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'13","last_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:17:35.738211+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":481,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283340+0000","last_change":"2024-03-11T22:20:29.423108+0000","last_active":"2024-03-11T22:20:32.283340+0000","last_peered":"2024-03-11T22:20:32.283340+0000","last_clean":"2024-03-11T22:20:32.283340+0000","last_became_active":"2024-03-11T22:20:29.422747+0000","last_became_peered":"2024-03-11T22:20:29.422747+0000","last_unstale":"2024-03-11T22:20:32.283340+0000","last_undegraded":"2024-03-11T22:20:32.283340+0000","last_fullsized":"2024-03-11T22:20:32.283340+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:44.563614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:44.563614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:46.079800+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":530,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284553+0000","last_change":"2024-03-11T22:19:39.445467+0000","last_active":"2024-03-11T22:20:32.284553+0000","last_peered":"2024-03-11T22:20:32.284553+0000","last_clean":"2024-03-11T22:20:32.284553+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:20:32.284553+0000","last_undegraded":"2024-03-11T22:20:32.284553+0000","last_fullsized":"2024-03-11T22:20:32.284553+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:39.445349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:19:39.445349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:56.972428+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":526,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.287130+0000","last_change":"2024-03-11T22:19:26.459086+0000","last_active":"2024-03-11T22:20:32.287130+0000","last_peered":"2024-03-11T22:20:32.287130+0000","last_clean":"2024-03-11T22:20:32.287130+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:20:32.287130+0000","last_undegraded":"2024-03-11T22:20:32.287130+0000","last_fullsized":"2024-03-11T22:20:32.287130+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:26.458937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:26.458937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:41.990792+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"93'130","reported_seq":846,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.279768+0000","last_change":"2024-03-11T22:18:43.722555+0000","last_active":"2024-03-11T22:20:32.279768+0000","last_peered":"2024-03-11T22:20:32.279768+0000","last_clean":"2024-03-11T22:20:32.279768+0000","last_became_active":"2024-03-11T22:14:43.194955+0000","last_became_peered":"2024-03-11T22:14:43.194955+0000","last_unstale":"2024-03-11T22:20:32.279768+0000","last_undegraded":"2024-03-11T22:20:32.279768+0000","last_fullsized":"2024-03-11T22:20:32.279768+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:43.722467+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:18:43.722467+0000","last_clean_scrub_stamp":"2024-03-11T22:18:43.722467+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"274'20","reported_seq":594,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284925+0000","last_change":"2024-03-11T22:20:29.487125+0000","last_active":"2024-03-11T22:20:32.284925+0000","last_peered":"2024-03-11T22:20:32.284925+0000","last_clean":"2024-03-11T22:20:32.284925+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:20:32.284925+0000","last_undegraded":"2024-03-11T22:20:32.284925+0000","last_fullsized":"2024-03-11T22:20:32.284925+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'20","last_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:18:42.135888+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":550,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.293061+0000","last_change":"2024-03-11T22:20:29.417149+0000","last_active":"2024-03-11T22:20:32.293061+0000","last_peered":"2024-03-11T22:20:32.293061+0000","last_clean":"2024-03-11T22:20:32.293061+0000","last_became_active":"2024-03-11T22:20:29.416792+0000","last_became_peered":"2024-03-11T22:20:29.416792+0000","last_unstale":"2024-03-11T22:20:32.293061+0000","last_undegraded":"2024-03-11T22:20:32.293061+0000","last_fullsized":"2024-03-11T22:20:32.293061+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:45.801375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:45.801375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":307,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283585+0000","last_change":"2024-03-11T22:17:19.160859+0000","last_active":"2024-03-11T22:20:32.283585+0000","last_peered":"2024-03-11T22:20:32.283585+0000","last_clean":"2024-03-11T22:20:32.283585+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:20:32.283585+0000","last_undegraded":"2024-03-11T22:20:32.283585+0000","last_fullsized":"2024-03-11T22:20:32.283585+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:19.160748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:17:19.160748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":518,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284840+0000","last_change":"2024-03-11T22:14:23.930193+0000","last_active":"2024-03-11T22:20:32.284840+0000","last_peered":"2024-03-11T22:20:32.284840+0000","last_clean":"2024-03-11T22:20:32.284840+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:20:32.284840+0000","last_undegraded":"2024-03-11T22:20:32.284840+0000","last_fullsized":"2024-03-11T22:20:32.284840+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.930091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.930091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"273'8","reported_seq":558,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283201+0000","last_change":"2024-03-11T22:18:53.157176+0000","last_active":"2024-03-11T22:20:32.283201+0000","last_peered":"2024-03-11T22:20:32.283201+0000","last_clean":"2024-03-11T22:20:32.283201+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:20:32.283201+0000","last_undegraded":"2024-03-11T22:20:32.283201+0000","last_fullsized":"2024-03-11T22:20:32.283201+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'8","last_scrub_stamp":"2024-03-11T22:18:53.157080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:53.157080+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":538,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.286387+0000","last_change":"2024-03-11T22:19:45.501704+0000","last_active":"2024-03-11T22:20:32.286387+0000","last_peered":"2024-03-11T22:20:32.286387+0000","last_clean":"2024-03-11T22:20:32.286387+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:20:32.286387+0000","last_undegraded":"2024-03-11T22:20:32.286387+0000","last_fullsized":"2024-03-11T22:20:32.286387+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.501593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.501593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:53.822220+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":124,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284431+0000","last_change":"2024-03-11T22:20:30.047479+0000","last_active":"2024-03-11T22:20:32.284431+0000","last_peered":"2024-03-11T22:20:32.284431+0000","last_clean":"2024-03-11T22:20:32.284431+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:20:32.284431+0000","last_undegraded":"2024-03-11T22:20:32.284431+0000","last_fullsized":"2024-03-11T22:20:32.284431+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:30.047368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:30.047368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:43.466695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":492,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.282349+0000","last_change":"2024-03-11T22:19:32.989760+0000","last_active":"2024-03-11T22:20:32.282349+0000","last_peered":"2024-03-11T22:20:32.282349+0000","last_clean":"2024-03-11T22:20:32.282349+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:20:32.282349+0000","last_undegraded":"2024-03-11T22:20:32.282349+0000","last_fullsized":"2024-03-11T22:20:32.282349+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.989348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.989348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:47.266882+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"273'19","reported_seq":573,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.286211+0000","last_change":"2024-03-11T22:18:27.901239+0000","last_active":"2024-03-11T22:20:32.286211+0000","last_peered":"2024-03-11T22:20:32.286211+0000","last_clean":"2024-03-11T22:20:32.286211+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:20:32.286211+0000","last_undegraded":"2024-03-11T22:20:32.286211+0000","last_fullsized":"2024-03-11T22:20:32.286211+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-11T22:18:27.901147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.901147+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":538,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283146+0000","last_change":"2024-03-11T22:19:25.669161+0000","last_active":"2024-03-11T22:20:32.283146+0000","last_peered":"2024-03-11T22:20:32.283146+0000","last_clean":"2024-03-11T22:20:32.283146+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:20:32.283146+0000","last_undegraded":"2024-03-11T22:20:32.283146+0000","last_fullsized":"2024-03-11T22:20:32.283146+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:25.668878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:25.668878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":538,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.289256+0000","last_change":"2024-03-11T22:20:27.333574+0000","last_active":"2024-03-11T22:20:32.289256+0000","last_peered":"2024-03-11T22:20:32.289256+0000","last_clean":"2024-03-11T22:20:32.289256+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:20:32.289256+0000","last_undegraded":"2024-03-11T22:20:32.289256+0000","last_fullsized":"2024-03-11T22:20:32.289256+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:27.333450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:27.333450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:54.425323+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":541,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.292167+0000","last_change":"2024-03-11T22:19:45.686859+0000","last_active":"2024-03-11T22:20:32.292167+0000","last_peered":"2024-03-11T22:20:32.292167+0000","last_clean":"2024-03-11T22:20:32.292167+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:20:32.292167+0000","last_undegraded":"2024-03-11T22:20:32.292167+0000","last_fullsized":"2024-03-11T22:20:32.292167+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.686738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.686738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:08.688833+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"273'20","reported_seq":601,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.288991+0000","last_change":"2024-03-11T22:20:29.460876+0000","last_active":"2024-03-11T22:20:32.288991+0000","last_peered":"2024-03-11T22:20:32.288991+0000","last_clean":"2024-03-11T22:20:32.288991+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:20:32.288991+0000","last_undegraded":"2024-03-11T22:20:32.288991+0000","last_fullsized":"2024-03-11T22:20:32.288991+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:19:36.469333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.469333+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":547,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284195+0000","last_change":"2024-03-11T22:17:36.687785+0000","last_active":"2024-03-11T22:20:32.284195+0000","last_peered":"2024-03-11T22:20:32.284195+0000","last_clean":"2024-03-11T22:20:32.284195+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:20:32.284195+0000","last_undegraded":"2024-03-11T22:20:32.284195+0000","last_fullsized":"2024-03-11T22:20:32.284195+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:17:36.687660+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"310'10","reported_seq":724,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:33.493980+0000","last_change":"2024-03-11T22:20:29.422187+0000","last_active":"2024-03-11T22:20:33.493980+0000","last_peered":"2024-03-11T22:20:33.493980+0000","last_clean":"2024-03-11T22:20:33.493980+0000","last_became_active":"2024-03-11T22:20:29.415781+0000","last_became_peered":"2024-03-11T22:20:29.415781+0000","last_unstale":"2024-03-11T22:20:33.493980+0000","last_undegraded":"2024-03-11T22:20:33.493980+0000","last_fullsized":"2024-03-11T22:20:33.493980+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'9","last_scrub_stamp":"2024-03-11T22:18:50.706782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:50.706782+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":487,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.292953+0000","last_change":"2024-03-11T22:19:27.754842+0000","last_active":"2024-03-11T22:20:32.292953+0000","last_peered":"2024-03-11T22:20:32.292953+0000","last_clean":"2024-03-11T22:20:32.292953+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:20:32.292953+0000","last_undegraded":"2024-03-11T22:20:32.292953+0000","last_fullsized":"2024-03-11T22:20:32.292953+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:19:27.754721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:27.754721+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:55.635621+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"273'18","reported_seq":557,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283599+0000","last_change":"2024-03-11T22:14:27.998984+0000","last_active":"2024-03-11T22:20:32.283599+0000","last_peered":"2024-03-11T22:20:32.283599+0000","last_clean":"2024-03-11T22:20:32.283599+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:20:32.283599+0000","last_undegraded":"2024-03-11T22:20:32.283599+0000","last_fullsized":"2024-03-11T22:20:32.283599+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:14:27.998819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:27.998819+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":313,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.290438+0000","last_change":"2024-03-11T22:19:36.471241+0000","last_active":"2024-03-11T22:20:32.290438+0000","last_peered":"2024-03-11T22:20:32.290438+0000","last_clean":"2024-03-11T22:20:32.290438+0000","last_became_active":"2024-03-11T22:14:52.245199+0000","last_became_peered":"2024-03-11T22:14:52.245199+0000","last_unstale":"2024-03-11T22:20:32.290438+0000","last_undegraded":"2024-03-11T22:20:32.290438+0000","last_fullsized":"2024-03-11T22:20:32.290438+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.471133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.471133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:00.098252+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":288,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.287073+0000","last_change":"2024-03-11T22:14:52.233517+0000","last_active":"2024-03-11T22:20:32.287073+0000","last_peered":"2024-03-11T22:20:32.287073+0000","last_clean":"2024-03-11T22:20:32.287073+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:20:32.287073+0000","last_undegraded":"2024-03-11T22:20:32.287073+0000","last_fullsized":"2024-03-11T22:20:32.287073+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.849679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.849679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":518,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.286117+0000","last_change":"2024-03-11T22:18:27.116987+0000","last_active":"2024-03-11T22:20:32.286117+0000","last_peered":"2024-03-11T22:20:32.286117+0000","last_clean":"2024-03-11T22:20:32.286117+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:20:32.286117+0000","last_undegraded":"2024-03-11T22:20:32.286117+0000","last_fullsized":"2024-03-11T22:20:32.286117+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:27.116858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.116858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"273'27","reported_seq":600,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.279866+0000","last_change":"2024-03-11T22:18:42.691421+0000","last_active":"2024-03-11T22:20:32.279866+0000","last_peered":"2024-03-11T22:20:32.279866+0000","last_clean":"2024-03-11T22:20:32.279866+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:20:32.279866+0000","last_undegraded":"2024-03-11T22:20:32.279866+0000","last_fullsized":"2024-03-11T22:20:32.279866+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:18:42.691319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:42.691319+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":549,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.287590+0000","last_change":"2024-03-11T22:20:29.414748+0000","last_active":"2024-03-11T22:20:32.287590+0000","last_peered":"2024-03-11T22:20:32.287590+0000","last_clean":"2024-03-11T22:20:32.287590+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:20:32.287590+0000","last_undegraded":"2024-03-11T22:20:32.287590+0000","last_fullsized":"2024-03-11T22:20:32.287590+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:18:43.931797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":537,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.292061+0000","last_change":"2024-03-11T22:18:31.120211+0000","last_active":"2024-03-11T22:20:32.292061+0000","last_peered":"2024-03-11T22:20:32.292061+0000","last_clean":"2024-03-11T22:20:32.292061+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:20:32.292061+0000","last_undegraded":"2024-03-11T22:20:32.292061+0000","last_fullsized":"2024-03-11T22:20:32.292061+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:31.120131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:31.120131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":304,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.286703+0000","last_change":"2024-03-11T22:18:28.862908+0000","last_active":"2024-03-11T22:20:32.286703+0000","last_peered":"2024-03-11T22:20:32.286703+0000","last_clean":"2024-03-11T22:20:32.286703+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:20:32.286703+0000","last_undegraded":"2024-03-11T22:20:32.286703+0000","last_fullsized":"2024-03-11T22:20:32.286703+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:28.862793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:28.862793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"273'9","reported_seq":373,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284369+0000","last_change":"2024-03-11T22:18:51.098736+0000","last_active":"2024-03-11T22:20:32.284369+0000","last_peered":"2024-03-11T22:20:32.284369+0000","last_clean":"2024-03-11T22:20:32.284369+0000","last_became_active":"2024-03-11T22:14:52.238411+0000","last_became_peered":"2024-03-11T22:14:52.238411+0000","last_unstale":"2024-03-11T22:20:32.284369+0000","last_undegraded":"2024-03-11T22:20:32.284369+0000","last_fullsized":"2024-03-11T22:20:32.284369+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9","last_scrub_stamp":"2024-03-11T22:18:51.098646+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:18:51.098646+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":537,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.293136+0000","last_change":"2024-03-11T22:18:29.763217+0000","last_active":"2024-03-11T22:20:32.293136+0000","last_peered":"2024-03-11T22:20:32.293136+0000","last_clean":"2024-03-11T22:20:32.293136+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:20:32.293136+0000","last_undegraded":"2024-03-11T22:20:32.293136+0000","last_fullsized":"2024-03-11T22:20:32.293136+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:29.739164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:18:29.739164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":307,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.289346+0000","last_change":"2024-03-11T22:17:27.798877+0000","last_active":"2024-03-11T22:20:32.289346+0000","last_peered":"2024-03-11T22:20:32.289346+0000","last_clean":"2024-03-11T22:20:32.289346+0000","last_became_active":"2024-03-11T22:14:52.239391+0000","last_became_peered":"2024-03-11T22:14:52.239391+0000","last_unstale":"2024-03-11T22:20:32.289346+0000","last_undegraded":"2024-03-11T22:20:32.289346+0000","last_fullsized":"2024-03-11T22:20:32.289346+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:27.798763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_clean_scrub_stamp":"2024-03-11T22:17:27.798763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":525,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.290466+0000","last_change":"2024-03-11T22:18:38.736070+0000","last_active":"2024-03-11T22:20:32.290466+0000","last_peered":"2024-03-11T22:20:32.290466+0000","last_clean":"2024-03-11T22:20:32.290466+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:20:32.290466+0000","last_undegraded":"2024-03-11T22:20:32.290466+0000","last_fullsized":"2024-03-11T22:20:32.290466+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:38.735937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.735937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"273'15","reported_seq":340,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283140+0000","last_change":"2024-03-11T22:15:18.401675+0000","last_active":"2024-03-11T22:20:32.283140+0000","last_peered":"2024-03-11T22:20:32.283140+0000","last_clean":"2024-03-11T22:20:32.283140+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:20:32.283140+0000","last_undegraded":"2024-03-11T22:20:32.283140+0000","last_fullsized":"2024-03-11T22:20:32.283140+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:15:18.401549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:18.401549+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":545,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.294573+0000","last_change":"2024-03-11T22:19:40.864768+0000","last_active":"2024-03-11T22:20:32.294573+0000","last_peered":"2024-03-11T22:20:32.294573+0000","last_clean":"2024-03-11T22:20:32.294573+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:20:32.294573+0000","last_undegraded":"2024-03-11T22:20:32.294573+0000","last_fullsized":"2024-03-11T22:20:32.294573+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:40.864689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:40.864689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:05.030010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":541,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.292715+0000","last_change":"2024-03-11T22:20:29.420120+0000","last_active":"2024-03-11T22:20:32.292715+0000","last_peered":"2024-03-11T22:20:32.292715+0000","last_clean":"2024-03-11T22:20:32.292715+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:20:32.292715+0000","last_undegraded":"2024-03-11T22:20:32.292715+0000","last_fullsized":"2024-03-11T22:20:32.292715+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:28.713390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:18:28.713390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":525,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.287521+0000","last_change":"2024-03-11T22:20:30.493247+0000","last_active":"2024-03-11T22:20:32.287521+0000","last_peered":"2024-03-11T22:20:32.287521+0000","last_clean":"2024-03-11T22:20:32.287521+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:20:32.287521+0000","last_undegraded":"2024-03-11T22:20:32.287521+0000","last_fullsized":"2024-03-11T22:20:32.287521+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:30.493146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:20:30.493146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:48.539526+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"273'27","reported_seq":568,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.285025+0000","last_change":"2024-03-11T22:18:37.066095+0000","last_active":"2024-03-11T22:20:32.285025+0000","last_peered":"2024-03-11T22:20:32.285025+0000","last_clean":"2024-03-11T22:20:32.285025+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:20:32.285025+0000","last_undegraded":"2024-03-11T22:20:32.285025+0000","last_fullsized":"2024-03-11T22:20:32.285025+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:18:37.065988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.065988+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":545,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283838+0000","last_change":"2024-03-11T22:18:38.095521+0000","last_active":"2024-03-11T22:20:32.283838+0000","last_peered":"2024-03-11T22:20:32.283838+0000","last_clean":"2024-03-11T22:20:32.283838+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:20:32.283838+0000","last_undegraded":"2024-03-11T22:20:32.283838+0000","last_fullsized":"2024-03-11T22:20:32.283838+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:18:38.095367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.095367+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":312,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283698+0000","last_change":"2024-03-11T22:20:26.125601+0000","last_active":"2024-03-11T22:20:32.283698+0000","last_peered":"2024-03-11T22:20:32.283698+0000","last_clean":"2024-03-11T22:20:32.283698+0000","last_became_active":"2024-03-11T22:14:52.247689+0000","last_became_peered":"2024-03-11T22:14:52.247689+0000","last_unstale":"2024-03-11T22:20:32.283698+0000","last_undegraded":"2024-03-11T22:20:32.283698+0000","last_fullsized":"2024-03-11T22:20:32.283698+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:26.125517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:26.125517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:42.748307+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":529,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284641+0000","last_change":"2024-03-11T22:20:29.436389+0000","last_active":"2024-03-11T22:20:32.284641+0000","last_peered":"2024-03-11T22:20:32.284641+0000","last_clean":"2024-03-11T22:20:32.284641+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:20:32.284641+0000","last_undegraded":"2024-03-11T22:20:32.284641+0000","last_fullsized":"2024-03-11T22:20:32.284641+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:24.023596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:24.023596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"273'19","reported_seq":352,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.283785+0000","last_change":"2024-03-11T22:19:30.613894+0000","last_active":"2024-03-11T22:20:32.283785+0000","last_peered":"2024-03-11T22:20:32.283785+0000","last_clean":"2024-03-11T22:20:32.283785+0000","last_became_active":"2024-03-11T22:14:52.247411+0000","last_became_peered":"2024-03-11T22:14:52.247411+0000","last_unstale":"2024-03-11T22:20:32.283785+0000","last_undegraded":"2024-03-11T22:20:32.283785+0000","last_fullsized":"2024-03-11T22:20:32.283785+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:19:30.613736+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:19:30.613736+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:33.474855+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":538,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.286101+0000","last_change":"2024-03-11T22:18:24.613444+0000","last_active":"2024-03-11T22:20:32.286101+0000","last_peered":"2024-03-11T22:20:32.286101+0000","last_clean":"2024-03-11T22:20:32.286101+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:20:32.286101+0000","last_undegraded":"2024-03-11T22:20:32.286101+0000","last_fullsized":"2024-03-11T22:20:32.286101+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:24.613372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:24.613372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":252,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.284478+0000","last_change":"2024-03-11T22:20:29.420332+0000","last_active":"2024-03-11T22:20:32.284478+0000","last_peered":"2024-03-11T22:20:32.284478+0000","last_clean":"2024-03-11T22:20:32.284478+0000","last_became_active":"2024-03-11T22:20:29.418227+0000","last_became_peered":"2024-03-11T22:20:29.418227+0000","last_unstale":"2024-03-11T22:20:32.284478+0000","last_undegraded":"2024-03-11T22:20:32.284478+0000","last_fullsized":"2024-03-11T22:20:32.284478+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.629853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.629853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:14.302678+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":304,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-11T22:20:32.289703+0000","last_change":"2024-03-11T22:19:26.746519+0000","last_active":"2024-03-11T22:20:32.289703+0000","last_peered":"2024-03-11T22:20:32.289703+0000","last_clean":"2024-03-11T22:20:32.289703+0000","last_became_active":"2024-03-11T22:14:52.247273+0000","last_became_peered":"2024-03-11T22:14:52.247273+0000","last_unstale":"2024-03-11T22:20:32.289703+0000","last_undegraded":"2024-03-11T22:20:32.289703+0000","last_fullsized":"2024-03-11T22:20:32.289703+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:26.746435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:26.746435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:38.021051+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6995836952,"num_objects":106749,"num_object_clones":0,"num_object_copies":320247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":41836,"num_objects_unfound":0,"num_objects_dirty":106749,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141487,"num_write_kb":7943489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5284,"num_bytes_recovered":346095616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21350264832,"data_stored":21350252616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72087,"ondisk_log_size":72087,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":693,"num_read_kb":688,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":1430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":717,"ondisk_log_size":717,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005206,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1533620,"kb_used_data":1223404,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92211532,"statfs":{"total":95995035648,"available":94424608768,"internally_reserved":0,"allocated":1252765696,"data_stored":1252438044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":317651403},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332251,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1627872,"kb_used_data":1289880,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92117280,"statfs":{"total":95995035648,"available":94328094720,"internally_reserved":0,"allocated":1320837120,"data_stored":1320529758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346094022},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4723924,"kb_used_data":4287372,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":89021228,"statfs":{"total":95995035648,"available":91157737472,"internally_reserved":0,"allocated":4390268928,"data_stored":4389908538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738555,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2525488,"kb_used_data":2137512,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":91219664,"statfs":{"total":95995035648,"available":93408935936,"internally_reserved":0,"allocated":2188812288,"data_stored":2188488064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411504,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2032864,"kb_used_data":1694872,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":91712288,"statfs":{"total":95995035648,"available":93913382912,"internally_reserved":0,"allocated":1735548928,"data_stored":1735241936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4306696,"kb_used_data":3860416,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":89438456,"statfs":{"total":95995035648,"available":91584978944,"internally_reserved":0,"allocated":3953065984,"data_stored":3952718712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":456980935},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920934,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2475580,"kb_used_data":2155636,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":91269572,"statfs":{"total":95995035648,"available":93460041728,"internally_reserved":0,"allocated":2207371264,"data_stored":2207034985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":327612870},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659295,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4824236,"kb_used_data":4351588,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":88920916,"statfs":{"total":95995035648,"available":91055017984,"internally_reserved":0,"allocated":4456026112,"data_stored":4455706303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":483981766},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2173763584,"data_stored":2173763584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4374597632,"data_stored":4374593560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1734017024,"data_stored":1734017024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3941859328,"data_stored":3941859328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2191265792,"data_stored":2191261720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4418768896,"data_stored":4418764824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1315176448,"data_stored":1315176448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1200816128,"data_stored":1200816128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:20:46.428 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:20:46.484 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:46.517 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:46.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:46.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:46.557 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:20:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:20:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:46 smithi144 ceph-mon[33984]: 6.1e scrub starts 2024-03-11T22:20:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:46 smithi144 ceph-mon[33984]: 6.1e scrub ok 2024-03-11T22:20:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:46 smithi144 ceph-mon[33984]: 3.9 scrub starts 2024-03-11T22:20:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:46 smithi144 ceph-mon[33984]: 3.9 scrub ok 2024-03-11T22:20:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:46 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2361091411' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:20:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:46 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:20:46.597 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:20:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:20:46.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:46.667 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:46.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:46.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:46.768 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:46.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:46.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:46.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:46.955 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:46.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:47.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:20:47.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:47.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[30178]: 6.1e scrub starts 2024-03-11T22:20:47.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[30178]: 6.1e scrub ok 2024-03-11T22:20:47.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[30178]: 3.9 scrub starts 2024-03-11T22:20:47.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[30178]: 3.9 scrub ok 2024-03-11T22:20:47.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2361091411' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:20:47.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:20:47.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:20:47.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:20:47.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[39715]: 6.1e scrub starts 2024-03-11T22:20:47.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[39715]: 6.1e scrub ok 2024-03-11T22:20:47.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[39715]: 3.9 scrub starts 2024-03-11T22:20:47.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[39715]: 3.9 scrub ok 2024-03-11T22:20:47.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2361091411' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:20:47.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:46 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:20:47.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:47.110 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:47.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:47.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:47.252 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:47.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:47.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:47.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:47.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:47.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:47.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:47.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:47.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:47.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:47 smithi144 ceph-mon[33984]: pgmap v576: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 37904/308148 objects misplaced (12.301%); 40 MiB/s, 646 objects/s recovering 2024-03-11T22:20:47.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:47.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:47 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:20:47.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:47 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:20:47.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:47 smithi144 ceph-mon[33984]: osdmap e314: 8 total, 8 up, 8 in 2024-03-11T22:20:47.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:47 smithi144 ceph-mon[33984]: 3.11 scrub starts 2024-03-11T22:20:47.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:47 smithi144 ceph-mon[33984]: 3.11 scrub ok 2024-03-11T22:20:47.639 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:47.670 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:47.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:47.772 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:47.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:47.809 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:47.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:47.875 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[39715]: pgmap v576: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 37904/308148 objects misplaced (12.301%); 40 MiB/s, 646 objects/s recovering 2024-03-11T22:20:47.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:47.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:20:47.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:20:47.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[39715]: osdmap e314: 8 total, 8 up, 8 in 2024-03-11T22:20:47.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[39715]: 3.11 scrub starts 2024-03-11T22:20:47.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[39715]: 3.11 scrub ok 2024-03-11T22:20:47.876 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[30178]: pgmap v576: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 37904/308148 objects misplaced (12.301%); 40 MiB/s, 646 objects/s recovering 2024-03-11T22:20:47.876 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:47.876 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:20:47.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:20:47.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[30178]: osdmap e314: 8 total, 8 up, 8 in 2024-03-11T22:20:47.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[30178]: 3.11 scrub starts 2024-03-11T22:20:47.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:47 smithi045 ceph-mon[30178]: 3.11 scrub ok 2024-03-11T22:20:47.910 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:47.944 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:47.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:48.045 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:48.082 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:48.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:48.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:48.220 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:48.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:48.321 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:48.351 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:48.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:48.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:48.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:48.504 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:48.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:48.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:48.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:48.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:48.754 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:48.786 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:48.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:48.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:48 smithi144 ceph-mon[33984]: 3.14 scrub starts 2024-03-11T22:20:48.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:48 smithi144 ceph-mon[33984]: 3.14 scrub ok 2024-03-11T22:20:48.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:48 smithi144 ceph-mon[33984]: 5.8 scrub starts 2024-03-11T22:20:48.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:48 smithi144 ceph-mon[33984]: 5.8 scrub ok 2024-03-11T22:20:48.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:48.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:48.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:48.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:49.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:49.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:48 smithi045 ceph-mon[39715]: 3.14 scrub starts 2024-03-11T22:20:49.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:48 smithi045 ceph-mon[39715]: 3.14 scrub ok 2024-03-11T22:20:49.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:48 smithi045 ceph-mon[39715]: 5.8 scrub starts 2024-03-11T22:20:49.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:48 smithi045 ceph-mon[39715]: 5.8 scrub ok 2024-03-11T22:20:49.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:48 smithi045 ceph-mon[30178]: 3.14 scrub starts 2024-03-11T22:20:49.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:48 smithi045 ceph-mon[30178]: 3.14 scrub ok 2024-03-11T22:20:49.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:48 smithi045 ceph-mon[30178]: 5.8 scrub starts 2024-03-11T22:20:49.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:48 smithi045 ceph-mon[30178]: 5.8 scrub ok 2024-03-11T22:20:49.059 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:49.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:49.160 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:49.197 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:49.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:49.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:49.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:49.336 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:49.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:49.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:49.438 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:49.470 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:49.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:49.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:49.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:49.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:49.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:49.749 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:49.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:49.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:49 smithi045 ceph-mon[39715]: pgmap v578: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 37277/305454 objects misplaced (12.204%); 32 MiB/s, 513 objects/s recovering 2024-03-11T22:20:49.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:49 smithi045 ceph-mon[30178]: pgmap v578: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 37277/305454 objects misplaced (12.204%); 32 MiB/s, 513 objects/s recovering 2024-03-11T22:20:49.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:49 smithi144 ceph-mon[33984]: pgmap v578: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 37277/305454 objects misplaced (12.204%); 32 MiB/s, 513 objects/s recovering 2024-03-11T22:20:49.851 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:49.881 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:49.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:49.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:49.982 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:50.017 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:50.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:50.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:50.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:50.169 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:50.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:50.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:50.271 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:50.309 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:50.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:50.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:50.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:50.483 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:50.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:50.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:50.585 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:50.627 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:50.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:50.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:50.723 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:50 smithi045 ceph-mon[30178]: 1.14 scrub starts 2024-03-11T22:20:50.723 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:50 smithi045 ceph-mon[30178]: 1.14 scrub ok 2024-03-11T22:20:50.723 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:50 smithi045 ceph-mon[39715]: 1.14 scrub starts 2024-03-11T22:20:50.723 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:50 smithi045 ceph-mon[39715]: 1.14 scrub ok 2024-03-11T22:20:50.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:50.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:50.773 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:50.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:50.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:50 smithi144 ceph-mon[33984]: 1.14 scrub starts 2024-03-11T22:20:50.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:50 smithi144 ceph-mon[33984]: 1.14 scrub ok 2024-03-11T22:20:50.874 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:50.907 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:50.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:50.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:51.009 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:51.039 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:51.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:51.141 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:51.188 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:51.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:51.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:51.290 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:51.320 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:51.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:51.371 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:20:51.371 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:20:51.371 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:20:51.371 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:20:51.371 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:20:51.381 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:51.395 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:20:51.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:51.436 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:20:51.468 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:51.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:51.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:51.478 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:20:51.481 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:20:51.482 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:20:51.482 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:20:51.482 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:20:51.482 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:20:51.482 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:20:51.482 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:20:51.482 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:20:51.571 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:51.603 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:51.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:51.704 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:51.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:51.735 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:51.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:51.836 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:51.841 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:20:51.841 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:20:51.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:51 smithi045 ceph-mon[30178]: pgmap v579: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 37277/305454 objects misplaced (12.204%); 32 MiB/s, 513 objects/s recovering 2024-03-11T22:20:51.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:51 smithi045 ceph-mon[30178]: 4.13 scrub starts 2024-03-11T22:20:51.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:51 smithi045 ceph-mon[30178]: 4.13 scrub ok 2024-03-11T22:20:51.842 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:51 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/467849803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:20:51.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:51 smithi045 ceph-mon[39715]: pgmap v579: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 37277/305454 objects misplaced (12.204%); 32 MiB/s, 513 objects/s recovering 2024-03-11T22:20:51.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:51 smithi045 ceph-mon[39715]: 4.13 scrub starts 2024-03-11T22:20:51.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:51 smithi045 ceph-mon[39715]: 4.13 scrub ok 2024-03-11T22:20:51.843 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:51 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/467849803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:20:51.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:51 smithi144 ceph-mon[33984]: pgmap v579: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 37277/305454 objects misplaced (12.204%); 32 MiB/s, 513 objects/s recovering 2024-03-11T22:20:51.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:51 smithi144 ceph-mon[33984]: 4.13 scrub starts 2024-03-11T22:20:51.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:51 smithi144 ceph-mon[33984]: 4.13 scrub ok 2024-03-11T22:20:51.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:51 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/467849803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:20:51.870 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:51.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:51.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:52.021 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:52.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:52.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:52.161 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:52.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:52.262 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:52.296 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:52.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:52.398 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:52.431 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:52.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:52.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:20:52.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:52.601 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:52.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:52.703 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:52.740 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:52.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:52.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:52.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:52 smithi045 ceph-mon[30178]: 4.1f scrub starts 2024-03-11T22:20:52.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:52 smithi045 ceph-mon[30178]: 4.1f scrub ok 2024-03-11T22:20:52.785 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:20:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:20:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:20:52.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:52 smithi045 ceph-mon[39715]: 4.1f scrub starts 2024-03-11T22:20:52.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:52 smithi045 ceph-mon[39715]: 4.1f scrub ok 2024-03-11T22:20:52.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:52.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:52 smithi144 ceph-mon[33984]: 4.1f scrub starts 2024-03-11T22:20:52.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:52 smithi144 ceph-mon[33984]: 4.1f scrub ok 2024-03-11T22:20:52.888 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:52.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:52.990 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:53.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:53.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:53.129 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:53.163 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:53.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:53.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:53.264 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:53.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:53.301 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:53.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:53.401 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:53.440 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:53.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:53.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:53.475 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T22:20:53.541 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:53.574 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:53.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:53.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:53.675 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:53.709 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:53.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:53.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:53 smithi045 ceph-mon[39715]: pgmap v580: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 32522/290274 objects misplaced (11.204%); 35 MiB/s, 558 objects/s recovering 2024-03-11T22:20:53.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:53 smithi045 ceph-mon[39715]: 3.a scrub starts 2024-03-11T22:20:53.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:53 smithi045 ceph-mon[39715]: 3.a scrub ok 2024-03-11T22:20:53.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:53 smithi045 ceph-mon[39715]: 2.1 scrub starts 2024-03-11T22:20:53.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:53 smithi045 ceph-mon[39715]: 2.1 scrub ok 2024-03-11T22:20:53.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:53 smithi045 ceph-mon[30178]: pgmap v580: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 32522/290274 objects misplaced (11.204%); 35 MiB/s, 558 objects/s recovering 2024-03-11T22:20:53.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:53 smithi045 ceph-mon[30178]: 3.a scrub starts 2024-03-11T22:20:53.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:53 smithi045 ceph-mon[30178]: 3.a scrub ok 2024-03-11T22:20:53.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:53 smithi045 ceph-mon[30178]: 2.1 scrub starts 2024-03-11T22:20:53.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:53 smithi045 ceph-mon[30178]: 2.1 scrub ok 2024-03-11T22:20:53.810 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:53.844 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:53.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:53.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:53 smithi144 ceph-mon[33984]: pgmap v580: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 32522/290274 objects misplaced (11.204%); 35 MiB/s, 558 objects/s recovering 2024-03-11T22:20:53.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:53 smithi144 ceph-mon[33984]: 3.a scrub starts 2024-03-11T22:20:53.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:53 smithi144 ceph-mon[33984]: 3.a scrub ok 2024-03-11T22:20:53.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:53 smithi144 ceph-mon[33984]: 2.1 scrub starts 2024-03-11T22:20:53.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:53 smithi144 ceph-mon[33984]: 2.1 scrub ok 2024-03-11T22:20:53.945 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:53.979 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:53.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:53.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:54.081 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:54.113 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:54.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:54.215 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:54.248 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:54.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:54.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:54.394 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:54.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:54.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:54.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:54.531 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:54.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:54.633 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:54.663 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:54.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:54.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:54.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:54.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:54.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:54.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:54.937 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:54.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:54.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:55.039 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:55.065 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:55.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:55.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:55.167 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:55.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:55.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:55.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:55.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:55.337 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:55.337 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:55.338 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:20:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:55.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:55.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:55.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:55.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:55.571 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:55.597 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:55.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:55.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:55.689 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:20:55.699 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:55.712 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:20:55.726 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:55.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:55.827 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:55.873 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:55.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:55.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:55.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:20:55.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:20:55.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[39715]: pgmap v581: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 32522/290274 objects misplaced (11.204%); 35 MiB/s, 558 objects/s recovering 2024-03-11T22:20:55.963 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:20:55.963 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:20:55.963 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[39715]: osdmap e315: 8 total, 8 up, 8 in 2024-03-11T22:20:55.963 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[39715]: 5.16 deep-scrub starts 2024-03-11T22:20:55.963 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[39715]: 5.16 deep-scrub ok 2024-03-11T22:20:55.963 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[39715]: 5.1f deep-scrub starts 2024-03-11T22:20:55.963 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[39715]: 5.1f deep-scrub ok 2024-03-11T22:20:55.963 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:20:55.963 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:20:55.963 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[30178]: pgmap v581: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 32522/290274 objects misplaced (11.204%); 35 MiB/s, 558 objects/s recovering 2024-03-11T22:20:55.963 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:20:55.964 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:20:55.964 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[30178]: osdmap e315: 8 total, 8 up, 8 in 2024-03-11T22:20:55.964 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[30178]: 5.16 deep-scrub starts 2024-03-11T22:20:55.964 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[30178]: 5.16 deep-scrub ok 2024-03-11T22:20:55.964 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[30178]: 5.1f deep-scrub starts 2024-03-11T22:20:55.964 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:55 smithi045 ceph-mon[30178]: 5.1f deep-scrub ok 2024-03-11T22:20:55.975 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:56.005 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:56.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:56.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:55 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:20:56.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:55 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:20:56.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:55 smithi144 ceph-mon[33984]: pgmap v581: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+remapped+backfill_wait, 161 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 32522/290274 objects misplaced (11.204%); 35 MiB/s, 558 objects/s recovering 2024-03-11T22:20:56.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:55 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:20:56.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:55 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:20:56.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:55 smithi144 ceph-mon[33984]: osdmap e315: 8 total, 8 up, 8 in 2024-03-11T22:20:56.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:55 smithi144 ceph-mon[33984]: 5.16 deep-scrub starts 2024-03-11T22:20:56.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:55 smithi144 ceph-mon[33984]: 5.16 deep-scrub ok 2024-03-11T22:20:56.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:55 smithi144 ceph-mon[33984]: 5.1f deep-scrub starts 2024-03-11T22:20:56.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:55 smithi144 ceph-mon[33984]: 5.1f deep-scrub ok 2024-03-11T22:20:56.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:56.150 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:56.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:56.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:56.252 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:56.284 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:56.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:56.387 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:56.414 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:56.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:56.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:56.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:56.544 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:56.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:56.614 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:20:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:20:56.646 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:56.683 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:56.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:56.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:20:56.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:56.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:56.854 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:56.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:56.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:20:57.002 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:57.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:57.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:20:57.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:56 smithi045 ceph-mon[30178]: osdmap e316: 8 total, 8 up, 8 in 2024-03-11T22:20:57.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:56 smithi045 ceph-mon[30178]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:20:57.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:56 smithi045 ceph-mon[30178]: 4.1c scrub starts 2024-03-11T22:20:57.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:56 smithi045 ceph-mon[30178]: 4.1c scrub ok 2024-03-11T22:20:57.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:56 smithi045 ceph-mon[39715]: osdmap e316: 8 total, 8 up, 8 in 2024-03-11T22:20:57.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:56 smithi045 ceph-mon[39715]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:20:57.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:56 smithi045 ceph-mon[39715]: 4.1c scrub starts 2024-03-11T22:20:57.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:56 smithi045 ceph-mon[39715]: 4.1c scrub ok 2024-03-11T22:20:57.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:56 smithi144 ceph-mon[33984]: osdmap e316: 8 total, 8 up, 8 in 2024-03-11T22:20:57.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:56 smithi144 ceph-mon[33984]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:20:57.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:56 smithi144 ceph-mon[33984]: 4.1c scrub starts 2024-03-11T22:20:57.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:56 smithi144 ceph-mon[33984]: 4.1c scrub ok 2024-03-11T22:20:57.103 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:57.140 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:57.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:57.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:57.281 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:57.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:57.383 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:57.420 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:57.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:57.522 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:57.555 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:57.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:57.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:57.657 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:20:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:57.708 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:57.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:20:57.809 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:57.839 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:57.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:57.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:57.941 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:57.972 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:57.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:58.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:57 smithi045 ceph-mon[30178]: pgmap v584: 168 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 160 active+clean; 5.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 28446/279132 objects misplaced (10.191%); 48 MiB/s, 769 objects/s recovering 2024-03-11T22:20:58.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:58.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:57 smithi045 ceph-mon[39715]: pgmap v584: 168 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 160 active+clean; 5.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 28446/279132 objects misplaced (10.191%); 48 MiB/s, 769 objects/s recovering 2024-03-11T22:20:58.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:58.075 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:58.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:57 smithi144 ceph-mon[33984]: pgmap v584: 168 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 160 active+clean; 5.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 28446/279132 objects misplaced (10.191%); 48 MiB/s, 769 objects/s recovering 2024-03-11T22:20:58.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:20:58.107 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:58.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:58.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:58.208 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:58.213 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":583,"stamp":"2024-03-11T22:20:55.612176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6255090036,"num_objects":95635,"num_object_clones":0,"num_object_copies":286905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":31375,"num_objects_unfound":0,"num_objects_dirty":95635,"num_whiteouts":0,"num_read":1828,"num_read_kb":1580,"num_write":153521,"num_write_kb":7943534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12109,"num_bytes_recovered":786303364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74640,"ondisk_log_size":74640,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":512,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22220152,"kb_used_data":19075460,"kb_used_omap":12,"kb_used_meta":3143923,"kb_avail":727741064,"statfs":{"total":767960285184,"available":745206849536,"internally_reserved":0,"allocated":19533271040,"data_stored":19530711836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":3219377715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,7,10,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-464060416,"num_objects":-7081,"num_object_clones":0,"num_object_copies":-21243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6529,"num_objects_unfound":0,"num_objects_dirty":-7081,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7081,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4105,"num_bytes_recovered":269025280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.072109"},"pg_stats":[{"pgid":"9.1","version":"314'9602","reported_seq":15018,"reported_epoch":314,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T22:20:51.165665+0000","last_change":"2024-03-11T22:20:30.592256+0000","last_active":"2024-03-11T22:20:51.165665+0000","last_peered":"2024-03-11T22:20:51.165665+0000","last_clean":"2024-03-11T22:20:28.399763+0000","last_became_active":"2024-03-11T22:20:30.566997+0000","last_became_peered":"2024-03-11T22:20:30.566997+0000","last_unstale":"2024-03-11T22:20:51.165665+0000","last_undegraded":"2024-03-11T22:20:51.165665+0000","last_fullsized":"2024-03-11T22:20:51.165665+0000","mapping_epoch":310,"log_start":"298'5500","ondisk_log_start":"298'5500","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4102,"log_dups_size":0,"ondisk_log_size":4102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393609216,"num_objects":6006,"num_object_clones":0,"num_object_copies":18018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3613,"num_objects_unfound":0,"num_objects_dirty":6006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9602,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2657,"num_bytes_recovered":174063616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":6006}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"294'25","reported_seq":352,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.577331+0000","last_change":"2024-03-11T22:19:41.508884+0000","last_active":"2024-03-11T22:20:46.577331+0000","last_peered":"2024-03-11T22:20:46.577331+0000","last_clean":"2024-03-11T22:20:46.577331+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:20:46.577331+0000","last_undegraded":"2024-03-11T22:20:46.577331+0000","last_fullsized":"2024-03-11T22:20:46.577331+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:19:41.508161+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":540,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.572644+0000","last_change":"2024-03-11T22:19:28.586829+0000","last_active":"2024-03-11T22:20:46.572644+0000","last_peered":"2024-03-11T22:20:46.572644+0000","last_clean":"2024-03-11T22:20:46.572644+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:20:46.572644+0000","last_undegraded":"2024-03-11T22:20:46.572644+0000","last_fullsized":"2024-03-11T22:20:46.572644+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:28.586614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:28.586614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":713,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.549928+0000","last_change":"2024-03-11T22:19:40.008490+0000","last_active":"2024-03-11T22:20:46.549928+0000","last_peered":"2024-03-11T22:20:46.549928+0000","last_clean":"2024-03-11T22:20:46.549928+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:20:46.549928+0000","last_undegraded":"2024-03-11T22:20:46.549928+0000","last_fullsized":"2024-03-11T22:20:46.549928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:19:40.008325+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:19:40.008325+0000","last_clean_scrub_stamp":"2024-03-11T22:19:40.008325+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":306,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.563464+0000","last_change":"2024-03-11T22:18:37.116172+0000","last_active":"2024-03-11T22:20:46.563464+0000","last_peered":"2024-03-11T22:20:46.563464+0000","last_clean":"2024-03-11T22:20:46.563464+0000","last_became_active":"2024-03-11T22:14:52.243690+0000","last_became_peered":"2024-03-11T22:14:52.243690+0000","last_unstale":"2024-03-11T22:20:46.563464+0000","last_undegraded":"2024-03-11T22:20:46.563464+0000","last_fullsized":"2024-03-11T22:20:46.563464+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:37.116093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.116093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":863,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.573967+0000","last_change":"2024-03-11T22:20:32.294412+0000","last_active":"2024-03-11T22:20:46.573967+0000","last_peered":"2024-03-11T22:20:46.573967+0000","last_clean":"2024-03-11T22:20:46.573967+0000","last_became_active":"2024-03-11T22:20:32.293356+0000","last_became_peered":"2024-03-11T22:20:32.293356+0000","last_unstale":"2024-03-11T22:20:46.573967+0000","last_undegraded":"2024-03-11T22:20:46.573967+0000","last_fullsized":"2024-03-11T22:20:46.573967+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:20.907743+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.907743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"314'9526","reported_seq":9619,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:51.135568+0000","last_change":"2024-03-11T22:18:41.783594+0000","last_active":"2024-03-11T22:20:51.135568+0000","last_peered":"2024-03-11T22:20:51.135568+0000","last_clean":"2024-03-11T22:20:51.135568+0000","last_became_active":"2024-03-11T22:18:41.783248+0000","last_became_peered":"2024-03-11T22:18:41.783248+0000","last_unstale":"2024-03-11T22:20:51.135568+0000","last_undegraded":"2024-03-11T22:20:51.135568+0000","last_fullsized":"2024-03-11T22:20:51.135568+0000","mapping_epoch":273,"log_start":"295'5000","ondisk_log_start":"295'5000","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4526,"log_dups_size":0,"ondisk_log_size":4526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395444224,"num_objects":6034,"num_object_clones":0,"num_object_copies":18102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9526,"num_write_kb":497920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"273'31","reported_seq":390,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.549679+0000","last_change":"2024-03-11T22:20:40.790500+0000","last_active":"2024-03-11T22:20:46.549679+0000","last_peered":"2024-03-11T22:20:46.549679+0000","last_clean":"2024-03-11T22:20:46.549679+0000","last_became_active":"2024-03-11T22:14:52.239649+0000","last_became_peered":"2024-03-11T22:14:52.239649+0000","last_unstale":"2024-03-11T22:20:46.549679+0000","last_undegraded":"2024-03-11T22:20:46.549679+0000","last_fullsized":"2024-03-11T22:20:46.549679+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'31","last_scrub_stamp":"2024-03-11T22:20:40.790339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:40.790339+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:48.818452+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":566,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.618539+0000","last_change":"2024-03-11T22:20:52.712468+0000","last_active":"2024-03-11T22:20:54.618539+0000","last_peered":"2024-03-11T22:20:54.618539+0000","last_clean":"2024-03-11T22:20:54.618539+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:20:54.618539+0000","last_undegraded":"2024-03-11T22:20:54.618539+0000","last_fullsized":"2024-03-11T22:20:54.618539+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:52.712329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:20:52.712329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:11.261543+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":540,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.557729+0000","last_change":"2024-03-11T22:18:44.130366+0000","last_active":"2024-03-11T22:20:46.557729+0000","last_peered":"2024-03-11T22:20:46.557729+0000","last_clean":"2024-03-11T22:20:46.557729+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:20:46.557729+0000","last_undegraded":"2024-03-11T22:20:46.557729+0000","last_fullsized":"2024-03-11T22:20:46.557729+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:44.130252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:18:44.130252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":544,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.550127+0000","last_change":"2024-03-11T22:18:51.794420+0000","last_active":"2024-03-11T22:20:46.550127+0000","last_peered":"2024-03-11T22:20:46.550127+0000","last_clean":"2024-03-11T22:20:46.550127+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:20:46.550127+0000","last_undegraded":"2024-03-11T22:20:46.550127+0000","last_fullsized":"2024-03-11T22:20:46.550127+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:51.794313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:51.794313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":841,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.611107+0000","last_change":"2024-03-11T22:18:21.646464+0000","last_active":"2024-03-11T22:20:54.611107+0000","last_peered":"2024-03-11T22:20:54.611107+0000","last_clean":"2024-03-11T22:20:54.611107+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:20:54.611107+0000","last_undegraded":"2024-03-11T22:20:54.611107+0000","last_fullsized":"2024-03-11T22:20:54.611107+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:21.646305+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.646305+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"314'9509","reported_seq":9651,"reported_epoch":314,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-11T22:20:51.150783+0000","last_change":"2024-03-11T22:20:30.647673+0000","last_active":"2024-03-11T22:20:51.150783+0000","last_peered":"2024-03-11T22:20:51.150783+0000","last_clean":"2024-03-11T22:20:28.402388+0000","last_became_active":"2024-03-11T22:20:30.515089+0000","last_became_peered":"2024-03-11T22:20:30.515089+0000","last_unstale":"2024-03-11T22:20:51.150783+0000","last_undegraded":"2024-03-11T22:20:51.150783+0000","last_fullsized":"2024-03-11T22:20:51.150783+0000","mapping_epoch":310,"log_start":"298'5400","ondisk_log_start":"298'5400","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405471232,"num_objects":6187,"num_object_clones":0,"num_object_copies":18561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6174,"num_objects_unfound":0,"num_objects_dirty":6187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9509,"num_write_kb":502272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,1,5],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":6187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"273'19","reported_seq":366,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.618953+0000","last_change":"2024-03-11T22:20:27.089007+0000","last_active":"2024-03-11T22:20:54.618953+0000","last_peered":"2024-03-11T22:20:54.618953+0000","last_clean":"2024-03-11T22:20:54.618953+0000","last_became_active":"2024-03-11T22:14:52.240437+0000","last_became_peered":"2024-03-11T22:14:52.240437+0000","last_unstale":"2024-03-11T22:20:54.618953+0000","last_undegraded":"2024-03-11T22:20:54.618953+0000","last_fullsized":"2024-03-11T22:20:54.618953+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:20:27.088856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:27.088856+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:30.994147+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":323,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.549821+0000","last_change":"2024-03-11T22:20:45.895055+0000","last_active":"2024-03-11T22:20:46.549821+0000","last_peered":"2024-03-11T22:20:46.549821+0000","last_clean":"2024-03-11T22:20:46.549821+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:20:46.549821+0000","last_undegraded":"2024-03-11T22:20:46.549821+0000","last_fullsized":"2024-03-11T22:20:46.549821+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:45.894940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:45.894940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:50.593037+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":534,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.613261+0000","last_change":"2024-03-11T22:18:37.705396+0000","last_active":"2024-03-11T22:20:54.613261+0000","last_peered":"2024-03-11T22:20:54.613261+0000","last_clean":"2024-03-11T22:20:54.613261+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:20:54.613261+0000","last_undegraded":"2024-03-11T22:20:54.613261+0000","last_fullsized":"2024-03-11T22:20:54.613261+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:37.705305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.705305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":520,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.574513+0000","last_change":"2024-03-11T22:18:48.086934+0000","last_active":"2024-03-11T22:20:46.574513+0000","last_peered":"2024-03-11T22:20:46.574513+0000","last_clean":"2024-03-11T22:20:46.574513+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:20:46.574513+0000","last_undegraded":"2024-03-11T22:20:46.574513+0000","last_fullsized":"2024-03-11T22:20:46.574513+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:48.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:48.086856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"93'130","reported_seq":867,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.554738+0000","last_change":"2024-03-11T22:18:20.968124+0000","last_active":"2024-03-11T22:20:46.554738+0000","last_peered":"2024-03-11T22:20:46.554738+0000","last_clean":"2024-03-11T22:20:46.554738+0000","last_became_active":"2024-03-11T22:18:00.271764+0000","last_became_peered":"2024-03-11T22:18:00.271764+0000","last_unstale":"2024-03-11T22:20:46.554738+0000","last_undegraded":"2024-03-11T22:20:46.554738+0000","last_fullsized":"2024-03-11T22:20:46.554738+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:20.968038+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.968038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"314'9399","reported_seq":9491,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:51.154631+0000","last_change":"2024-03-11T22:18:41.238608+0000","last_active":"2024-03-11T22:20:51.154631+0000","last_peered":"2024-03-11T22:20:51.154631+0000","last_clean":"2024-03-11T22:20:51.154631+0000","last_became_active":"2024-03-11T22:18:41.238269+0000","last_became_peered":"2024-03-11T22:18:41.238269+0000","last_unstale":"2024-03-11T22:20:51.154631+0000","last_undegraded":"2024-03-11T22:20:51.154631+0000","last_fullsized":"2024-03-11T22:20:51.154631+0000","mapping_epoch":273,"log_start":"298'5300","ondisk_log_start":"298'5300","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387776512,"num_objects":5917,"num_object_clones":0,"num_object_copies":17751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9399,"num_write_kb":490112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"273'34","reported_seq":602,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.557897+0000","last_change":"2024-03-11T22:20:26.599126+0000","last_active":"2024-03-11T22:20:46.557897+0000","last_peered":"2024-03-11T22:20:46.557897+0000","last_clean":"2024-03-11T22:20:46.557897+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:20:46.557897+0000","last_undegraded":"2024-03-11T22:20:46.557897+0000","last_fullsized":"2024-03-11T22:20:46.557897+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:20:26.598991+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:20:26.598991+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:48.861807+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":544,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.569648+0000","last_change":"2024-03-11T22:20:29.432709+0000","last_active":"2024-03-11T22:20:46.569648+0000","last_peered":"2024-03-11T22:20:46.569648+0000","last_clean":"2024-03-11T22:20:46.569648+0000","last_became_active":"2024-03-11T22:20:29.426224+0000","last_became_peered":"2024-03-11T22:20:29.426224+0000","last_unstale":"2024-03-11T22:20:46.569648+0000","last_undegraded":"2024-03-11T22:20:46.569648+0000","last_fullsized":"2024-03-11T22:20:46.569648+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:38.013058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:38.013058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":399,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.612155+0000","last_change":"2024-03-11T22:15:39.663447+0000","last_active":"2024-03-11T22:20:54.612155+0000","last_peered":"2024-03-11T22:20:54.612155+0000","last_clean":"2024-03-11T22:20:54.612155+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:20:54.612155+0000","last_undegraded":"2024-03-11T22:20:54.612155+0000","last_fullsized":"2024-03-11T22:20:54.612155+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:15:39.663324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:39.663324+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":530,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.612125+0000","last_change":"2024-03-11T22:18:50.805816+0000","last_active":"2024-03-11T22:20:54.612125+0000","last_peered":"2024-03-11T22:20:54.612125+0000","last_clean":"2024-03-11T22:20:54.612125+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:20:54.612125+0000","last_undegraded":"2024-03-11T22:20:54.612125+0000","last_fullsized":"2024-03-11T22:20:54.612125+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:50.805698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:18:50.805698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":842,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.611936+0000","last_change":"2024-03-11T22:18:20.663763+0000","last_active":"2024-03-11T22:20:54.611936+0000","last_peered":"2024-03-11T22:20:54.611936+0000","last_clean":"2024-03-11T22:20:54.611936+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:20:54.611936+0000","last_undegraded":"2024-03-11T22:20:54.611936+0000","last_fullsized":"2024-03-11T22:20:54.611936+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:20.663661+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.663661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"314'9517","reported_seq":9615,"reported_epoch":314,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-11T22:20:51.115099+0000","last_change":"2024-03-11T22:20:30.592868+0000","last_active":"2024-03-11T22:20:51.115099+0000","last_peered":"2024-03-11T22:20:51.115099+0000","last_clean":"2024-03-11T22:20:28.382555+0000","last_became_active":"2024-03-11T22:20:30.580794+0000","last_became_peered":"2024-03-11T22:20:30.580794+0000","last_unstale":"2024-03-11T22:20:51.115099+0000","last_undegraded":"2024-03-11T22:20:51.115099+0000","last_fullsized":"2024-03-11T22:20:51.115099+0000","mapping_epoch":310,"log_start":"295'5000","ondisk_log_start":"295'5000","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4517,"log_dups_size":0,"ondisk_log_size":4517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390201368,"num_objects":5955,"num_object_clones":0,"num_object_copies":17865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5955,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9517,"num_write_kb":495041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,5,4],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":5955}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":316,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.576204+0000","last_change":"2024-03-11T22:20:29.441577+0000","last_active":"2024-03-11T22:20:46.576204+0000","last_peered":"2024-03-11T22:20:46.576204+0000","last_clean":"2024-03-11T22:20:46.576204+0000","last_became_active":"2024-03-11T22:20:29.438896+0000","last_became_peered":"2024-03-11T22:20:29.438896+0000","last_unstale":"2024-03-11T22:20:46.576204+0000","last_undegraded":"2024-03-11T22:20:46.576204+0000","last_fullsized":"2024-03-11T22:20:46.576204+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.293977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.293977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"273'30","reported_seq":630,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.569742+0000","last_change":"2024-03-11T22:20:29.473203+0000","last_active":"2024-03-11T22:20:46.569742+0000","last_peered":"2024-03-11T22:20:46.569742+0000","last_clean":"2024-03-11T22:20:46.569742+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:20:46.569742+0000","last_undegraded":"2024-03-11T22:20:46.569742+0000","last_fullsized":"2024-03-11T22:20:46.569742+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:15:33.497732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:33.497732+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":552,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.572977+0000","last_change":"2024-03-11T22:20:29.441068+0000","last_active":"2024-03-11T22:20:46.572977+0000","last_peered":"2024-03-11T22:20:46.572977+0000","last_clean":"2024-03-11T22:20:46.572977+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:20:46.572977+0000","last_undegraded":"2024-03-11T22:20:46.572977+0000","last_fullsized":"2024-03-11T22:20:46.572977+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.133631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.133631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":298,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.573540+0000","last_change":"2024-03-11T22:15:12.339167+0000","last_active":"2024-03-11T22:20:46.573540+0000","last_peered":"2024-03-11T22:20:46.573540+0000","last_clean":"2024-03-11T22:20:46.573540+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:20:46.573540+0000","last_undegraded":"2024-03-11T22:20:46.573540+0000","last_fullsized":"2024-03-11T22:20:46.573540+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:12.339050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:12.339050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":792,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.549985+0000","last_change":"2024-03-11T22:18:36.683358+0000","last_active":"2024-03-11T22:20:46.549985+0000","last_peered":"2024-03-11T22:20:46.549985+0000","last_clean":"2024-03-11T22:20:46.549985+0000","last_became_active":"2024-03-11T22:18:20.240728+0000","last_became_peered":"2024-03-11T22:18:20.240728+0000","last_unstale":"2024-03-11T22:20:46.549985+0000","last_undegraded":"2024-03-11T22:20:46.549985+0000","last_fullsized":"2024-03-11T22:20:46.549985+0000","mapping_epoch":175,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:36.683219+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:36.683219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.c","version":"314'9386","reported_seq":9477,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:51.149955+0000","last_change":"2024-03-11T22:18:41.260878+0000","last_active":"2024-03-11T22:20:51.149955+0000","last_peered":"2024-03-11T22:20:51.149955+0000","last_clean":"2024-03-11T22:20:51.149955+0000","last_became_active":"2024-03-11T22:18:41.260611+0000","last_became_peered":"2024-03-11T22:18:41.260611+0000","last_unstale":"2024-03-11T22:20:51.149955+0000","last_undegraded":"2024-03-11T22:20:51.149955+0000","last_fullsized":"2024-03-11T22:20:51.149955+0000","mapping_epoch":273,"log_start":"298'5300","ondisk_log_start":"298'5300","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4086,"log_dups_size":0,"ondisk_log_size":4086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396230656,"num_objects":6046,"num_object_clones":0,"num_object_copies":18138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9386,"num_write_kb":493824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":681,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.569898+0000","last_change":"2024-03-11T22:15:31.519229+0000","last_active":"2024-03-11T22:20:46.569898+0000","last_peered":"2024-03-11T22:20:46.569898+0000","last_clean":"2024-03-11T22:20:46.569898+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:20:46.569898+0000","last_undegraded":"2024-03-11T22:20:46.569898+0000","last_fullsized":"2024-03-11T22:20:46.569898+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:15:31.513336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:31.513336+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"273'28","reported_seq":361,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.611105+0000","last_change":"2024-03-11T22:15:36.626072+0000","last_active":"2024-03-11T22:20:54.611105+0000","last_peered":"2024-03-11T22:20:54.611105+0000","last_clean":"2024-03-11T22:20:54.611105+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:20:54.611105+0000","last_undegraded":"2024-03-11T22:20:54.611105+0000","last_fullsized":"2024-03-11T22:20:54.611105+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:15:36.625916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:36.625916+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":541,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.547975+0000","last_change":"2024-03-11T22:19:28.428409+0000","last_active":"2024-03-11T22:20:46.547975+0000","last_peered":"2024-03-11T22:20:46.547975+0000","last_clean":"2024-03-11T22:20:46.547975+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:20:46.547975+0000","last_undegraded":"2024-03-11T22:20:46.547975+0000","last_fullsized":"2024-03-11T22:20:46.547975+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:19:28.428283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:28.428283+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":488,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.619704+0000","last_change":"2024-03-11T22:19:35.746810+0000","last_active":"2024-03-11T22:20:54.619704+0000","last_peered":"2024-03-11T22:20:54.619704+0000","last_clean":"2024-03-11T22:20:54.619704+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:20:54.619704+0000","last_undegraded":"2024-03-11T22:20:54.619704+0000","last_fullsized":"2024-03-11T22:20:54.619704+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:35.746693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:19:35.746693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":856,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.574360+0000","last_change":"2024-03-11T22:20:32.293956+0000","last_active":"2024-03-11T22:20:46.574360+0000","last_peered":"2024-03-11T22:20:46.574360+0000","last_clean":"2024-03-11T22:20:46.574360+0000","last_became_active":"2024-03-11T22:20:32.293085+0000","last_became_peered":"2024-03-11T22:20:32.293085+0000","last_unstale":"2024-03-11T22:20:46.574360+0000","last_undegraded":"2024-03-11T22:20:46.574360+0000","last_fullsized":"2024-03-11T22:20:46.574360+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:18.905939+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:18.905939+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"315'9842","reported_seq":9937,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:55.032176+0000","last_change":"2024-03-11T22:18:41.236691+0000","last_active":"2024-03-11T22:20:55.032176+0000","last_peered":"2024-03-11T22:20:55.032176+0000","last_clean":"2024-03-11T22:20:55.032176+0000","last_became_active":"2024-03-11T22:18:41.236462+0000","last_became_peered":"2024-03-11T22:18:41.236462+0000","last_unstale":"2024-03-11T22:20:55.032176+0000","last_undegraded":"2024-03-11T22:20:55.032176+0000","last_fullsized":"2024-03-11T22:20:55.032176+0000","mapping_epoch":273,"log_start":"296'5300","ondisk_log_start":"296'5300","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389414912,"num_objects":5942,"num_object_clones":0,"num_object_copies":17826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9842,"num_write_kb":505088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":688,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.549296+0000","last_change":"2024-03-11T22:18:35.815130+0000","last_active":"2024-03-11T22:20:46.549296+0000","last_peered":"2024-03-11T22:20:46.549296+0000","last_clean":"2024-03-11T22:20:46.549296+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:20:46.549296+0000","last_undegraded":"2024-03-11T22:20:46.549296+0000","last_fullsized":"2024-03-11T22:20:46.549296+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:18:35.815031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:35.815031+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"273'20","reported_seq":582,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.555663+0000","last_change":"2024-03-11T22:19:31.503395+0000","last_active":"2024-03-11T22:20:46.555663+0000","last_peered":"2024-03-11T22:20:46.555663+0000","last_clean":"2024-03-11T22:20:46.555663+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:20:46.555663+0000","last_undegraded":"2024-03-11T22:20:46.555663+0000","last_fullsized":"2024-03-11T22:20:46.555663+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:19:31.502993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:31.502993+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:53.451821+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":550,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.612280+0000","last_change":"2024-03-11T22:19:34.706343+0000","last_active":"2024-03-11T22:20:54.612280+0000","last_peered":"2024-03-11T22:20:54.612280+0000","last_clean":"2024-03-11T22:20:54.612280+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:20:54.612280+0000","last_undegraded":"2024-03-11T22:20:54.612280+0000","last_fullsized":"2024-03-11T22:20:54.612280+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.706222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.706222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":529,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.547001+0000","last_change":"2024-03-11T22:19:34.477472+0000","last_active":"2024-03-11T22:20:46.547001+0000","last_peered":"2024-03-11T22:20:46.547001+0000","last_clean":"2024-03-11T22:20:46.547001+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:20:46.547001+0000","last_undegraded":"2024-03-11T22:20:46.547001+0000","last_fullsized":"2024-03-11T22:20:46.547001+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.477349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.477349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":783,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.619300+0000","last_change":"2024-03-11T22:20:43.803186+0000","last_active":"2024-03-11T22:20:54.619300+0000","last_peered":"2024-03-11T22:20:54.619300+0000","last_clean":"2024-03-11T22:20:54.619300+0000","last_became_active":"2024-03-11T22:20:32.290512+0000","last_became_peered":"2024-03-11T22:20:32.290512+0000","last_unstale":"2024-03-11T22:20:54.619300+0000","last_undegraded":"2024-03-11T22:20:54.619300+0000","last_fullsized":"2024-03-11T22:20:54.619300+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:43.803067+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:20:43.803067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:49.661991+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"314'9435","reported_seq":14901,"reported_epoch":314,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T22:20:51.150509+0000","last_change":"2024-03-11T22:20:30.553695+0000","last_active":"2024-03-11T22:20:51.150509+0000","last_peered":"2024-03-11T22:20:51.150509+0000","last_clean":"2024-03-11T22:20:28.378192+0000","last_became_active":"2024-03-11T22:20:30.544603+0000","last_became_peered":"2024-03-11T22:20:30.544603+0000","last_unstale":"2024-03-11T22:20:51.150509+0000","last_undegraded":"2024-03-11T22:20:51.150509+0000","last_fullsized":"2024-03-11T22:20:51.150509+0000","mapping_epoch":310,"log_start":"295'4900","ondisk_log_start":"295'4900","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390529024,"num_objects":5959,"num_object_clones":0,"num_object_copies":17877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3528,"num_objects_unfound":0,"num_objects_dirty":5959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9435,"num_write_kb":492608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2684,"num_bytes_recovered":175898624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,4,0],"avail_no_missing":["1","0","4"],"object_location_counts":[{"shards":"0,1,4","objects":5959}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":550,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.618456+0000","last_change":"2024-03-11T22:19:43.625744+0000","last_active":"2024-03-11T22:20:54.618456+0000","last_peered":"2024-03-11T22:20:54.618456+0000","last_clean":"2024-03-11T22:20:54.618456+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:20:54.618456+0000","last_undegraded":"2024-03-11T22:20:54.618456+0000","last_fullsized":"2024-03-11T22:20:54.618456+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:43.625634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:19:43.625634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"273'29","reported_seq":587,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.549094+0000","last_change":"2024-03-11T22:19:38.450843+0000","last_active":"2024-03-11T22:20:46.549094+0000","last_peered":"2024-03-11T22:20:46.549094+0000","last_clean":"2024-03-11T22:20:46.549094+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:20:46.549094+0000","last_undegraded":"2024-03-11T22:20:46.549094+0000","last_fullsized":"2024-03-11T22:20:46.549094+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'29","last_scrub_stamp":"2024-03-11T22:19:38.450581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.450581+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:05.577079+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":548,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.557428+0000","last_change":"2024-03-11T22:18:38.153695+0000","last_active":"2024-03-11T22:20:46.557428+0000","last_peered":"2024-03-11T22:20:46.557428+0000","last_clean":"2024-03-11T22:20:46.557428+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:20:46.557428+0000","last_undegraded":"2024-03-11T22:20:46.557428+0000","last_fullsized":"2024-03-11T22:20:46.557428+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:38.153614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.153614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":536,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.552194+0000","last_change":"2024-03-11T22:18:23.998360+0000","last_active":"2024-03-11T22:20:46.552194+0000","last_peered":"2024-03-11T22:20:46.552194+0000","last_clean":"2024-03-11T22:20:46.552194+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:20:46.552194+0000","last_undegraded":"2024-03-11T22:20:46.552194+0000","last_fullsized":"2024-03-11T22:20:46.552194+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:23.998251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:23.998251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":758,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.619214+0000","last_change":"2024-03-11T22:20:44.849410+0000","last_active":"2024-03-11T22:20:54.619214+0000","last_peered":"2024-03-11T22:20:54.619214+0000","last_clean":"2024-03-11T22:20:54.619214+0000","last_became_active":"2024-03-11T22:18:17.238247+0000","last_became_peered":"2024-03-11T22:18:17.238247+0000","last_unstale":"2024-03-11T22:20:54.619214+0000","last_undegraded":"2024-03-11T22:20:54.619214+0000","last_fullsized":"2024-03-11T22:20:54.619214+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:44.849279+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:20:44.849279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:04.442646+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"314'9336","reported_seq":9428,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:51.780464+0000","last_change":"2024-03-11T22:18:41.783592+0000","last_active":"2024-03-11T22:20:51.780464+0000","last_peered":"2024-03-11T22:20:51.780464+0000","last_clean":"2024-03-11T22:20:51.780464+0000","last_became_active":"2024-03-11T22:18:41.783277+0000","last_became_peered":"2024-03-11T22:18:41.783277+0000","last_unstale":"2024-03-11T22:20:51.780464+0000","last_undegraded":"2024-03-11T22:20:51.780464+0000","last_fullsized":"2024-03-11T22:20:51.780464+0000","mapping_epoch":273,"log_start":"291'4100","ondisk_log_start":"291'4100","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389808128,"num_objects":5948,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9336,"num_write_kb":489088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":534,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.613343+0000","last_change":"2024-03-11T22:18:25.668662+0000","last_active":"2024-03-11T22:20:54.613343+0000","last_peered":"2024-03-11T22:20:54.613343+0000","last_clean":"2024-03-11T22:20:54.613343+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:20:54.613343+0000","last_undegraded":"2024-03-11T22:20:54.613343+0000","last_fullsized":"2024-03-11T22:20:54.613343+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:25.668581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:25.668581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"273'24","reported_seq":580,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.547876+0000","last_change":"2024-03-11T22:20:29.380624+0000","last_active":"2024-03-11T22:20:46.547876+0000","last_peered":"2024-03-11T22:20:46.547876+0000","last_clean":"2024-03-11T22:20:46.547876+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:20:46.547876+0000","last_undegraded":"2024-03-11T22:20:46.547876+0000","last_fullsized":"2024-03-11T22:20:46.547876+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'24","last_scrub_stamp":"2024-03-11T22:20:29.380527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:29.380527+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:49.652759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":1267,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:49.446301+0000","last_change":"2024-03-11T22:18:27.077438+0000","last_active":"2024-03-11T22:20:49.446301+0000","last_peered":"2024-03-11T22:20:49.446301+0000","last_clean":"2024-03-11T22:20:49.446301+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:20:49.446301+0000","last_undegraded":"2024-03-11T22:20:49.446301+0000","last_fullsized":"2024-03-11T22:20:49.446301+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:18:27.077275+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.077275+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":703,"num_read_kb":698,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":540,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.546668+0000","last_change":"2024-03-11T22:18:28.722689+0000","last_active":"2024-03-11T22:20:46.546668+0000","last_peered":"2024-03-11T22:20:46.546668+0000","last_clean":"2024-03-11T22:20:46.546668+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:20:46.546668+0000","last_undegraded":"2024-03-11T22:20:46.546668+0000","last_fullsized":"2024-03-11T22:20:46.546668+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:28.722615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:28.722615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":544,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.554235+0000","last_change":"2024-03-11T22:18:36.127513+0000","last_active":"2024-03-11T22:20:46.554235+0000","last_peered":"2024-03-11T22:20:46.554235+0000","last_clean":"2024-03-11T22:20:46.554235+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:20:46.554235+0000","last_undegraded":"2024-03-11T22:20:46.554235+0000","last_fullsized":"2024-03-11T22:20:46.554235+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:36.127435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:36.127435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":846,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.554970+0000","last_change":"2024-03-11T22:19:29.435468+0000","last_active":"2024-03-11T22:20:46.554970+0000","last_peered":"2024-03-11T22:20:46.554970+0000","last_clean":"2024-03-11T22:20:46.554970+0000","last_became_active":"2024-03-11T22:14:38.164101+0000","last_became_peered":"2024-03-11T22:14:38.164101+0000","last_unstale":"2024-03-11T22:20:46.554970+0000","last_undegraded":"2024-03-11T22:20:46.554970+0000","last_fullsized":"2024-03-11T22:20:46.554970+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:29.435334+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.435334+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:50.439118+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"314'9571","reported_seq":9663,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:51.138227+0000","last_change":"2024-03-11T22:18:41.240089+0000","last_active":"2024-03-11T22:20:51.138227+0000","last_peered":"2024-03-11T22:20:51.138227+0000","last_clean":"2024-03-11T22:20:51.138227+0000","last_became_active":"2024-03-11T22:18:41.239829+0000","last_became_peered":"2024-03-11T22:18:41.239829+0000","last_unstale":"2024-03-11T22:20:51.138227+0000","last_undegraded":"2024-03-11T22:20:51.138227+0000","last_fullsized":"2024-03-11T22:20:51.138227+0000","mapping_epoch":273,"log_start":"298'5500","ondisk_log_start":"298'5500","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4071,"log_dups_size":0,"ondisk_log_size":4071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394067968,"num_objects":6013,"num_object_clones":0,"num_object_copies":18039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9571,"num_write_kb":498688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":506,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.551309+0000","last_change":"2024-03-11T22:20:29.420829+0000","last_active":"2024-03-11T22:20:46.551309+0000","last_peered":"2024-03-11T22:20:46.551309+0000","last_clean":"2024-03-11T22:20:46.551309+0000","last_became_active":"2024-03-11T22:20:29.420497+0000","last_became_peered":"2024-03-11T22:20:29.420497+0000","last_unstale":"2024-03-11T22:20:46.551309+0000","last_undegraded":"2024-03-11T22:20:46.551309+0000","last_fullsized":"2024-03-11T22:20:46.551309+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:24.875284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:19:24.875284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"273'28","reported_seq":743,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.618969+0000","last_change":"2024-03-11T22:19:34.671571+0000","last_active":"2024-03-11T22:20:54.618969+0000","last_peered":"2024-03-11T22:20:54.618969+0000","last_clean":"2024-03-11T22:20:54.618969+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:20:54.618969+0000","last_undegraded":"2024-03-11T22:20:54.618969+0000","last_fullsized":"2024-03-11T22:20:54.618969+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:19:34.671335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.671335+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":540,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.569767+0000","last_change":"2024-03-11T22:19:24.533661+0000","last_active":"2024-03-11T22:20:46.569767+0000","last_peered":"2024-03-11T22:20:46.569767+0000","last_clean":"2024-03-11T22:20:46.569767+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:20:46.569767+0000","last_undegraded":"2024-03-11T22:20:46.569767+0000","last_fullsized":"2024-03-11T22:20:46.569767+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:24.533532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:19:24.533532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":856,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.547110+0000","last_change":"2024-03-11T22:20:32.294325+0000","last_active":"2024-03-11T22:20:46.547110+0000","last_peered":"2024-03-11T22:20:46.547110+0000","last_clean":"2024-03-11T22:20:46.547110+0000","last_became_active":"2024-03-11T22:20:32.293740+0000","last_became_peered":"2024-03-11T22:20:32.293740+0000","last_unstale":"2024-03-11T22:20:46.547110+0000","last_undegraded":"2024-03-11T22:20:46.547110+0000","last_fullsized":"2024-03-11T22:20:46.547110+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:44.952136+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:18:44.952136+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":530,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.613196+0000","last_change":"2024-03-11T22:18:22.688031+0000","last_active":"2024-03-11T22:20:54.613196+0000","last_peered":"2024-03-11T22:20:54.613196+0000","last_clean":"2024-03-11T22:20:54.613196+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:20:54.613196+0000","last_undegraded":"2024-03-11T22:20:54.613196+0000","last_fullsized":"2024-03-11T22:20:54.613196+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:22.687922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.687922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"315'9830","reported_seq":9925,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.959941+0000","last_change":"2024-03-11T22:18:41.260551+0000","last_active":"2024-03-11T22:20:54.959941+0000","last_peered":"2024-03-11T22:20:54.959941+0000","last_clean":"2024-03-11T22:20:54.959941+0000","last_became_active":"2024-03-11T22:18:41.260185+0000","last_became_peered":"2024-03-11T22:18:41.260185+0000","last_unstale":"2024-03-11T22:20:54.959941+0000","last_undegraded":"2024-03-11T22:20:54.959941+0000","last_fullsized":"2024-03-11T22:20:54.959941+0000","mapping_epoch":273,"log_start":"298'5300","ondisk_log_start":"298'5300","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386400256,"num_objects":5896,"num_object_clones":0,"num_object_copies":17688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9830,"num_write_kb":503232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":556,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.550334+0000","last_change":"2024-03-11T22:19:37.032204+0000","last_active":"2024-03-11T22:20:46.550334+0000","last_peered":"2024-03-11T22:20:46.550334+0000","last_clean":"2024-03-11T22:20:46.550334+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:20:46.550334+0000","last_undegraded":"2024-03-11T22:20:46.550334+0000","last_fullsized":"2024-03-11T22:20:46.550334+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.032058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.032058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:54.420326+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"273'17","reported_seq":585,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.558114+0000","last_change":"2024-03-11T22:19:34.486422+0000","last_active":"2024-03-11T22:20:46.558114+0000","last_peered":"2024-03-11T22:20:46.558114+0000","last_clean":"2024-03-11T22:20:46.558114+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:20:46.558114+0000","last_undegraded":"2024-03-11T22:20:46.558114+0000","last_fullsized":"2024-03-11T22:20:46.558114+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'17","last_scrub_stamp":"2024-03-11T22:19:34.486239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.486239+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:49.516370+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":573,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.549932+0000","last_change":"2024-03-11T22:18:31.559422+0000","last_active":"2024-03-11T22:20:46.549932+0000","last_peered":"2024-03-11T22:20:46.549932+0000","last_clean":"2024-03-11T22:20:46.549932+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:20:46.549932+0000","last_undegraded":"2024-03-11T22:20:46.549932+0000","last_fullsized":"2024-03-11T22:20:46.549932+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:31.559330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:18:31.559330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":557,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.553034+0000","last_change":"2024-03-11T22:19:46.587019+0000","last_active":"2024-03-11T22:20:46.553034+0000","last_peered":"2024-03-11T22:20:46.553034+0000","last_clean":"2024-03-11T22:20:46.553034+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:20:46.553034+0000","last_undegraded":"2024-03-11T22:20:46.553034+0000","last_fullsized":"2024-03-11T22:20:46.553034+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:19:46.586446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:48.778337+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":488,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.618998+0000","last_change":"2024-03-11T22:19:37.776868+0000","last_active":"2024-03-11T22:20:54.618998+0000","last_peered":"2024-03-11T22:20:54.618998+0000","last_clean":"2024-03-11T22:20:54.618998+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:20:54.618998+0000","last_undegraded":"2024-03-11T22:20:54.618998+0000","last_fullsized":"2024-03-11T22:20:54.618998+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.776600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.776600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:58.256015+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":863,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.546620+0000","last_change":"2024-03-11T22:20:32.295561+0000","last_active":"2024-03-11T22:20:46.546620+0000","last_peered":"2024-03-11T22:20:46.546620+0000","last_clean":"2024-03-11T22:20:46.546620+0000","last_became_active":"2024-03-11T22:20:32.295175+0000","last_became_peered":"2024-03-11T22:20:32.295175+0000","last_unstale":"2024-03-11T22:20:46.546620+0000","last_undegraded":"2024-03-11T22:20:46.546620+0000","last_fullsized":"2024-03-11T22:20:46.546620+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:25.392244+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:25.392244+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:48.078541+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"314'9519","reported_seq":9613,"reported_epoch":314,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-11T22:20:51.774280+0000","last_change":"2024-03-11T22:20:30.589165+0000","last_active":"2024-03-11T22:20:51.774280+0000","last_peered":"2024-03-11T22:20:51.774280+0000","last_clean":"2024-03-11T22:20:28.362243+0000","last_became_active":"2024-03-11T22:20:30.567189+0000","last_became_peered":"2024-03-11T22:20:30.567189+0000","last_unstale":"2024-03-11T22:20:51.774280+0000","last_undegraded":"2024-03-11T22:20:51.774280+0000","last_fullsized":"2024-03-11T22:20:51.774280+0000","mapping_epoch":310,"log_start":"292'4300","ondisk_log_start":"292'4300","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5219,"log_dups_size":0,"ondisk_log_size":5219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390791168,"num_objects":5963,"num_object_clones":0,"num_object_copies":17889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5963,"num_objects_unfound":0,"num_objects_dirty":5963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9519,"num_write_kb":495424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,5,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":5963}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":532,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.546736+0000","last_change":"2024-03-11T22:15:30.922162+0000","last_active":"2024-03-11T22:20:46.546736+0000","last_peered":"2024-03-11T22:20:46.546736+0000","last_clean":"2024-03-11T22:20:46.546736+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:20:46.546736+0000","last_undegraded":"2024-03-11T22:20:46.546736+0000","last_fullsized":"2024-03-11T22:20:46.546736+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:30.922001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:30.922001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"273'40","reported_seq":766,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.557639+0000","last_change":"2024-03-11T22:19:39.417690+0000","last_active":"2024-03-11T22:20:46.557639+0000","last_peered":"2024-03-11T22:20:46.557639+0000","last_clean":"2024-03-11T22:20:46.557639+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:20:46.557639+0000","last_undegraded":"2024-03-11T22:20:46.557639+0000","last_fullsized":"2024-03-11T22:20:46.557639+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'40","last_scrub_stamp":"2024-03-11T22:19:39.417411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:39.417411+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":299,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.549036+0000","last_change":"2024-03-11T22:15:24.899285+0000","last_active":"2024-03-11T22:20:46.549036+0000","last_peered":"2024-03-11T22:20:46.549036+0000","last_clean":"2024-03-11T22:20:46.549036+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:20:46.549036+0000","last_undegraded":"2024-03-11T22:20:46.549036+0000","last_fullsized":"2024-03-11T22:20:46.549036+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:24.899168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:24.899168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":554,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.612191+0000","last_change":"2024-03-11T22:18:36.684061+0000","last_active":"2024-03-11T22:20:54.612191+0000","last_peered":"2024-03-11T22:20:54.612191+0000","last_clean":"2024-03-11T22:20:54.612191+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:20:54.612191+0000","last_undegraded":"2024-03-11T22:20:54.612191+0000","last_fullsized":"2024-03-11T22:20:54.612191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:18:36.683930+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:18:36.683930+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":532,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.570564+0000","last_change":"2024-03-11T22:20:29.434967+0000","last_active":"2024-03-11T22:20:46.570564+0000","last_peered":"2024-03-11T22:20:46.570564+0000","last_clean":"2024-03-11T22:20:46.570564+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:20:46.570564+0000","last_undegraded":"2024-03-11T22:20:46.570564+0000","last_fullsized":"2024-03-11T22:20:46.570564+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:32.518054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:15:32.518054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":839,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.610959+0000","last_change":"2024-03-11T22:18:41.790977+0000","last_active":"2024-03-11T22:20:54.610959+0000","last_peered":"2024-03-11T22:20:54.610959+0000","last_clean":"2024-03-11T22:20:54.610959+0000","last_became_active":"2024-03-11T22:14:38.164859+0000","last_became_peered":"2024-03-11T22:14:38.164859+0000","last_unstale":"2024-03-11T22:20:54.610959+0000","last_undegraded":"2024-03-11T22:20:54.610959+0000","last_fullsized":"2024-03-11T22:20:54.610959+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:41.790896+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:41.790896+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"315'9438","reported_seq":9533,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.891065+0000","last_change":"2024-03-11T22:18:41.233217+0000","last_active":"2024-03-11T22:20:54.891065+0000","last_peered":"2024-03-11T22:20:54.891065+0000","last_clean":"2024-03-11T22:20:54.891065+0000","last_became_active":"2024-03-11T22:18:41.232989+0000","last_became_peered":"2024-03-11T22:18:41.232989+0000","last_unstale":"2024-03-11T22:20:54.891065+0000","last_undegraded":"2024-03-11T22:20:54.891065+0000","last_fullsized":"2024-03-11T22:20:54.891065+0000","mapping_epoch":273,"log_start":"295'4900","ondisk_log_start":"295'4900","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363462656,"num_objects":5546,"num_object_clones":0,"num_object_copies":16638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9438,"num_write_kb":479488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"273'15","reported_seq":577,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.549225+0000","last_change":"2024-03-11T22:19:32.486126+0000","last_active":"2024-03-11T22:20:46.549225+0000","last_peered":"2024-03-11T22:20:46.549225+0000","last_clean":"2024-03-11T22:20:46.549225+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:20:46.549225+0000","last_undegraded":"2024-03-11T22:20:46.549225+0000","last_fullsized":"2024-03-11T22:20:46.549225+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:19:32.485571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.485571+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:57.322796+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":542,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.611732+0000","last_change":"2024-03-11T22:18:40.745227+0000","last_active":"2024-03-11T22:20:54.611732+0000","last_peered":"2024-03-11T22:20:54.611732+0000","last_clean":"2024-03-11T22:20:54.611732+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:20:54.611732+0000","last_undegraded":"2024-03-11T22:20:54.611732+0000","last_fullsized":"2024-03-11T22:20:54.611732+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.745151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.745151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":556,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.556829+0000","last_change":"2024-03-11T22:19:42.927329+0000","last_active":"2024-03-11T22:20:46.556829+0000","last_peered":"2024-03-11T22:20:46.556829+0000","last_clean":"2024-03-11T22:20:46.556829+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:20:46.556829+0000","last_undegraded":"2024-03-11T22:20:46.556829+0000","last_fullsized":"2024-03-11T22:20:46.556829+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:42.926178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:19:42.926178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:49.199630+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":540,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.546950+0000","last_change":"2024-03-11T22:20:29.418105+0000","last_active":"2024-03-11T22:20:46.546950+0000","last_peered":"2024-03-11T22:20:46.546950+0000","last_clean":"2024-03-11T22:20:46.546950+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:20:46.546950+0000","last_undegraded":"2024-03-11T22:20:46.546950+0000","last_fullsized":"2024-03-11T22:20:46.546950+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:27.721245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.721245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"93'130","reported_seq":690,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.569807+0000","last_change":"2024-03-11T22:20:32.293722+0000","last_active":"2024-03-11T22:20:46.569807+0000","last_peered":"2024-03-11T22:20:46.569807+0000","last_clean":"2024-03-11T22:20:46.569807+0000","last_became_active":"2024-03-11T22:20:32.292676+0000","last_became_peered":"2024-03-11T22:20:32.292676+0000","last_unstale":"2024-03-11T22:20:46.569807+0000","last_undegraded":"2024-03-11T22:20:46.569807+0000","last_fullsized":"2024-03-11T22:20:46.569807+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:22.939363+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.939363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.7","version":"314'9711","reported_seq":23098,"reported_epoch":314,"state":"peering","last_fresh":"2024-03-11T22:20:54.591876+0000","last_change":"2024-03-11T22:20:54.612650+0000","last_active":"2024-03-11T22:20:54.612650+0000","last_peered":"2024-03-11T22:20:54.591876+0000","last_clean":"2024-03-11T22:20:54.591876+0000","last_became_active":"2024-03-11T22:20:30.524339+0000","last_became_peered":"2024-03-11T22:20:30.524339+0000","last_unstale":"2024-03-11T22:20:54.591876+0000","last_undegraded":"2024-03-11T22:20:54.591876+0000","last_fullsized":"2024-03-11T22:20:54.591876+0000","mapping_epoch":315,"log_start":"297'5200","ondisk_log_start":"297'5200","created":273,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4511,"log_dups_size":0,"ondisk_log_size":4511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386465792,"num_objects":5897,"num_object_clones":0,"num_object_copies":17691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9711,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6642,"num_bytes_recovered":435290112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"273'35","reported_seq":661,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.618392+0000","last_change":"2024-03-11T22:20:29.437134+0000","last_active":"2024-03-11T22:20:54.618392+0000","last_peered":"2024-03-11T22:20:54.618392+0000","last_clean":"2024-03-11T22:20:54.618392+0000","last_became_active":"2024-03-11T22:20:29.418041+0000","last_became_peered":"2024-03-11T22:20:29.418041+0000","last_unstale":"2024-03-11T22:20:54.618392+0000","last_undegraded":"2024-03-11T22:20:54.618392+0000","last_fullsized":"2024-03-11T22:20:54.618392+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'35","last_scrub_stamp":"2024-03-11T22:19:46.640720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:46.640720+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":493,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.612367+0000","last_change":"2024-03-11T22:20:29.414029+0000","last_active":"2024-03-11T22:20:54.612367+0000","last_peered":"2024-03-11T22:20:54.612367+0000","last_clean":"2024-03-11T22:20:54.612367+0000","last_became_active":"2024-03-11T22:20:29.413664+0000","last_became_peered":"2024-03-11T22:20:29.413664+0000","last_unstale":"2024-03-11T22:20:54.612367+0000","last_undegraded":"2024-03-11T22:20:54.612367+0000","last_fullsized":"2024-03-11T22:20:54.612367+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:32.682157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:32.682157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":556,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.551072+0000","last_change":"2024-03-11T22:20:39.805771+0000","last_active":"2024-03-11T22:20:46.551072+0000","last_peered":"2024-03-11T22:20:46.551072+0000","last_clean":"2024-03-11T22:20:46.551072+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:20:46.551072+0000","last_undegraded":"2024-03-11T22:20:46.551072+0000","last_fullsized":"2024-03-11T22:20:46.551072+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:39.805647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:20:39.805647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:55.656366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":506,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.549494+0000","last_change":"2024-03-11T22:20:29.416481+0000","last_active":"2024-03-11T22:20:46.549494+0000","last_peered":"2024-03-11T22:20:46.549494+0000","last_clean":"2024-03-11T22:20:46.549494+0000","last_became_active":"2024-03-11T22:20:29.415565+0000","last_became_peered":"2024-03-11T22:20:29.415565+0000","last_unstale":"2024-03-11T22:20:46.549494+0000","last_undegraded":"2024-03-11T22:20:46.549494+0000","last_fullsized":"2024-03-11T22:20:46.549494+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:44.717740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:20.931122+0000","last_clean_scrub_stamp":"2024-03-11T22:18:44.717740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"93'130","reported_seq":743,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.619295+0000","last_change":"2024-03-11T22:20:36.022151+0000","last_active":"2024-03-11T22:20:54.619295+0000","last_peered":"2024-03-11T22:20:54.619295+0000","last_clean":"2024-03-11T22:20:54.619295+0000","last_became_active":"2024-03-11T22:14:43.195433+0000","last_became_peered":"2024-03-11T22:14:43.195433+0000","last_unstale":"2024-03-11T22:20:54.619295+0000","last_undegraded":"2024-03-11T22:20:54.619295+0000","last_fullsized":"2024-03-11T22:20:54.619295+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:36.022039+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:20:36.022039+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:45.766952+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"314'9440","reported_seq":9532,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:51.093371+0000","last_change":"2024-03-11T22:18:41.237962+0000","last_active":"2024-03-11T22:20:51.093371+0000","last_peered":"2024-03-11T22:20:51.093371+0000","last_clean":"2024-03-11T22:20:51.093371+0000","last_became_active":"2024-03-11T22:18:41.237565+0000","last_became_peered":"2024-03-11T22:18:41.237565+0000","last_unstale":"2024-03-11T22:20:51.093371+0000","last_undegraded":"2024-03-11T22:20:51.093371+0000","last_fullsized":"2024-03-11T22:20:51.093371+0000","mapping_epoch":273,"log_start":"295'4900","ondisk_log_start":"295'4900","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4540,"log_dups_size":0,"ondisk_log_size":4540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392298496,"num_objects":5986,"num_object_clones":0,"num_object_copies":17958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9440,"num_write_kb":493632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":556,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:47.845492+0000","last_change":"2024-03-11T22:20:47.845641+0000","last_active":"2024-03-11T22:20:47.845492+0000","last_peered":"2024-03-11T22:20:47.845492+0000","last_clean":"2024-03-11T22:20:47.845492+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:20:47.845492+0000","last_undegraded":"2024-03-11T22:20:47.845492+0000","last_fullsized":"2024-03-11T22:20:47.845492+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:47.845156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:20:47.845156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:48.717654+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"273'18","reported_seq":366,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.611681+0000","last_change":"2024-03-11T22:19:38.795869+0000","last_active":"2024-03-11T22:20:54.611681+0000","last_peered":"2024-03-11T22:20:54.611681+0000","last_clean":"2024-03-11T22:20:54.611681+0000","last_became_active":"2024-03-11T22:14:52.247400+0000","last_became_peered":"2024-03-11T22:14:52.247400+0000","last_unstale":"2024-03-11T22:20:54.611681+0000","last_undegraded":"2024-03-11T22:20:54.611681+0000","last_fullsized":"2024-03-11T22:20:54.611681+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:19:38.795650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.795650+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":330,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.564601+0000","last_change":"2024-03-11T22:20:29.417391+0000","last_active":"2024-03-11T22:20:46.564601+0000","last_peered":"2024-03-11T22:20:46.564601+0000","last_clean":"2024-03-11T22:20:46.564601+0000","last_became_active":"2024-03-11T22:20:29.416729+0000","last_became_peered":"2024-03-11T22:20:29.416729+0000","last_unstale":"2024-03-11T22:20:46.564601+0000","last_undegraded":"2024-03-11T22:20:46.564601+0000","last_fullsized":"2024-03-11T22:20:46.564601+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.439756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.439756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:56.331649+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":542,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.572909+0000","last_change":"2024-03-11T22:20:29.470787+0000","last_active":"2024-03-11T22:20:46.572909+0000","last_peered":"2024-03-11T22:20:46.572909+0000","last_clean":"2024-03-11T22:20:46.572909+0000","last_became_active":"2024-03-11T22:20:29.438413+0000","last_became_peered":"2024-03-11T22:20:29.438413+0000","last_unstale":"2024-03-11T22:20:46.572909+0000","last_undegraded":"2024-03-11T22:20:46.572909+0000","last_fullsized":"2024-03-11T22:20:46.572909+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:15:40.678629+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:15:40.678629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"93'130","reported_seq":866,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.574134+0000","last_change":"2024-03-11T22:18:21.956432+0000","last_active":"2024-03-11T22:20:46.574134+0000","last_peered":"2024-03-11T22:20:46.574134+0000","last_clean":"2024-03-11T22:20:46.574134+0000","last_became_active":"2024-03-11T22:17:57.246241+0000","last_became_peered":"2024-03-11T22:17:57.246241+0000","last_unstale":"2024-03-11T22:20:46.574134+0000","last_undegraded":"2024-03-11T22:20:46.574134+0000","last_fullsized":"2024-03-11T22:20:46.574134+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:21.956334+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.956334+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"314'9738","reported_seq":9836,"reported_epoch":314,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-11T22:20:51.757483+0000","last_change":"2024-03-11T22:20:30.593378+0000","last_active":"2024-03-11T22:20:51.757483+0000","last_peered":"2024-03-11T22:20:51.757483+0000","last_clean":"2024-03-11T22:20:28.376799+0000","last_became_active":"2024-03-11T22:20:30.582612+0000","last_became_peered":"2024-03-11T22:20:30.582612+0000","last_unstale":"2024-03-11T22:20:51.757483+0000","last_undegraded":"2024-03-11T22:20:51.757483+0000","last_fullsized":"2024-03-11T22:20:51.757483+0000","mapping_epoch":310,"log_start":"292'4500","ondisk_log_start":"292'4500","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402522112,"num_objects":6142,"num_object_clones":0,"num_object_copies":18426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6142,"num_objects_unfound":0,"num_objects_dirty":6142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9738,"num_write_kb":508160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":6142}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"310'11","reported_seq":659,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.618469+0000","last_change":"2024-03-11T22:20:29.432306+0000","last_active":"2024-03-11T22:20:54.618469+0000","last_peered":"2024-03-11T22:20:54.618469+0000","last_clean":"2024-03-11T22:20:54.618469+0000","last_became_active":"2024-03-11T22:20:29.418355+0000","last_became_peered":"2024-03-11T22:20:29.418355+0000","last_unstale":"2024-03-11T22:20:54.618469+0000","last_undegraded":"2024-03-11T22:20:54.618469+0000","last_fullsized":"2024-03-11T22:20:54.618469+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'10","last_scrub_stamp":"2024-03-11T22:18:52.719695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:52.719695+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"273'34","reported_seq":640,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.610884+0000","last_change":"2024-03-11T22:20:29.431072+0000","last_active":"2024-03-11T22:20:54.610884+0000","last_peered":"2024-03-11T22:20:54.610884+0000","last_clean":"2024-03-11T22:20:54.610884+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:20:54.610884+0000","last_undegraded":"2024-03-11T22:20:54.610884+0000","last_fullsized":"2024-03-11T22:20:54.610884+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:19:46.888741+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:19:46.888741+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":340,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.547284+0000","last_change":"2024-03-11T22:20:29.440395+0000","last_active":"2024-03-11T22:20:46.547284+0000","last_peered":"2024-03-11T22:20:46.547284+0000","last_clean":"2024-03-11T22:20:46.547284+0000","last_became_active":"2024-03-11T22:20:29.415226+0000","last_became_peered":"2024-03-11T22:20:29.415226+0000","last_unstale":"2024-03-11T22:20:46.547284+0000","last_undegraded":"2024-03-11T22:20:46.547284+0000","last_fullsized":"2024-03-11T22:20:46.547284+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:20:26.335219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:26.335219+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:47.255101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":530,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.612572+0000","last_change":"2024-03-11T22:19:33.760079+0000","last_active":"2024-03-11T22:20:54.612572+0000","last_peered":"2024-03-11T22:20:54.612572+0000","last_clean":"2024-03-11T22:20:54.612572+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:20:54.612572+0000","last_undegraded":"2024-03-11T22:20:54.612572+0000","last_fullsized":"2024-03-11T22:20:54.612572+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:33.759955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:33.759955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"93'130","reported_seq":883,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.564685+0000","last_change":"2024-03-11T22:20:32.300771+0000","last_active":"2024-03-11T22:20:46.564685+0000","last_peered":"2024-03-11T22:20:46.564685+0000","last_clean":"2024-03-11T22:20:46.564685+0000","last_became_active":"2024-03-11T22:20:32.295606+0000","last_became_peered":"2024-03-11T22:20:32.295606+0000","last_unstale":"2024-03-11T22:20:46.564685+0000","last_undegraded":"2024-03-11T22:20:46.564685+0000","last_fullsized":"2024-03-11T22:20:46.564685+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:41.158328+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:41.158328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"273'12","reported_seq":574,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.573284+0000","last_change":"2024-03-11T22:20:29.483060+0000","last_active":"2024-03-11T22:20:46.573284+0000","last_peered":"2024-03-11T22:20:46.573284+0000","last_clean":"2024-03-11T22:20:46.573284+0000","last_became_active":"2024-03-11T22:20:29.441599+0000","last_became_peered":"2024-03-11T22:20:29.441599+0000","last_unstale":"2024-03-11T22:20:46.573284+0000","last_undegraded":"2024-03-11T22:20:46.573284+0000","last_fullsized":"2024-03-11T22:20:46.573284+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:15:27.448002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:27.448002+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":558,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.619676+0000","last_change":"2024-03-11T22:20:42.845142+0000","last_active":"2024-03-11T22:20:54.619676+0000","last_peered":"2024-03-11T22:20:54.619676+0000","last_clean":"2024-03-11T22:20:54.619676+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:20:54.619676+0000","last_undegraded":"2024-03-11T22:20:54.619676+0000","last_fullsized":"2024-03-11T22:20:54.619676+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:42.844876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:20:42.844876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:08.076350+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":542,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.635461+0000","last_change":"2024-03-11T22:20:54.635534+0000","last_active":"2024-03-11T22:20:54.635461+0000","last_peered":"2024-03-11T22:20:54.635461+0000","last_clean":"2024-03-11T22:20:54.635461+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:20:54.635461+0000","last_undegraded":"2024-03-11T22:20:54.635461+0000","last_fullsized":"2024-03-11T22:20:54.635461+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:20:54.635309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:15.693951+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":488,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.619083+0000","last_change":"2024-03-11T22:20:32.993441+0000","last_active":"2024-03-11T22:20:54.619083+0000","last_peered":"2024-03-11T22:20:54.619083+0000","last_clean":"2024-03-11T22:20:54.619083+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:20:54.619083+0000","last_undegraded":"2024-03-11T22:20:54.619083+0000","last_fullsized":"2024-03-11T22:20:54.619083+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:32.993301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:20:32.993301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:01.295107+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"93'130","reported_seq":848,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.611061+0000","last_change":"2024-03-11T22:18:45.781426+0000","last_active":"2024-03-11T22:20:54.611061+0000","last_peered":"2024-03-11T22:20:54.611061+0000","last_clean":"2024-03-11T22:20:54.611061+0000","last_became_active":"2024-03-11T22:14:43.194413+0000","last_became_peered":"2024-03-11T22:14:43.194413+0000","last_unstale":"2024-03-11T22:20:54.611061+0000","last_undegraded":"2024-03-11T22:20:54.611061+0000","last_fullsized":"2024-03-11T22:20:54.611061+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:45.781296+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:45.781296+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"93'130","reported_seq":853,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.554557+0000","last_change":"2024-03-11T22:19:27.413677+0000","last_active":"2024-03-11T22:20:46.554557+0000","last_peered":"2024-03-11T22:20:46.554557+0000","last_clean":"2024-03-11T22:20:46.554557+0000","last_became_active":"2024-03-11T22:14:43.198216+0000","last_became_peered":"2024-03-11T22:14:43.198216+0000","last_unstale":"2024-03-11T22:20:46.554557+0000","last_undegraded":"2024-03-11T22:20:46.554557+0000","last_fullsized":"2024-03-11T22:20:46.554557+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:27.413560+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:27.413560+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"273'19","reported_seq":597,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.548600+0000","last_change":"2024-03-11T22:20:29.451236+0000","last_active":"2024-03-11T22:20:46.548600+0000","last_peered":"2024-03-11T22:20:46.548600+0000","last_clean":"2024-03-11T22:20:46.548600+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:20:46.548600+0000","last_undegraded":"2024-03-11T22:20:46.548600+0000","last_fullsized":"2024-03-11T22:20:46.548600+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:18:22.657089+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.657089+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":320,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.618383+0000","last_change":"2024-03-11T22:20:46.807212+0000","last_active":"2024-03-11T22:20:54.618383+0000","last_peered":"2024-03-11T22:20:54.618383+0000","last_clean":"2024-03-11T22:20:54.618383+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:20:54.618383+0000","last_undegraded":"2024-03-11T22:20:54.618383+0000","last_fullsized":"2024-03-11T22:20:54.618383+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:46.807096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:46.807096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:52.495224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":544,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.573192+0000","last_change":"2024-03-11T22:20:29.433304+0000","last_active":"2024-03-11T22:20:46.573192+0000","last_peered":"2024-03-11T22:20:46.573192+0000","last_clean":"2024-03-11T22:20:46.573192+0000","last_became_active":"2024-03-11T22:20:29.433005+0000","last_became_peered":"2024-03-11T22:20:29.433005+0000","last_unstale":"2024-03-11T22:20:46.573192+0000","last_undegraded":"2024-03-11T22:20:46.573192+0000","last_fullsized":"2024-03-11T22:20:46.573192+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:43.544959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:43.544959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:58.636623+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":552,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.551218+0000","last_change":"2024-03-11T22:19:33.966233+0000","last_active":"2024-03-11T22:20:46.551218+0000","last_peered":"2024-03-11T22:20:46.551218+0000","last_clean":"2024-03-11T22:20:46.551218+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:20:46.551218+0000","last_undegraded":"2024-03-11T22:20:46.551218+0000","last_fullsized":"2024-03-11T22:20:46.551218+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:33.966085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:19:33.966085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"273'19","reported_seq":575,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.612456+0000","last_change":"2024-03-11T22:19:29.865779+0000","last_active":"2024-03-11T22:20:54.612456+0000","last_peered":"2024-03-11T22:20:54.612456+0000","last_clean":"2024-03-11T22:20:54.612456+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:20:54.612456+0000","last_undegraded":"2024-03-11T22:20:54.612456+0000","last_fullsized":"2024-03-11T22:20:54.612456+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:19:29.865528+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.865528+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":552,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.546518+0000","last_change":"2024-03-11T22:20:29.417404+0000","last_active":"2024-03-11T22:20:46.546518+0000","last_peered":"2024-03-11T22:20:46.546518+0000","last_clean":"2024-03-11T22:20:46.546518+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:20:46.546518+0000","last_undegraded":"2024-03-11T22:20:46.546518+0000","last_fullsized":"2024-03-11T22:20:46.546518+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:21.693554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.693554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'11","reported_seq":723,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.577142+0000","last_change":"2024-03-11T22:20:29.445996+0000","last_active":"2024-03-11T22:20:46.577142+0000","last_peered":"2024-03-11T22:20:46.577142+0000","last_clean":"2024-03-11T22:20:46.577142+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:20:46.577142+0000","last_undegraded":"2024-03-11T22:20:46.577142+0000","last_fullsized":"2024-03-11T22:20:46.577142+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'10","last_scrub_stamp":"2024-03-11T22:19:32.461683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.461683+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:01.555630+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":314,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.549900+0000","last_change":"2024-03-11T22:18:35.713146+0000","last_active":"2024-03-11T22:20:46.549900+0000","last_peered":"2024-03-11T22:20:46.549900+0000","last_clean":"2024-03-11T22:20:46.549900+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:20:46.549900+0000","last_undegraded":"2024-03-11T22:20:46.549900+0000","last_fullsized":"2024-03-11T22:20:46.549900+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:35.713009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:35.713009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"93'130","reported_seq":841,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.611161+0000","last_change":"2024-03-11T22:18:53.739248+0000","last_active":"2024-03-11T22:20:54.611161+0000","last_peered":"2024-03-11T22:20:54.611161+0000","last_clean":"2024-03-11T22:20:54.611161+0000","last_became_active":"2024-03-11T22:14:43.194350+0000","last_became_peered":"2024-03-11T22:14:43.194350+0000","last_unstale":"2024-03-11T22:20:54.611161+0000","last_undegraded":"2024-03-11T22:20:54.611161+0000","last_fullsized":"2024-03-11T22:20:54.611161+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:53.738316+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:18:53.738316+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"273'20","reported_seq":576,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.576972+0000","last_change":"2024-03-11T22:16:16.055622+0000","last_active":"2024-03-11T22:20:46.576972+0000","last_peered":"2024-03-11T22:20:46.576972+0000","last_clean":"2024-03-11T22:20:46.576972+0000","last_became_active":"2024-03-11T22:16:15.815814+0000","last_became_peered":"2024-03-11T22:16:15.815814+0000","last_unstale":"2024-03-11T22:20:46.576972+0000","last_undegraded":"2024-03-11T22:20:46.576972+0000","last_fullsized":"2024-03-11T22:20:46.576972+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:15:29.460297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.460297+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":338,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.550399+0000","last_change":"2024-03-11T22:20:31.797304+0000","last_active":"2024-03-11T22:20:46.550399+0000","last_peered":"2024-03-11T22:20:46.550399+0000","last_clean":"2024-03-11T22:20:46.550399+0000","last_became_active":"2024-03-11T22:20:29.415019+0000","last_became_peered":"2024-03-11T22:20:29.415019+0000","last_unstale":"2024-03-11T22:20:46.550399+0000","last_undegraded":"2024-03-11T22:20:46.550399+0000","last_fullsized":"2024-03-11T22:20:46.550399+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:31.796712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:31.796712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:47.886151+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":699,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.612187+0000","last_change":"2024-03-11T22:19:42.854028+0000","last_active":"2024-03-11T22:20:54.612187+0000","last_peered":"2024-03-11T22:20:54.612187+0000","last_clean":"2024-03-11T22:20:54.612187+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:20:54.612187+0000","last_undegraded":"2024-03-11T22:20:54.612187+0000","last_fullsized":"2024-03-11T22:20:54.612187+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:19:42.853428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:42.853428+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:57.865194+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":532,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.547816+0000","last_change":"2024-03-11T22:20:29.431434+0000","last_active":"2024-03-11T22:20:46.547816+0000","last_peered":"2024-03-11T22:20:46.547816+0000","last_clean":"2024-03-11T22:20:46.547816+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:20:46.547816+0000","last_undegraded":"2024-03-11T22:20:46.547816+0000","last_fullsized":"2024-03-11T22:20:46.547816+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:20.763181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.763181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"93'130","reported_seq":795,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.546077+0000","last_change":"2024-03-11T22:19:29.435235+0000","last_active":"2024-03-11T22:20:46.546077+0000","last_peered":"2024-03-11T22:20:46.546077+0000","last_clean":"2024-03-11T22:20:46.546077+0000","last_became_active":"2024-03-11T22:14:43.191297+0000","last_became_peered":"2024-03-11T22:14:43.191297+0000","last_unstale":"2024-03-11T22:20:46.546077+0000","last_undegraded":"2024-03-11T22:20:46.546077+0000","last_fullsized":"2024-03-11T22:20:46.546077+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:29.434647+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.434647+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"273'19","reported_seq":366,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.619667+0000","last_change":"2024-03-11T22:20:50.729986+0000","last_active":"2024-03-11T22:20:54.619667+0000","last_peered":"2024-03-11T22:20:54.619667+0000","last_clean":"2024-03-11T22:20:54.619667+0000","last_became_active":"2024-03-11T22:14:52.239810+0000","last_became_peered":"2024-03-11T22:14:52.239810+0000","last_unstale":"2024-03-11T22:20:54.619667+0000","last_undegraded":"2024-03-11T22:20:54.619667+0000","last_fullsized":"2024-03-11T22:20:54.619667+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:20:50.729848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:50.729848+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:58.488489+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":345,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.618704+0000","last_change":"2024-03-11T22:20:47.791284+0000","last_active":"2024-03-11T22:20:54.618704+0000","last_peered":"2024-03-11T22:20:54.618704+0000","last_clean":"2024-03-11T22:20:54.618704+0000","last_became_active":"2024-03-11T22:20:29.420867+0000","last_became_peered":"2024-03-11T22:20:29.420867+0000","last_unstale":"2024-03-11T22:20:54.618704+0000","last_undegraded":"2024-03-11T22:20:54.618704+0000","last_fullsized":"2024-03-11T22:20:54.618704+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:20:47.791128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:47.791128+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:56.876944+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":540,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.555576+0000","last_change":"2024-03-11T22:17:24.726548+0000","last_active":"2024-03-11T22:20:46.555576+0000","last_peered":"2024-03-11T22:20:46.555576+0000","last_clean":"2024-03-11T22:20:46.555576+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:20:46.555576+0000","last_undegraded":"2024-03-11T22:20:46.555576+0000","last_fullsized":"2024-03-11T22:20:46.555576+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:24.726421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:17:24.726421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":520,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.575968+0000","last_change":"2024-03-11T22:18:29.014246+0000","last_active":"2024-03-11T22:20:46.575968+0000","last_peered":"2024-03-11T22:20:46.575968+0000","last_clean":"2024-03-11T22:20:46.575968+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:20:46.575968+0000","last_undegraded":"2024-03-11T22:20:46.575968+0000","last_fullsized":"2024-03-11T22:20:46.575968+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:29.014149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:29.014149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":573,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.555744+0000","last_change":"2024-03-11T22:17:35.738522+0000","last_active":"2024-03-11T22:20:46.555744+0000","last_peered":"2024-03-11T22:20:46.555744+0000","last_clean":"2024-03-11T22:20:46.555744+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:20:46.555744+0000","last_undegraded":"2024-03-11T22:20:46.555744+0000","last_fullsized":"2024-03-11T22:20:46.555744+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'13","last_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:17:35.738211+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":483,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.569406+0000","last_change":"2024-03-11T22:20:29.423108+0000","last_active":"2024-03-11T22:20:46.569406+0000","last_peered":"2024-03-11T22:20:46.569406+0000","last_clean":"2024-03-11T22:20:46.569406+0000","last_became_active":"2024-03-11T22:20:29.422747+0000","last_became_peered":"2024-03-11T22:20:29.422747+0000","last_unstale":"2024-03-11T22:20:46.569406+0000","last_undegraded":"2024-03-11T22:20:46.569406+0000","last_fullsized":"2024-03-11T22:20:46.569406+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:44.563614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:44.563614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":532,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.572729+0000","last_change":"2024-03-11T22:19:39.445467+0000","last_active":"2024-03-11T22:20:46.572729+0000","last_peered":"2024-03-11T22:20:46.572729+0000","last_clean":"2024-03-11T22:20:46.572729+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:20:46.572729+0000","last_undegraded":"2024-03-11T22:20:46.572729+0000","last_fullsized":"2024-03-11T22:20:46.572729+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:39.445349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:19:39.445349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:56.972428+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":528,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.546873+0000","last_change":"2024-03-11T22:19:26.459086+0000","last_active":"2024-03-11T22:20:46.546873+0000","last_peered":"2024-03-11T22:20:46.546873+0000","last_clean":"2024-03-11T22:20:46.546873+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:20:46.546873+0000","last_undegraded":"2024-03-11T22:20:46.546873+0000","last_fullsized":"2024-03-11T22:20:46.546873+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:26.458937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:26.458937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"93'130","reported_seq":856,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:49.901380+0000","last_change":"2024-03-11T22:20:49.901428+0000","last_active":"2024-03-11T22:20:49.901380+0000","last_peered":"2024-03-11T22:20:49.901380+0000","last_clean":"2024-03-11T22:20:49.901380+0000","last_became_active":"2024-03-11T22:14:43.194955+0000","last_became_peered":"2024-03-11T22:14:43.194955+0000","last_unstale":"2024-03-11T22:20:49.901380+0000","last_undegraded":"2024-03-11T22:20:49.901380+0000","last_fullsized":"2024-03-11T22:20:49.901380+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:49.901270+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:18:43.722467+0000","last_clean_scrub_stamp":"2024-03-11T22:20:49.901270+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:57.047219+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"274'20","reported_seq":596,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.576297+0000","last_change":"2024-03-11T22:20:29.487125+0000","last_active":"2024-03-11T22:20:46.576297+0000","last_peered":"2024-03-11T22:20:46.576297+0000","last_clean":"2024-03-11T22:20:46.576297+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:20:46.576297+0000","last_undegraded":"2024-03-11T22:20:46.576297+0000","last_fullsized":"2024-03-11T22:20:46.576297+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'20","last_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:18:42.135888+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":554,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.612383+0000","last_change":"2024-03-11T22:20:29.417149+0000","last_active":"2024-03-11T22:20:54.612383+0000","last_peered":"2024-03-11T22:20:54.612383+0000","last_clean":"2024-03-11T22:20:54.612383+0000","last_became_active":"2024-03-11T22:20:29.416792+0000","last_became_peered":"2024-03-11T22:20:29.416792+0000","last_unstale":"2024-03-11T22:20:54.612383+0000","last_undegraded":"2024-03-11T22:20:54.612383+0000","last_fullsized":"2024-03-11T22:20:54.612383+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:45.801375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:45.801375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":319,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.619579+0000","last_change":"2024-03-11T22:20:40.885587+0000","last_active":"2024-03-11T22:20:54.619579+0000","last_peered":"2024-03-11T22:20:54.619579+0000","last_clean":"2024-03-11T22:20:54.619579+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:20:54.619579+0000","last_undegraded":"2024-03-11T22:20:54.619579+0000","last_fullsized":"2024-03-11T22:20:54.619579+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:40.885360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:20:40.885360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:51.440703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":520,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.573003+0000","last_change":"2024-03-11T22:14:23.930193+0000","last_active":"2024-03-11T22:20:46.573003+0000","last_peered":"2024-03-11T22:20:46.573003+0000","last_clean":"2024-03-11T22:20:46.573003+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:20:46.573003+0000","last_undegraded":"2024-03-11T22:20:46.573003+0000","last_fullsized":"2024-03-11T22:20:46.573003+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.930091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.930091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"273'8","reported_seq":560,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.569543+0000","last_change":"2024-03-11T22:18:53.157176+0000","last_active":"2024-03-11T22:20:46.569543+0000","last_peered":"2024-03-11T22:20:46.569543+0000","last_clean":"2024-03-11T22:20:46.569543+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:20:46.569543+0000","last_undegraded":"2024-03-11T22:20:46.569543+0000","last_fullsized":"2024-03-11T22:20:46.569543+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'8","last_scrub_stamp":"2024-03-11T22:18:53.157080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:53.157080+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":540,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.547385+0000","last_change":"2024-03-11T22:19:45.501704+0000","last_active":"2024-03-11T22:20:46.547385+0000","last_peered":"2024-03-11T22:20:46.547385+0000","last_clean":"2024-03-11T22:20:46.547385+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:20:46.547385+0000","last_undegraded":"2024-03-11T22:20:46.547385+0000","last_fullsized":"2024-03-11T22:20:46.547385+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.501593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.501593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:53.822220+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":128,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.618234+0000","last_change":"2024-03-11T22:20:30.047479+0000","last_active":"2024-03-11T22:20:54.618234+0000","last_peered":"2024-03-11T22:20:54.618234+0000","last_clean":"2024-03-11T22:20:54.618234+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:20:54.618234+0000","last_undegraded":"2024-03-11T22:20:54.618234+0000","last_fullsized":"2024-03-11T22:20:54.618234+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:30.047368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:30.047368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:43.466695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":494,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.557158+0000","last_change":"2024-03-11T22:19:32.989760+0000","last_active":"2024-03-11T22:20:46.557158+0000","last_peered":"2024-03-11T22:20:46.557158+0000","last_clean":"2024-03-11T22:20:46.557158+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:20:46.557158+0000","last_undegraded":"2024-03-11T22:20:46.557158+0000","last_fullsized":"2024-03-11T22:20:46.557158+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.989348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.989348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:47.266882+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"273'19","reported_seq":585,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.619393+0000","last_change":"2024-03-11T22:20:51.762478+0000","last_active":"2024-03-11T22:20:54.619393+0000","last_peered":"2024-03-11T22:20:54.619393+0000","last_clean":"2024-03-11T22:20:54.619393+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:20:54.619393+0000","last_undegraded":"2024-03-11T22:20:54.619393+0000","last_fullsized":"2024-03-11T22:20:54.619393+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:20:51.762318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:51.762318+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:54.676538+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":540,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.569445+0000","last_change":"2024-03-11T22:19:25.669161+0000","last_active":"2024-03-11T22:20:46.569445+0000","last_peered":"2024-03-11T22:20:46.569445+0000","last_clean":"2024-03-11T22:20:46.569445+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:20:46.569445+0000","last_undegraded":"2024-03-11T22:20:46.569445+0000","last_fullsized":"2024-03-11T22:20:46.569445+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:25.668878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:25.668878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":541,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.548810+0000","last_change":"2024-03-11T22:20:27.333574+0000","last_active":"2024-03-11T22:20:46.548810+0000","last_peered":"2024-03-11T22:20:46.548810+0000","last_clean":"2024-03-11T22:20:46.548810+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:20:46.548810+0000","last_undegraded":"2024-03-11T22:20:46.548810+0000","last_fullsized":"2024-03-11T22:20:46.548810+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:27.333450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:27.333450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:54.425323+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":543,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.565231+0000","last_change":"2024-03-11T22:19:45.686859+0000","last_active":"2024-03-11T22:20:46.565231+0000","last_peered":"2024-03-11T22:20:46.565231+0000","last_clean":"2024-03-11T22:20:46.565231+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:20:46.565231+0000","last_undegraded":"2024-03-11T22:20:46.565231+0000","last_fullsized":"2024-03-11T22:20:46.565231+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.686738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.686738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:08.688833+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"273'20","reported_seq":603,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.548067+0000","last_change":"2024-03-11T22:20:29.460876+0000","last_active":"2024-03-11T22:20:46.548067+0000","last_peered":"2024-03-11T22:20:46.548067+0000","last_clean":"2024-03-11T22:20:46.548067+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:20:46.548067+0000","last_undegraded":"2024-03-11T22:20:46.548067+0000","last_fullsized":"2024-03-11T22:20:46.548067+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:19:36.469333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.469333+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":550,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.554296+0000","last_change":"2024-03-11T22:17:36.687785+0000","last_active":"2024-03-11T22:20:46.554296+0000","last_peered":"2024-03-11T22:20:46.554296+0000","last_clean":"2024-03-11T22:20:46.554296+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:20:46.554296+0000","last_undegraded":"2024-03-11T22:20:46.554296+0000","last_fullsized":"2024-03-11T22:20:46.554296+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:17:36.687660+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'11","reported_seq":740,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.741729+0000","last_change":"2024-03-11T22:20:54.741772+0000","last_active":"2024-03-11T22:20:54.741729+0000","last_peered":"2024-03-11T22:20:54.741729+0000","last_clean":"2024-03-11T22:20:54.741729+0000","last_became_active":"2024-03-11T22:20:29.415781+0000","last_became_peered":"2024-03-11T22:20:29.415781+0000","last_unstale":"2024-03-11T22:20:54.741729+0000","last_undegraded":"2024-03-11T22:20:54.741729+0000","last_fullsized":"2024-03-11T22:20:54.741729+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_deep_scrub":"315'11","last_deep_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_clean_scrub_stamp":"2024-03-11T22:20:54.741633+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:10.626413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":491,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.611300+0000","last_change":"2024-03-11T22:19:27.754842+0000","last_active":"2024-03-11T22:20:54.611300+0000","last_peered":"2024-03-11T22:20:54.611300+0000","last_clean":"2024-03-11T22:20:54.611300+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:20:54.611300+0000","last_undegraded":"2024-03-11T22:20:54.611300+0000","last_fullsized":"2024-03-11T22:20:54.611300+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:19:27.754721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:27.754721+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:20:55.635621+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"273'18","reported_seq":559,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.569747+0000","last_change":"2024-03-11T22:14:27.998984+0000","last_active":"2024-03-11T22:20:46.569747+0000","last_peered":"2024-03-11T22:20:46.569747+0000","last_clean":"2024-03-11T22:20:46.569747+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:20:46.569747+0000","last_undegraded":"2024-03-11T22:20:46.569747+0000","last_fullsized":"2024-03-11T22:20:46.569747+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:14:27.998819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:27.998819+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":315,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.555428+0000","last_change":"2024-03-11T22:19:36.471241+0000","last_active":"2024-03-11T22:20:46.555428+0000","last_peered":"2024-03-11T22:20:46.555428+0000","last_clean":"2024-03-11T22:20:46.555428+0000","last_became_active":"2024-03-11T22:14:52.245199+0000","last_became_peered":"2024-03-11T22:14:52.245199+0000","last_unstale":"2024-03-11T22:20:46.555428+0000","last_undegraded":"2024-03-11T22:20:46.555428+0000","last_fullsized":"2024-03-11T22:20:46.555428+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.471133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.471133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:00.098252+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":290,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.577250+0000","last_change":"2024-03-11T22:14:52.233517+0000","last_active":"2024-03-11T22:20:46.577250+0000","last_peered":"2024-03-11T22:20:46.577250+0000","last_clean":"2024-03-11T22:20:46.577250+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:20:46.577250+0000","last_undegraded":"2024-03-11T22:20:46.577250+0000","last_fullsized":"2024-03-11T22:20:46.577250+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.849679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.849679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":520,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.572798+0000","last_change":"2024-03-11T22:18:27.116987+0000","last_active":"2024-03-11T22:20:46.572798+0000","last_peered":"2024-03-11T22:20:46.572798+0000","last_clean":"2024-03-11T22:20:46.572798+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:20:46.572798+0000","last_undegraded":"2024-03-11T22:20:46.572798+0000","last_fullsized":"2024-03-11T22:20:46.572798+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:27.116858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.116858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"273'27","reported_seq":602,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.551121+0000","last_change":"2024-03-11T22:18:42.691421+0000","last_active":"2024-03-11T22:20:46.551121+0000","last_peered":"2024-03-11T22:20:46.551121+0000","last_clean":"2024-03-11T22:20:46.551121+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:20:46.551121+0000","last_undegraded":"2024-03-11T22:20:46.551121+0000","last_fullsized":"2024-03-11T22:20:46.551121+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:18:42.691319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:42.691319+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":551,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.549135+0000","last_change":"2024-03-11T22:20:29.414748+0000","last_active":"2024-03-11T22:20:46.549135+0000","last_peered":"2024-03-11T22:20:46.549135+0000","last_clean":"2024-03-11T22:20:46.549135+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:20:46.549135+0000","last_undegraded":"2024-03-11T22:20:46.549135+0000","last_fullsized":"2024-03-11T22:20:46.549135+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:18:43.931797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":539,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.565548+0000","last_change":"2024-03-11T22:18:31.120211+0000","last_active":"2024-03-11T22:20:46.565548+0000","last_peered":"2024-03-11T22:20:46.565548+0000","last_clean":"2024-03-11T22:20:46.565548+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:20:46.565548+0000","last_undegraded":"2024-03-11T22:20:46.565548+0000","last_fullsized":"2024-03-11T22:20:46.565548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:31.120131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:31.120131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":317,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.619212+0000","last_change":"2024-03-11T22:20:45.819718+0000","last_active":"2024-03-11T22:20:54.619212+0000","last_peered":"2024-03-11T22:20:54.619212+0000","last_clean":"2024-03-11T22:20:54.619212+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:20:54.619212+0000","last_undegraded":"2024-03-11T22:20:54.619212+0000","last_fullsized":"2024-03-11T22:20:54.619212+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:45.819593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:20:45.819593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:49.197505+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"273'9","reported_seq":375,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.572545+0000","last_change":"2024-03-11T22:18:51.098736+0000","last_active":"2024-03-11T22:20:46.572545+0000","last_peered":"2024-03-11T22:20:46.572545+0000","last_clean":"2024-03-11T22:20:46.572545+0000","last_became_active":"2024-03-11T22:14:52.238411+0000","last_became_peered":"2024-03-11T22:14:52.238411+0000","last_unstale":"2024-03-11T22:20:46.572545+0000","last_undegraded":"2024-03-11T22:20:46.572545+0000","last_fullsized":"2024-03-11T22:20:46.572545+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9","last_scrub_stamp":"2024-03-11T22:18:51.098646+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:18:51.098646+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":541,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.611436+0000","last_change":"2024-03-11T22:18:29.763217+0000","last_active":"2024-03-11T22:20:54.611436+0000","last_peered":"2024-03-11T22:20:54.611436+0000","last_clean":"2024-03-11T22:20:54.611436+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:20:54.611436+0000","last_undegraded":"2024-03-11T22:20:54.611436+0000","last_fullsized":"2024-03-11T22:20:54.611436+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:29.739164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:18:29.739164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":309,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.555268+0000","last_change":"2024-03-11T22:17:27.798877+0000","last_active":"2024-03-11T22:20:46.555268+0000","last_peered":"2024-03-11T22:20:46.555268+0000","last_clean":"2024-03-11T22:20:46.555268+0000","last_became_active":"2024-03-11T22:14:52.239391+0000","last_became_peered":"2024-03-11T22:14:52.239391+0000","last_unstale":"2024-03-11T22:20:46.555268+0000","last_undegraded":"2024-03-11T22:20:46.555268+0000","last_fullsized":"2024-03-11T22:20:46.555268+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:27.798763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_clean_scrub_stamp":"2024-03-11T22:17:27.798763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":530,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.611471+0000","last_change":"2024-03-11T22:18:38.736070+0000","last_active":"2024-03-11T22:20:54.611471+0000","last_peered":"2024-03-11T22:20:54.611471+0000","last_clean":"2024-03-11T22:20:54.611471+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:20:54.611471+0000","last_undegraded":"2024-03-11T22:20:54.611471+0000","last_fullsized":"2024-03-11T22:20:54.611471+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:38.735937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.735937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"273'15","reported_seq":342,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.569920+0000","last_change":"2024-03-11T22:15:18.401675+0000","last_active":"2024-03-11T22:20:46.569920+0000","last_peered":"2024-03-11T22:20:46.569920+0000","last_clean":"2024-03-11T22:20:46.569920+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:20:46.569920+0000","last_undegraded":"2024-03-11T22:20:46.569920+0000","last_fullsized":"2024-03-11T22:20:46.569920+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:15:18.401549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:18.401549+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":549,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.612043+0000","last_change":"2024-03-11T22:19:40.864768+0000","last_active":"2024-03-11T22:20:54.612043+0000","last_peered":"2024-03-11T22:20:54.612043+0000","last_clean":"2024-03-11T22:20:54.612043+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:20:54.612043+0000","last_undegraded":"2024-03-11T22:20:54.612043+0000","last_fullsized":"2024-03-11T22:20:54.612043+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:40.864689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:40.864689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:05.030010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":545,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.611990+0000","last_change":"2024-03-11T22:20:29.420120+0000","last_active":"2024-03-11T22:20:54.611990+0000","last_peered":"2024-03-11T22:20:54.611990+0000","last_clean":"2024-03-11T22:20:54.611990+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:20:54.611990+0000","last_undegraded":"2024-03-11T22:20:54.611990+0000","last_fullsized":"2024-03-11T22:20:54.611990+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:28.713390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:18:28.713390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":528,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.547284+0000","last_change":"2024-03-11T22:20:30.493247+0000","last_active":"2024-03-11T22:20:46.547284+0000","last_peered":"2024-03-11T22:20:46.547284+0000","last_clean":"2024-03-11T22:20:46.547284+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:20:46.547284+0000","last_undegraded":"2024-03-11T22:20:46.547284+0000","last_fullsized":"2024-03-11T22:20:46.547284+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:30.493146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:20:30.493146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:48.539526+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"273'27","reported_seq":571,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.576391+0000","last_change":"2024-03-11T22:18:37.066095+0000","last_active":"2024-03-11T22:20:46.576391+0000","last_peered":"2024-03-11T22:20:46.576391+0000","last_clean":"2024-03-11T22:20:46.576391+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:20:46.576391+0000","last_undegraded":"2024-03-11T22:20:46.576391+0000","last_fullsized":"2024-03-11T22:20:46.576391+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:18:37.065988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.065988+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":547,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.570256+0000","last_change":"2024-03-11T22:18:38.095521+0000","last_active":"2024-03-11T22:20:46.570256+0000","last_peered":"2024-03-11T22:20:46.570256+0000","last_clean":"2024-03-11T22:20:46.570256+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:20:46.570256+0000","last_undegraded":"2024-03-11T22:20:46.570256+0000","last_fullsized":"2024-03-11T22:20:46.570256+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:18:38.095367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.095367+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":316,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.619837+0000","last_change":"2024-03-11T22:20:26.125601+0000","last_active":"2024-03-11T22:20:54.619837+0000","last_peered":"2024-03-11T22:20:54.619837+0000","last_clean":"2024-03-11T22:20:54.619837+0000","last_became_active":"2024-03-11T22:14:52.247689+0000","last_became_peered":"2024-03-11T22:14:52.247689+0000","last_unstale":"2024-03-11T22:20:54.619837+0000","last_undegraded":"2024-03-11T22:20:54.619837+0000","last_fullsized":"2024-03-11T22:20:54.619837+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:26.125517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:26.125517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:42.748307+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":531,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.572809+0000","last_change":"2024-03-11T22:20:29.436389+0000","last_active":"2024-03-11T22:20:46.572809+0000","last_peered":"2024-03-11T22:20:46.572809+0000","last_clean":"2024-03-11T22:20:46.572809+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:20:46.572809+0000","last_undegraded":"2024-03-11T22:20:46.572809+0000","last_fullsized":"2024-03-11T22:20:46.572809+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:24.023596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:24.023596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"273'19","reported_seq":356,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.619952+0000","last_change":"2024-03-11T22:19:30.613894+0000","last_active":"2024-03-11T22:20:54.619952+0000","last_peered":"2024-03-11T22:20:54.619952+0000","last_clean":"2024-03-11T22:20:54.619952+0000","last_became_active":"2024-03-11T22:14:52.247411+0000","last_became_peered":"2024-03-11T22:14:52.247411+0000","last_unstale":"2024-03-11T22:20:54.619952+0000","last_undegraded":"2024-03-11T22:20:54.619952+0000","last_fullsized":"2024-03-11T22:20:54.619952+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:19:30.613736+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:19:30.613736+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":540,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T22:20:46.547138+0000","last_change":"2024-03-11T22:18:24.613444+0000","last_active":"2024-03-11T22:20:46.547138+0000","last_peered":"2024-03-11T22:20:46.547138+0000","last_clean":"2024-03-11T22:20:46.547138+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:20:46.547138+0000","last_undegraded":"2024-03-11T22:20:46.547138+0000","last_fullsized":"2024-03-11T22:20:46.547138+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:24.613372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:24.613372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":256,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.618236+0000","last_change":"2024-03-11T22:20:29.420332+0000","last_active":"2024-03-11T22:20:54.618236+0000","last_peered":"2024-03-11T22:20:54.618236+0000","last_clean":"2024-03-11T22:20:54.618236+0000","last_became_active":"2024-03-11T22:20:29.418227+0000","last_became_peered":"2024-03-11T22:20:29.418227+0000","last_unstale":"2024-03-11T22:20:54.618236+0000","last_undegraded":"2024-03-11T22:20:54.618236+0000","last_fullsized":"2024-03-11T22:20:54.618236+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.629853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.629853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:14.302678+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":308,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-11T22:20:54.612077+0000","last_change":"2024-03-11T22:19:26.746519+0000","last_active":"2024-03-11T22:20:54.612077+0000","last_peered":"2024-03-11T22:20:54.612077+0000","last_clean":"2024-03-11T22:20:54.612077+0000","last_became_active":"2024-03-11T22:14:52.247273+0000","last_became_peered":"2024-03-11T22:14:52.247273+0000","last_unstale":"2024-03-11T22:20:54.612077+0000","last_undegraded":"2024-03-11T22:20:54.612077+0000","last_fullsized":"2024-03-11T22:20:54.612077+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:26.746435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:26.746435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6254493720,"num_objects":95437,"num_object_clones":0,"num_object_copies":286311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":31375,"num_objects_unfound":0,"num_objects_dirty":95437,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152799,"num_write_kb":7943489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12001,"num_bytes_recovered":786300928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19443494912,"data_stored":19443482696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72399,"ondisk_log_size":72399,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":703,"num_read_kb":698,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":1430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":717,"ondisk_log_size":717,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005209,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1888336,"kb_used_data":1541900,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91856816,"statfs":{"total":95995035648,"available":94061379584,"internally_reserved":0,"allocated":1578905600,"data_stored":1578608030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":354613707},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332254,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1440892,"kb_used_data":1106936,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92304260,"statfs":{"total":95995035648,"available":94519562240,"internally_reserved":0,"allocated":1133502464,"data_stored":1133225184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":341834182},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575031,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4300832,"kb_used_data":3872600,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":89444320,"statfs":{"total":95995035648,"available":91590983680,"internally_reserved":0,"allocated":3965542400,"data_stored":3965178116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738557,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2349436,"kb_used_data":1930932,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":91395716,"statfs":{"total":95995035648,"available":93589213184,"internally_reserved":0,"allocated":1977274368,"data_stored":1976946250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411506,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1896940,"kb_used_data":1550628,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91848212,"statfs":{"total":95995035648,"available":94052569088,"internally_reserved":0,"allocated":1587843072,"data_stored":1587532186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3946836,"kb_used_data":3496076,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":89798316,"statfs":{"total":95995035648,"available":91953475584,"internally_reserved":0,"allocated":3579981824,"data_stored":3579630658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":461240775},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920937,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2171336,"kb_used_data":1847232,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91573816,"statfs":{"total":95995035648,"available":93771587584,"internally_reserved":0,"allocated":1891565568,"data_stored":1891225395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":331872710},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659298,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4225544,"kb_used_data":3729156,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":89519608,"statfs":{"total":95995035648,"available":91668078592,"internally_reserved":0,"allocated":3818655744,"data_stored":3818366017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1962999808,"data_stored":1962999808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3954970624,"data_stored":3954966552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1569980416,"data_stored":1569980416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3566600192,"data_stored":3566600192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1882394624,"data_stored":1882390552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3808563200,"data_stored":3808559128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1130692608,"data_stored":1130692608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1567293440,"data_stored":1567293440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:20:58.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph status --format=json 2024-03-11T22:20:58.253 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:58.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:58.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:58.355 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:20:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:58.388 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:58.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:20:58.489 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:58.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:58.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:58.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:58.697 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:58.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:58.798 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:20:58.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:58.835 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:58.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:20:58.936 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:20:58.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:58.973 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:58.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:20:59.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:58 smithi045 ceph-mon[30178]: 5.9 scrub starts 2024-03-11T22:20:59.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:58 smithi045 ceph-mon[30178]: 5.9 scrub ok 2024-03-11T22:20:59.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:58 smithi045 ceph-mon[30178]: 1.5 deep-scrub starts 2024-03-11T22:20:59.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:58 smithi045 ceph-mon[30178]: 1.5 deep-scrub ok 2024-03-11T22:20:59.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:58 smithi045 ceph-mon[39715]: 5.9 scrub starts 2024-03-11T22:20:59.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:58 smithi045 ceph-mon[39715]: 5.9 scrub ok 2024-03-11T22:20:59.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:58 smithi045 ceph-mon[39715]: 1.5 deep-scrub starts 2024-03-11T22:20:59.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:58 smithi045 ceph-mon[39715]: 1.5 deep-scrub ok 2024-03-11T22:20:59.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:59.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:58 smithi144 ceph-mon[33984]: 5.9 scrub starts 2024-03-11T22:20:59.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:58 smithi144 ceph-mon[33984]: 5.9 scrub ok 2024-03-11T22:20:59.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:58 smithi144 ceph-mon[33984]: 1.5 deep-scrub starts 2024-03-11T22:20:59.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:58 smithi144 ceph-mon[33984]: 1.5 deep-scrub ok 2024-03-11T22:20:59.120 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:59.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:59.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:59.221 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:59.244 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:59.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:59.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:59.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:59.391 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:59.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:59.492 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:59.520 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:59.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:59.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:59.622 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:20:59.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:59 smithi045 ceph-mon[30178]: pgmap v585: 168 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 160 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 27116/276549 objects misplaced (9.805%); 54 MiB/s, 864 objects/s recovering 2024-03-11T22:20:59.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:59 smithi045 ceph-mon[30178]: 6.17 scrub starts 2024-03-11T22:20:59.629 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:20:59 smithi045 ceph-mon[30178]: 6.17 scrub ok 2024-03-11T22:20:59.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:59.662 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:59.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:20:59.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:20:59.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:59.802 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:20:59.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:20:59.903 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:20:59.936 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:20:59.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:20:59.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:20:59.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:59 smithi045 ceph-mon[39715]: pgmap v585: 168 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 160 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 27116/276549 objects misplaced (9.805%); 54 MiB/s, 864 objects/s recovering 2024-03-11T22:20:59.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:59 smithi045 ceph-mon[39715]: 6.17 scrub starts 2024-03-11T22:20:59.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:20:59 smithi045 ceph-mon[39715]: 6.17 scrub ok 2024-03-11T22:21:00.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:00.076 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:00.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:00.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:59 smithi144 ceph-mon[33984]: pgmap v585: 168 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 160 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 27116/276549 objects misplaced (9.805%); 54 MiB/s, 864 objects/s recovering 2024-03-11T22:21:00.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:59 smithi144 ceph-mon[33984]: 6.17 scrub starts 2024-03-11T22:21:00.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:20:59 smithi144 ceph-mon[33984]: 6.17 scrub ok 2024-03-11T22:21:00.144 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:21:00.178 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:00.207 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:00.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:00.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:00.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:00.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:00.367 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:00.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:00.468 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:00.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:00.515 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:00.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:00.616 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:00.660 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:00.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:00.762 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:00.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:00.799 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:00.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:00.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:00 smithi045 ceph-mon[39715]: 5.e scrub starts 2024-03-11T22:21:00.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:00 smithi045 ceph-mon[39715]: 5.e scrub ok 2024-03-11T22:21:00.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:21:00.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:21:00.828 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:00 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4014193920' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:21:00.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:00 smithi045 ceph-mon[30178]: 5.e scrub starts 2024-03-11T22:21:00.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:00 smithi045 ceph-mon[30178]: 5.e scrub ok 2024-03-11T22:21:00.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:21:00.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:21:00.829 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:00 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4014193920' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:21:00.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:00.928 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:21:00.928 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:21:00.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:00.949 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:00.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:01.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:01.097 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:01.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:01.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:00 smithi144 ceph-mon[33984]: 5.e scrub starts 2024-03-11T22:21:01.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:00 smithi144 ceph-mon[33984]: 5.e scrub ok 2024-03-11T22:21:01.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:21:01.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:21:01.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:00 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4014193920' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:21:01.197 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:01.225 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:01.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:01.326 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:01.355 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:01.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:01.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:01.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:01.494 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:01.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:01.575 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:21:01.596 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:01.629 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:01.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:01.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:01.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:01.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:01.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:01.872 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:01.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:01 smithi045 ceph-mon[30178]: pgmap v586: 168 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 160 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 572 op/s; 27116/276549 objects misplaced (9.805%); 31 MiB/s, 493 objects/s recovering 2024-03-11T22:21:01.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:01 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:21:01.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:01 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4014193920' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:21:01.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:01 smithi045 ceph-mon[30178]: osdmap e317: 8 total, 8 up, 8 in 2024-03-11T22:21:01.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:01 smithi045 ceph-mon[39715]: pgmap v586: 168 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 160 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 572 op/s; 27116/276549 objects misplaced (9.805%); 31 MiB/s, 493 objects/s recovering 2024-03-11T22:21:01.880 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:01 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:21:01.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:01 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4014193920' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:21:01.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:01 smithi045 ceph-mon[39715]: osdmap e317: 8 total, 8 up, 8 in 2024-03-11T22:21:01.903 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:01.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:02.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:02.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:02.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:02.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:02.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:01 smithi144 ceph-mon[33984]: pgmap v586: 168 pgs: 1 active+clean+scrubbing, 1 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 160 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 572 op/s; 27116/276549 objects misplaced (9.805%); 31 MiB/s, 493 objects/s recovering 2024-03-11T22:21:02.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:01 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:21:02.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:01 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4014193920' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:21:02.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:01 smithi144 ceph-mon[33984]: osdmap e317: 8 total, 8 up, 8 in 2024-03-11T22:21:02.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:02.170 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:02.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:02.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:02.272 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:02.307 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:02.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:02.409 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:02.443 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:02.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:02.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:02.545 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:02.575 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:02.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:02.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:02.677 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:02.708 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:02.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:02.783 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:21:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:21:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:21:02.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:02.853 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:02.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:02.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:02.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:02.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:02.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:02.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:03.050 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:21:03.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:03.151 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:03.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:03.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:03.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:03.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:03.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:03.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:03.434 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:03.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:03.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:03.578 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:03.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:03.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:03.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:03.717 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:03.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:03.819 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:03.858 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:03.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:03.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:03.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:03.990 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:03.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:04.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:03 smithi045 ceph-mon[30178]: pgmap v588: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.28k op/s; 23538/260631 objects misplaced (9.031%); 49 MiB/s, 780 objects/s recovering 2024-03-11T22:21:04.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:03 smithi045 ceph-mon[39715]: pgmap v588: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.28k op/s; 23538/260631 objects misplaced (9.031%); 49 MiB/s, 780 objects/s recovering 2024-03-11T22:21:04.092 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:04.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:03 smithi144 ceph-mon[33984]: pgmap v588: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.28k op/s; 23538/260631 objects misplaced (9.031%); 49 MiB/s, 780 objects/s recovering 2024-03-11T22:21:04.127 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:04.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:04.229 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:04.266 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:04.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:04.367 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:04.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:04.404 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:04.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:04.505 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:04.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:04.539 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:04.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:04.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:04.683 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:04.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:04.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:04.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:04.824 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:04.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:04.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:04.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:04.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:05.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:05.104 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:05.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:05.205 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:05.235 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:05.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:05.337 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:05.371 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:05.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:05.472 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:05.507 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:05.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:05.608 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:05.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:05.637 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:05.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:05.738 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:05.772 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:05.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:05.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:05.874 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:05.898 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:05 smithi144 ceph-mon[33984]: pgmap v589: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 23538/260631 objects misplaced (9.031%); 35 MiB/s, 561 objects/s recovering 2024-03-11T22:21:05.907 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:05.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:06.009 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:06.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:05 smithi045 ceph-mon[39715]: pgmap v589: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 23538/260631 objects misplaced (9.031%); 35 MiB/s, 561 objects/s recovering 2024-03-11T22:21:06.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:05 smithi045 ceph-mon[30178]: pgmap v589: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 23538/260631 objects misplaced (9.031%); 35 MiB/s, 561 objects/s recovering 2024-03-11T22:21:06.043 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:06.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:06.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:06.145 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:06.178 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:06.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:06.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:06.280 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:06.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:06.305 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:06.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:06.406 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:06.439 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:06.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:06.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:06.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:06.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:06.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:06.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:06.718 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:06.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:06.819 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:06.847 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:21:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:21:06.851 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:06.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:06.952 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:06.975 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:06.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:06.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:07.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:07.115 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:07.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:07.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:07.217 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:07.250 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:07.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:07.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:07.352 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:07.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:07.390 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:07.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:07.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:07.531 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:07.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:07.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:07.596 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:07 smithi144 ceph-mon[33984]: pgmap v590: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 21464/247662 objects misplaced (8.667%); 26 MiB/s, 408 objects/s recovering 2024-03-11T22:21:07.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:07.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:07.673 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:07.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:07.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:07.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:07 smithi045 ceph-mon[39715]: pgmap v590: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 21464/247662 objects misplaced (8.667%); 26 MiB/s, 408 objects/s recovering 2024-03-11T22:21:07.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:07.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:07 smithi045 ceph-mon[30178]: pgmap v590: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 21464/247662 objects misplaced (8.667%); 26 MiB/s, 408 objects/s recovering 2024-03-11T22:21:07.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:08.016 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:08.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:08.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:08.160 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:08.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:08.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:08.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:08.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:08.400 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:08.440 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:08.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:08.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:08.542 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:08.575 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:08.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:08.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:08.677 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:08.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:08.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:08.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:08.856 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:08.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:08.958 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:08.994 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:08.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:08.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:09.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:09.140 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:09.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:09.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:09.242 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:09.278 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:09.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:09.378 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:09.412 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:09.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:09.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:09.513 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:09.542 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:09.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:09.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:09.643 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:09.680 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:09.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:09.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:09.782 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:09.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:09 smithi045 ceph-mon[39715]: pgmap v591: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 20187/244824 objects misplaced (8.246%); 26 MiB/s, 410 objects/s recovering 2024-03-11T22:21:09.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:09 smithi045 ceph-mon[30178]: pgmap v591: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 20187/244824 objects misplaced (8.246%); 26 MiB/s, 410 objects/s recovering 2024-03-11T22:21:09.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:09.819 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:09.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:09.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:09 smithi144 ceph-mon[33984]: pgmap v591: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 20187/244824 objects misplaced (8.246%); 26 MiB/s, 410 objects/s recovering 2024-03-11T22:21:09.920 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:09.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:09.956 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:09.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:10.057 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:10.094 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:10.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:10.195 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:10.229 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:10.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:10.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:10.331 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:10.363 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:10.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:10.465 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:10.502 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:10.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:10.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:10.604 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:10.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T22:21:10.636 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:10.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:10.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:10.739 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:10.773 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:10.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:10.874 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:10.903 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:10.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:10.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:11.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:11.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:11.058 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:11.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:11.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:11.214 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:11.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:11.317 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:11.350 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:11.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:11.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:11.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:11.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:11.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:11.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:11 smithi045 ceph-mon[39715]: pgmap v592: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 20187/244824 objects misplaced (8.246%); 26 MiB/s, 410 objects/s recovering 2024-03-11T22:21:11.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:11 smithi045 ceph-mon[30178]: pgmap v592: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 20187/244824 objects misplaced (8.246%); 26 MiB/s, 410 objects/s recovering 2024-03-11T22:21:11.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:11.630 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:11.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:11.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:11.781 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:11.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:11.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:11.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:11 smithi144 ceph-mon[33984]: pgmap v592: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s; 20187/244824 objects misplaced (8.246%); 26 MiB/s, 410 objects/s recovering 2024-03-11T22:21:11.883 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:11.921 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:11.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:12.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:12.070 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:12.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:12.172 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:12.202 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:12.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:12.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:12.355 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:12.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:12.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:12.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:12.491 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:12.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:12.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:12.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:12.630 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:12.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:12.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:12.732 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:12.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:12.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:12.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:12.937 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:12.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:13.033 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:21:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:21:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:21:13.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:13.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:13.085 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:13.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:13.106 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:21:13.185 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:13.216 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:13.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:13.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:13.318 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:13.341 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:13.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:13.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:13.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:13.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:13.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:13.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:13.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:13 smithi045 ceph-mon[39715]: pgmap v593: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 953 B/s rd, 0 B/s wr, 1.07k op/s; 17120/229164 objects misplaced (7.471%); 27 MiB/s, 429 objects/s recovering 2024-03-11T22:21:13.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:13 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:21:13.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:13 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:21:13.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:13 smithi045 ceph-mon[30178]: pgmap v593: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 953 B/s rd, 0 B/s wr, 1.07k op/s; 17120/229164 objects misplaced (7.471%); 27 MiB/s, 429 objects/s recovering 2024-03-11T22:21:13.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:13 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:21:13.533 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:13 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:21:13.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:13.617 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:13.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:13.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:13.718 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:13.750 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:13.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:13.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:13 smithi144 ceph-mon[33984]: pgmap v593: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 953 B/s rd, 0 B/s wr, 1.07k op/s; 17120/229164 objects misplaced (7.471%); 27 MiB/s, 429 objects/s recovering 2024-03-11T22:21:13.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:13 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:21:13.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:13 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:21:13.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:13.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:13.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:13.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:14.013 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:14.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:14.114 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:14.141 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:14.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:14.243 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:14.271 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:14.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:14.373 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:14.397 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:14.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:14.499 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:14.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:14.531 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:14.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:14.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:14.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:14.670 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:14.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:14.772 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:14.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:14 smithi045 ceph-mon[39715]: osdmap e318: 8 total, 8 up, 8 in 2024-03-11T22:21:14.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:14 smithi045 ceph-mon[30178]: osdmap e318: 8 total, 8 up, 8 in 2024-03-11T22:21:14.795 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:14.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:14.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:14.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:14 smithi144 ceph-mon[33984]: osdmap e318: 8 total, 8 up, 8 in 2024-03-11T22:21:14.898 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:14.924 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:14.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:15.026 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:15.058 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:15.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:15.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:15.160 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:15.186 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:15.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:15.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:15.316 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:15.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:15.418 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:15.443 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:15.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:15.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:15.544 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:15.574 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:15.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:15.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:15.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:15.720 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:15.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:15.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:15 smithi045 ceph-mon[30178]: pgmap v595: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 17120/229164 objects misplaced (7.471%); 24 MiB/s, 388 objects/s recovering 2024-03-11T22:21:15.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:15 smithi045 ceph-mon[30178]: osdmap e319: 8 total, 8 up, 8 in 2024-03-11T22:21:15.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:15 smithi045 ceph-mon[39715]: pgmap v595: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 17120/229164 objects misplaced (7.471%); 24 MiB/s, 388 objects/s recovering 2024-03-11T22:21:15.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:15 smithi045 ceph-mon[39715]: osdmap e319: 8 total, 8 up, 8 in 2024-03-11T22:21:15.822 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:15.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:15 smithi144 ceph-mon[33984]: pgmap v595: 168 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 162 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 17120/229164 objects misplaced (7.471%); 24 MiB/s, 388 objects/s recovering 2024-03-11T22:21:15.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:15 smithi144 ceph-mon[33984]: osdmap e319: 8 total, 8 up, 8 in 2024-03-11T22:21:15.868 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:15.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:15.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:15.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:15.997 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:15.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:16.099 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:16.129 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:16.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:16.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:16.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:16.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:16.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:16.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:16.370 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:16.403 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:16.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:16.505 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:16.537 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:16.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:16.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:16.639 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:16.671 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:16.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:16.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:16.685 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:16 smithi045 ceph-mon[30178]: osdmap e320: 8 total, 8 up, 8 in 2024-03-11T22:21:16.685 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:21:16.685 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:21:16.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:16 smithi045 ceph-mon[39715]: osdmap e320: 8 total, 8 up, 8 in 2024-03-11T22:21:16.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:21:16.685 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:21:16.774 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:16.804 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:16.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:16.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:16 smithi144 ceph-mon[33984]: osdmap e320: 8 total, 8 up, 8 in 2024-03-11T22:21:16.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:21:16.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:21:16.847 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:21:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:21:16.906 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:16.933 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:16.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:17.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:17.059 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:17.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:17.160 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:17.192 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:17.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:17.294 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:17.327 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:17.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:17.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:17.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:17.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:17.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:17.559 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:21:17.559 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:21:17.559 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:21:17.559 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:21:17.559 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:21:17.559 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:21:17.559 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:21:17.559 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:21:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:21:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:21:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:21:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:21:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:21:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:21:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:21:17.560 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:21:17.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:17.598 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:17.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:17.699 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:17.734 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:17.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:17.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:17 smithi045 ceph-mon[39715]: pgmap v598: 168 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.36k op/s; 14584/220281 objects misplaced (6.621%); 41 MiB/s, 653 objects/s recovering 2024-03-11T22:21:17.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:17.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:17 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3237058424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:21:17.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:17 smithi045 ceph-mon[39715]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:21:17.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:17 smithi045 ceph-mon[30178]: pgmap v598: 168 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.36k op/s; 14584/220281 objects misplaced (6.621%); 41 MiB/s, 653 objects/s recovering 2024-03-11T22:21:17.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:17.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:17 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3237058424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:21:17.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:17 smithi045 ceph-mon[30178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:21:17.836 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:17.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:17 smithi144 ceph-mon[33984]: pgmap v598: 168 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.36k op/s; 14584/220281 objects misplaced (6.621%); 41 MiB/s, 653 objects/s recovering 2024-03-11T22:21:17.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:17.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:17 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3237058424' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:21:17.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:17 smithi144 ceph-mon[33984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:21:17.866 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:17.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:17.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:17.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:18.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:18.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:18.149 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:18.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:18.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:18.290 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:18.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:18.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:18.392 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:18.424 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:18.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:18.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:18.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:18.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:18.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:18.710 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:18.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:18.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:18.811 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:18.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:18.847 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:18.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:18.947 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:18.983 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:18.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:19.085 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:19.109 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:19.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:19.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:19.253 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:19.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:19.355 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:19.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:19.387 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:19.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:19.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:19.526 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:19.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:19.629 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:19.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:19.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:19.772 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:19.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:19 smithi045 ceph-mon[30178]: pgmap v599: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 609 op/s; 12552/218205 objects misplaced (5.752%); 39 MiB/s, 631 objects/s recovering 2024-03-11T22:21:19.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:19 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:21:19.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:19 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:21:19.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:19 smithi045 ceph-mon[39715]: pgmap v599: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 609 op/s; 12552/218205 objects misplaced (5.752%); 39 MiB/s, 631 objects/s recovering 2024-03-11T22:21:19.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:19 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:21:19.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:19 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:21:19.804 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:19.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:19.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:19.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:19 smithi144 ceph-mon[33984]: pgmap v599: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 609 op/s; 12552/218205 objects misplaced (5.752%); 39 MiB/s, 631 objects/s recovering 2024-03-11T22:21:19.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:19 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:21:19.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:19 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:21:19.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:19.947 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:19.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:20.048 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:20.084 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:20.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:20.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:20.186 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:20.209 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:20.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:20.311 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:20.349 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:20.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:20.450 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:20.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:20.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:20.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:20.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:20.614 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:20.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:20.716 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:20.749 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:20.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:20.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:20.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:20 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:21:20.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:20 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:21:20.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:20 smithi045 ceph-mon[39715]: osdmap e321: 8 total, 8 up, 8 in 2024-03-11T22:21:20.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:20 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:21:20.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:20 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:21:20.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:20 smithi045 ceph-mon[30178]: osdmap e321: 8 total, 8 up, 8 in 2024-03-11T22:21:20.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:20 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:21:20.881 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:20 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:21:20.881 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:20 smithi144 ceph-mon[33984]: osdmap e321: 8 total, 8 up, 8 in 2024-03-11T22:21:20.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:20.917 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:20.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:21.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:21.055 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:21.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:21.157 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:21.190 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:21.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:21.291 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:21.321 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:21.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:21.423 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:21.464 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:21.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:21.565 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:21.597 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:21.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:21.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:21.700 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:21.743 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:21.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:21.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:21.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:21 smithi045 ceph-mon[39715]: pgmap v601: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 609 op/s; 12552/218205 objects misplaced (5.752%); 39 MiB/s, 631 objects/s recovering 2024-03-11T22:21:21.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:21 smithi045 ceph-mon[30178]: pgmap v601: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 609 op/s; 12552/218205 objects misplaced (5.752%); 39 MiB/s, 631 objects/s recovering 2024-03-11T22:21:21.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:21.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:21 smithi144 ceph-mon[33984]: pgmap v601: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 609 op/s; 12552/218205 objects misplaced (5.752%); 39 MiB/s, 631 objects/s recovering 2024-03-11T22:21:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:21.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:21.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:21.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:22.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:22.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:22.134 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:22.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:22.159 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:22.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:22.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:22.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:22.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:22.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:22.398 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:22.426 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:22.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:22.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:22.563 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:22.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:22.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:22.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:22.704 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:22.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:22.806 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:22.831 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:22.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:22.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:22.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:22.976 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:22.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:23.033 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:21:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:21:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:21:23.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:23.110 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:23.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:23.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:23.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:23.257 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:23.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:23.359 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:23.398 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:23.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:23.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:23.499 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:23.524 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:23.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:23.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:23.665 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:23.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:23.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:23.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:23 smithi045 ceph-mon[39715]: pgmap v602: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 9744/203199 objects misplaced (4.795%); 46 MiB/s, 732 objects/s recovering 2024-03-11T22:21:23.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:23 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:21:23.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:23 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:21:23.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:23 smithi045 ceph-mon[39715]: osdmap e322: 8 total, 8 up, 8 in 2024-03-11T22:21:23.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:23 smithi045 ceph-mon[30178]: pgmap v602: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 9744/203199 objects misplaced (4.795%); 46 MiB/s, 732 objects/s recovering 2024-03-11T22:21:23.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:23 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:21:23.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:23 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:21:23.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:23 smithi045 ceph-mon[30178]: osdmap e322: 8 total, 8 up, 8 in 2024-03-11T22:21:23.805 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:23.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:23.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:23.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:23 smithi144 ceph-mon[33984]: pgmap v602: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 9744/203199 objects misplaced (4.795%); 46 MiB/s, 732 objects/s recovering 2024-03-11T22:21:23.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:23 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:21:23.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:23 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:21:23.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:23 smithi144 ceph-mon[33984]: osdmap e322: 8 total, 8 up, 8 in 2024-03-11T22:21:23.907 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:23.937 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:23.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:23.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:24.039 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:24.070 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:24.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:24.172 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:24.196 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:24.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:24.297 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:24.330 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:24.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:24.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:24.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:24.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:24.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:24.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:24.561 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:24.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:24.592 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:24.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:24.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:24.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:24.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:24.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:24.864 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:24.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:24.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:25.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:25.013 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:25.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:25.114 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:25.154 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:25.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:25.213 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:21:25.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:25.298 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:25.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:25.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:25.445 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:25.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:25.500 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:21:25.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:25 smithi045 ceph-mon[30178]: pgmap v604: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 712 op/s; 9744/203199 objects misplaced (4.795%); 29 MiB/s, 464 objects/s recovering 2024-03-11T22:21:25.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:25 smithi045 ceph-mon[39715]: pgmap v604: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 712 op/s; 9744/203199 objects misplaced (4.795%); 29 MiB/s, 464 objects/s recovering 2024-03-11T22:21:25.545 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:25.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:25.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:25.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:25.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:25.746 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:25.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:25.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:25 smithi144 ceph-mon[33984]: pgmap v604: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 712 op/s; 9744/203199 objects misplaced (4.795%); 29 MiB/s, 464 objects/s recovering 2024-03-11T22:21:25.848 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:25.877 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:25.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:25.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:26.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:26.023 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:26.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:26.124 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:26.125 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:21:26.125 INFO:teuthology.orchestra.run.smithi045.stdout:{"fsid":"f75aafa4-dff2-11ee-95c8-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1003,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":322,"num_osds":8,"num_up_osds":8,"osd_up_since":1710194897,"num_in_osds":8,"osd_in_since":1710194880,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":164},{"state_name":"active+remapped+backfilling","count":3},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":168,"num_pools":7,"num_objects":67733,"data_bytes":4426504564,"bytes_used":17272651776,"bytes_avail":750687633408,"bytes_total":767960285184,"misplaced_objects":9744,"misplaced_total":203199,"misplaced_ratio":0.04795299189464515,"recovering_objects_per_sec":464,"recovering_bytes_per_sec":30412522,"recovering_keys_per_sec":0,"num_objects_recovered":3714,"num_bytes_recovered":243400704,"num_keys_recovered":0,"read_bytes_sec":639,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":711},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.45:8443/","prometheus":"http://172.21.15.45:9283/"}},"servicemap":{"epoch":14,"modified":"2024-03-11T22:10:16.051510+0000","services":{"rgw":{"daemons":{"summary":"","24335":{"start_epoch":12,"start_stamp":"2024-03-11T22:08:40.174093+0000","gid":24335,"addr":"172.21.15.45:0/3033943457","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)","ceph_version_short":"19.0.0-1607-g94a53970","container_hostname":"smithi045","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi045","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024","kernel_version":"5.14.0-427.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32494648","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"fa98086e-7b4b-4ee6-a4b3-5c5d86ddb897","zone_name":"default","zonegroup_id":"b0e3bd88-a162-4db6-9277-27555b40e874","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-03-11T22:21:26.169 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:26.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:26.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:26.271 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:26.313 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:26.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:26.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:26.415 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:26.445 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:26.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:26.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:26.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:26.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:26.587 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:26.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:26.689 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:26.716 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:26.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:26.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:26.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:26 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1013499336' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T22:21:26.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:26 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1013499336' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T22:21:26.818 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:26.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:26 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1013499336' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T22:21:26.848 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:21:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:21:26.852 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:26.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:26.954 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:26.985 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:26.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:27.087 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:27.123 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:27.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:27.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:27.255 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:21:27.259 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:27.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:27.361 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:27.399 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:27.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:27.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:27.501 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:27.537 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:27.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:27.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:27.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:27.674 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:27.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:27.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:27.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:27 smithi045 ceph-mon[39715]: pgmap v605: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 8153/191424 objects misplaced (4.259%); 25 MiB/s, 404 objects/s recovering 2024-03-11T22:21:27.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:27.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:27 smithi045 ceph-mon[30178]: pgmap v605: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 8153/191424 objects misplaced (4.259%); 25 MiB/s, 404 objects/s recovering 2024-03-11T22:21:27.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:27.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:27.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:27.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:27.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:27 smithi144 ceph-mon[33984]: pgmap v605: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 8153/191424 objects misplaced (4.259%); 25 MiB/s, 404 objects/s recovering 2024-03-11T22:21:27.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:27.908 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:27.941 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:27.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:28.043 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:28.072 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:28.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:28.173 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:28.204 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:28.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:28.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:28.305 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:28.350 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:28.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:28.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:28.514 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:28.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:28.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:28.615 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:28.651 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:28.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:28.753 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:28.789 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:28.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:28.890 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:28.924 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:28.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:28.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:29.025 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:29.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:29.057 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:29.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:29.157 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:29.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:29.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:29.299 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:29.331 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:29.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:29.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:29.433 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:29.463 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:29.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:29.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:29.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:29.608 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-03-11T22:21:29.608 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:21:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:29.609 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:29.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:29.704 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:29 smithi045 ceph-mon[30178]: pgmap v606: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s; 7035/188904 objects misplaced (3.724%); 29 MiB/s, 461 objects/s recovering 2024-03-11T22:21:29.704 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:29 smithi045 ceph-mon[39715]: pgmap v606: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s; 7035/188904 objects misplaced (3.724%); 29 MiB/s, 461 objects/s recovering 2024-03-11T22:21:29.710 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:29.742 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:29.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:29.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:29.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:29 smithi144 ceph-mon[33984]: pgmap v606: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s; 7035/188904 objects misplaced (3.724%); 29 MiB/s, 461 objects/s recovering 2024-03-11T22:21:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:29.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:29.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:29.990 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:30.040 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:30.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:30.141 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:30.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:30.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:30.343 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:30.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:30.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:30.444 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:30.478 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:30.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:30.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:30.580 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:30.613 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:30.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:30.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:30.715 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:30.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:30.792 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:30.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:30.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:30.958 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:30.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:31.059 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:31.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:31.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:31.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:31.231 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:31.264 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:31.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:31.365 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:31.393 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:31.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:31.462 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:21:31.464 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:21:31.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:31.501 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:31 smithi045 ceph-mon[30178]: pgmap v607: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 976 op/s; 7035/188904 objects misplaced (3.724%); 26 MiB/s, 410 objects/s recovering 2024-03-11T22:21:31.501 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:31.501 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:21:31.501 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:31 smithi045 ceph-mon[39715]: pgmap v607: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 976 op/s; 7035/188904 objects misplaced (3.724%); 26 MiB/s, 410 objects/s recovering 2024-03-11T22:21:31.501 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:31.501 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:21:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:31.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:31.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:31.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:31.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:31.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:31.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:31 smithi144 ceph-mon[33984]: pgmap v607: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 976 op/s; 7035/188904 objects misplaced (3.724%); 26 MiB/s, 410 objects/s recovering 2024-03-11T22:21:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:21:31.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:21:31.856 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:31.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:31.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:31.959 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:32.000 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:32.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:32.101 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:32.132 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:32.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:32.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:32.234 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:32.265 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:32.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:32.367 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:32.409 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:32.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:32.510 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:32.546 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:32.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:32.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:32.648 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:32.683 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:32.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:32.783 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:21:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:21:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:21:32.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:32 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:32.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:32 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:21:32.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:32 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:21:32.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:32 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1356528796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:21:32.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:32 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:32.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:32 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:21:32.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:32 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:21:32.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:32 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1356528796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:21:32.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:32.821 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:32.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:32.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:32.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:32 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:21:32.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:32 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:21:32.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:32 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:21:32.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:32 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1356528796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:21:32.854 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:21:32.923 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:32.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:32.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:32.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:33.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:33.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:33.113 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:33.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:33.214 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:33.249 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:33.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:33.351 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:33.383 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:33.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:33.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:33.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:33.525 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:33.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:33.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:33.627 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:33.662 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:33.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:33.763 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:33.766 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:33 smithi045 ceph-mon[39715]: pgmap v608: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s; 4431/173127 objects misplaced (2.559%); 26 MiB/s, 418 objects/s recovering 2024-03-11T22:21:33.767 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:33 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:21:33.767 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:33 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1356528796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:21:33.767 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:33 smithi045 ceph-mon[39715]: osdmap e323: 8 total, 8 up, 8 in 2024-03-11T22:21:33.767 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:33 smithi045 ceph-mon[30178]: pgmap v608: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s; 4431/173127 objects misplaced (2.559%); 26 MiB/s, 418 objects/s recovering 2024-03-11T22:21:33.767 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:33 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:21:33.767 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:33 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1356528796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:21:33.767 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:33 smithi045 ceph-mon[30178]: osdmap e323: 8 total, 8 up, 8 in 2024-03-11T22:21:33.792 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:33.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:33.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:33 smithi144 ceph-mon[33984]: pgmap v608: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s; 4431/173127 objects misplaced (2.559%); 26 MiB/s, 418 objects/s recovering 2024-03-11T22:21:33.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:33 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:21:33.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:33 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1356528796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:21:33.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:33 smithi144 ceph-mon[33984]: osdmap e323: 8 total, 8 up, 8 in 2024-03-11T22:21:33.866 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:21:33.866 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:21:33.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.869 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25142.0:191665 9.0 e322/273 9:03d57f70:::benchmark_data_smithi045_141553_object67545:head v 322'12048, mlcod=322'12044)", 2024-03-11T22:21:33.869 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:32.247484+0000", 2024-03-11T22:21:33.869 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0047182300000000003, 2024-03-11T22:21:33.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.004751674, 2024-03-11T22:21:33.869 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:21:33.869 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:21:33.869 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:21:33.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:21:33.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.247484+0000", 2024-03-11T22:21:33.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:21:33.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:21:33.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.247484+0000", 2024-03-11T22:21:33.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:21:33.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:21:33.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.247485+0000", 2024-03-11T22:21:33.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3900000000000003e-07 2024-03-11T22:21:33.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:21:33.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.247491+0000", 2024-03-11T22:21:33.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.28e-06 2024-03-11T22:21:33.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:21:33.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.247505+0000", 2024-03-11T22:21:33.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4122e-05 2024-03-11T22:21:33.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:21:33.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.247513+0000", 2024-03-11T22:21:33.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0809999999999994e-06 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.247688+0000", 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017502500000000001 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.247739+0000", 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0373e-05 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25142.0:191673 9.6 e322/273 9:6814bb0b:::benchmark_data_smithi045_141553_object67553:head v 322'11613, mlcod=322'11611)", 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:32.249736+0000", 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0024658430000000001, 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025693320000000001, 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.249736+0000", 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:21:33.873 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.249736+0000", 2024-03-11T22:21:33.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:21:33.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:21:33.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.249737+0000", 2024-03-11T22:21:33.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0529999999999999e-06 2024-03-11T22:21:33.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:21:33.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.249745+0000", 2024-03-11T22:21:33.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.323e-06 2024-03-11T22:21:33.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:21:33.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.249760+0000", 2024-03-11T22:21:33.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5325e-05 2024-03-11T22:21:33.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:21:33.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.249769+0000", 2024-03-11T22:21:33.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4740000000000005e-06 2024-03-11T22:21:33.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:21:33.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.249986+0000", 2024-03-11T22:21:33.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021753500000000001 2024-03-11T22:21:33.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:21:33.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.250038+0000", 2024-03-11T22:21:33.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1567999999999998e-05 2024-03-11T22:21:33.876 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:21:33.876 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:21:33.876 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:21:33.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.876 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25142.0:191669 9.2 e322/273 9:42d84eed:::benchmark_data_smithi045_141553_object67549:head v 322'11834, mlcod=322'11832)", 2024-03-11T22:21:33.876 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:32.250493+0000", 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0017088069999999999, 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0018746800000000001, 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.250493+0000", 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.250493+0000", 2024-03-11T22:21:33.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.250494+0000", 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.47e-07 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.250501+0000", 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1060000000000002e-06 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:21:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.250518+0000", 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6075000000000001e-05 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.250527+0000", 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0580000000000004e-06 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.250701+0000", 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017431099999999999 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:21:33.879 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:21:33.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:32.250748+0000", 2024-03-11T22:21:33.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7162e-05 2024-03-11T22:21:33.880 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:21:33.880 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:21:33.880 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:21:33.880 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:21:33.880 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:21:33.880 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 3 2024-03-11T22:21:33.880 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:21:33.894 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:33.931 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:33.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:34.032 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:34.064 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:34.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:34.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:34.165 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:34.199 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:34.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:34.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:34.342 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:34.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:34.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:34.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:34.483 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:34.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:34.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:34.585 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:34.619 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:34.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:34.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:34.721 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:34.754 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:34.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:34.855 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:34.885 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:34.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:34.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:34.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T22:21:34.987 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:35.021 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:35.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:35.123 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:35.155 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:35.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:35.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:35.303 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:35.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:35.404 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:35.434 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:35.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:35.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:35.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:35.585 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:35.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:35.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:35.714 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:35.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:35.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:35.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:35 smithi045 ceph-mon[39715]: pgmap v610: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s; 4431/173127 objects misplaced (2.559%); 26 MiB/s, 418 objects/s recovering 2024-03-11T22:21:35.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:35 smithi045 ceph-mon[30178]: pgmap v610: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s; 4431/173127 objects misplaced (2.559%); 26 MiB/s, 418 objects/s recovering 2024-03-11T22:21:35.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:35.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:21:35.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:35.843 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:35.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:35.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:35 smithi144 ceph-mon[33984]: pgmap v610: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s; 4431/173127 objects misplaced (2.559%); 26 MiB/s, 418 objects/s recovering 2024-03-11T22:21:35.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:35.982 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:35.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:35.992 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:21:36.084 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:36.108 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:36.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:36.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:36.263 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:36.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:36.364 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:36.389 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:36.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:36.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:36.491 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:36.517 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:36.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:36.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:36.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:36.657 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:36.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:36.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:36 smithi045 ceph-mon[39715]: osdmap e324: 8 total, 8 up, 8 in 2024-03-11T22:21:36.748 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:36 smithi045 ceph-mon[30178]: osdmap e324: 8 total, 8 up, 8 in 2024-03-11T22:21:36.759 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:36.785 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:36.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:36.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:36.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:36 smithi144 ceph-mon[33984]: osdmap e324: 8 total, 8 up, 8 in 2024-03-11T22:21:36.847 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:21:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:21:36.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:36.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:36.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:37.000 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:21:37.003 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:21:37.014 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:37.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:37.038 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:37.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:37.139 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:37.163 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:37.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:37.264 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:37.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:37.289 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:37.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:37.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:37.415 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:37.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:37.501 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:37 smithi045 ceph-mon[30178]: pgmap v612: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 2171/161223 objects misplaced (1.347%); 33 MiB/s, 521 objects/s recovering 2024-03-11T22:21:37.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:37.545 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:37.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:37.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:37.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:37.679 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:37.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:37.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:37.780 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:37.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:37 smithi045 ceph-mon[39715]: pgmap v612: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 2171/161223 objects misplaced (1.347%); 33 MiB/s, 521 objects/s recovering 2024-03-11T22:21:37.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:37.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:37 smithi045 ceph-mon[39715]: osdmap e325: 8 total, 8 up, 8 in 2024-03-11T22:21:37.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:37 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1066397005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:21:37.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:37.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:37 smithi045 ceph-mon[30178]: osdmap e325: 8 total, 8 up, 8 in 2024-03-11T22:21:37.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:37 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1066397005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:21:37.805 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:37.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:37.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:37 smithi144 ceph-mon[33984]: pgmap v612: 168 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 164 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 2171/161223 objects misplaced (1.347%); 33 MiB/s, 521 objects/s recovering 2024-03-11T22:21:37.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:37.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:37 smithi144 ceph-mon[33984]: osdmap e325: 8 total, 8 up, 8 in 2024-03-11T22:21:37.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:37 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1066397005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:21:37.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:37.930 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:37.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:38.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:38.056 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:38.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:38.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:38.157 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:38.182 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:38.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:38.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:38.200 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":613,"stamp":"2024-03-11T22:21:36.486652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3509524852,"num_objects":53741,"num_object_clones":0,"num_object_copies":161223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2171,"num_objects_unfound":0,"num_objects_dirty":53741,"num_whiteouts":0,"num_read":1873,"num_read_kb":1625,"num_write":195415,"num_write_kb":7943534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32953,"num_bytes_recovered":2152204700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75034,"ondisk_log_size":75034,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":508,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14387520,"kb_used_data":10958584,"kb_used_omap":12,"kb_used_meta":3428531,"kb_avail":735573696,"statfs":{"total":767960285184,"available":753227464704,"internally_reserved":0,"allocated":11221590016,"data_stored":11218940643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":3510816307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,4,6,12,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-604700672,"num_objects":-9227,"num_object_clones":0,"num_object_copies":-27681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4864,"num_objects_unfound":0,"num_objects_dirty":-9227,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4177,"num_bytes_recovered":273678360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.168408"},"pg_stats":[{"pgid":"9.1","version":"324'12276","reported_seq":24193,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.637583+0000","last_change":"2024-03-11T22:21:14.515399+0000","last_active":"2024-03-11T22:21:35.637583+0000","last_peered":"2024-03-11T22:21:35.637583+0000","last_clean":"2024-03-11T22:21:35.637583+0000","last_became_active":"2024-03-11T22:21:14.450983+0000","last_became_peered":"2024-03-11T22:21:14.450983+0000","last_unstale":"2024-03-11T22:21:35.637583+0000","last_undegraded":"2024-03-11T22:21:35.637583+0000","last_fullsized":"2024-03-11T22:21:35.637583+0000","mapping_epoch":318,"log_start":"308'8200","ondisk_log_start":"308'8200","created":273,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218365952,"num_objects":3332,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12276,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5892,"num_bytes_recovered":386007040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"294'25","reported_seq":372,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506750+0000","last_change":"2024-03-11T22:19:41.508884+0000","last_active":"2024-03-11T22:21:35.506750+0000","last_peered":"2024-03-11T22:21:35.506750+0000","last_clean":"2024-03-11T22:21:35.506750+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:21:35.506750+0000","last_undegraded":"2024-03-11T22:21:35.506750+0000","last_fullsized":"2024-03-11T22:21:35.506750+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:19:41.508161+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":560,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.499585+0000","last_change":"2024-03-11T22:19:28.586829+0000","last_active":"2024-03-11T22:21:35.499585+0000","last_peered":"2024-03-11T22:21:35.499585+0000","last_clean":"2024-03-11T22:21:35.499585+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:21:35.499585+0000","last_undegraded":"2024-03-11T22:21:35.499585+0000","last_fullsized":"2024-03-11T22:21:35.499585+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:28.586614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:28.586614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":742,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506043+0000","last_change":"2024-03-11T22:19:40.008490+0000","last_active":"2024-03-11T22:21:35.506043+0000","last_peered":"2024-03-11T22:21:35.506043+0000","last_clean":"2024-03-11T22:21:35.506043+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:21:35.506043+0000","last_undegraded":"2024-03-11T22:21:35.506043+0000","last_fullsized":"2024-03-11T22:21:35.506043+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:19:40.008325+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:19:40.008325+0000","last_clean_scrub_stamp":"2024-03-11T22:19:40.008325+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":326,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.511694+0000","last_change":"2024-03-11T22:18:37.116172+0000","last_active":"2024-03-11T22:21:35.511694+0000","last_peered":"2024-03-11T22:21:35.511694+0000","last_clean":"2024-03-11T22:21:35.511694+0000","last_became_active":"2024-03-11T22:14:52.243690+0000","last_became_peered":"2024-03-11T22:14:52.243690+0000","last_unstale":"2024-03-11T22:21:35.511694+0000","last_undegraded":"2024-03-11T22:21:35.511694+0000","last_fullsized":"2024-03-11T22:21:35.511694+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:37.116093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.116093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":883,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.501907+0000","last_change":"2024-03-11T22:20:32.294412+0000","last_active":"2024-03-11T22:21:35.501907+0000","last_peered":"2024-03-11T22:21:35.501907+0000","last_clean":"2024-03-11T22:21:35.501907+0000","last_became_active":"2024-03-11T22:20:32.293356+0000","last_became_peered":"2024-03-11T22:20:32.293356+0000","last_unstale":"2024-03-11T22:21:35.501907+0000","last_undegraded":"2024-03-11T22:21:35.501907+0000","last_fullsized":"2024-03-11T22:21:35.501907+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:20.907743+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.907743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"324'12293","reported_seq":12406,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.639686+0000","last_change":"2024-03-11T22:18:41.783594+0000","last_active":"2024-03-11T22:21:35.639686+0000","last_peered":"2024-03-11T22:21:35.639686+0000","last_clean":"2024-03-11T22:21:35.639686+0000","last_became_active":"2024-03-11T22:18:41.783248+0000","last_became_peered":"2024-03-11T22:18:41.783248+0000","last_unstale":"2024-03-11T22:21:35.639686+0000","last_undegraded":"2024-03-11T22:21:35.639686+0000","last_fullsized":"2024-03-11T22:21:35.639686+0000","mapping_epoch":273,"log_start":"307'7800","ondisk_log_start":"307'7800","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214106112,"num_objects":3267,"num_object_clones":0,"num_object_copies":9801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12293,"num_write_kb":497920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"273'31","reported_seq":410,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.505886+0000","last_change":"2024-03-11T22:20:40.790500+0000","last_active":"2024-03-11T22:21:35.505886+0000","last_peered":"2024-03-11T22:21:35.505886+0000","last_clean":"2024-03-11T22:21:35.505886+0000","last_became_active":"2024-03-11T22:14:52.239649+0000","last_became_peered":"2024-03-11T22:14:52.239649+0000","last_unstale":"2024-03-11T22:21:35.505886+0000","last_undegraded":"2024-03-11T22:21:35.505886+0000","last_fullsized":"2024-03-11T22:21:35.505886+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'31","last_scrub_stamp":"2024-03-11T22:20:40.790339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:40.790339+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:48.818452+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":583,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.848512+0000","last_change":"2024-03-11T22:20:52.712468+0000","last_active":"2024-03-11T22:21:32.848512+0000","last_peered":"2024-03-11T22:21:32.848512+0000","last_clean":"2024-03-11T22:21:32.848512+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:21:32.848512+0000","last_undegraded":"2024-03-11T22:21:32.848512+0000","last_fullsized":"2024-03-11T22:21:32.848512+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:52.712329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:20:52.712329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:11.261543+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":560,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.505002+0000","last_change":"2024-03-11T22:18:44.130366+0000","last_active":"2024-03-11T22:21:35.505002+0000","last_peered":"2024-03-11T22:21:35.505002+0000","last_clean":"2024-03-11T22:21:35.505002+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:21:35.505002+0000","last_undegraded":"2024-03-11T22:21:35.505002+0000","last_fullsized":"2024-03-11T22:21:35.505002+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:44.130252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:18:44.130252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":564,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506546+0000","last_change":"2024-03-11T22:18:51.794420+0000","last_active":"2024-03-11T22:21:35.506546+0000","last_peered":"2024-03-11T22:21:35.506546+0000","last_clean":"2024-03-11T22:21:35.506546+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:21:35.506546+0000","last_undegraded":"2024-03-11T22:21:35.506546+0000","last_fullsized":"2024-03-11T22:21:35.506546+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:51.794313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:51.794313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":859,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494461+0000","last_change":"2024-03-11T22:18:21.646464+0000","last_active":"2024-03-11T22:21:35.494461+0000","last_peered":"2024-03-11T22:21:35.494461+0000","last_clean":"2024-03-11T22:21:35.494461+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:21:35.494461+0000","last_undegraded":"2024-03-11T22:21:35.494461+0000","last_fullsized":"2024-03-11T22:21:35.494461+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:21.646305+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.646305+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"324'12299","reported_seq":19948,"reported_epoch":324,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T22:21:36.280419+0000","last_change":"2024-03-11T22:21:14.518776+0000","last_active":"2024-03-11T22:21:36.280419+0000","last_peered":"2024-03-11T22:21:36.280419+0000","last_clean":"2024-03-11T22:20:28.402388+0000","last_became_active":"2024-03-11T22:20:30.515089+0000","last_became_peered":"2024-03-11T22:20:30.515089+0000","last_unstale":"2024-03-11T22:21:36.280419+0000","last_undegraded":"2024-03-11T22:21:36.280419+0000","last_fullsized":"2024-03-11T22:21:36.280419+0000","mapping_epoch":310,"log_start":"308'8200","ondisk_log_start":"308'8200","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222625792,"num_objects":3397,"num_object_clones":0,"num_object_copies":10191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":231,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12299,"num_write_kb":502272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3760,"num_bytes_recovered":246284288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,1,5],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":3397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"273'19","reported_seq":382,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.848802+0000","last_change":"2024-03-11T22:20:27.089007+0000","last_active":"2024-03-11T22:21:32.848802+0000","last_peered":"2024-03-11T22:21:32.848802+0000","last_clean":"2024-03-11T22:21:32.848802+0000","last_became_active":"2024-03-11T22:14:52.240437+0000","last_became_peered":"2024-03-11T22:14:52.240437+0000","last_unstale":"2024-03-11T22:21:32.848802+0000","last_undegraded":"2024-03-11T22:21:32.848802+0000","last_fullsized":"2024-03-11T22:21:32.848802+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:20:27.088856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:27.088856+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":344,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506057+0000","last_change":"2024-03-11T22:20:45.895055+0000","last_active":"2024-03-11T22:21:35.506057+0000","last_peered":"2024-03-11T22:21:35.506057+0000","last_clean":"2024-03-11T22:21:35.506057+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:21:35.506057+0000","last_undegraded":"2024-03-11T22:21:35.506057+0000","last_fullsized":"2024-03-11T22:21:35.506057+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:45.894940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:45.894940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:50.593037+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":552,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.495319+0000","last_change":"2024-03-11T22:18:37.705396+0000","last_active":"2024-03-11T22:21:35.495319+0000","last_peered":"2024-03-11T22:21:35.495319+0000","last_clean":"2024-03-11T22:21:35.495319+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:21:35.495319+0000","last_undegraded":"2024-03-11T22:21:35.495319+0000","last_fullsized":"2024-03-11T22:21:35.495319+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:37.705305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.705305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":540,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506205+0000","last_change":"2024-03-11T22:18:48.086934+0000","last_active":"2024-03-11T22:21:35.506205+0000","last_peered":"2024-03-11T22:21:35.506205+0000","last_clean":"2024-03-11T22:21:35.506205+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:21:35.506205+0000","last_undegraded":"2024-03-11T22:21:35.506205+0000","last_fullsized":"2024-03-11T22:21:35.506205+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:48.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:48.086856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"93'130","reported_seq":888,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.503206+0000","last_change":"2024-03-11T22:18:20.968124+0000","last_active":"2024-03-11T22:21:35.503206+0000","last_peered":"2024-03-11T22:21:35.503206+0000","last_clean":"2024-03-11T22:21:35.503206+0000","last_became_active":"2024-03-11T22:18:00.271764+0000","last_became_peered":"2024-03-11T22:18:00.271764+0000","last_unstale":"2024-03-11T22:21:35.503206+0000","last_undegraded":"2024-03-11T22:21:35.503206+0000","last_fullsized":"2024-03-11T22:21:35.503206+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:20.968038+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.968038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"324'12072","reported_seq":12185,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.645258+0000","last_change":"2024-03-11T22:18:41.238608+0000","last_active":"2024-03-11T22:21:35.645258+0000","last_peered":"2024-03-11T22:21:35.645258+0000","last_clean":"2024-03-11T22:21:35.645258+0000","last_became_active":"2024-03-11T22:18:41.238269+0000","last_became_peered":"2024-03-11T22:18:41.238269+0000","last_unstale":"2024-03-11T22:21:35.645258+0000","last_undegraded":"2024-03-11T22:21:35.645258+0000","last_fullsized":"2024-03-11T22:21:35.645258+0000","mapping_epoch":273,"log_start":"308'8000","ondisk_log_start":"308'8000","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4072,"log_dups_size":0,"ondisk_log_size":4072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212598784,"num_objects":3244,"num_object_clones":0,"num_object_copies":9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":490112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"273'34","reported_seq":622,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.511355+0000","last_change":"2024-03-11T22:20:26.599126+0000","last_active":"2024-03-11T22:21:35.511355+0000","last_peered":"2024-03-11T22:21:35.511355+0000","last_clean":"2024-03-11T22:21:35.511355+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:21:35.511355+0000","last_undegraded":"2024-03-11T22:21:35.511355+0000","last_fullsized":"2024-03-11T22:21:35.511355+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:20:26.598991+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:20:26.598991+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:48.861807+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":565,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.499338+0000","last_change":"2024-03-11T22:20:29.432709+0000","last_active":"2024-03-11T22:21:35.499338+0000","last_peered":"2024-03-11T22:21:35.499338+0000","last_clean":"2024-03-11T22:21:35.499338+0000","last_became_active":"2024-03-11T22:20:29.426224+0000","last_became_peered":"2024-03-11T22:20:29.426224+0000","last_unstale":"2024-03-11T22:21:35.499338+0000","last_undegraded":"2024-03-11T22:21:35.499338+0000","last_fullsized":"2024-03-11T22:21:35.499338+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:38.013058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:14:38.013058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":433,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.493569+0000","last_change":"2024-03-11T22:20:59.636170+0000","last_active":"2024-03-11T22:21:35.493569+0000","last_peered":"2024-03-11T22:21:35.493569+0000","last_clean":"2024-03-11T22:21:35.493569+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:21:35.493569+0000","last_undegraded":"2024-03-11T22:21:35.493569+0000","last_fullsized":"2024-03-11T22:21:35.493569+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:20:59.636041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:59.636041+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:24.576756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":548,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.493254+0000","last_change":"2024-03-11T22:18:50.805816+0000","last_active":"2024-03-11T22:21:35.493254+0000","last_peered":"2024-03-11T22:21:35.493254+0000","last_clean":"2024-03-11T22:21:35.493254+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:21:35.493254+0000","last_undegraded":"2024-03-11T22:21:35.493254+0000","last_fullsized":"2024-03-11T22:21:35.493254+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:50.805698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:18:50.805698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":860,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494930+0000","last_change":"2024-03-11T22:18:20.663763+0000","last_active":"2024-03-11T22:21:35.494930+0000","last_peered":"2024-03-11T22:21:35.494930+0000","last_clean":"2024-03-11T22:21:35.494930+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:21:35.494930+0000","last_undegraded":"2024-03-11T22:21:35.494930+0000","last_fullsized":"2024-03-11T22:21:35.494930+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:20.663661+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.663661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"324'12264","reported_seq":19772,"reported_epoch":324,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T22:21:36.275575+0000","last_change":"2024-03-11T22:21:13.302337+0000","last_active":"2024-03-11T22:21:36.275575+0000","last_peered":"2024-03-11T22:21:36.275575+0000","last_clean":"2024-03-11T22:20:28.382555+0000","last_became_active":"2024-03-11T22:20:30.580794+0000","last_became_peered":"2024-03-11T22:20:30.580794+0000","last_unstale":"2024-03-11T22:21:36.275575+0000","last_undegraded":"2024-03-11T22:21:36.275575+0000","last_fullsized":"2024-03-11T22:21:36.275575+0000","mapping_epoch":310,"log_start":"307'7700","ondisk_log_start":"307'7700","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210173976,"num_objects":3208,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":147,"num_objects_unfound":0,"num_objects_dirty":3208,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12264,"num_write_kb":495041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3694,"num_bytes_recovered":242024472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,5,4],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":3208}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":337,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.507704+0000","last_change":"2024-03-11T22:20:29.441577+0000","last_active":"2024-03-11T22:21:35.507704+0000","last_peered":"2024-03-11T22:21:35.507704+0000","last_clean":"2024-03-11T22:21:35.507704+0000","last_became_active":"2024-03-11T22:20:29.438896+0000","last_became_peered":"2024-03-11T22:20:29.438896+0000","last_unstale":"2024-03-11T22:21:35.507704+0000","last_undegraded":"2024-03-11T22:21:35.507704+0000","last_fullsized":"2024-03-11T22:21:35.507704+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.293977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.293977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"273'30","reported_seq":650,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.499391+0000","last_change":"2024-03-11T22:20:29.473203+0000","last_active":"2024-03-11T22:21:35.499391+0000","last_peered":"2024-03-11T22:21:35.499391+0000","last_clean":"2024-03-11T22:21:35.499391+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:21:35.499391+0000","last_undegraded":"2024-03-11T22:21:35.499391+0000","last_fullsized":"2024-03-11T22:21:35.499391+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:15:33.497732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:33.497732+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":573,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.500674+0000","last_change":"2024-03-11T22:20:29.441068+0000","last_active":"2024-03-11T22:21:35.500674+0000","last_peered":"2024-03-11T22:21:35.500674+0000","last_clean":"2024-03-11T22:21:35.500674+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:21:35.500674+0000","last_undegraded":"2024-03-11T22:21:35.500674+0000","last_fullsized":"2024-03-11T22:21:35.500674+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.133631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.133631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":319,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.499040+0000","last_change":"2024-03-11T22:15:12.339167+0000","last_active":"2024-03-11T22:21:35.499040+0000","last_peered":"2024-03-11T22:21:35.499040+0000","last_clean":"2024-03-11T22:21:35.499040+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:21:35.499040+0000","last_undegraded":"2024-03-11T22:21:35.499040+0000","last_fullsized":"2024-03-11T22:21:35.499040+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:12.339050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:12.339050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":821,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506146+0000","last_change":"2024-03-11T22:20:57.831615+0000","last_active":"2024-03-11T22:21:35.506146+0000","last_peered":"2024-03-11T22:21:35.506146+0000","last_clean":"2024-03-11T22:21:35.506146+0000","last_became_active":"2024-03-11T22:18:20.240728+0000","last_became_peered":"2024-03-11T22:18:20.240728+0000","last_unstale":"2024-03-11T22:21:35.506146+0000","last_undegraded":"2024-03-11T22:21:35.506146+0000","last_fullsized":"2024-03-11T22:21:35.506146+0000","mapping_epoch":175,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:57.831334+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:20:57.831334+0000","last_clean_scrub_stamp":"2024-03-11T22:20:57.831334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:02.019183+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.c","version":"324'12103","reported_seq":12216,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.612694+0000","last_change":"2024-03-11T22:18:41.260878+0000","last_active":"2024-03-11T22:21:35.612694+0000","last_peered":"2024-03-11T22:21:35.612694+0000","last_clean":"2024-03-11T22:21:35.612694+0000","last_became_active":"2024-03-11T22:18:41.260611+0000","last_became_peered":"2024-03-11T22:18:41.260611+0000","last_unstale":"2024-03-11T22:21:35.612694+0000","last_undegraded":"2024-03-11T22:21:35.612694+0000","last_fullsized":"2024-03-11T22:21:35.612694+0000","mapping_epoch":273,"log_start":"308'8000","ondisk_log_start":"308'8000","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4103,"log_dups_size":0,"ondisk_log_size":4103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218169344,"num_objects":3329,"num_object_clones":0,"num_object_copies":9987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12103,"num_write_kb":493824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":710,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.499493+0000","last_change":"2024-03-11T22:15:31.519229+0000","last_active":"2024-03-11T22:21:35.499493+0000","last_peered":"2024-03-11T22:21:35.499493+0000","last_clean":"2024-03-11T22:21:35.499493+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:21:35.499493+0000","last_undegraded":"2024-03-11T22:21:35.499493+0000","last_fullsized":"2024-03-11T22:21:35.499493+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:15:31.513336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:31.513336+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"273'28","reported_seq":380,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.493804+0000","last_change":"2024-03-11T22:15:36.626072+0000","last_active":"2024-03-11T22:21:35.493804+0000","last_peered":"2024-03-11T22:21:35.493804+0000","last_clean":"2024-03-11T22:21:35.493804+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:21:35.493804+0000","last_undegraded":"2024-03-11T22:21:35.493804+0000","last_fullsized":"2024-03-11T22:21:35.493804+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'14","last_scrub_stamp":"2024-03-11T22:15:36.625916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:36.625916+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":557,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.457939+0000","last_change":"2024-03-11T22:19:28.428409+0000","last_active":"2024-03-11T22:21:22.457939+0000","last_peered":"2024-03-11T22:21:22.457939+0000","last_clean":"2024-03-11T22:21:22.457939+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:21:22.457939+0000","last_undegraded":"2024-03-11T22:21:22.457939+0000","last_fullsized":"2024-03-11T22:21:22.457939+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:19:28.428283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:28.428283+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":505,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.848690+0000","last_change":"2024-03-11T22:19:35.746810+0000","last_active":"2024-03-11T22:21:32.848690+0000","last_peered":"2024-03-11T22:21:32.848690+0000","last_clean":"2024-03-11T22:21:32.848690+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:21:32.848690+0000","last_undegraded":"2024-03-11T22:21:32.848690+0000","last_fullsized":"2024-03-11T22:21:32.848690+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:35.746693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:19:35.746693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":876,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.505992+0000","last_change":"2024-03-11T22:20:32.293956+0000","last_active":"2024-03-11T22:21:35.505992+0000","last_peered":"2024-03-11T22:21:35.505992+0000","last_clean":"2024-03-11T22:21:35.505992+0000","last_became_active":"2024-03-11T22:20:32.293085+0000","last_became_peered":"2024-03-11T22:20:32.293085+0000","last_unstale":"2024-03-11T22:21:35.505992+0000","last_undegraded":"2024-03-11T22:21:35.505992+0000","last_fullsized":"2024-03-11T22:21:35.505992+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:18.905939+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:18.905939+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"323'12439","reported_seq":12552,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494518+0000","last_change":"2024-03-11T22:18:41.236691+0000","last_active":"2024-03-11T22:21:35.494518+0000","last_peered":"2024-03-11T22:21:35.494518+0000","last_clean":"2024-03-11T22:21:35.494518+0000","last_became_active":"2024-03-11T22:18:41.236462+0000","last_became_peered":"2024-03-11T22:18:41.236462+0000","last_unstale":"2024-03-11T22:21:35.494518+0000","last_undegraded":"2024-03-11T22:21:35.494518+0000","last_fullsized":"2024-03-11T22:21:35.494518+0000","mapping_epoch":273,"log_start":"307'7900","ondisk_log_start":"307'7900","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4539,"log_dups_size":0,"ondisk_log_size":4539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219217920,"num_objects":3345,"num_object_clones":0,"num_object_copies":10035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12439,"num_write_kb":505088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":713,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:23.515977+0000","last_change":"2024-03-11T22:18:35.815130+0000","last_active":"2024-03-11T22:21:23.515977+0000","last_peered":"2024-03-11T22:21:23.515977+0000","last_clean":"2024-03-11T22:21:23.515977+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:21:23.515977+0000","last_undegraded":"2024-03-11T22:21:23.515977+0000","last_fullsized":"2024-03-11T22:21:23.515977+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:18:35.815031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:35.815031+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"273'20","reported_seq":603,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.503346+0000","last_change":"2024-03-11T22:19:31.503395+0000","last_active":"2024-03-11T22:21:35.503346+0000","last_peered":"2024-03-11T22:21:35.503346+0000","last_clean":"2024-03-11T22:21:35.503346+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:21:35.503346+0000","last_undegraded":"2024-03-11T22:21:35.503346+0000","last_fullsized":"2024-03-11T22:21:35.503346+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:19:31.502993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:31.502993+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":569,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.493512+0000","last_change":"2024-03-11T22:19:34.706343+0000","last_active":"2024-03-11T22:21:35.493512+0000","last_peered":"2024-03-11T22:21:35.493512+0000","last_clean":"2024-03-11T22:21:35.493512+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:21:35.493512+0000","last_undegraded":"2024-03-11T22:21:35.493512+0000","last_fullsized":"2024-03-11T22:21:35.493512+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.706222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.706222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":545,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.458537+0000","last_change":"2024-03-11T22:19:34.477472+0000","last_active":"2024-03-11T22:21:22.458537+0000","last_peered":"2024-03-11T22:21:22.458537+0000","last_clean":"2024-03-11T22:21:22.458537+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:21:22.458537+0000","last_undegraded":"2024-03-11T22:21:22.458537+0000","last_fullsized":"2024-03-11T22:21:22.458537+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.477349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.477349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":799,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.849355+0000","last_change":"2024-03-11T22:20:43.803186+0000","last_active":"2024-03-11T22:21:32.849355+0000","last_peered":"2024-03-11T22:21:32.849355+0000","last_clean":"2024-03-11T22:21:32.849355+0000","last_became_active":"2024-03-11T22:20:32.290512+0000","last_became_peered":"2024-03-11T22:20:32.290512+0000","last_unstale":"2024-03-11T22:21:32.849355+0000","last_undegraded":"2024-03-11T22:21:32.849355+0000","last_fullsized":"2024-03-11T22:21:32.849355+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:43.803067+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:20:43.803067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:49.661991+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"324'12157","reported_seq":23979,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.614203+0000","last_change":"2024-03-11T22:21:15.469013+0000","last_active":"2024-03-11T22:21:35.614203+0000","last_peered":"2024-03-11T22:21:35.614203+0000","last_clean":"2024-03-11T22:21:35.614203+0000","last_became_active":"2024-03-11T22:21:15.407140+0000","last_became_peered":"2024-03-11T22:21:15.407140+0000","last_unstale":"2024-03-11T22:21:35.614203+0000","last_undegraded":"2024-03-11T22:21:35.614203+0000","last_fullsized":"2024-03-11T22:21:35.614203+0000","mapping_epoch":319,"log_start":"306'7600","ondisk_log_start":"306'7600","created":273,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4557,"log_dups_size":0,"ondisk_log_size":4557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212140032,"num_objects":3237,"num_object_clones":0,"num_object_copies":9711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12157,"num_write_kb":492608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5847,"num_bytes_recovered":383188992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":567,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.848387+0000","last_change":"2024-03-11T22:19:43.625744+0000","last_active":"2024-03-11T22:21:32.848387+0000","last_peered":"2024-03-11T22:21:32.848387+0000","last_clean":"2024-03-11T22:21:32.848387+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:21:32.848387+0000","last_undegraded":"2024-03-11T22:21:32.848387+0000","last_fullsized":"2024-03-11T22:21:32.848387+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:43.625634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:19:43.625634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"273'29","reported_seq":604,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.457052+0000","last_change":"2024-03-11T22:19:38.450843+0000","last_active":"2024-03-11T22:21:22.457052+0000","last_peered":"2024-03-11T22:21:22.457052+0000","last_clean":"2024-03-11T22:21:22.457052+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:21:22.457052+0000","last_undegraded":"2024-03-11T22:21:22.457052+0000","last_fullsized":"2024-03-11T22:21:22.457052+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'29","last_scrub_stamp":"2024-03-11T22:19:38.450581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.450581+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":569,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.504824+0000","last_change":"2024-03-11T22:18:38.153695+0000","last_active":"2024-03-11T22:21:35.504824+0000","last_peered":"2024-03-11T22:21:35.504824+0000","last_clean":"2024-03-11T22:21:35.504824+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:21:35.504824+0000","last_undegraded":"2024-03-11T22:21:35.504824+0000","last_fullsized":"2024-03-11T22:21:35.504824+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:38.153614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.153614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":557,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.502215+0000","last_change":"2024-03-11T22:18:23.998360+0000","last_active":"2024-03-11T22:21:35.502215+0000","last_peered":"2024-03-11T22:21:35.502215+0000","last_clean":"2024-03-11T22:21:35.502215+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:21:35.502215+0000","last_undegraded":"2024-03-11T22:21:35.502215+0000","last_fullsized":"2024-03-11T22:21:35.502215+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:23.998251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:23.998251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":774,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.849061+0000","last_change":"2024-03-11T22:20:44.849410+0000","last_active":"2024-03-11T22:21:32.849061+0000","last_peered":"2024-03-11T22:21:32.849061+0000","last_clean":"2024-03-11T22:21:32.849061+0000","last_became_active":"2024-03-11T22:18:17.238247+0000","last_became_peered":"2024-03-11T22:18:17.238247+0000","last_unstale":"2024-03-11T22:21:32.849061+0000","last_undegraded":"2024-03-11T22:21:32.849061+0000","last_fullsized":"2024-03-11T22:21:32.849061+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:44.849279+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:20:44.849279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:04.442646+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"322'11745","reported_seq":11854,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:31.807433+0000","last_change":"2024-03-11T22:18:41.783592+0000","last_active":"2024-03-11T22:21:31.807433+0000","last_peered":"2024-03-11T22:21:31.807433+0000","last_clean":"2024-03-11T22:21:31.807433+0000","last_became_active":"2024-03-11T22:18:41.783277+0000","last_became_peered":"2024-03-11T22:18:41.783277+0000","last_unstale":"2024-03-11T22:21:31.807433+0000","last_undegraded":"2024-03-11T22:21:31.807433+0000","last_fullsized":"2024-03-11T22:21:31.807433+0000","mapping_epoch":273,"log_start":"304'6400","ondisk_log_start":"304'6400","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231931904,"num_objects":3539,"num_object_clones":0,"num_object_copies":10617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11745,"num_write_kb":489088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":553,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.495383+0000","last_change":"2024-03-11T22:18:25.668662+0000","last_active":"2024-03-11T22:21:35.495383+0000","last_peered":"2024-03-11T22:21:35.495383+0000","last_clean":"2024-03-11T22:21:35.495383+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:21:35.495383+0000","last_undegraded":"2024-03-11T22:21:35.495383+0000","last_fullsized":"2024-03-11T22:21:35.495383+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:25.668581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:25.668581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"273'24","reported_seq":596,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.457836+0000","last_change":"2024-03-11T22:20:29.380624+0000","last_active":"2024-03-11T22:21:22.457836+0000","last_peered":"2024-03-11T22:21:22.457836+0000","last_clean":"2024-03-11T22:21:22.457836+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:21:22.457836+0000","last_undegraded":"2024-03-11T22:21:22.457836+0000","last_fullsized":"2024-03-11T22:21:22.457836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'24","last_scrub_stamp":"2024-03-11T22:20:29.380527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:29.380527+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:49.652759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":1332,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.502709+0000","last_change":"2024-03-11T22:18:27.077438+0000","last_active":"2024-03-11T22:21:35.502709+0000","last_peered":"2024-03-11T22:21:35.502709+0000","last_clean":"2024-03-11T22:21:35.502709+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:21:35.502709+0000","last_undegraded":"2024-03-11T22:21:35.502709+0000","last_fullsized":"2024-03-11T22:21:35.502709+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:18:27.077275+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.077275+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":748,"num_read_kb":743,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":557,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.456504+0000","last_change":"2024-03-11T22:18:28.722689+0000","last_active":"2024-03-11T22:21:22.456504+0000","last_peered":"2024-03-11T22:21:22.456504+0000","last_clean":"2024-03-11T22:21:22.456504+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:21:22.456504+0000","last_undegraded":"2024-03-11T22:21:22.456504+0000","last_fullsized":"2024-03-11T22:21:22.456504+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:28.722615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:28.722615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":565,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.504620+0000","last_change":"2024-03-11T22:18:36.127513+0000","last_active":"2024-03-11T22:21:35.504620+0000","last_peered":"2024-03-11T22:21:35.504620+0000","last_clean":"2024-03-11T22:21:35.504620+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:21:35.504620+0000","last_undegraded":"2024-03-11T22:21:35.504620+0000","last_fullsized":"2024-03-11T22:21:35.504620+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:36.127435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:36.127435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":866,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.503430+0000","last_change":"2024-03-11T22:19:29.435468+0000","last_active":"2024-03-11T22:21:35.503430+0000","last_peered":"2024-03-11T22:21:35.503430+0000","last_clean":"2024-03-11T22:21:35.503430+0000","last_became_active":"2024-03-11T22:14:38.164101+0000","last_became_peered":"2024-03-11T22:14:38.164101+0000","last_unstale":"2024-03-11T22:21:35.503430+0000","last_undegraded":"2024-03-11T22:21:35.503430+0000","last_fullsized":"2024-03-11T22:21:35.503430+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:29.435334+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.435334+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"324'12328","reported_seq":12441,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.657803+0000","last_change":"2024-03-11T22:18:41.240089+0000","last_active":"2024-03-11T22:21:35.657803+0000","last_peered":"2024-03-11T22:21:35.657803+0000","last_clean":"2024-03-11T22:21:35.657803+0000","last_became_active":"2024-03-11T22:18:41.239829+0000","last_became_peered":"2024-03-11T22:18:41.239829+0000","last_unstale":"2024-03-11T22:21:35.657803+0000","last_undegraded":"2024-03-11T22:21:35.657803+0000","last_fullsized":"2024-03-11T22:21:35.657803+0000","mapping_epoch":273,"log_start":"311'8200","ondisk_log_start":"311'8200","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213385216,"num_objects":3256,"num_object_clones":0,"num_object_copies":9768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12328,"num_write_kb":498688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":527,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506258+0000","last_change":"2024-03-11T22:20:29.420829+0000","last_active":"2024-03-11T22:21:35.506258+0000","last_peered":"2024-03-11T22:21:35.506258+0000","last_clean":"2024-03-11T22:21:35.506258+0000","last_became_active":"2024-03-11T22:20:29.420497+0000","last_became_peered":"2024-03-11T22:20:29.420497+0000","last_unstale":"2024-03-11T22:21:35.506258+0000","last_undegraded":"2024-03-11T22:21:35.506258+0000","last_fullsized":"2024-03-11T22:21:35.506258+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:24.875284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:19:24.875284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"273'28","reported_seq":768,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.848999+0000","last_change":"2024-03-11T22:19:34.671571+0000","last_active":"2024-03-11T22:21:32.848999+0000","last_peered":"2024-03-11T22:21:32.848999+0000","last_clean":"2024-03-11T22:21:32.848999+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:21:32.848999+0000","last_undegraded":"2024-03-11T22:21:32.848999+0000","last_fullsized":"2024-03-11T22:21:32.848999+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:19:34.671335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.671335+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":561,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.498739+0000","last_change":"2024-03-11T22:19:24.533661+0000","last_active":"2024-03-11T22:21:35.498739+0000","last_peered":"2024-03-11T22:21:35.498739+0000","last_clean":"2024-03-11T22:21:35.498739+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:21:35.498739+0000","last_undegraded":"2024-03-11T22:21:35.498739+0000","last_fullsized":"2024-03-11T22:21:35.498739+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:24.533532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:19:24.533532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":872,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.458699+0000","last_change":"2024-03-11T22:20:32.294325+0000","last_active":"2024-03-11T22:21:22.458699+0000","last_peered":"2024-03-11T22:21:22.458699+0000","last_clean":"2024-03-11T22:21:22.458699+0000","last_became_active":"2024-03-11T22:20:32.293740+0000","last_became_peered":"2024-03-11T22:20:32.293740+0000","last_unstale":"2024-03-11T22:21:22.458699+0000","last_undegraded":"2024-03-11T22:21:22.458699+0000","last_fullsized":"2024-03-11T22:21:22.458699+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:44.952136+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:18:44.952136+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":548,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494913+0000","last_change":"2024-03-11T22:18:22.688031+0000","last_active":"2024-03-11T22:21:35.494913+0000","last_peered":"2024-03-11T22:21:35.494913+0000","last_clean":"2024-03-11T22:21:35.494913+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:21:35.494913+0000","last_undegraded":"2024-03-11T22:21:35.494913+0000","last_fullsized":"2024-03-11T22:21:35.494913+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:22.687922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.687922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"323'12373","reported_seq":12486,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494778+0000","last_change":"2024-03-11T22:18:41.260551+0000","last_active":"2024-03-11T22:21:35.494778+0000","last_peered":"2024-03-11T22:21:35.494778+0000","last_clean":"2024-03-11T22:21:35.494778+0000","last_became_active":"2024-03-11T22:18:41.260185+0000","last_became_peered":"2024-03-11T22:18:41.260185+0000","last_unstale":"2024-03-11T22:21:35.494778+0000","last_undegraded":"2024-03-11T22:21:35.494778+0000","last_fullsized":"2024-03-11T22:21:35.494778+0000","mapping_epoch":273,"log_start":"307'7800","ondisk_log_start":"307'7800","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4573,"log_dups_size":0,"ondisk_log_size":4573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219742208,"num_objects":3353,"num_object_clones":0,"num_object_copies":10059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12373,"num_write_kb":503232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":577,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.507147+0000","last_change":"2024-03-11T22:19:37.032204+0000","last_active":"2024-03-11T22:21:35.507147+0000","last_peered":"2024-03-11T22:21:35.507147+0000","last_clean":"2024-03-11T22:21:35.507147+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:21:35.507147+0000","last_undegraded":"2024-03-11T22:21:35.507147+0000","last_fullsized":"2024-03-11T22:21:35.507147+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.032058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.032058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"273'17","reported_seq":606,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.512769+0000","last_change":"2024-03-11T22:19:34.486422+0000","last_active":"2024-03-11T22:21:35.512769+0000","last_peered":"2024-03-11T22:21:35.512769+0000","last_clean":"2024-03-11T22:21:35.512769+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:21:35.512769+0000","last_undegraded":"2024-03-11T22:21:35.512769+0000","last_fullsized":"2024-03-11T22:21:35.512769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'17","last_scrub_stamp":"2024-03-11T22:19:34.486239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.486239+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":602,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506157+0000","last_change":"2024-03-11T22:20:52.869557+0000","last_active":"2024-03-11T22:21:35.506157+0000","last_peered":"2024-03-11T22:21:35.506157+0000","last_clean":"2024-03-11T22:21:35.506157+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:21:35.506157+0000","last_undegraded":"2024-03-11T22:21:35.506157+0000","last_fullsized":"2024-03-11T22:21:35.506157+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:52.869444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:20:52.869444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:57.295297+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":577,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.502817+0000","last_change":"2024-03-11T22:19:46.587019+0000","last_active":"2024-03-11T22:21:35.502817+0000","last_peered":"2024-03-11T22:21:35.502817+0000","last_clean":"2024-03-11T22:21:35.502817+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:21:35.502817+0000","last_undegraded":"2024-03-11T22:21:35.502817+0000","last_fullsized":"2024-03-11T22:21:35.502817+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:19:46.586446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":505,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.849053+0000","last_change":"2024-03-11T22:19:37.776868+0000","last_active":"2024-03-11T22:21:32.849053+0000","last_peered":"2024-03-11T22:21:32.849053+0000","last_clean":"2024-03-11T22:21:32.849053+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:21:32.849053+0000","last_undegraded":"2024-03-11T22:21:32.849053+0000","last_fullsized":"2024-03-11T22:21:32.849053+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.776600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.776600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":879,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.456364+0000","last_change":"2024-03-11T22:20:32.295561+0000","last_active":"2024-03-11T22:21:22.456364+0000","last_peered":"2024-03-11T22:21:22.456364+0000","last_clean":"2024-03-11T22:21:22.456364+0000","last_became_active":"2024-03-11T22:20:32.295175+0000","last_became_peered":"2024-03-11T22:20:32.295175+0000","last_unstale":"2024-03-11T22:21:22.456364+0000","last_undegraded":"2024-03-11T22:21:22.456364+0000","last_fullsized":"2024-03-11T22:21:22.456364+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:25.392244+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:25.392244+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"322'11924","reported_seq":18894,"reported_epoch":322,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T22:21:31.887136+0000","last_change":"2024-03-11T22:21:13.432152+0000","last_active":"2024-03-11T22:21:31.887136+0000","last_peered":"2024-03-11T22:21:31.887136+0000","last_clean":"2024-03-11T22:20:28.362243+0000","last_became_active":"2024-03-11T22:20:30.567189+0000","last_became_peered":"2024-03-11T22:20:30.567189+0000","last_unstale":"2024-03-11T22:21:31.887136+0000","last_undegraded":"2024-03-11T22:21:31.887136+0000","last_fullsized":"2024-03-11T22:21:31.887136+0000","mapping_epoch":310,"log_start":"304'6600","ondisk_log_start":"304'6600","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233177088,"num_objects":3558,"num_object_clones":0,"num_object_copies":10674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":603,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11924,"num_write_kb":495424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3429,"num_bytes_recovered":224722944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,5,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":3558}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":549,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.458546+0000","last_change":"2024-03-11T22:15:30.922162+0000","last_active":"2024-03-11T22:21:22.458546+0000","last_peered":"2024-03-11T22:21:22.458546+0000","last_clean":"2024-03-11T22:21:22.458546+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:21:22.458546+0000","last_undegraded":"2024-03-11T22:21:22.458546+0000","last_fullsized":"2024-03-11T22:21:22.458546+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:30.922001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:30.922001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"273'40","reported_seq":796,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.504916+0000","last_change":"2024-03-11T22:19:39.417690+0000","last_active":"2024-03-11T22:21:35.504916+0000","last_peered":"2024-03-11T22:21:35.504916+0000","last_clean":"2024-03-11T22:21:35.504916+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:21:35.504916+0000","last_undegraded":"2024-03-11T22:21:35.504916+0000","last_fullsized":"2024-03-11T22:21:35.504916+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'40","last_scrub_stamp":"2024-03-11T22:19:39.417411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:39.417411+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":316,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.456967+0000","last_change":"2024-03-11T22:15:24.899285+0000","last_active":"2024-03-11T22:21:22.456967+0000","last_peered":"2024-03-11T22:21:22.456967+0000","last_clean":"2024-03-11T22:21:22.456967+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:21:22.456967+0000","last_undegraded":"2024-03-11T22:21:22.456967+0000","last_fullsized":"2024-03-11T22:21:22.456967+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:24.899168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:24.899168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":573,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.495475+0000","last_change":"2024-03-11T22:18:36.684061+0000","last_active":"2024-03-11T22:21:35.495475+0000","last_peered":"2024-03-11T22:21:35.495475+0000","last_clean":"2024-03-11T22:21:35.495475+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:21:35.495475+0000","last_undegraded":"2024-03-11T22:21:35.495475+0000","last_fullsized":"2024-03-11T22:21:35.495475+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:18:36.683930+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:18:36.683930+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":553,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.497531+0000","last_change":"2024-03-11T22:20:29.434967+0000","last_active":"2024-03-11T22:21:35.497531+0000","last_peered":"2024-03-11T22:21:35.497531+0000","last_clean":"2024-03-11T22:21:35.497531+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:21:35.497531+0000","last_undegraded":"2024-03-11T22:21:35.497531+0000","last_fullsized":"2024-03-11T22:21:35.497531+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:32.518054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:15:32.518054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":857,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494646+0000","last_change":"2024-03-11T22:18:41.790977+0000","last_active":"2024-03-11T22:21:35.494646+0000","last_peered":"2024-03-11T22:21:35.494646+0000","last_clean":"2024-03-11T22:21:35.494646+0000","last_became_active":"2024-03-11T22:14:38.164859+0000","last_became_peered":"2024-03-11T22:14:38.164859+0000","last_unstale":"2024-03-11T22:21:35.494646+0000","last_undegraded":"2024-03-11T22:21:35.494646+0000","last_fullsized":"2024-03-11T22:21:35.494646+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:41.790896+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:41.790896+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"323'11847","reported_seq":11960,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.492943+0000","last_change":"2024-03-11T22:18:41.233217+0000","last_active":"2024-03-11T22:21:35.492943+0000","last_peered":"2024-03-11T22:21:35.492943+0000","last_clean":"2024-03-11T22:21:35.492943+0000","last_became_active":"2024-03-11T22:18:41.232989+0000","last_became_peered":"2024-03-11T22:18:41.232989+0000","last_unstale":"2024-03-11T22:21:35.492943+0000","last_undegraded":"2024-03-11T22:21:35.492943+0000","last_fullsized":"2024-03-11T22:21:35.492943+0000","mapping_epoch":273,"log_start":"306'7300","ondisk_log_start":"306'7300","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205586432,"num_objects":3137,"num_object_clones":0,"num_object_copies":9411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11847,"num_write_kb":479488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"273'15","reported_seq":593,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.457270+0000","last_change":"2024-03-11T22:19:32.486126+0000","last_active":"2024-03-11T22:21:22.457270+0000","last_peered":"2024-03-11T22:21:22.457270+0000","last_clean":"2024-03-11T22:21:22.457270+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:21:22.457270+0000","last_undegraded":"2024-03-11T22:21:22.457270+0000","last_fullsized":"2024-03-11T22:21:22.457270+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:19:32.485571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.485571+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":560,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.493332+0000","last_change":"2024-03-11T22:18:40.745227+0000","last_active":"2024-03-11T22:21:35.493332+0000","last_peered":"2024-03-11T22:21:35.493332+0000","last_clean":"2024-03-11T22:21:35.493332+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:21:35.493332+0000","last_undegraded":"2024-03-11T22:21:35.493332+0000","last_fullsized":"2024-03-11T22:21:35.493332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.745151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.745151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":576,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.505923+0000","last_change":"2024-03-11T22:19:42.927329+0000","last_active":"2024-03-11T22:21:35.505923+0000","last_peered":"2024-03-11T22:21:35.505923+0000","last_clean":"2024-03-11T22:21:35.505923+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:21:35.505923+0000","last_undegraded":"2024-03-11T22:21:35.505923+0000","last_fullsized":"2024-03-11T22:21:35.505923+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:42.926178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:19:42.926178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":556,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.456714+0000","last_change":"2024-03-11T22:20:29.418105+0000","last_active":"2024-03-11T22:21:22.456714+0000","last_peered":"2024-03-11T22:21:22.456714+0000","last_clean":"2024-03-11T22:21:22.456714+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:21:22.456714+0000","last_undegraded":"2024-03-11T22:21:22.456714+0000","last_fullsized":"2024-03-11T22:21:22.456714+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:27.721245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.721245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"93'130","reported_seq":710,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.497049+0000","last_change":"2024-03-11T22:20:32.293722+0000","last_active":"2024-03-11T22:21:35.497049+0000","last_peered":"2024-03-11T22:21:35.497049+0000","last_clean":"2024-03-11T22:21:35.497049+0000","last_became_active":"2024-03-11T22:20:32.292676+0000","last_became_peered":"2024-03-11T22:20:32.292676+0000","last_unstale":"2024-03-11T22:21:35.497049+0000","last_undegraded":"2024-03-11T22:21:35.497049+0000","last_fullsized":"2024-03-11T22:21:35.497049+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:22.939363+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.939363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.7","version":"323'12210","reported_seq":25622,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.495100+0000","last_change":"2024-03-11T22:20:55.630922+0000","last_active":"2024-03-11T22:21:35.495100+0000","last_peered":"2024-03-11T22:21:35.495100+0000","last_clean":"2024-03-11T22:21:35.495100+0000","last_became_active":"2024-03-11T22:20:55.617058+0000","last_became_peered":"2024-03-11T22:20:55.617058+0000","last_unstale":"2024-03-11T22:21:35.495100+0000","last_undegraded":"2024-03-11T22:21:35.495100+0000","last_fullsized":"2024-03-11T22:21:35.495100+0000","mapping_epoch":315,"log_start":"306'7700","ondisk_log_start":"306'7700","created":273,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4510,"log_dups_size":0,"ondisk_log_size":4510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222691328,"num_objects":3398,"num_object_clones":0,"num_object_copies":10194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12210,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6642,"num_bytes_recovered":435290112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"273'35","reported_seq":677,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.848280+0000","last_change":"2024-03-11T22:20:29.437134+0000","last_active":"2024-03-11T22:21:32.848280+0000","last_peered":"2024-03-11T22:21:32.848280+0000","last_clean":"2024-03-11T22:21:32.848280+0000","last_became_active":"2024-03-11T22:20:29.418041+0000","last_became_peered":"2024-03-11T22:20:29.418041+0000","last_unstale":"2024-03-11T22:21:32.848280+0000","last_undegraded":"2024-03-11T22:21:32.848280+0000","last_fullsized":"2024-03-11T22:21:32.848280+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'35","last_scrub_stamp":"2024-03-11T22:19:46.640720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:46.640720+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":511,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.497620+0000","last_change":"2024-03-11T22:20:29.414029+0000","last_active":"2024-03-11T22:21:35.497620+0000","last_peered":"2024-03-11T22:21:35.497620+0000","last_clean":"2024-03-11T22:21:35.497620+0000","last_became_active":"2024-03-11T22:20:29.413664+0000","last_became_peered":"2024-03-11T22:20:29.413664+0000","last_unstale":"2024-03-11T22:21:35.497620+0000","last_undegraded":"2024-03-11T22:21:35.497620+0000","last_fullsized":"2024-03-11T22:21:35.497620+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:32.682157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:32.682157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":576,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.505751+0000","last_change":"2024-03-11T22:20:39.805771+0000","last_active":"2024-03-11T22:21:35.505751+0000","last_peered":"2024-03-11T22:21:35.505751+0000","last_clean":"2024-03-11T22:21:35.505751+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:21:35.505751+0000","last_undegraded":"2024-03-11T22:21:35.505751+0000","last_fullsized":"2024-03-11T22:21:35.505751+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:39.805647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:20:39.805647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:55.656366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":527,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.505641+0000","last_change":"2024-03-11T22:20:29.416481+0000","last_active":"2024-03-11T22:21:35.505641+0000","last_peered":"2024-03-11T22:21:35.505641+0000","last_clean":"2024-03-11T22:21:35.505641+0000","last_became_active":"2024-03-11T22:20:29.415565+0000","last_became_peered":"2024-03-11T22:20:29.415565+0000","last_unstale":"2024-03-11T22:21:35.505641+0000","last_undegraded":"2024-03-11T22:21:35.505641+0000","last_fullsized":"2024-03-11T22:21:35.505641+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:44.717740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:20.931122+0000","last_clean_scrub_stamp":"2024-03-11T22:18:44.717740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"93'130","reported_seq":760,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.849067+0000","last_change":"2024-03-11T22:20:36.022151+0000","last_active":"2024-03-11T22:21:32.849067+0000","last_peered":"2024-03-11T22:21:32.849067+0000","last_clean":"2024-03-11T22:21:32.849067+0000","last_became_active":"2024-03-11T22:14:43.195433+0000","last_became_peered":"2024-03-11T22:14:43.195433+0000","last_unstale":"2024-03-11T22:21:32.849067+0000","last_undegraded":"2024-03-11T22:21:32.849067+0000","last_fullsized":"2024-03-11T22:21:32.849067+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:36.022039+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:20:36.022039+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:45.766952+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"324'12133","reported_seq":12246,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.643324+0000","last_change":"2024-03-11T22:18:41.237962+0000","last_active":"2024-03-11T22:21:35.643324+0000","last_peered":"2024-03-11T22:21:35.643324+0000","last_clean":"2024-03-11T22:21:35.643324+0000","last_became_active":"2024-03-11T22:18:41.237565+0000","last_became_peered":"2024-03-11T22:18:41.237565+0000","last_unstale":"2024-03-11T22:21:35.643324+0000","last_undegraded":"2024-03-11T22:21:35.643324+0000","last_fullsized":"2024-03-11T22:21:35.643324+0000","mapping_epoch":273,"log_start":"306'7600","ondisk_log_start":"306'7600","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4533,"log_dups_size":0,"ondisk_log_size":4533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215810048,"num_objects":3293,"num_object_clones":0,"num_object_copies":9879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12133,"num_write_kb":493632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":576,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.505331+0000","last_change":"2024-03-11T22:20:47.845641+0000","last_active":"2024-03-11T22:21:35.505331+0000","last_peered":"2024-03-11T22:21:35.505331+0000","last_clean":"2024-03-11T22:21:35.505331+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:21:35.505331+0000","last_undegraded":"2024-03-11T22:21:35.505331+0000","last_fullsized":"2024-03-11T22:21:35.505331+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:47.845156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:20:47.845156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:48.717654+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"273'18","reported_seq":385,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494894+0000","last_change":"2024-03-11T22:19:38.795869+0000","last_active":"2024-03-11T22:21:35.494894+0000","last_peered":"2024-03-11T22:21:35.494894+0000","last_clean":"2024-03-11T22:21:35.494894+0000","last_became_active":"2024-03-11T22:14:52.247400+0000","last_became_peered":"2024-03-11T22:14:52.247400+0000","last_unstale":"2024-03-11T22:21:35.494894+0000","last_undegraded":"2024-03-11T22:21:35.494894+0000","last_fullsized":"2024-03-11T22:21:35.494894+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:19:38.795650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.795650+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":350,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.511863+0000","last_change":"2024-03-11T22:20:29.417391+0000","last_active":"2024-03-11T22:21:35.511863+0000","last_peered":"2024-03-11T22:21:35.511863+0000","last_clean":"2024-03-11T22:21:35.511863+0000","last_became_active":"2024-03-11T22:20:29.416729+0000","last_became_peered":"2024-03-11T22:20:29.416729+0000","last_unstale":"2024-03-11T22:21:35.511863+0000","last_undegraded":"2024-03-11T22:21:35.511863+0000","last_fullsized":"2024-03-11T22:21:35.511863+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.439756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.439756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":562,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.499789+0000","last_change":"2024-03-11T22:20:29.470787+0000","last_active":"2024-03-11T22:21:35.499789+0000","last_peered":"2024-03-11T22:21:35.499789+0000","last_clean":"2024-03-11T22:21:35.499789+0000","last_became_active":"2024-03-11T22:20:29.438413+0000","last_became_peered":"2024-03-11T22:20:29.438413+0000","last_unstale":"2024-03-11T22:21:35.499789+0000","last_undegraded":"2024-03-11T22:21:35.499789+0000","last_fullsized":"2024-03-11T22:21:35.499789+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:15:40.678629+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:15:40.678629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"93'130","reported_seq":887,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.501813+0000","last_change":"2024-03-11T22:18:21.956432+0000","last_active":"2024-03-11T22:21:35.501813+0000","last_peered":"2024-03-11T22:21:35.501813+0000","last_clean":"2024-03-11T22:21:35.501813+0000","last_became_active":"2024-03-11T22:17:57.246241+0000","last_became_peered":"2024-03-11T22:17:57.246241+0000","last_unstale":"2024-03-11T22:21:35.501813+0000","last_undegraded":"2024-03-11T22:21:35.501813+0000","last_fullsized":"2024-03-11T22:21:35.501813+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:21.956334+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.956334+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"322'12230","reported_seq":19511,"reported_epoch":322,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-11T22:21:31.856117+0000","last_change":"2024-03-11T22:21:14.484934+0000","last_active":"2024-03-11T22:21:31.856117+0000","last_peered":"2024-03-11T22:21:31.856117+0000","last_clean":"2024-03-11T22:20:28.376799+0000","last_became_active":"2024-03-11T22:20:30.582612+0000","last_became_peered":"2024-03-11T22:20:30.582612+0000","last_unstale":"2024-03-11T22:21:31.856117+0000","last_undegraded":"2024-03-11T22:21:31.856117+0000","last_fullsized":"2024-03-11T22:21:31.856117+0000","mapping_epoch":310,"log_start":"304'6900","ondisk_log_start":"304'6900","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5330,"log_dups_size":0,"ondisk_log_size":5330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239206400,"num_objects":3650,"num_object_clones":0,"num_object_copies":10950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1190,"num_objects_unfound":0,"num_objects_dirty":3650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12230,"num_write_kb":508160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3581,"num_bytes_recovered":234684416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":3650}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"310'11","reported_seq":692,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.848392+0000","last_change":"2024-03-11T22:20:57.698062+0000","last_active":"2024-03-11T22:21:32.848392+0000","last_peered":"2024-03-11T22:21:32.848392+0000","last_clean":"2024-03-11T22:21:32.848392+0000","last_became_active":"2024-03-11T22:20:29.418355+0000","last_became_peered":"2024-03-11T22:20:29.418355+0000","last_unstale":"2024-03-11T22:21:32.848392+0000","last_undegraded":"2024-03-11T22:21:32.848392+0000","last_fullsized":"2024-03-11T22:21:32.848392+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:20:57.697970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:57.697970+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:27.002078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"273'34","reported_seq":658,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494268+0000","last_change":"2024-03-11T22:20:29.431072+0000","last_active":"2024-03-11T22:21:35.494268+0000","last_peered":"2024-03-11T22:21:35.494268+0000","last_clean":"2024-03-11T22:21:35.494268+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:21:35.494268+0000","last_undegraded":"2024-03-11T22:21:35.494268+0000","last_fullsized":"2024-03-11T22:21:35.494268+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:19:46.888741+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:19:46.888741+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":356,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.456453+0000","last_change":"2024-03-11T22:20:29.440395+0000","last_active":"2024-03-11T22:21:22.456453+0000","last_peered":"2024-03-11T22:21:22.456453+0000","last_clean":"2024-03-11T22:21:22.456453+0000","last_became_active":"2024-03-11T22:20:29.415226+0000","last_became_peered":"2024-03-11T22:20:29.415226+0000","last_unstale":"2024-03-11T22:21:22.456453+0000","last_undegraded":"2024-03-11T22:21:22.456453+0000","last_fullsized":"2024-03-11T22:21:22.456453+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:20:26.335219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:26.335219+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:47.255101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":548,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.495638+0000","last_change":"2024-03-11T22:19:33.760079+0000","last_active":"2024-03-11T22:21:35.495638+0000","last_peered":"2024-03-11T22:21:35.495638+0000","last_clean":"2024-03-11T22:21:35.495638+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:21:35.495638+0000","last_undegraded":"2024-03-11T22:21:35.495638+0000","last_fullsized":"2024-03-11T22:21:35.495638+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:33.759955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:33.759955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"93'130","reported_seq":903,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.508236+0000","last_change":"2024-03-11T22:20:32.300771+0000","last_active":"2024-03-11T22:21:35.508236+0000","last_peered":"2024-03-11T22:21:35.508236+0000","last_clean":"2024-03-11T22:21:35.508236+0000","last_became_active":"2024-03-11T22:20:32.295606+0000","last_became_peered":"2024-03-11T22:20:32.295606+0000","last_unstale":"2024-03-11T22:21:35.508236+0000","last_undegraded":"2024-03-11T22:21:35.508236+0000","last_fullsized":"2024-03-11T22:21:35.508236+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:41.158328+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:41.158328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"273'12","reported_seq":594,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.500975+0000","last_change":"2024-03-11T22:20:29.483060+0000","last_active":"2024-03-11T22:21:35.500975+0000","last_peered":"2024-03-11T22:21:35.500975+0000","last_clean":"2024-03-11T22:21:35.500975+0000","last_became_active":"2024-03-11T22:20:29.441599+0000","last_became_peered":"2024-03-11T22:20:29.441599+0000","last_unstale":"2024-03-11T22:21:35.500975+0000","last_undegraded":"2024-03-11T22:21:35.500975+0000","last_fullsized":"2024-03-11T22:21:35.500975+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:15:27.448002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:27.448002+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":574,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.847869+0000","last_change":"2024-03-11T22:20:42.845142+0000","last_active":"2024-03-11T22:21:32.847869+0000","last_peered":"2024-03-11T22:21:32.847869+0000","last_clean":"2024-03-11T22:21:32.847869+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:21:32.847869+0000","last_undegraded":"2024-03-11T22:21:32.847869+0000","last_fullsized":"2024-03-11T22:21:32.847869+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:42.844876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:20:42.844876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:08.076350+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":560,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494436+0000","last_change":"2024-03-11T22:20:54.635534+0000","last_active":"2024-03-11T22:21:35.494436+0000","last_peered":"2024-03-11T22:21:35.494436+0000","last_clean":"2024-03-11T22:21:35.494436+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:21:35.494436+0000","last_undegraded":"2024-03-11T22:21:35.494436+0000","last_fullsized":"2024-03-11T22:21:35.494436+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:20:54.635309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:15.693951+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":504,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.849652+0000","last_change":"2024-03-11T22:20:32.993441+0000","last_active":"2024-03-11T22:21:32.849652+0000","last_peered":"2024-03-11T22:21:32.849652+0000","last_clean":"2024-03-11T22:21:32.849652+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:21:32.849652+0000","last_undegraded":"2024-03-11T22:21:32.849652+0000","last_fullsized":"2024-03-11T22:21:32.849652+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:32.993301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:20:32.993301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:01.295107+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"93'130","reported_seq":866,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494825+0000","last_change":"2024-03-11T22:18:45.781426+0000","last_active":"2024-03-11T22:21:35.494825+0000","last_peered":"2024-03-11T22:21:35.494825+0000","last_clean":"2024-03-11T22:21:35.494825+0000","last_became_active":"2024-03-11T22:14:43.194413+0000","last_became_peered":"2024-03-11T22:14:43.194413+0000","last_unstale":"2024-03-11T22:21:35.494825+0000","last_undegraded":"2024-03-11T22:21:35.494825+0000","last_fullsized":"2024-03-11T22:21:35.494825+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:45.781296+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:45.781296+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"93'130","reported_seq":873,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.503031+0000","last_change":"2024-03-11T22:19:27.413677+0000","last_active":"2024-03-11T22:21:35.503031+0000","last_peered":"2024-03-11T22:21:35.503031+0000","last_clean":"2024-03-11T22:21:35.503031+0000","last_became_active":"2024-03-11T22:14:43.198216+0000","last_became_peered":"2024-03-11T22:14:43.198216+0000","last_unstale":"2024-03-11T22:21:35.503031+0000","last_undegraded":"2024-03-11T22:21:35.503031+0000","last_fullsized":"2024-03-11T22:21:35.503031+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:27.413560+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:27.413560+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"273'19","reported_seq":613,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.458133+0000","last_change":"2024-03-11T22:20:29.451236+0000","last_active":"2024-03-11T22:21:22.458133+0000","last_peered":"2024-03-11T22:21:22.458133+0000","last_clean":"2024-03-11T22:21:22.458133+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:21:22.458133+0000","last_undegraded":"2024-03-11T22:21:22.458133+0000","last_fullsized":"2024-03-11T22:21:22.458133+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:18:22.657089+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.657089+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":336,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.848213+0000","last_change":"2024-03-11T22:20:46.807212+0000","last_active":"2024-03-11T22:21:32.848213+0000","last_peered":"2024-03-11T22:21:32.848213+0000","last_clean":"2024-03-11T22:21:32.848213+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:21:32.848213+0000","last_undegraded":"2024-03-11T22:21:32.848213+0000","last_fullsized":"2024-03-11T22:21:32.848213+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:46.807096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:46.807096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:52.495224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":564,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.500892+0000","last_change":"2024-03-11T22:20:29.433304+0000","last_active":"2024-03-11T22:21:35.500892+0000","last_peered":"2024-03-11T22:21:35.500892+0000","last_clean":"2024-03-11T22:21:35.500892+0000","last_became_active":"2024-03-11T22:20:29.433005+0000","last_became_peered":"2024-03-11T22:20:29.433005+0000","last_unstale":"2024-03-11T22:21:35.500892+0000","last_undegraded":"2024-03-11T22:21:35.500892+0000","last_fullsized":"2024-03-11T22:21:35.500892+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:43.544959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:43.544959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":572,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506225+0000","last_change":"2024-03-11T22:19:33.966233+0000","last_active":"2024-03-11T22:21:35.506225+0000","last_peered":"2024-03-11T22:21:35.506225+0000","last_clean":"2024-03-11T22:21:35.506225+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:21:35.506225+0000","last_undegraded":"2024-03-11T22:21:35.506225+0000","last_fullsized":"2024-03-11T22:21:35.506225+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:33.966085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:19:33.966085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"273'19","reported_seq":593,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494081+0000","last_change":"2024-03-11T22:19:29.865779+0000","last_active":"2024-03-11T22:21:35.494081+0000","last_peered":"2024-03-11T22:21:35.494081+0000","last_clean":"2024-03-11T22:21:35.494081+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:21:35.494081+0000","last_undegraded":"2024-03-11T22:21:35.494081+0000","last_fullsized":"2024-03-11T22:21:35.494081+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:19:29.865528+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.865528+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":568,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.456317+0000","last_change":"2024-03-11T22:20:29.417404+0000","last_active":"2024-03-11T22:21:22.456317+0000","last_peered":"2024-03-11T22:21:22.456317+0000","last_clean":"2024-03-11T22:21:22.456317+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:21:22.456317+0000","last_undegraded":"2024-03-11T22:21:22.456317+0000","last_fullsized":"2024-03-11T22:21:22.456317+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:21.693554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.693554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'11","reported_seq":752,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506480+0000","last_change":"2024-03-11T22:20:29.445996+0000","last_active":"2024-03-11T22:21:35.506480+0000","last_peered":"2024-03-11T22:21:35.506480+0000","last_clean":"2024-03-11T22:21:35.506480+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:21:35.506480+0000","last_undegraded":"2024-03-11T22:21:35.506480+0000","last_fullsized":"2024-03-11T22:21:35.506480+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'10","last_scrub_stamp":"2024-03-11T22:19:32.461683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.461683+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":342,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506401+0000","last_change":"2024-03-11T22:20:58.956323+0000","last_active":"2024-03-11T22:21:35.506401+0000","last_peered":"2024-03-11T22:21:35.506401+0000","last_clean":"2024-03-11T22:21:35.506401+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:21:35.506401+0000","last_undegraded":"2024-03-11T22:21:35.506401+0000","last_fullsized":"2024-03-11T22:21:35.506401+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:58.956196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:20:58.956196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:23.295311+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"93'130","reported_seq":859,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494548+0000","last_change":"2024-03-11T22:18:53.739248+0000","last_active":"2024-03-11T22:21:35.494548+0000","last_peered":"2024-03-11T22:21:35.494548+0000","last_clean":"2024-03-11T22:21:35.494548+0000","last_became_active":"2024-03-11T22:14:43.194350+0000","last_became_peered":"2024-03-11T22:14:43.194350+0000","last_unstale":"2024-03-11T22:21:35.494548+0000","last_undegraded":"2024-03-11T22:21:35.494548+0000","last_fullsized":"2024-03-11T22:21:35.494548+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:53.738316+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:18:53.738316+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"273'20","reported_seq":596,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506390+0000","last_change":"2024-03-11T22:16:16.055622+0000","last_active":"2024-03-11T22:21:35.506390+0000","last_peered":"2024-03-11T22:21:35.506390+0000","last_clean":"2024-03-11T22:21:35.506390+0000","last_became_active":"2024-03-11T22:16:15.815814+0000","last_became_peered":"2024-03-11T22:16:15.815814+0000","last_unstale":"2024-03-11T22:21:35.506390+0000","last_undegraded":"2024-03-11T22:21:35.506390+0000","last_fullsized":"2024-03-11T22:21:35.506390+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:15:29.460297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.460297+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":358,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.505651+0000","last_change":"2024-03-11T22:20:31.797304+0000","last_active":"2024-03-11T22:21:35.505651+0000","last_peered":"2024-03-11T22:21:35.505651+0000","last_clean":"2024-03-11T22:21:35.505651+0000","last_became_active":"2024-03-11T22:20:29.415019+0000","last_became_peered":"2024-03-11T22:20:29.415019+0000","last_unstale":"2024-03-11T22:21:35.505651+0000","last_undegraded":"2024-03-11T22:21:35.505651+0000","last_fullsized":"2024-03-11T22:21:35.505651+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:31.796712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:31.796712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:47.886151+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":725,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.493428+0000","last_change":"2024-03-11T22:19:42.854028+0000","last_active":"2024-03-11T22:21:35.493428+0000","last_peered":"2024-03-11T22:21:35.493428+0000","last_clean":"2024-03-11T22:21:35.493428+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:21:35.493428+0000","last_undegraded":"2024-03-11T22:21:35.493428+0000","last_fullsized":"2024-03-11T22:21:35.493428+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:19:42.853428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:42.853428+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":548,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.457733+0000","last_change":"2024-03-11T22:20:29.431434+0000","last_active":"2024-03-11T22:21:22.457733+0000","last_peered":"2024-03-11T22:21:22.457733+0000","last_clean":"2024-03-11T22:21:22.457733+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:21:22.457733+0000","last_undegraded":"2024-03-11T22:21:22.457733+0000","last_fullsized":"2024-03-11T22:21:22.457733+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:20.763181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.763181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"93'130","reported_seq":811,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.455973+0000","last_change":"2024-03-11T22:19:29.435235+0000","last_active":"2024-03-11T22:21:22.455973+0000","last_peered":"2024-03-11T22:21:22.455973+0000","last_clean":"2024-03-11T22:21:22.455973+0000","last_became_active":"2024-03-11T22:14:43.191297+0000","last_became_peered":"2024-03-11T22:14:43.191297+0000","last_unstale":"2024-03-11T22:21:22.455973+0000","last_undegraded":"2024-03-11T22:21:22.455973+0000","last_fullsized":"2024-03-11T22:21:22.455973+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:29.434647+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.434647+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"273'19","reported_seq":382,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.848539+0000","last_change":"2024-03-11T22:20:50.729986+0000","last_active":"2024-03-11T22:21:32.848539+0000","last_peered":"2024-03-11T22:21:32.848539+0000","last_clean":"2024-03-11T22:21:32.848539+0000","last_became_active":"2024-03-11T22:14:52.239810+0000","last_became_peered":"2024-03-11T22:14:52.239810+0000","last_unstale":"2024-03-11T22:21:32.848539+0000","last_undegraded":"2024-03-11T22:21:32.848539+0000","last_fullsized":"2024-03-11T22:21:32.848539+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:20:50.729848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:50.729848+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:58.488489+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":361,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.849243+0000","last_change":"2024-03-11T22:20:47.791284+0000","last_active":"2024-03-11T22:21:32.849243+0000","last_peered":"2024-03-11T22:21:32.849243+0000","last_clean":"2024-03-11T22:21:32.849243+0000","last_became_active":"2024-03-11T22:20:29.420867+0000","last_became_peered":"2024-03-11T22:20:29.420867+0000","last_unstale":"2024-03-11T22:21:32.849243+0000","last_undegraded":"2024-03-11T22:21:32.849243+0000","last_fullsized":"2024-03-11T22:21:32.849243+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:20:47.791128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:47.791128+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:56.876944+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":560,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.503193+0000","last_change":"2024-03-11T22:17:24.726548+0000","last_active":"2024-03-11T22:21:35.503193+0000","last_peered":"2024-03-11T22:21:35.503193+0000","last_clean":"2024-03-11T22:21:35.503193+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:21:35.503193+0000","last_undegraded":"2024-03-11T22:21:35.503193+0000","last_fullsized":"2024-03-11T22:21:35.503193+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:24.726421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:17:24.726421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":540,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.507520+0000","last_change":"2024-03-11T22:18:29.014246+0000","last_active":"2024-03-11T22:21:35.507520+0000","last_peered":"2024-03-11T22:21:35.507520+0000","last_clean":"2024-03-11T22:21:35.507520+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:21:35.507520+0000","last_undegraded":"2024-03-11T22:21:35.507520+0000","last_fullsized":"2024-03-11T22:21:35.507520+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:29.014149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:29.014149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":593,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.503444+0000","last_change":"2024-03-11T22:17:35.738522+0000","last_active":"2024-03-11T22:21:35.503444+0000","last_peered":"2024-03-11T22:21:35.503444+0000","last_clean":"2024-03-11T22:21:35.503444+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:21:35.503444+0000","last_undegraded":"2024-03-11T22:21:35.503444+0000","last_fullsized":"2024-03-11T22:21:35.503444+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'13","last_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:17:35.738211+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":503,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.496627+0000","last_change":"2024-03-11T22:20:29.423108+0000","last_active":"2024-03-11T22:21:35.496627+0000","last_peered":"2024-03-11T22:21:35.496627+0000","last_clean":"2024-03-11T22:21:35.496627+0000","last_became_active":"2024-03-11T22:20:29.422747+0000","last_became_peered":"2024-03-11T22:20:29.422747+0000","last_unstale":"2024-03-11T22:21:35.496627+0000","last_undegraded":"2024-03-11T22:21:35.496627+0000","last_fullsized":"2024-03-11T22:21:35.496627+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:44.563614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:44.563614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":552,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.499661+0000","last_change":"2024-03-11T22:19:39.445467+0000","last_active":"2024-03-11T22:21:35.499661+0000","last_peered":"2024-03-11T22:21:35.499661+0000","last_clean":"2024-03-11T22:21:35.499661+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:21:35.499661+0000","last_undegraded":"2024-03-11T22:21:35.499661+0000","last_fullsized":"2024-03-11T22:21:35.499661+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:39.445349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:19:39.445349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":544,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.458445+0000","last_change":"2024-03-11T22:19:26.459086+0000","last_active":"2024-03-11T22:21:22.458445+0000","last_peered":"2024-03-11T22:21:22.458445+0000","last_clean":"2024-03-11T22:21:22.458445+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:21:22.458445+0000","last_undegraded":"2024-03-11T22:21:22.458445+0000","last_fullsized":"2024-03-11T22:21:22.458445+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:26.458937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:26.458937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"93'130","reported_seq":876,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.505754+0000","last_change":"2024-03-11T22:20:49.901428+0000","last_active":"2024-03-11T22:21:35.505754+0000","last_peered":"2024-03-11T22:21:35.505754+0000","last_clean":"2024-03-11T22:21:35.505754+0000","last_became_active":"2024-03-11T22:14:43.194955+0000","last_became_peered":"2024-03-11T22:14:43.194955+0000","last_unstale":"2024-03-11T22:21:35.505754+0000","last_undegraded":"2024-03-11T22:21:35.505754+0000","last_fullsized":"2024-03-11T22:21:35.505754+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:49.901270+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:18:43.722467+0000","last_clean_scrub_stamp":"2024-03-11T22:20:49.901270+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:57.047219+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"274'20","reported_seq":616,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.507799+0000","last_change":"2024-03-11T22:20:29.487125+0000","last_active":"2024-03-11T22:21:35.507799+0000","last_peered":"2024-03-11T22:21:35.507799+0000","last_clean":"2024-03-11T22:21:35.507799+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:21:35.507799+0000","last_undegraded":"2024-03-11T22:21:35.507799+0000","last_fullsized":"2024-03-11T22:21:35.507799+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'20","last_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:18:42.135888+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":572,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494358+0000","last_change":"2024-03-11T22:20:29.417149+0000","last_active":"2024-03-11T22:21:35.494358+0000","last_peered":"2024-03-11T22:21:35.494358+0000","last_clean":"2024-03-11T22:21:35.494358+0000","last_became_active":"2024-03-11T22:20:29.416792+0000","last_became_peered":"2024-03-11T22:20:29.416792+0000","last_unstale":"2024-03-11T22:21:35.494358+0000","last_undegraded":"2024-03-11T22:21:35.494358+0000","last_fullsized":"2024-03-11T22:21:35.494358+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:45.801375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:15:45.801375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":335,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.847851+0000","last_change":"2024-03-11T22:20:40.885587+0000","last_active":"2024-03-11T22:21:32.847851+0000","last_peered":"2024-03-11T22:21:32.847851+0000","last_clean":"2024-03-11T22:21:32.847851+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:21:32.847851+0000","last_undegraded":"2024-03-11T22:21:32.847851+0000","last_fullsized":"2024-03-11T22:21:32.847851+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:40.885360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:20:40.885360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:51.440703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":540,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.499843+0000","last_change":"2024-03-11T22:14:23.930193+0000","last_active":"2024-03-11T22:21:35.499843+0000","last_peered":"2024-03-11T22:21:35.499843+0000","last_clean":"2024-03-11T22:21:35.499843+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:21:35.499843+0000","last_undegraded":"2024-03-11T22:21:35.499843+0000","last_fullsized":"2024-03-11T22:21:35.499843+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:23.930091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:14:23.930091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"273'8","reported_seq":580,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.499199+0000","last_change":"2024-03-11T22:18:53.157176+0000","last_active":"2024-03-11T22:21:35.499199+0000","last_peered":"2024-03-11T22:21:35.499199+0000","last_clean":"2024-03-11T22:21:35.499199+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:21:35.499199+0000","last_undegraded":"2024-03-11T22:21:35.499199+0000","last_fullsized":"2024-03-11T22:21:35.499199+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'8","last_scrub_stamp":"2024-03-11T22:18:53.157080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:53.157080+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":556,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.456825+0000","last_change":"2024-03-11T22:19:45.501704+0000","last_active":"2024-03-11T22:21:22.456825+0000","last_peered":"2024-03-11T22:21:22.456825+0000","last_clean":"2024-03-11T22:21:22.456825+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:21:22.456825+0000","last_undegraded":"2024-03-11T22:21:22.456825+0000","last_fullsized":"2024-03-11T22:21:22.456825+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.501593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.501593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":144,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.848049+0000","last_change":"2024-03-11T22:20:30.047479+0000","last_active":"2024-03-11T22:21:32.848049+0000","last_peered":"2024-03-11T22:21:32.848049+0000","last_clean":"2024-03-11T22:21:32.848049+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:21:32.848049+0000","last_undegraded":"2024-03-11T22:21:32.848049+0000","last_fullsized":"2024-03-11T22:21:32.848049+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:30.047368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:30.047368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:43.466695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":514,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506116+0000","last_change":"2024-03-11T22:19:32.989760+0000","last_active":"2024-03-11T22:21:35.506116+0000","last_peered":"2024-03-11T22:21:35.506116+0000","last_clean":"2024-03-11T22:21:35.506116+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:21:35.506116+0000","last_undegraded":"2024-03-11T22:21:35.506116+0000","last_fullsized":"2024-03-11T22:21:35.506116+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.989348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.989348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"273'19","reported_seq":602,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.849449+0000","last_change":"2024-03-11T22:20:51.762478+0000","last_active":"2024-03-11T22:21:32.849449+0000","last_peered":"2024-03-11T22:21:32.849449+0000","last_clean":"2024-03-11T22:21:32.849449+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:21:32.849449+0000","last_undegraded":"2024-03-11T22:21:32.849449+0000","last_fullsized":"2024-03-11T22:21:32.849449+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:20:51.762318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:51.762318+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:54.676538+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":560,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.499102+0000","last_change":"2024-03-11T22:19:25.669161+0000","last_active":"2024-03-11T22:21:35.499102+0000","last_peered":"2024-03-11T22:21:35.499102+0000","last_clean":"2024-03-11T22:21:35.499102+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:21:35.499102+0000","last_undegraded":"2024-03-11T22:21:35.499102+0000","last_fullsized":"2024-03-11T22:21:35.499102+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:25.668878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:25.668878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":557,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.458310+0000","last_change":"2024-03-11T22:20:27.333574+0000","last_active":"2024-03-11T22:21:22.458310+0000","last_peered":"2024-03-11T22:21:22.458310+0000","last_clean":"2024-03-11T22:21:22.458310+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:21:22.458310+0000","last_undegraded":"2024-03-11T22:21:22.458310+0000","last_fullsized":"2024-03-11T22:21:22.458310+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:27.333450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:27.333450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:54.425323+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":564,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.511904+0000","last_change":"2024-03-11T22:19:45.686859+0000","last_active":"2024-03-11T22:21:35.511904+0000","last_peered":"2024-03-11T22:21:35.511904+0000","last_clean":"2024-03-11T22:21:35.511904+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:21:35.511904+0000","last_undegraded":"2024-03-11T22:21:35.511904+0000","last_fullsized":"2024-03-11T22:21:35.511904+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.686738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.686738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"273'20","reported_seq":619,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.458037+0000","last_change":"2024-03-11T22:20:29.460876+0000","last_active":"2024-03-11T22:21:22.458037+0000","last_peered":"2024-03-11T22:21:22.458037+0000","last_clean":"2024-03-11T22:21:22.458037+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:21:22.458037+0000","last_undegraded":"2024-03-11T22:21:22.458037+0000","last_fullsized":"2024-03-11T22:21:22.458037+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:19:36.469333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.469333+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":570,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.502383+0000","last_change":"2024-03-11T22:17:36.687785+0000","last_active":"2024-03-11T22:21:35.502383+0000","last_peered":"2024-03-11T22:21:35.502383+0000","last_clean":"2024-03-11T22:21:35.502383+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:21:35.502383+0000","last_undegraded":"2024-03-11T22:21:35.502383+0000","last_fullsized":"2024-03-11T22:21:35.502383+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:17:36.687660+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'11","reported_seq":765,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.849287+0000","last_change":"2024-03-11T22:20:54.741772+0000","last_active":"2024-03-11T22:21:32.849287+0000","last_peered":"2024-03-11T22:21:32.849287+0000","last_clean":"2024-03-11T22:21:32.849287+0000","last_became_active":"2024-03-11T22:20:29.415781+0000","last_became_peered":"2024-03-11T22:20:29.415781+0000","last_unstale":"2024-03-11T22:21:32.849287+0000","last_undegraded":"2024-03-11T22:21:32.849287+0000","last_fullsized":"2024-03-11T22:21:32.849287+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_deep_scrub":"315'11","last_deep_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_clean_scrub_stamp":"2024-03-11T22:20:54.741633+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:10.626413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":509,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.493012+0000","last_change":"2024-03-11T22:19:27.754842+0000","last_active":"2024-03-11T22:21:35.493012+0000","last_peered":"2024-03-11T22:21:35.493012+0000","last_clean":"2024-03-11T22:21:35.493012+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:21:35.493012+0000","last_undegraded":"2024-03-11T22:21:35.493012+0000","last_fullsized":"2024-03-11T22:21:35.493012+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:19:27.754721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:27.754721+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"273'18","reported_seq":580,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.496931+0000","last_change":"2024-03-11T22:14:27.998984+0000","last_active":"2024-03-11T22:21:35.496931+0000","last_peered":"2024-03-11T22:21:35.496931+0000","last_clean":"2024-03-11T22:21:35.496931+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:21:35.496931+0000","last_undegraded":"2024-03-11T22:21:35.496931+0000","last_fullsized":"2024-03-11T22:21:35.496931+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-11T22:14:27.998819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:14:27.998819+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":335,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.503013+0000","last_change":"2024-03-11T22:19:36.471241+0000","last_active":"2024-03-11T22:21:35.503013+0000","last_peered":"2024-03-11T22:21:35.503013+0000","last_clean":"2024-03-11T22:21:35.503013+0000","last_became_active":"2024-03-11T22:14:52.245199+0000","last_became_peered":"2024-03-11T22:14:52.245199+0000","last_unstale":"2024-03-11T22:21:35.503013+0000","last_undegraded":"2024-03-11T22:21:35.503013+0000","last_fullsized":"2024-03-11T22:21:35.503013+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.471133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.471133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":310,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.506663+0000","last_change":"2024-03-11T22:14:52.233517+0000","last_active":"2024-03-11T22:21:35.506663+0000","last_peered":"2024-03-11T22:21:35.506663+0000","last_clean":"2024-03-11T22:21:35.506663+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:21:35.506663+0000","last_undegraded":"2024-03-11T22:21:35.506663+0000","last_fullsized":"2024-03-11T22:21:35.506663+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:14:34.849679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:14:34.849679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":541,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.497496+0000","last_change":"2024-03-11T22:18:27.116987+0000","last_active":"2024-03-11T22:21:35.497496+0000","last_peered":"2024-03-11T22:21:35.497496+0000","last_clean":"2024-03-11T22:21:35.497496+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:21:35.497496+0000","last_undegraded":"2024-03-11T22:21:35.497496+0000","last_fullsized":"2024-03-11T22:21:35.497496+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:27.116858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.116858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"273'27","reported_seq":631,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.505937+0000","last_change":"2024-03-11T22:20:55.875480+0000","last_active":"2024-03-11T22:21:35.505937+0000","last_peered":"2024-03-11T22:21:35.505937+0000","last_clean":"2024-03-11T22:21:35.505937+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:21:35.505937+0000","last_undegraded":"2024-03-11T22:21:35.505937+0000","last_fullsized":"2024-03-11T22:21:35.505937+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:20:55.875312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:55.875312+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:57.302857+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":568,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.457138+0000","last_change":"2024-03-11T22:20:29.414748+0000","last_active":"2024-03-11T22:21:22.457138+0000","last_peered":"2024-03-11T22:21:22.457138+0000","last_clean":"2024-03-11T22:21:22.457138+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:21:22.457138+0000","last_undegraded":"2024-03-11T22:21:22.457138+0000","last_fullsized":"2024-03-11T22:21:22.457138+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:18:43.931797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":560,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.511752+0000","last_change":"2024-03-11T22:18:31.120211+0000","last_active":"2024-03-11T22:21:35.511752+0000","last_peered":"2024-03-11T22:21:35.511752+0000","last_clean":"2024-03-11T22:21:35.511752+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:21:35.511752+0000","last_undegraded":"2024-03-11T22:21:35.511752+0000","last_fullsized":"2024-03-11T22:21:35.511752+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:31.120131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:31.120131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":333,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.849011+0000","last_change":"2024-03-11T22:20:45.819718+0000","last_active":"2024-03-11T22:21:32.849011+0000","last_peered":"2024-03-11T22:21:32.849011+0000","last_clean":"2024-03-11T22:21:32.849011+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:21:32.849011+0000","last_undegraded":"2024-03-11T22:21:32.849011+0000","last_fullsized":"2024-03-11T22:21:32.849011+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:45.819593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:20:45.819593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:49.197505+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"273'9","reported_seq":395,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.499497+0000","last_change":"2024-03-11T22:18:51.098736+0000","last_active":"2024-03-11T22:21:35.499497+0000","last_peered":"2024-03-11T22:21:35.499497+0000","last_clean":"2024-03-11T22:21:35.499497+0000","last_became_active":"2024-03-11T22:14:52.238411+0000","last_became_peered":"2024-03-11T22:14:52.238411+0000","last_unstale":"2024-03-11T22:21:35.499497+0000","last_undegraded":"2024-03-11T22:21:35.499497+0000","last_fullsized":"2024-03-11T22:21:35.499497+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9","last_scrub_stamp":"2024-03-11T22:18:51.098646+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:18:51.098646+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":560,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.493238+0000","last_change":"2024-03-11T22:18:29.763217+0000","last_active":"2024-03-11T22:21:35.493238+0000","last_peered":"2024-03-11T22:21:35.493238+0000","last_clean":"2024-03-11T22:21:35.493238+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:21:35.493238+0000","last_undegraded":"2024-03-11T22:21:35.493238+0000","last_fullsized":"2024-03-11T22:21:35.493238+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:29.739164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:18:29.739164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":329,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.502819+0000","last_change":"2024-03-11T22:17:27.798877+0000","last_active":"2024-03-11T22:21:35.502819+0000","last_peered":"2024-03-11T22:21:35.502819+0000","last_clean":"2024-03-11T22:21:35.502819+0000","last_became_active":"2024-03-11T22:14:52.239391+0000","last_became_peered":"2024-03-11T22:14:52.239391+0000","last_unstale":"2024-03-11T22:21:35.502819+0000","last_undegraded":"2024-03-11T22:21:35.502819+0000","last_fullsized":"2024-03-11T22:21:35.502819+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:27.798763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_clean_scrub_stamp":"2024-03-11T22:17:27.798763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":548,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494739+0000","last_change":"2024-03-11T22:18:38.736070+0000","last_active":"2024-03-11T22:21:35.494739+0000","last_peered":"2024-03-11T22:21:35.494739+0000","last_clean":"2024-03-11T22:21:35.494739+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:21:35.494739+0000","last_undegraded":"2024-03-11T22:21:35.494739+0000","last_fullsized":"2024-03-11T22:21:35.494739+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:38.735937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.735937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"273'15","reported_seq":362,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.498887+0000","last_change":"2024-03-11T22:15:18.401675+0000","last_active":"2024-03-11T22:21:35.498887+0000","last_peered":"2024-03-11T22:21:35.498887+0000","last_clean":"2024-03-11T22:21:35.498887+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:21:35.498887+0000","last_undegraded":"2024-03-11T22:21:35.498887+0000","last_fullsized":"2024-03-11T22:21:35.498887+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:15:18.401549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:18.401549+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":568,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.493345+0000","last_change":"2024-03-11T22:19:40.864768+0000","last_active":"2024-03-11T22:21:35.493345+0000","last_peered":"2024-03-11T22:21:35.493345+0000","last_clean":"2024-03-11T22:21:35.493345+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:21:35.493345+0000","last_undegraded":"2024-03-11T22:21:35.493345+0000","last_fullsized":"2024-03-11T22:21:35.493345+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:40.864689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:40.864689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":564,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.494707+0000","last_change":"2024-03-11T22:20:29.420120+0000","last_active":"2024-03-11T22:21:35.494707+0000","last_peered":"2024-03-11T22:21:35.494707+0000","last_clean":"2024-03-11T22:21:35.494707+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:21:35.494707+0000","last_undegraded":"2024-03-11T22:21:35.494707+0000","last_fullsized":"2024-03-11T22:21:35.494707+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:28.713390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:18:28.713390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":544,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.458876+0000","last_change":"2024-03-11T22:20:30.493247+0000","last_active":"2024-03-11T22:21:22.458876+0000","last_peered":"2024-03-11T22:21:22.458876+0000","last_clean":"2024-03-11T22:21:22.458876+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:21:22.458876+0000","last_undegraded":"2024-03-11T22:21:22.458876+0000","last_fullsized":"2024-03-11T22:21:22.458876+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:30.493146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:20:30.493146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:48.539526+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"273'27","reported_seq":591,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.507918+0000","last_change":"2024-03-11T22:18:37.066095+0000","last_active":"2024-03-11T22:21:35.507918+0000","last_peered":"2024-03-11T22:21:35.507918+0000","last_clean":"2024-03-11T22:21:35.507918+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:21:35.507918+0000","last_undegraded":"2024-03-11T22:21:35.507918+0000","last_fullsized":"2024-03-11T22:21:35.507918+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:18:37.065988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.065988+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":567,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.497211+0000","last_change":"2024-03-11T22:18:38.095521+0000","last_active":"2024-03-11T22:21:35.497211+0000","last_peered":"2024-03-11T22:21:35.497211+0000","last_clean":"2024-03-11T22:21:35.497211+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:21:35.497211+0000","last_undegraded":"2024-03-11T22:21:35.497211+0000","last_fullsized":"2024-03-11T22:21:35.497211+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:18:38.095367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.095367+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":332,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.848538+0000","last_change":"2024-03-11T22:20:26.125601+0000","last_active":"2024-03-11T22:21:32.848538+0000","last_peered":"2024-03-11T22:21:32.848538+0000","last_clean":"2024-03-11T22:21:32.848538+0000","last_became_active":"2024-03-11T22:14:52.247689+0000","last_became_peered":"2024-03-11T22:14:52.247689+0000","last_unstale":"2024-03-11T22:21:32.848538+0000","last_undegraded":"2024-03-11T22:21:32.848538+0000","last_fullsized":"2024-03-11T22:21:32.848538+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:26.125517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:26.125517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:42.748307+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":552,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.499733+0000","last_change":"2024-03-11T22:20:29.436389+0000","last_active":"2024-03-11T22:21:35.499733+0000","last_peered":"2024-03-11T22:21:35.499733+0000","last_clean":"2024-03-11T22:21:35.499733+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:21:35.499733+0000","last_undegraded":"2024-03-11T22:21:35.499733+0000","last_fullsized":"2024-03-11T22:21:35.499733+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:24.023596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:24.023596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"273'19","reported_seq":372,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.848102+0000","last_change":"2024-03-11T22:19:30.613894+0000","last_active":"2024-03-11T22:21:32.848102+0000","last_peered":"2024-03-11T22:21:32.848102+0000","last_clean":"2024-03-11T22:21:32.848102+0000","last_became_active":"2024-03-11T22:14:52.247411+0000","last_became_peered":"2024-03-11T22:14:52.247411+0000","last_unstale":"2024-03-11T22:21:32.848102+0000","last_undegraded":"2024-03-11T22:21:32.848102+0000","last_fullsized":"2024-03-11T22:21:32.848102+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:19:30.613736+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:19:30.613736+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":557,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T22:21:22.456299+0000","last_change":"2024-03-11T22:18:24.613444+0000","last_active":"2024-03-11T22:21:22.456299+0000","last_peered":"2024-03-11T22:21:22.456299+0000","last_clean":"2024-03-11T22:21:22.456299+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:21:22.456299+0000","last_undegraded":"2024-03-11T22:21:22.456299+0000","last_fullsized":"2024-03-11T22:21:22.456299+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:24.613372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:24.613372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":273,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-03-11T22:21:32.848132+0000","last_change":"2024-03-11T22:20:29.420332+0000","last_active":"2024-03-11T22:21:32.848132+0000","last_peered":"2024-03-11T22:21:32.848132+0000","last_clean":"2024-03-11T22:21:32.848132+0000","last_became_active":"2024-03-11T22:20:29.418227+0000","last_became_peered":"2024-03-11T22:20:29.418227+0000","last_unstale":"2024-03-11T22:21:32.848132+0000","last_undegraded":"2024-03-11T22:21:32.848132+0000","last_fullsized":"2024-03-11T22:21:32.848132+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.629853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.629853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":326,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T22:21:35.495041+0000","last_change":"2024-03-11T22:19:26.746519+0000","last_active":"2024-03-11T22:21:35.495041+0000","last_peered":"2024-03-11T22:21:35.495041+0000","last_clean":"2024-03-11T22:21:35.495041+0000","last_became_active":"2024-03-11T22:14:52.247273+0000","last_became_peered":"2024-03-11T22:14:52.247273+0000","last_unstale":"2024-03-11T22:21:35.495041+0000","last_undegraded":"2024-03-11T22:21:35.495041+0000","last_fullsized":"2024-03-11T22:21:35.495041+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:26.746435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:26.746435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3508928536,"num_objects":53543,"num_object_clones":0,"num_object_copies":160629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2171,"num_objects_unfound":0,"num_objects_dirty":53543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194693,"num_write_kb":7943489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32845,"num_bytes_recovered":2152202264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11164794880,"data_stored":11164778592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72793,"ondisk_log_size":72793,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":748,"num_read_kb":743,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":1430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":717,"ondisk_log_size":717,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005217,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2256376,"kb_used_data":1773808,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":91488776,"statfs":{"total":95995035648,"available":93684506624,"internally_reserved":0,"allocated":1816379392,"data_stored":1816046253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":493943243},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332262,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":992352,"kb_used_data":646040,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92752800,"statfs":{"total":95995035648,"available":94978867200,"internally_reserved":0,"allocated":661544960,"data_stored":661236183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":354613702},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575040,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2592228,"kb_used_data":2101528,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":91152924,"statfs":{"total":95995035648,"available":93340594176,"internally_reserved":0,"allocated":2151964672,"data_stored":2151598870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738565,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1330020,"kb_used_data":919708,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92415132,"statfs":{"total":95995035648,"available":94633095168,"internally_reserved":0,"allocated":941780992,"data_stored":941459346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411515,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":955872,"kb_used_data":622040,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92789280,"statfs":{"total":95995035648,"available":95016222720,"internally_reserved":0,"allocated":636968960,"data_stored":636660553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149877,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2417496,"kb_used_data":1912908,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":91327656,"statfs":{"total":95995035648,"available":93519519744,"internally_reserved":0,"allocated":1958817792,"data_stored":1958465108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":516684231},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920945,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1191484,"kb_used_data":857652,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92553668,"statfs":{"total":95995035648,"available":94774956032,"internally_reserved":0,"allocated":878235648,"data_stored":877897954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":341834182},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659306,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2651692,"kb_used_data":2124900,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91093460,"statfs":{"total":95995035648,"available":93279703040,"internally_reserved":0,"allocated":2175897600,"data_stored":2175576376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":539425222},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":934019072,"data_stored":934019072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2151092224,"data_stored":2151088152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":631701504,"data_stored":631701504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1957363712,"data_stored":1957363712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":863637504,"data_stored":863633432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2167738368,"data_stored":2167734296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":652738560,"data_stored":652738560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1806503936,"data_stored":1806499864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:21:38.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:38.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:21:38.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:21:38.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:21:38.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:21:38.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:21:38.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:21:38.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:21:38.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:21:38.293 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:21:38.293 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:21:38.294 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:21:38.294 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:21:38.294 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:21:38.294 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:21:38.294 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:21:38.294 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:21:38.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:38.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:38.409 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:38.433 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:38.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:38.474 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:21:38.504 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:38 smithi045 ceph-mon[30178]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:21:38.504 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:38 smithi045 ceph-mon[30178]: osdmap e326: 8 total, 8 up, 8 in 2024-03-11T22:21:38.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:38.571 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:38.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:38.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:38.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:38.713 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:38.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:38.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:38 smithi045 ceph-mon[39715]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:21:38.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:38 smithi045 ceph-mon[39715]: osdmap e326: 8 total, 8 up, 8 in 2024-03-11T22:21:38.813 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:38.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:38 smithi144 ceph-mon[33984]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:21:38.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:38 smithi144 ceph-mon[33984]: osdmap e326: 8 total, 8 up, 8 in 2024-03-11T22:21:38.849 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:38.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:38.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:38.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:38.994 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:38.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:39.097 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:39.127 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:39.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:39.229 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:39.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:39.264 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:39.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:39.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:39.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:39.404 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:39.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:39.505 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:39.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:39.541 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:39.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:39.642 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:39.673 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:39.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:39.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:39.768 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:39 smithi045 ceph-mon[39715]: pgmap v615: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 165 active+clean; 3.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 936 B/s rd, 0 B/s wr, 888 op/s; 1480/158571 objects misplaced (0.933%); 31 MiB/s, 489 objects/s recovering 2024-03-11T22:21:39.768 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:39 smithi045 ceph-mon[39715]: osdmap e327: 8 total, 8 up, 8 in 2024-03-11T22:21:39.769 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:39 smithi045 ceph-mon[30178]: pgmap v615: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 165 active+clean; 3.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 936 B/s rd, 0 B/s wr, 888 op/s; 1480/158571 objects misplaced (0.933%); 31 MiB/s, 489 objects/s recovering 2024-03-11T22:21:39.769 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:39 smithi045 ceph-mon[30178]: osdmap e327: 8 total, 8 up, 8 in 2024-03-11T22:21:39.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:39.814 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:39.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:39.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:39.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:39 smithi144 ceph-mon[33984]: pgmap v615: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 165 active+clean; 3.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 936 B/s rd, 0 B/s wr, 888 op/s; 1480/158571 objects misplaced (0.933%); 31 MiB/s, 489 objects/s recovering 2024-03-11T22:21:39.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:39 smithi144 ceph-mon[33984]: osdmap e327: 8 total, 8 up, 8 in 2024-03-11T22:21:39.915 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:39.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:39.950 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:39.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:40.051 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:40.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:40.087 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:40.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:40.188 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:40.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:40.220 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:40.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:40.321 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:40.363 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:40.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:40.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:40.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:40.502 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:40.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:40.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:40.603 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:40.638 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:40.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:40.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:40.784 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:40.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:40.886 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:40.923 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:40.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:41.024 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:41.057 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:41.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:41.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:21:41.160 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:41.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.206 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:41.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:41.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:21:41.307 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:41.337 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:41.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:41.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:41.491 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:41.593 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:41.626 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:41.728 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:41.757 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:41.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:41 smithi045 ceph-mon[39715]: pgmap v617: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 165 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.00k op/s; 1480/158571 objects misplaced (0.933%); 35 MiB/s, 552 objects/s recovering 2024-03-11T22:21:41.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:41 smithi045 ceph-mon[30178]: pgmap v617: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 165 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.00k op/s; 1480/158571 objects misplaced (0.933%); 35 MiB/s, 552 objects/s recovering 2024-03-11T22:21:41.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:41 smithi144 ceph-mon[33984]: pgmap v617: 168 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 165 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.00k op/s; 1480/158571 objects misplaced (0.933%); 35 MiB/s, 552 objects/s recovering 2024-03-11T22:21:41.859 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:41.892 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:41.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:41.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:41.994 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:42.055 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:42.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:42.157 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:42.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:42.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:42.302 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:42.333 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:42.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:42.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:42.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:42.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:42.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:42.576 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:42.607 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:42.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:42.709 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:42.720 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:21:42.738 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:42.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:42.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:42.839 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:42.841 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:21:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:21:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:21:42.843 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:21:42.843 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:21:42.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:42.875 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:42.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:42.976 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:43.010 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:43.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:43.111 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:43.144 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:43.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:43.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:43.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:43.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:43.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:43.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:43.437 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:43.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:43.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:43.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:43.576 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:43.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:43.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:43.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:43.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:43.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:43.733 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:43 smithi045 ceph-mon[39715]: pgmap v618: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 834 op/s; 999/146214 objects misplaced (0.683%); 9.9 MiB/s, 159 objects/s recovering 2024-03-11T22:21:43.733 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:43 smithi045 ceph-mon[30178]: pgmap v618: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 834 op/s; 999/146214 objects misplaced (0.683%); 9.9 MiB/s, 159 objects/s recovering 2024-03-11T22:21:43.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:43.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:43 smithi144 ceph-mon[33984]: pgmap v618: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 834 op/s; 999/146214 objects misplaced (0.683%); 9.9 MiB/s, 159 objects/s recovering 2024-03-11T22:21:43.853 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:43.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:43.954 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:43.986 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:43.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:44.088 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:44.122 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:44.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:44.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:44.224 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:44.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T22:21:44.261 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:44.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:44.362 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:44.394 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:44.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:44.496 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:44.529 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:44.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:44.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:44.676 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:44.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:44.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:44.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:44.826 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:44.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:44.927 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:44.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:44.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:45.068 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:45.101 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:45.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:45.203 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:45.231 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:45.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:45.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:45.333 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:45.364 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:45.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:45.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:45.466 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:45.499 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:45.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:45.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:45 smithi045 ceph-mon[39715]: pgmap v619: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 653 B/s rd, 0 B/s wr, 639 op/s; 999/146214 objects misplaced (0.683%); 7.6 MiB/s, 121 objects/s recovering 2024-03-11T22:21:45.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:45 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:21:45.547 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:45 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:21:45.547 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:45 smithi045 ceph-mon[30178]: pgmap v619: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 653 B/s rd, 0 B/s wr, 639 op/s; 999/146214 objects misplaced (0.683%); 7.6 MiB/s, 121 objects/s recovering 2024-03-11T22:21:45.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:45 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:21:45.548 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:45 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:21:45.600 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:45.623 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:45.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:45.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:45.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:45.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:45.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:45 smithi144 ceph-mon[33984]: pgmap v619: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 3.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 653 B/s rd, 0 B/s wr, 639 op/s; 999/146214 objects misplaced (0.683%); 7.6 MiB/s, 121 objects/s recovering 2024-03-11T22:21:45.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:45 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:21:45.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:45 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:21:45.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:45.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:45.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:45.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:46.006 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:46.041 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:46.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:46.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:46.143 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:46.172 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:46.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:46.274 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:46.307 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:46.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:46.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:46.408 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:46.440 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:46.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:46.533 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:21:46.541 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:46.564 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:46.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:46.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:46.707 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:46.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:46.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:46 smithi045 ceph-mon[39715]: osdmap e328: 8 total, 8 up, 8 in 2024-03-11T22:21:46.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:21:46.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:46 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2701466512' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:21:46.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:46 smithi045 ceph-mon[30178]: osdmap e328: 8 total, 8 up, 8 in 2024-03-11T22:21:46.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:21:46.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:46 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2701466512' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:21:46.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:46.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:46 smithi144 ceph-mon[33984]: osdmap e328: 8 total, 8 up, 8 in 2024-03-11T22:21:46.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:21:46.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:46 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2701466512' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:21:46.847 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:21:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:21:46.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:46.848 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:46.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:46.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:46.987 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:46.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:46.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:47.089 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:47.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:47.122 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:47.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:47.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:47.267 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:47.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:47.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:47.369 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:47.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:47.401 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:47.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:47.502 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:47.535 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:47.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:47.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:47.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:47.676 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:47.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:47.778 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:47 smithi045 ceph-mon[39715]: pgmap v621: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 999/130677 objects misplaced (0.764%); 2.9 MiB/s, 46 objects/s recovering 2024-03-11T22:21:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:47 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:21:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:47 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2701466512' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:21:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:47 smithi045 ceph-mon[39715]: osdmap e329: 8 total, 8 up, 8 in 2024-03-11T22:21:47.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:47 smithi045 ceph-mon[30178]: pgmap v621: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 999/130677 objects misplaced (0.764%); 2.9 MiB/s, 46 objects/s recovering 2024-03-11T22:21:47.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:47.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:47 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:21:47.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2701466512' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:21:47.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:47 smithi045 ceph-mon[30178]: osdmap e329: 8 total, 8 up, 8 in 2024-03-11T22:21:47.804 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:47.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:47.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:47.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:47 smithi144 ceph-mon[33984]: pgmap v621: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 999/130677 objects misplaced (0.764%); 2.9 MiB/s, 46 objects/s recovering 2024-03-11T22:21:47.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:47.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:47 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:21:47.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:47 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2701466512' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:21:47.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:47 smithi144 ceph-mon[33984]: osdmap e329: 8 total, 8 up, 8 in 2024-03-11T22:21:47.906 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:47.941 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:47.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:48.042 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:48.076 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:48.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:48.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:48.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:48.219 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:48.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:48.320 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:48.361 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:48.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:48.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:48.502 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:48.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:48.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:48.629 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:48.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:48.731 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:48.760 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:48.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:48.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:48 smithi045 ceph-mon[39715]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T22:21:48.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:48 smithi045 ceph-mon[30178]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T22:21:48.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:48 smithi144 ceph-mon[33984]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T22:21:48.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:48.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:48.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:49.004 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:49.037 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:49.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:49.138 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:49.170 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:49.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:49.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:49.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:49.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:49.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:49.416 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:49.444 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:49.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:49.545 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:49.589 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:49.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:49.691 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:49.723 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:49.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:49.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:49.825 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:49.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:49 smithi144 ceph-mon[33984]: pgmap v623: 168 pgs: 1 active+remapped+backfilling, 167 active+clean; 2.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 511/127266 objects misplaced (0.402%); 5.8 MiB/s, 92 objects/s recovering 2024-03-11T22:21:49.859 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:49.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:49.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:49.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:50.000 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:50.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:50.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:49 smithi045 ceph-mon[39715]: pgmap v623: 168 pgs: 1 active+remapped+backfilling, 167 active+clean; 2.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 511/127266 objects misplaced (0.402%); 5.8 MiB/s, 92 objects/s recovering 2024-03-11T22:21:50.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:49 smithi045 ceph-mon[30178]: pgmap v623: 168 pgs: 1 active+remapped+backfilling, 167 active+clean; 2.6 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 511/127266 objects misplaced (0.402%); 5.8 MiB/s, 92 objects/s recovering 2024-03-11T22:21:50.101 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:50.134 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:50.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:50.236 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:50.268 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:50.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:50.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:50.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:50.402 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:50.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:50.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:50.544 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:50.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:50.646 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:50.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:50.670 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:50.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:50.771 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:50.800 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:50.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:50.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:50.901 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:50.926 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:50.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:50.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:51.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:51.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:21:51.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:21:51.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[30178]: osdmap e330: 8 total, 8 up, 8 in 2024-03-11T22:21:51.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[30178]: 4.f scrub starts 2024-03-11T22:21:51.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[30178]: 4.f scrub ok 2024-03-11T22:21:51.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[30178]: 5.1c scrub starts 2024-03-11T22:21:51.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[30178]: 5.1c scrub ok 2024-03-11T22:21:51.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:21:51.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:21:51.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[39715]: osdmap e330: 8 total, 8 up, 8 in 2024-03-11T22:21:51.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[39715]: 4.f scrub starts 2024-03-11T22:21:51.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[39715]: 4.f scrub ok 2024-03-11T22:21:51.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[39715]: 5.1c scrub starts 2024-03-11T22:21:51.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:50 smithi045 ceph-mon[39715]: 5.1c scrub ok 2024-03-11T22:21:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:51.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:51.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:51.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:50 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:21:51.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:50 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:21:51.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:50 smithi144 ceph-mon[33984]: osdmap e330: 8 total, 8 up, 8 in 2024-03-11T22:21:51.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:50 smithi144 ceph-mon[33984]: 4.f scrub starts 2024-03-11T22:21:51.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:50 smithi144 ceph-mon[33984]: 4.f scrub ok 2024-03-11T22:21:51.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:50 smithi144 ceph-mon[33984]: 5.1c scrub starts 2024-03-11T22:21:51.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:50 smithi144 ceph-mon[33984]: 5.1c scrub ok 2024-03-11T22:21:51.153 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:51.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:51.177 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:51.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:51.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:51.302 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:51.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:51.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:51.404 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:51.427 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:51.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:51.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:51.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:51.694 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:51.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:51.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:51.835 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:51.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:51.937 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:51.961 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:51.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:51.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:52.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:51 smithi045 ceph-mon[39715]: 4.1 scrub starts 2024-03-11T22:21:52.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:51 smithi045 ceph-mon[39715]: 4.1 scrub ok 2024-03-11T22:21:52.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:51 smithi045 ceph-mon[39715]: pgmap v625: 168 pgs: 1 active+remapped+backfilling, 167 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 511/127266 objects misplaced (0.402%); 3.8 MiB/s, 60 objects/s recovering 2024-03-11T22:21:52.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:51 smithi045 ceph-mon[39715]: 4.18 scrub starts 2024-03-11T22:21:52.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:51 smithi045 ceph-mon[39715]: 4.18 scrub ok 2024-03-11T22:21:52.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:51 smithi045 ceph-mon[39715]: osdmap e331: 8 total, 8 up, 8 in 2024-03-11T22:21:52.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:51 smithi045 ceph-mon[30178]: 4.1 scrub starts 2024-03-11T22:21:52.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:51 smithi045 ceph-mon[30178]: 4.1 scrub ok 2024-03-11T22:21:52.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:51 smithi045 ceph-mon[30178]: pgmap v625: 168 pgs: 1 active+remapped+backfilling, 167 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 511/127266 objects misplaced (0.402%); 3.8 MiB/s, 60 objects/s recovering 2024-03-11T22:21:52.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:51 smithi045 ceph-mon[30178]: 4.18 scrub starts 2024-03-11T22:21:52.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:51 smithi045 ceph-mon[30178]: 4.18 scrub ok 2024-03-11T22:21:52.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:51 smithi045 ceph-mon[30178]: osdmap e331: 8 total, 8 up, 8 in 2024-03-11T22:21:52.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:52.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:51 smithi144 ceph-mon[33984]: 4.1 scrub starts 2024-03-11T22:21:52.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:51 smithi144 ceph-mon[33984]: 4.1 scrub ok 2024-03-11T22:21:52.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:51 smithi144 ceph-mon[33984]: pgmap v625: 168 pgs: 1 active+remapped+backfilling, 167 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 511/127266 objects misplaced (0.402%); 3.8 MiB/s, 60 objects/s recovering 2024-03-11T22:21:52.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:51 smithi144 ceph-mon[33984]: 4.18 scrub starts 2024-03-11T22:21:52.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:51 smithi144 ceph-mon[33984]: 4.18 scrub ok 2024-03-11T22:21:52.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:51 smithi144 ceph-mon[33984]: osdmap e331: 8 total, 8 up, 8 in 2024-03-11T22:21:52.100 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:52.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:52.201 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:52.231 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:52.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:52.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:52.355 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:52.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:52.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:52.457 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:52.481 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:52.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:52.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:52.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:52.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:52.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:52.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:52.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:52.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:52.898 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:52.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:53.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:53.033 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:21:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:21:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:21:53.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:52 smithi045 ceph-mon[39715]: osdmap e332: 8 total, 8 up, 8 in 2024-03-11T22:21:53.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:52 smithi045 ceph-mon[39715]: 5.12 scrub starts 2024-03-11T22:21:53.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:52 smithi045 ceph-mon[39715]: 5.12 scrub ok 2024-03-11T22:21:53.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:52 smithi045 ceph-mon[30178]: osdmap e332: 8 total, 8 up, 8 in 2024-03-11T22:21:53.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:52 smithi045 ceph-mon[30178]: 5.12 scrub starts 2024-03-11T22:21:53.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:52 smithi045 ceph-mon[30178]: 5.12 scrub ok 2024-03-11T22:21:53.040 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:53.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:53.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:52 smithi144 ceph-mon[33984]: osdmap e332: 8 total, 8 up, 8 in 2024-03-11T22:21:53.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:52 smithi144 ceph-mon[33984]: 5.12 scrub starts 2024-03-11T22:21:53.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:52 smithi144 ceph-mon[33984]: 5.12 scrub ok 2024-03-11T22:21:53.142 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:53.174 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:53.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:53.275 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:53.308 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:53.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:53.409 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:53.441 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:53.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:53.544 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:53.573 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:53.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:53.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:53.675 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:53.710 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:53.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:53.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:53.814 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:53.837 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:53.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:53.939 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:53.974 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:53.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:53.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:54.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:53 smithi045 ceph-mon[39715]: pgmap v628: 168 pgs: 168 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 883 B/s rd, 0 B/s wr, 1.08k op/s; 9.2 MiB/s, 147 objects/s recovering 2024-03-11T22:21:54.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:53 smithi045 ceph-mon[39715]: 4.1d scrub starts 2024-03-11T22:21:54.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:53 smithi045 ceph-mon[39715]: 4.1d scrub ok 2024-03-11T22:21:54.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:53 smithi045 ceph-mon[39715]: 6.8 scrub starts 2024-03-11T22:21:54.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:53 smithi045 ceph-mon[39715]: 6.8 scrub ok 2024-03-11T22:21:54.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:53 smithi045 ceph-mon[30178]: pgmap v628: 168 pgs: 168 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 883 B/s rd, 0 B/s wr, 1.08k op/s; 9.2 MiB/s, 147 objects/s recovering 2024-03-11T22:21:54.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:53 smithi045 ceph-mon[30178]: 4.1d scrub starts 2024-03-11T22:21:54.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:53 smithi045 ceph-mon[30178]: 4.1d scrub ok 2024-03-11T22:21:54.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:53 smithi045 ceph-mon[30178]: 6.8 scrub starts 2024-03-11T22:21:54.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:53 smithi045 ceph-mon[30178]: 6.8 scrub ok 2024-03-11T22:21:54.075 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:54.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:53 smithi144 ceph-mon[33984]: pgmap v628: 168 pgs: 168 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 883 B/s rd, 0 B/s wr, 1.08k op/s; 9.2 MiB/s, 147 objects/s recovering 2024-03-11T22:21:54.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:53 smithi144 ceph-mon[33984]: 4.1d scrub starts 2024-03-11T22:21:54.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:53 smithi144 ceph-mon[33984]: 4.1d scrub ok 2024-03-11T22:21:54.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:53 smithi144 ceph-mon[33984]: 6.8 scrub starts 2024-03-11T22:21:54.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:53 smithi144 ceph-mon[33984]: 6.8 scrub ok 2024-03-11T22:21:54.109 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:54.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:54.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:54.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:54.248 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:54.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:54.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:54.520 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:54.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:54.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:54.622 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:54.662 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:54.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:54.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:54.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:54.787 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:54.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:54.889 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:54.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:54.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:54.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:55.028 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:55.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:54 smithi045 ceph-mon[39715]: 3.3 scrub starts 2024-03-11T22:21:55.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:54 smithi045 ceph-mon[39715]: 3.3 scrub ok 2024-03-11T22:21:55.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:54 smithi045 ceph-mon[30178]: 3.3 scrub starts 2024-03-11T22:21:55.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:54 smithi045 ceph-mon[30178]: 3.3 scrub ok 2024-03-11T22:21:55.059 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:55.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:55.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:55.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:54 smithi144 ceph-mon[33984]: 3.3 scrub starts 2024-03-11T22:21:55.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:54 smithi144 ceph-mon[33984]: 3.3 scrub ok 2024-03-11T22:21:55.161 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:55.194 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:55.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:55.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:55.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:55.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:55.336 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:55.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:55.437 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:55.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:55.460 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:55.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:55.561 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:55.593 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:55.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:55.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:55.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:55.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:55.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:55.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:55.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:55.873 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:55.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:55.975 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:55.998 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:55.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:56.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:55 smithi045 ceph-mon[39715]: pgmap v629: 168 pgs: 168 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 853 op/s; 5.1 MiB/s, 81 objects/s recovering 2024-03-11T22:21:56.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:55 smithi045 ceph-mon[39715]: 6.13 scrub starts 2024-03-11T22:21:56.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:55 smithi045 ceph-mon[39715]: 6.13 scrub ok 2024-03-11T22:21:56.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:55 smithi045 ceph-mon[30178]: pgmap v629: 168 pgs: 168 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 853 op/s; 5.1 MiB/s, 81 objects/s recovering 2024-03-11T22:21:56.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:55 smithi045 ceph-mon[30178]: 6.13 scrub starts 2024-03-11T22:21:56.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:55 smithi045 ceph-mon[30178]: 6.13 scrub ok 2024-03-11T22:21:56.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:55 smithi144 ceph-mon[33984]: pgmap v629: 168 pgs: 168 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 853 op/s; 5.1 MiB/s, 81 objects/s recovering 2024-03-11T22:21:56.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:55 smithi144 ceph-mon[33984]: 6.13 scrub starts 2024-03-11T22:21:56.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:55 smithi144 ceph-mon[33984]: 6.13 scrub ok 2024-03-11T22:21:56.100 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:56.132 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:56.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:56.233 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:56.267 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:56.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:56.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:56.369 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:56.395 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:56.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:56.497 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:56.533 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:56.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:56.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:56.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:56.672 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:56.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:56.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:56.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:56.813 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:56.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:56.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:56.847 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:21:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:21:56.915 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:56.955 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:56.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:56.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:57.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:21:57.093 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:57.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:21:57.195 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:57.229 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:57.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:57.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:57.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:57.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:57.507 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:57.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:57.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:57.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:57 smithi144 ceph-mon[33984]: 3.8 scrub starts 2024-03-11T22:21:57.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:57 smithi144 ceph-mon[33984]: 3.8 scrub ok 2024-03-11T22:21:57.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:57 smithi144 ceph-mon[33984]: 3.16 scrub starts 2024-03-11T22:21:57.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:57 smithi144 ceph-mon[33984]: pgmap v630: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 760 B/s rd, 0 B/s wr, 934 op/s; 4.5 MiB/s, 72 objects/s recovering 2024-03-11T22:21:57.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:57 smithi144 ceph-mon[33984]: 3.16 scrub ok 2024-03-11T22:21:57.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:57.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:57 smithi144 ceph-mon[33984]: 5.5 scrub starts 2024-03-11T22:21:57.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:57 smithi144 ceph-mon[33984]: 5.5 scrub ok 2024-03-11T22:21:57.609 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:57.645 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:57.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:57.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:57.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:21:57.775 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:57.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:57.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:21:57.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[39715]: 3.8 scrub starts 2024-03-11T22:21:57.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[39715]: 3.8 scrub ok 2024-03-11T22:21:57.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[39715]: 3.16 scrub starts 2024-03-11T22:21:57.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[39715]: pgmap v630: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 760 B/s rd, 0 B/s wr, 934 op/s; 4.5 MiB/s, 72 objects/s recovering 2024-03-11T22:21:57.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[39715]: 3.16 scrub ok 2024-03-11T22:21:57.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:57.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[39715]: 5.5 scrub starts 2024-03-11T22:21:57.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[39715]: 5.5 scrub ok 2024-03-11T22:21:57.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[30178]: 3.8 scrub starts 2024-03-11T22:21:57.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[30178]: 3.8 scrub ok 2024-03-11T22:21:57.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[30178]: 3.16 scrub starts 2024-03-11T22:21:57.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[30178]: pgmap v630: 168 pgs: 168 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 760 B/s rd, 0 B/s wr, 934 op/s; 4.5 MiB/s, 72 objects/s recovering 2024-03-11T22:21:57.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[30178]: 3.16 scrub ok 2024-03-11T22:21:57.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:21:57.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[30178]: 5.5 scrub starts 2024-03-11T22:21:57.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:57 smithi045 ceph-mon[30178]: 5.5 scrub ok 2024-03-11T22:21:57.877 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:21:57.907 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:57.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:57.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:21:58.009 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:58.038 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:58.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:58.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:58.140 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:58.179 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:58.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:58.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:58.281 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:58.317 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:58.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:58.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:58.419 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:58.442 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:58.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:58.543 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:58.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:58.580 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:58.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:58.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:58 smithi144 ceph-mon[33984]: 1.a scrub starts 2024-03-11T22:21:58.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:58 smithi144 ceph-mon[33984]: 1.a scrub ok 2024-03-11T22:21:58.682 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:58.706 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:58.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:58.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:58.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:58 smithi045 ceph-mon[30178]: 1.a scrub starts 2024-03-11T22:21:58.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:58 smithi045 ceph-mon[30178]: 1.a scrub ok 2024-03-11T22:21:58.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:58 smithi045 ceph-mon[39715]: 1.a scrub starts 2024-03-11T22:21:58.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:58 smithi045 ceph-mon[39715]: 1.a scrub ok 2024-03-11T22:21:58.808 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:58.839 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:58.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:58.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:58.941 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:58.976 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:58.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:58.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:21:59.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:59.108 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:59.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:59.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:59.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:59.257 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:59.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:59.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:21:59.402 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:59.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:21:59.504 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:21:59.531 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:59.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:21:59.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[39715]: 3.1c scrub starts 2024-03-11T22:21:59.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[39715]: 3.1c scrub ok 2024-03-11T22:21:59.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[39715]: pgmap v631: 168 pgs: 168 active+clean; 2.0 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 3.8 MiB/s, 61 objects/s recovering 2024-03-11T22:21:59.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[39715]: 3.1 scrub starts 2024-03-11T22:21:59.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[39715]: 3.1 scrub ok 2024-03-11T22:21:59.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[39715]: 4.12 scrub starts 2024-03-11T22:21:59.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[39715]: 4.12 scrub ok 2024-03-11T22:21:59.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3192774302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:21:59.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[30178]: 3.1c scrub starts 2024-03-11T22:21:59.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[30178]: 3.1c scrub ok 2024-03-11T22:21:59.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[30178]: pgmap v631: 168 pgs: 168 active+clean; 2.0 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 3.8 MiB/s, 61 objects/s recovering 2024-03-11T22:21:59.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[30178]: 3.1 scrub starts 2024-03-11T22:21:59.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[30178]: 3.1 scrub ok 2024-03-11T22:21:59.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[30178]: 4.12 scrub starts 2024-03-11T22:21:59.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[30178]: 4.12 scrub ok 2024-03-11T22:21:59.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:21:59 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3192774302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:21:59.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:59 smithi144 ceph-mon[33984]: 3.1c scrub starts 2024-03-11T22:21:59.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:59 smithi144 ceph-mon[33984]: 3.1c scrub ok 2024-03-11T22:21:59.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:59 smithi144 ceph-mon[33984]: pgmap v631: 168 pgs: 168 active+clean; 2.0 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s; 3.8 MiB/s, 61 objects/s recovering 2024-03-11T22:21:59.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:59 smithi144 ceph-mon[33984]: 3.1 scrub starts 2024-03-11T22:21:59.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:59 smithi144 ceph-mon[33984]: 3.1 scrub ok 2024-03-11T22:21:59.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:59 smithi144 ceph-mon[33984]: 4.12 scrub starts 2024-03-11T22:21:59.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:59 smithi144 ceph-mon[33984]: 4.12 scrub ok 2024-03-11T22:21:59.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:21:59 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3192774302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:21:59.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:21:59.675 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:21:59.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:59.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:21:59.776 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:21:59.800 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:59.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:21:59.901 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:21:59.925 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:21:59.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:21:59.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:00.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:00.070 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:00.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:22:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:22:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:22:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:22:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:22:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:22:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:22:00.134 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:22:00.135 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:22:00.135 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:22:00.135 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:22:00.135 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:22:00.135 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:22:00.135 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:22:00.135 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:22:00.135 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:22:00.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:00.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:00.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:00.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:00.340 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:00.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:00.442 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:00.466 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:00.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:00.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:00 smithi045 ceph-mon[39715]: 6.3 scrub starts 2024-03-11T22:22:00.533 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:00 smithi045 ceph-mon[39715]: 6.3 scrub ok 2024-03-11T22:22:00.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:00 smithi045 ceph-mon[30178]: 6.3 scrub starts 2024-03-11T22:22:00.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:00 smithi045 ceph-mon[30178]: 6.3 scrub ok 2024-03-11T22:22:00.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:00.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:00 smithi144 ceph-mon[33984]: 6.3 scrub starts 2024-03-11T22:22:00.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:00 smithi144 ceph-mon[33984]: 6.3 scrub ok 2024-03-11T22:22:00.606 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:00.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:00.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:00.708 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:00.738 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:00.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:00.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:00.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:00.878 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:00.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:00.980 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:01.003 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:22:01.014 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:01.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:01.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:01.116 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:01.142 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:01.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:01.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:01.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:01.267 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:01.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:01.369 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:01.401 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:01.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:01.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:01.505 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:01.531 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:01.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:01.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:01.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:01.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:01.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:01.761 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:01.800 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:01.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:01.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:01.943 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:01.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:01.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:02.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[39715]: 6.f scrub starts 2024-03-11T22:22:02.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[39715]: 6.f scrub ok 2024-03-11T22:22:02.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[39715]: pgmap v632: 168 pgs: 168 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 914 op/s; 3.1 MiB/s, 50 objects/s recovering 2024-03-11T22:22:02.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[39715]: 5.2 scrub starts 2024-03-11T22:22:02.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[39715]: 5.2 scrub ok 2024-03-11T22:22:02.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:22:02.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:22:02.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[30178]: 6.f scrub starts 2024-03-11T22:22:02.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[30178]: 6.f scrub ok 2024-03-11T22:22:02.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[30178]: pgmap v632: 168 pgs: 168 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 914 op/s; 3.1 MiB/s, 50 objects/s recovering 2024-03-11T22:22:02.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[30178]: 5.2 scrub starts 2024-03-11T22:22:02.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[30178]: 5.2 scrub ok 2024-03-11T22:22:02.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:22:02.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:22:02.045 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:02.079 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:02.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:02.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:02.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:01 smithi144 ceph-mon[33984]: 6.f scrub starts 2024-03-11T22:22:02.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:01 smithi144 ceph-mon[33984]: 6.f scrub ok 2024-03-11T22:22:02.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:01 smithi144 ceph-mon[33984]: pgmap v632: 168 pgs: 168 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 914 op/s; 3.1 MiB/s, 50 objects/s recovering 2024-03-11T22:22:02.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:01 smithi144 ceph-mon[33984]: 5.2 scrub starts 2024-03-11T22:22:02.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:01 smithi144 ceph-mon[33984]: 5.2 scrub ok 2024-03-11T22:22:02.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:22:02.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:22:02.181 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:02.214 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:02.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:02.315 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:02.347 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:02.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:02.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:02.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:02.489 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:02.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:02.591 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:02.621 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:02.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:02.722 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:02.753 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:02.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:02.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:02.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:02 smithi045 ceph-mon[30178]: 6.14 scrub starts 2024-03-11T22:22:02.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:02 smithi045 ceph-mon[30178]: 6.14 scrub ok 2024-03-11T22:22:02.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:02 smithi045 ceph-mon[30178]: 4.5 scrub starts 2024-03-11T22:22:02.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:02 smithi045 ceph-mon[30178]: 4.5 scrub ok 2024-03-11T22:22:02.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:02 smithi045 ceph-mon[30178]: 6.16 scrub starts 2024-03-11T22:22:02.894 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:02 smithi045 ceph-mon[30178]: 6.16 scrub ok 2024-03-11T22:22:02.894 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:22:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:22:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:22:02.895 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:02 smithi045 ceph-mon[39715]: 6.14 scrub starts 2024-03-11T22:22:02.895 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:02 smithi045 ceph-mon[39715]: 6.14 scrub ok 2024-03-11T22:22:02.895 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:02 smithi045 ceph-mon[39715]: 4.5 scrub starts 2024-03-11T22:22:02.895 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:02 smithi045 ceph-mon[39715]: 4.5 scrub ok 2024-03-11T22:22:02.895 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:02 smithi045 ceph-mon[39715]: 6.16 scrub starts 2024-03-11T22:22:02.895 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:02 smithi045 ceph-mon[39715]: 6.16 scrub ok 2024-03-11T22:22:02.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:02.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:02.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:02.904 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:02.936 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:02.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:03.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:03.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:03.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:02 smithi144 ceph-mon[33984]: 6.14 scrub starts 2024-03-11T22:22:03.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:02 smithi144 ceph-mon[33984]: 6.14 scrub ok 2024-03-11T22:22:03.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:02 smithi144 ceph-mon[33984]: 4.5 scrub starts 2024-03-11T22:22:03.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:02 smithi144 ceph-mon[33984]: 4.5 scrub ok 2024-03-11T22:22:03.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:02 smithi144 ceph-mon[33984]: 6.16 scrub starts 2024-03-11T22:22:03.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:02 smithi144 ceph-mon[33984]: 6.16 scrub ok 2024-03-11T22:22:03.150 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:03.186 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:03.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:03.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:03.288 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:03.324 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:03.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:03.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:03.425 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:03.461 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:03.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:03.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:03.562 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:03.593 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:03.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:03.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:03.735 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:03.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:03.836 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:03.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:03.869 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:03.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:03.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:03 smithi045 ceph-mon[30178]: pgmap v633: 168 pgs: 168 active+clean; 1.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 954 B/s rd, 0 B/s wr, 1.07k op/s; 2.9 MiB/s, 45 objects/s recovering 2024-03-11T22:22:03.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:03 smithi045 ceph-mon[30178]: 4.a scrub starts 2024-03-11T22:22:03.882 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:03 smithi045 ceph-mon[30178]: 4.a scrub ok 2024-03-11T22:22:03.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:03 smithi045 ceph-mon[39715]: pgmap v633: 168 pgs: 168 active+clean; 1.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 954 B/s rd, 0 B/s wr, 1.07k op/s; 2.9 MiB/s, 45 objects/s recovering 2024-03-11T22:22:03.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:03 smithi045 ceph-mon[39715]: 4.a scrub starts 2024-03-11T22:22:03.882 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:03 smithi045 ceph-mon[39715]: 4.a scrub ok 2024-03-11T22:22:03.970 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:04.006 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:04.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:04.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:04.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:03 smithi144 ceph-mon[33984]: pgmap v633: 168 pgs: 168 active+clean; 1.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 954 B/s rd, 0 B/s wr, 1.07k op/s; 2.9 MiB/s, 45 objects/s recovering 2024-03-11T22:22:04.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:03 smithi144 ceph-mon[33984]: 4.a scrub starts 2024-03-11T22:22:04.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:03 smithi144 ceph-mon[33984]: 4.a scrub ok 2024-03-11T22:22:04.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:04.150 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:04.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:04.251 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:04.289 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:04.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:04.391 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:04.425 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:04.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:04.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:04.527 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:04.566 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:04.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:04.667 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:04.699 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:04.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:04.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:04.839 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:04.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:04.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:04.941 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:04.948 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:22:04.976 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:04.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:04.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:04.990 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:04 smithi045 ceph-mon[30178]: 3.12 scrub starts 2024-03-11T22:22:04.991 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:04 smithi045 ceph-mon[30178]: 3.12 scrub ok 2024-03-11T22:22:04.991 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:04 smithi045 ceph-mon[30178]: 5.1a scrub starts 2024-03-11T22:22:04.991 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:04 smithi045 ceph-mon[30178]: 5.1a scrub ok 2024-03-11T22:22:04.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:04 smithi045 ceph-mon[39715]: 3.12 scrub starts 2024-03-11T22:22:04.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:04 smithi045 ceph-mon[39715]: 3.12 scrub ok 2024-03-11T22:22:04.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:04 smithi045 ceph-mon[39715]: 5.1a scrub starts 2024-03-11T22:22:04.992 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:04 smithi045 ceph-mon[39715]: 5.1a scrub ok 2024-03-11T22:22:05.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:05.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:04 smithi144 ceph-mon[33984]: 3.12 scrub starts 2024-03-11T22:22:05.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:04 smithi144 ceph-mon[33984]: 3.12 scrub ok 2024-03-11T22:22:05.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:04 smithi144 ceph-mon[33984]: 5.1a scrub starts 2024-03-11T22:22:05.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:04 smithi144 ceph-mon[33984]: 5.1a scrub ok 2024-03-11T22:22:05.136 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:05.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:05.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:05.238 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:05.270 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:05.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:05.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:05.410 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:05.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:05.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:05.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:05.541 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:05.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:05.585 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:05.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:05.686 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:05.717 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:05.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:05.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:05.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:05.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:05.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:05.976 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:05.998 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:05 smithi144 ceph-mon[33984]: pgmap v634: 168 pgs: 168 active+clean; 1.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 881 op/s 2024-03-11T22:22:05.998 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:05 smithi144 ceph-mon[33984]: 6.2 scrub starts 2024-03-11T22:22:05.998 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:05 smithi144 ceph-mon[33984]: 6.2 scrub ok 2024-03-11T22:22:06.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:05 smithi045 ceph-mon[30178]: pgmap v634: 168 pgs: 168 active+clean; 1.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 881 op/s 2024-03-11T22:22:06.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:05 smithi045 ceph-mon[30178]: 6.2 scrub starts 2024-03-11T22:22:06.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:05 smithi045 ceph-mon[30178]: 6.2 scrub ok 2024-03-11T22:22:06.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:05 smithi045 ceph-mon[39715]: pgmap v634: 168 pgs: 168 active+clean; 1.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 881 op/s 2024-03-11T22:22:06.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:05 smithi045 ceph-mon[39715]: 6.2 scrub starts 2024-03-11T22:22:06.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:05 smithi045 ceph-mon[39715]: 6.2 scrub ok 2024-03-11T22:22:06.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:06.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:06.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:06.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:06.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:06.161 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:06.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:06.262 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:06.292 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:06.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:06.393 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:06.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:06.422 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:06.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:06.523 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:06.553 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:06.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:06.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:06.654 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:06.686 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:06.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:06.734 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:22:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:22:06.788 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:06.820 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:06.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:06.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:06.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:06 smithi045 ceph-mon[30178]: 6.9 scrub starts 2024-03-11T22:22:06.890 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:06 smithi045 ceph-mon[30178]: 6.9 scrub ok 2024-03-11T22:22:06.891 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:06 smithi045 ceph-mon[39715]: 6.9 scrub starts 2024-03-11T22:22:06.891 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:06 smithi045 ceph-mon[39715]: 6.9 scrub ok 2024-03-11T22:22:06.922 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:06.955 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:06.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:07.056 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:07.087 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:07.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:07.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:06 smithi144 ceph-mon[33984]: 6.9 scrub starts 2024-03-11T22:22:07.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:06 smithi144 ceph-mon[33984]: 6.9 scrub ok 2024-03-11T22:22:07.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:07.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:07.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:07.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:07.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:07.390 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:07.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:07.492 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:07.527 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:07.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:07.628 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:07.654 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:07.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:07.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:07.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:07.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:07.796 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:07.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:07.897 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:07.900 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:22:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-03-11T22:22:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-03-11T22:22:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:22:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205592 7.e 7:78d8296a:::benchmark_data_smithi045_83007_object87121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:22:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.361866+0000", 2024-03-11T22:22:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.98937256099998, 2024-03-11T22:22:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23581138800000001, 2024-03-11T22:22:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:22:07.900 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205592 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.361866+0000", 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.361866+0000", 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.361867+0000", 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.299e-06 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.361870+0000", 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4290000000000001e-06 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.361880+0000", 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2939999999999994e-06 2024-03-11T22:22:07.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.361887+0000", 2024-03-11T22:22:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7389999999999999e-06 2024-03-11T22:22:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.362227+0000", 2024-03-11T22:22:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00033996299999999998 2024-03-11T22:22:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.362357+0000", 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000130001 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.362492+0000", 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013507600000000001 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.444224+0000", 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.081731649000000003 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.444244+0000", 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9990999999999999e-05 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.597523+0000", 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15327923099999999 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.597595+0000", 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1928999999999995e-05 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.597677+0000", 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1786999999999995e-05 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205756 7.e 7:76db1325:::benchmark_data_smithi045_83007_object87285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.650961+0000", 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.700276901, 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.33229374, 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:22:07.906 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205756 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.650961+0000", 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.650961+0000", 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.650963+0000", 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2759999999999999e-06 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.650967+0000", 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8389999999999997e-06 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.650979+0000", 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1386000000000001e-05 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.650987+0000", 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0150000000000003e-06 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.911846+0000", 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26085960400000002 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.911983+0000", 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000136756 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.912124+0000", 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014117599999999999 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915845+0000", 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037210559999999999 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916955+0000", 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0011094259999999999 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.983174+0000", 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.066219263 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.983213+0000", 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8688e-05 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.983255+0000", 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2255000000000001e-05 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.911 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40360 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.665971+0000", 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.68526712099998, 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.249558946, 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.665971+0000", 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.665971+0000", 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.665972+0000", 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8000000000000004e-07 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.665981+0000", 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8950000000000004e-06 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.665991+0000", 2024-03-11T22:22:07.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0689999999999999e-05 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.665999+0000", 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6359999999999994e-06 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915503+0000", 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24950422899999999 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915512+0000", 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2849999999999995e-06 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915530+0000", 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8331000000000001e-05 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40360 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.668874+0000", 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.68236450500001, 2024-03-11T22:22:07.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.246778778, 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668874+0000", 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668874+0000", 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668874+0000", 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0100000000000004e-07 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668879+0000", 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5709999999999999e-06 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668889+0000", 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7000000000000003e-06 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668895+0000", 2024-03-11T22:22:07.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7819999999999996e-06 2024-03-11T22:22:07.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915554+0000", 2024-03-11T22:22:07.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24665883599999999 2024-03-11T22:22:07.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915561+0000", 2024-03-11T22:22:07.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3099999999999997e-06 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915652+0000", 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1878000000000004e-05 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40361 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.668998+0000", 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.68224059300002, 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.246704801, 2024-03-11T22:22:07.919 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668998+0000", 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668998+0000", 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.668998+0000", 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1899999999999998e-07 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.669001+0000", 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.683e-06 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.669008+0000", 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3409999999999997e-06 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.669013+0000", 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9579999999999998e-06 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915676+0000", 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24666300999999999 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915688+0000", 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1669e-05 2024-03-11T22:22:07.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915702+0000", 2024-03-11T22:22:07.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4820999999999999e-05 2024-03-11T22:22:07.923 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.923 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.923 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.923 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40363 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:22:07.923 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.675731+0000", 2024-03-11T22:22:07.923 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.67550765300001, 2024-03-11T22:22:07.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24011618100000001, 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.675731+0000", 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.675731+0000", 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.675732+0000", 2024-03-11T22:22:07.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.082e-06 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.675738+0000", 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6610000000000003e-06 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.675752+0000", 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3913e-05 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.675761+0000", 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3769999999999996e-06 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915807+0000", 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.240046381 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915815+0000", 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5739999999999996e-06 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915847+0000", 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1192999999999998e-05 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205779 7.e 7:7733b75f:::benchmark_data_smithi045_83007_object87308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.678063+0000", 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.67317553399999, 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30589906300000003, 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:22:07.927 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205779 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678063+0000", 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678063+0000", 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678065+0000", 2024-03-11T22:22:07.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0590000000000001e-06 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678071+0000", 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9499999999999998e-06 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678085+0000", 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4847e-05 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.678097+0000", 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1124000000000001e-05 2024-03-11T22:22:07.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916985+0000", 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23888874500000001 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.917115+0000", 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012928500000000001 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.917246+0000", 2024-03-11T22:22:07.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013107799999999999 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.924476+0000", 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072302360000000001 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.930799+0000", 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0063235510000000002 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.983827+0000", 2024-03-11T22:22:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.053027815999999998 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.983895+0000", 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7939000000000001e-05 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.983962+0000", 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6433000000000006e-05 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40364 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.679313+0000", 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.671924993, 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23669196200000001, 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679313+0000", 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679313+0000", 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679314+0000", 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2800000000000005e-07 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679323+0000", 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3190000000000001e-06 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679336+0000", 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2721999999999999e-05 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679345+0000", 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6379999999999993e-06 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915980+0000", 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23663551199999999 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.915988+0000", 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7069999999999992e-06 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916005+0000", 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7136e-05 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40361 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.679621+0000", 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.671616847, 2024-03-11T22:22:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23650580700000001, 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679621+0000", 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679621+0000", 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679622+0000", 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4899999999999995e-07 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679626+0000", 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0280000000000002e-06 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679636+0000", 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7520000000000008e-06 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.679642+0000", 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.319e-06 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916039+0000", 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23639743099999999 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916046+0000", 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9759999999999998e-06 2024-03-11T22:22:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916127+0000", 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0752000000000002e-05 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(osd.2.0:40365 7.e e150/99 ondisk, result = 0)", 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:45.683465+0000", 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.667773431, 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23270696499999999, 2024-03-11T22:22:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.683465+0000", 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.683465+0000", 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.683466+0000", 2024-03-11T22:22:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7699999999999992e-07 2024-03-11T22:22:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.683472+0000", 2024-03-11T22:22:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8889999999999999e-06 2024-03-11T22:22:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.683484+0000", 2024-03-11T22:22:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2340999999999999e-05 2024-03-11T22:22:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.683492+0000", 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9109999999999993e-06 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916152+0000", 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23266065399999999 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916160+0000", 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.328e-06 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:45.916172+0000", 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1865000000000001e-05 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205921 7.e 7:7bd432cf:::benchmark_data_smithi045_83007_object87450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.033967+0000", 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.31727128300003, 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.242674745, 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:22:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205921 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.033967+0000", 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.033967+0000", 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.033968+0000", 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.093e-06 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.033972+0000", 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5559999999999999e-06 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.033982+0000", 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0745e-05 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.033990+0000", 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5449999999999998e-06 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.034047+0000", 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7414999999999998e-05 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.034196+0000", 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000148465 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.034443+0000", 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000247366 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.038503+0000", 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040595529999999996 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.046484+0000", 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0079812549999999996 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.276516+0000", 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23003189099999999 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.276576+0000", 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0503e-05 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.276642+0000", 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5357999999999993e-05 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205927 7.e 7:72966589:::benchmark_data_smithi045_83007_object87456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.041175+0000", 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.31006289800001, 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23561992600000001, 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205927 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041175+0000", 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041175+0000", 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041177+0000", 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5960000000000001e-06 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041182+0000", 2024-03-11T22:22:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0499999999999999e-06 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041203+0000", 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1063000000000001e-05 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041213+0000", 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0322e-05 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041628+0000", 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000414666 2024-03-11T22:22:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041762+0000", 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013364599999999999 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.041905+0000", 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014386600000000001 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.045920+0000", 2024-03-11T22:22:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040150120000000001 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.046605+0000", 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00068451600000000003 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.276686+0000", 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23008144899999999 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.276741+0000", 2024-03-11T22:22:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4286999999999997e-05 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.276795+0000", 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4453000000000002e-05 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205934 7.e 7:79ae0710:::benchmark_data_smithi045_83007_object87463:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.051197+0000", 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.300041432, 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26231636600000002, 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205934 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.051197+0000", 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.051197+0000", 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.051198+0000", 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.389e-06 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.051203+0000", 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8740000000000003e-06 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.051215+0000", 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2131e-05 2024-03-11T22:22:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.051223+0000", 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1839999999999999e-06 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.272457+0000", 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22123373399999999 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.272609+0000", 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000152075 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.272782+0000", 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017291799999999999 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.280086+0000", 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0073040580000000004 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.306915+0000", 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.026828538999999998 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:22:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313321+0000", 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0064061220000000002 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313453+0000", 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013265299999999999 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313513+0000", 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9688999999999997e-05 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205936 7.e 7:7d99f5bd:::benchmark_data_smithi045_83007_object87465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.054435+0000", 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.29680290499999, 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.259277386, 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205936 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.054435+0000", 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.054435+0000", 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.054437+0000", 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6249999999999999e-06 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.054442+0000", 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2349999999999999e-06 2024-03-11T22:22:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.054455+0000", 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2476999999999999e-05 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.054463+0000", 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4819999999999994e-06 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.280116+0000", 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22565284499999999 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.280240+0000", 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000124079 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.280367+0000", 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000126608 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.293440+0000", 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013073342999999999 2024-03-11T22:22:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.306951+0000", 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.01351135 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313567+0000", 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.006616116 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313648+0000", 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0630999999999994e-05 2024-03-11T22:22:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313713+0000", 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4595000000000001e-05 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:205940 7.e 7:7935b90f:::benchmark_data_smithi045_83007_object87469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.055176+0000", 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.29606245600002, 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25869365, 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 205940 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.055176+0000", 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.055176+0000", 2024-03-11T22:22:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.055176+0000", 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3000000000000001e-07 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.055179+0000", 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2200000000000001e-06 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.055186+0000", 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8349999999999998e-06 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.055192+0000", 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4999999999999999e-06 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.288665+0000", 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.233473074 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.288787+0000", 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012224 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.288958+0000", 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000171206 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.293503+0000", 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0045447270000000001 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.306991+0000", 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.013488348000000001 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313721+0000", 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0067295289999999997 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313799+0000", 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7831999999999996e-05 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.313869+0000", 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0709000000000004e-05 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206080 7.e 7:70863e2d:::benchmark_data_smithi045_83007_object87609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.349294+0000", 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 501.00194378200001, 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25670769999999998, 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:22:07.972 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206080 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.349294+0000", 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.349294+0000", 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.349295+0000", 2024-03-11T22:22:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7299999999999995e-07 2024-03-11T22:22:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.349298+0000", 2024-03-11T22:22:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.298e-06 2024-03-11T22:22:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.349306+0000", 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0439999999999996e-06 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.349311+0000", 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0359999999999997e-06 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.519694+0000", 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17038214800000001 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.519821+0000", 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012704799999999999 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.519969+0000", 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014822699999999999 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.524895+0000", 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0049257370000000003 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.533760+0000", 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0088654259999999992 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.605877+0000", 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.072117052000000001 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.605939+0000", 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2008000000000007e-05 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.606002+0000", 2024-03-11T22:22:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3003000000000006e-05 2024-03-11T22:22:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206085 7.e 7:7809afd0:::benchmark_data_smithi045_83007_object87614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:22:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.361807+0000", 2024-03-11T22:22:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 500.98943157000002, 2024-03-11T22:22:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.246123166, 2024-03-11T22:22:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206085 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.361807+0000", 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.361807+0000", 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.361808+0000", 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3689999999999999e-06 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.361812+0000", 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1710000000000004e-06 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.361822+0000", 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7969999999999995e-06 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.361830+0000", 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6199999999999999e-06 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.524930+0000", 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16310034100000001 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.525065+0000", 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013499200000000001 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.525212+0000", 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014669300000000001 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.533369+0000", 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0081575659999999998 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.533765+0000", 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00039629299999999998 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.607806+0000", 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.074040410000000001 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.607875+0000", 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8998000000000006e-05 2024-03-11T22:22:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.607930+0000", 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4916000000000001e-05 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206224 7.e 7:7359e978:::benchmark_data_smithi045_83007_object87753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.663615+0000", 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 500.68762288599999, 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25281374200000001, 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206224 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.663615+0000", 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.663615+0000", 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.663617+0000", 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2890000000000001e-06 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.663621+0000", 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.685e-06 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.663632+0000", 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0957e-05 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.663641+0000", 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7979999999999995e-06 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.760335+0000", 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.096694374999999999 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.760460+0000", 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012459299999999999 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.760587+0000", 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012663900000000001 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.773226+0000", 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012639188000000001 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.773239+0000", 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3212e-05 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.916329+0000", 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.143089889 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.916378+0000", 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9440000000000001e-05 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.916429+0000", 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0677000000000003e-05 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206230 7.e 7:7bd288ae:::benchmark_data_smithi045_83007_object87759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.670446+0000", 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 500.68079226600003, 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25676780700000001, 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:22:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206230 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.670446+0000", 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.670446+0000", 2024-03-11T22:22:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.670447+0000", 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.265e-06 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.670452+0000", 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0050000000000004e-06 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.670467+0000", 2024-03-11T22:22:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4878000000000001e-05 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.670475+0000", 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0010000000000001e-06 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.766547+0000", 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.096072195999999999 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.766667+0000", 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000119776 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.766791+0000", 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000123499 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.773354+0000", 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0065630130000000004 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.773385+0000", 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1952000000000002e-05 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.926976+0000", 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15359084000000001 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.927123+0000", 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014644199999999999 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.927214+0000", 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0939999999999993e-05 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.14790.0:206249 7.e 7:79d284c3:::benchmark_data_smithi045_83007_object87778:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:13:46.695778+0000", 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 500.655460339, 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23172493399999999, 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:22:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.14790", 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 206249 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.695778+0000", 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.695778+0000", 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:22:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.695780+0000", 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0619999999999999e-06 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.695787+0000", 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8789999999999997e-06 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.695802+0000", 2024-03-11T22:22:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5362999999999998e-05 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.695818+0000", 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6124999999999999e-05 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.769867+0000", 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.074049078000000004 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.770014+0000", 2024-03-11T22:22:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014628500000000001 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.770137+0000", 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000122922 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.773419+0000", 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032821349999999998 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.783366+0000", 2024-03-11T22:22:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0099476949999999995 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.927268+0000", 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.14390134299999999 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.927431+0000", 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016337400000000001 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:13:46.927503+0000", 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1673000000000003e-05 2024-03-11T22:22:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:22:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:22:08.002 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:22:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:08.004 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:08.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:08.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[30178]: pgmap v635: 168 pgs: 168 active+clean; 1.5 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T22:22:08.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[30178]: 5.4 scrub starts 2024-03-11T22:22:08.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[30178]: 5.4 scrub ok 2024-03-11T22:22:08.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:08.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[30178]: 4.1a scrub starts 2024-03-11T22:22:08.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[30178]: 4.1a scrub ok 2024-03-11T22:22:08.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[30178]: 3.19 scrub starts 2024-03-11T22:22:08.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[30178]: 3.19 scrub ok 2024-03-11T22:22:08.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[39715]: pgmap v635: 168 pgs: 168 active+clean; 1.5 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T22:22:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[39715]: 5.4 scrub starts 2024-03-11T22:22:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[39715]: 5.4 scrub ok 2024-03-11T22:22:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[39715]: 4.1a scrub starts 2024-03-11T22:22:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[39715]: 4.1a scrub ok 2024-03-11T22:22:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[39715]: 3.19 scrub starts 2024-03-11T22:22:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:07 smithi045 ceph-mon[39715]: 3.19 scrub ok 2024-03-11T22:22:08.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:07 smithi144 ceph-mon[33984]: pgmap v635: 168 pgs: 168 active+clean; 1.5 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T22:22:08.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:07 smithi144 ceph-mon[33984]: 5.4 scrub starts 2024-03-11T22:22:08.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:07 smithi144 ceph-mon[33984]: 5.4 scrub ok 2024-03-11T22:22:08.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:08.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:07 smithi144 ceph-mon[33984]: 4.1a scrub starts 2024-03-11T22:22:08.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:07 smithi144 ceph-mon[33984]: 4.1a scrub ok 2024-03-11T22:22:08.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:07 smithi144 ceph-mon[33984]: 3.19 scrub starts 2024-03-11T22:22:08.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:07 smithi144 ceph-mon[33984]: 3.19 scrub ok 2024-03-11T22:22:08.104 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:08.144 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:08.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:08.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:08.246 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:08.279 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:08.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:08.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:08.417 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:08.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:08.519 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:08.554 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:08.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:08.655 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:08.692 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:08.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:08.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:08.794 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:08.829 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:08.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:08.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:08.977 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:08.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:09.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:08 smithi045 ceph-mon[30178]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:22:09.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:08 smithi045 ceph-mon[30178]: 1.1 scrub starts 2024-03-11T22:22:09.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:08 smithi045 ceph-mon[30178]: 1.1 scrub ok 2024-03-11T22:22:09.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:08 smithi045 ceph-mon[30178]: 4.16 scrub starts 2024-03-11T22:22:09.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:08 smithi045 ceph-mon[30178]: 4.16 scrub ok 2024-03-11T22:22:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:08 smithi045 ceph-mon[39715]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:22:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:08 smithi045 ceph-mon[39715]: 1.1 scrub starts 2024-03-11T22:22:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:08 smithi045 ceph-mon[39715]: 1.1 scrub ok 2024-03-11T22:22:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:08 smithi045 ceph-mon[39715]: 4.16 scrub starts 2024-03-11T22:22:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:08 smithi045 ceph-mon[39715]: 4.16 scrub ok 2024-03-11T22:22:09.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:09.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:08 smithi144 ceph-mon[33984]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:22:09.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:08 smithi144 ceph-mon[33984]: 1.1 scrub starts 2024-03-11T22:22:09.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:08 smithi144 ceph-mon[33984]: 1.1 scrub ok 2024-03-11T22:22:09.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:08 smithi144 ceph-mon[33984]: 4.16 scrub starts 2024-03-11T22:22:09.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:08 smithi144 ceph-mon[33984]: 4.16 scrub ok 2024-03-11T22:22:09.114 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:09.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:09.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:09.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:09.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:09.255 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:09.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:09.356 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:09.398 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:09.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:09.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:09.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:09.545 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:09.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:09.614 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:22:09.628 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:22:09.647 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:09.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:09.694 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:09.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:09.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:09 smithi045 ceph-mon[39715]: pgmap v636: 168 pgs: 168 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T22:22:09.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:09 smithi045 ceph-mon[30178]: pgmap v636: 168 pgs: 168 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T22:22:09.795 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:09.829 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:09.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:09.931 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:09.961 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:09.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:09.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:10.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:10.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:09 smithi144 ceph-mon[33984]: pgmap v636: 168 pgs: 168 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T22:22:10.102 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:10.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:10.204 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:10.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:10.239 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:10.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:10.340 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:10.374 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:10.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:10.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T22:22:10.476 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:10.513 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:10.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:10.614 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:10.669 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:10.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:10.770 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:10.826 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:10.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:10.927 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:10.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:10.967 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:10.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:11.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:10 smithi045 ceph-mon[39715]: 5.7 scrub starts 2024-03-11T22:22:11.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:10 smithi045 ceph-mon[39715]: 5.7 scrub ok 2024-03-11T22:22:11.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:10 smithi045 ceph-mon[30178]: 5.7 scrub starts 2024-03-11T22:22:11.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:10 smithi045 ceph-mon[30178]: 5.7 scrub ok 2024-03-11T22:22:11.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:11.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:10 smithi144 ceph-mon[33984]: 5.7 scrub starts 2024-03-11T22:22:11.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:10 smithi144 ceph-mon[33984]: 5.7 scrub ok 2024-03-11T22:22:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:11.109 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:11.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:11.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:11.234 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:11.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:11.264 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:11.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:11.280 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":635,"stamp":"2024-03-11T22:22:06.330209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1637095796,"num_objects":25170,"num_object_clones":0,"num_object_copies":75510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25170,"num_whiteouts":0,"num_read":1898,"num_read_kb":1650,"num_write":223986,"num_write_kb":7943534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34765,"num_bytes_recovered":2270890396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75205,"ondisk_log_size":75205,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":506,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8483912,"kb_used_data":4974152,"kb_used_omap":12,"kb_used_meta":3509043,"kb_avail":741477304,"statfs":{"total":767960285184,"available":759272759296,"internally_reserved":0,"allocated":5093531648,"data_stored":5090927100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":3593260595},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,0,0,0,0,0,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-795082752,"num_objects":-12132,"num_object_clones":0,"num_object_copies":-36396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12132,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12132,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003802"},"pg_stats":[{"pgid":"9.1","version":"332'13933","reported_seq":25866,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.376846+0000","last_change":"2024-03-11T22:21:14.515399+0000","last_active":"2024-03-11T22:22:01.376846+0000","last_peered":"2024-03-11T22:22:01.376846+0000","last_clean":"2024-03-11T22:22:01.376846+0000","last_became_active":"2024-03-11T22:21:14.450983+0000","last_became_peered":"2024-03-11T22:21:14.450983+0000","last_unstale":"2024-03-11T22:22:01.376846+0000","last_undegraded":"2024-03-11T22:22:01.376846+0000","last_fullsized":"2024-03-11T22:22:01.376846+0000","mapping_epoch":318,"log_start":"314'9800","ondisk_log_start":"314'9800","created":273,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4133,"log_dups_size":0,"ondisk_log_size":4133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109772800,"num_objects":1675,"num_object_clones":0,"num_object_copies":5025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13933,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5892,"num_bytes_recovered":386007040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"294'25","reported_seq":389,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.637492+0000","last_change":"2024-03-11T22:19:41.508884+0000","last_active":"2024-03-11T22:21:51.637492+0000","last_peered":"2024-03-11T22:21:51.637492+0000","last_clean":"2024-03-11T22:21:51.637492+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:21:51.637492+0000","last_undegraded":"2024-03-11T22:21:51.637492+0000","last_fullsized":"2024-03-11T22:21:51.637492+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:19:41.508161+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":577,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.636027+0000","last_change":"2024-03-11T22:19:28.586829+0000","last_active":"2024-03-11T22:21:51.636027+0000","last_peered":"2024-03-11T22:21:51.636027+0000","last_clean":"2024-03-11T22:21:51.636027+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:21:51.636027+0000","last_undegraded":"2024-03-11T22:21:51.636027+0000","last_fullsized":"2024-03-11T22:21:51.636027+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:28.586614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:28.586614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":764,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:58.510608+0000","last_change":"2024-03-11T22:19:40.008490+0000","last_active":"2024-03-11T22:21:58.510608+0000","last_peered":"2024-03-11T22:21:58.510608+0000","last_clean":"2024-03-11T22:21:58.510608+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:21:58.510608+0000","last_undegraded":"2024-03-11T22:21:58.510608+0000","last_fullsized":"2024-03-11T22:21:58.510608+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:19:40.008325+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:19:40.008325+0000","last_clean_scrub_stamp":"2024-03-11T22:19:40.008325+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":343,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.603246+0000","last_change":"2024-03-11T22:18:37.116172+0000","last_active":"2024-03-11T22:21:51.603246+0000","last_peered":"2024-03-11T22:21:51.603246+0000","last_clean":"2024-03-11T22:21:51.603246+0000","last_became_active":"2024-03-11T22:14:52.243690+0000","last_became_peered":"2024-03-11T22:14:52.243690+0000","last_unstale":"2024-03-11T22:21:51.603246+0000","last_undegraded":"2024-03-11T22:21:51.603246+0000","last_fullsized":"2024-03-11T22:21:51.603246+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:37.116093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.116093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":899,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.635622+0000","last_change":"2024-03-11T22:20:32.294412+0000","last_active":"2024-03-11T22:21:51.635622+0000","last_peered":"2024-03-11T22:21:51.635622+0000","last_clean":"2024-03-11T22:21:51.635622+0000","last_became_active":"2024-03-11T22:20:32.293356+0000","last_became_peered":"2024-03-11T22:20:32.293356+0000","last_unstale":"2024-03-11T22:21:51.635622+0000","last_undegraded":"2024-03-11T22:21:51.635622+0000","last_fullsized":"2024-03-11T22:21:51.635622+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:20.907743+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.907743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"332'13917","reported_seq":14046,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.337147+0000","last_change":"2024-03-11T22:18:41.783594+0000","last_active":"2024-03-11T22:22:01.337147+0000","last_peered":"2024-03-11T22:22:01.337147+0000","last_clean":"2024-03-11T22:22:01.337147+0000","last_became_active":"2024-03-11T22:18:41.783248+0000","last_became_peered":"2024-03-11T22:18:41.783248+0000","last_unstale":"2024-03-11T22:22:01.337147+0000","last_undegraded":"2024-03-11T22:22:01.337147+0000","last_fullsized":"2024-03-11T22:22:01.337147+0000","mapping_epoch":273,"log_start":"314'9300","ondisk_log_start":"314'9300","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4617,"log_dups_size":0,"ondisk_log_size":4617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107675648,"num_objects":1643,"num_object_clones":0,"num_object_copies":4929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13917,"num_write_kb":497920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"273'31","reported_seq":427,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.603302+0000","last_change":"2024-03-11T22:20:40.790500+0000","last_active":"2024-03-11T22:21:51.603302+0000","last_peered":"2024-03-11T22:21:51.603302+0000","last_clean":"2024-03-11T22:21:51.603302+0000","last_became_active":"2024-03-11T22:14:52.239649+0000","last_became_peered":"2024-03-11T22:14:52.239649+0000","last_unstale":"2024-03-11T22:21:51.603302+0000","last_undegraded":"2024-03-11T22:21:51.603302+0000","last_fullsized":"2024-03-11T22:21:51.603302+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'31","last_scrub_stamp":"2024-03-11T22:20:40.790339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:40.790339+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":601,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602393+0000","last_change":"2024-03-11T22:20:52.712468+0000","last_active":"2024-03-11T22:21:51.602393+0000","last_peered":"2024-03-11T22:21:51.602393+0000","last_clean":"2024-03-11T22:21:51.602393+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:21:51.602393+0000","last_undegraded":"2024-03-11T22:21:51.602393+0000","last_fullsized":"2024-03-11T22:21:51.602393+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:52.712329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:20:52.712329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:11.261543+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":576,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602941+0000","last_change":"2024-03-11T22:18:44.130366+0000","last_active":"2024-03-11T22:21:51.602941+0000","last_peered":"2024-03-11T22:21:51.602941+0000","last_clean":"2024-03-11T22:21:51.602941+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:21:51.602941+0000","last_undegraded":"2024-03-11T22:21:51.602941+0000","last_fullsized":"2024-03-11T22:21:51.602941+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:44.130252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:18:44.130252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":588,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:00.325304+0000","last_change":"2024-03-11T22:22:00.325341+0000","last_active":"2024-03-11T22:22:00.325304+0000","last_peered":"2024-03-11T22:22:00.325304+0000","last_clean":"2024-03-11T22:22:00.325304+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:22:00.325304+0000","last_undegraded":"2024-03-11T22:22:00.325304+0000","last_fullsized":"2024-03-11T22:22:00.325304+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:00.325095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:00.325095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:06.335725+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":875,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.607599+0000","last_change":"2024-03-11T22:18:21.646464+0000","last_active":"2024-03-11T22:21:51.607599+0000","last_peered":"2024-03-11T22:21:51.607599+0000","last_clean":"2024-03-11T22:21:51.607599+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:21:51.607599+0000","last_undegraded":"2024-03-11T22:21:51.607599+0000","last_fullsized":"2024-03-11T22:21:51.607599+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:21.646305+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.646305+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"332'13999","reported_seq":22133,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.462904+0000","last_change":"2024-03-11T22:21:38.531313+0000","last_active":"2024-03-11T22:22:01.462904+0000","last_peered":"2024-03-11T22:22:01.462904+0000","last_clean":"2024-03-11T22:22:01.462904+0000","last_became_active":"2024-03-11T22:21:38.510998+0000","last_became_peered":"2024-03-11T22:21:38.510998+0000","last_unstale":"2024-03-11T22:22:01.462904+0000","last_undegraded":"2024-03-11T22:22:01.462904+0000","last_fullsized":"2024-03-11T22:22:01.462904+0000","mapping_epoch":326,"log_start":"316'9900","ondisk_log_start":"316'9900","created":273,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111214592,"num_objects":1697,"num_object_clones":0,"num_object_copies":5091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13999,"num_write_kb":502272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3990,"num_bytes_recovered":261357568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"273'19","reported_seq":401,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602320+0000","last_change":"2024-03-11T22:20:27.089007+0000","last_active":"2024-03-11T22:21:51.602320+0000","last_peered":"2024-03-11T22:21:51.602320+0000","last_clean":"2024-03-11T22:21:51.602320+0000","last_became_active":"2024-03-11T22:14:52.240437+0000","last_became_peered":"2024-03-11T22:14:52.240437+0000","last_unstale":"2024-03-11T22:21:51.602320+0000","last_undegraded":"2024-03-11T22:21:51.602320+0000","last_fullsized":"2024-03-11T22:21:51.602320+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:20:27.088856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:27.088856+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":360,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601470+0000","last_change":"2024-03-11T22:20:45.895055+0000","last_active":"2024-03-11T22:21:51.601470+0000","last_peered":"2024-03-11T22:21:51.601470+0000","last_clean":"2024-03-11T22:21:51.601470+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:21:51.601470+0000","last_undegraded":"2024-03-11T22:21:51.601470+0000","last_fullsized":"2024-03-11T22:21:51.601470+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:45.894940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:45.894940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":568,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.608804+0000","last_change":"2024-03-11T22:18:37.705396+0000","last_active":"2024-03-11T22:21:51.608804+0000","last_peered":"2024-03-11T22:21:51.608804+0000","last_clean":"2024-03-11T22:21:51.608804+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:21:51.608804+0000","last_undegraded":"2024-03-11T22:21:51.608804+0000","last_fullsized":"2024-03-11T22:21:51.608804+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:37.705305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.705305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":556,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.636463+0000","last_change":"2024-03-11T22:18:48.086934+0000","last_active":"2024-03-11T22:21:51.636463+0000","last_peered":"2024-03-11T22:21:51.636463+0000","last_clean":"2024-03-11T22:21:51.636463+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:21:51.636463+0000","last_undegraded":"2024-03-11T22:21:51.636463+0000","last_fullsized":"2024-03-11T22:21:51.636463+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:48.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:48.086856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"93'130","reported_seq":904,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602777+0000","last_change":"2024-03-11T22:18:20.968124+0000","last_active":"2024-03-11T22:21:51.602777+0000","last_peered":"2024-03-11T22:21:51.602777+0000","last_clean":"2024-03-11T22:21:51.602777+0000","last_became_active":"2024-03-11T22:18:00.271764+0000","last_became_peered":"2024-03-11T22:18:00.271764+0000","last_unstale":"2024-03-11T22:21:51.602777+0000","last_undegraded":"2024-03-11T22:21:51.602777+0000","last_fullsized":"2024-03-11T22:21:51.602777+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:20.968038+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.968038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"332'13718","reported_seq":13846,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.377581+0000","last_change":"2024-03-11T22:18:41.238608+0000","last_active":"2024-03-11T22:22:01.377581+0000","last_peered":"2024-03-11T22:22:01.377581+0000","last_clean":"2024-03-11T22:22:01.377581+0000","last_became_active":"2024-03-11T22:18:41.238269+0000","last_became_peered":"2024-03-11T22:18:41.238269+0000","last_unstale":"2024-03-11T22:22:01.377581+0000","last_undegraded":"2024-03-11T22:22:01.377581+0000","last_fullsized":"2024-03-11T22:22:01.377581+0000","mapping_epoch":273,"log_start":"314'9600","ondisk_log_start":"314'9600","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4118,"log_dups_size":0,"ondisk_log_size":4118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104726528,"num_objects":1598,"num_object_clones":0,"num_object_copies":4794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13718,"num_write_kb":490112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"273'34","reported_seq":646,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602557+0000","last_change":"2024-03-11T22:21:49.811476+0000","last_active":"2024-03-11T22:21:51.602557+0000","last_peered":"2024-03-11T22:21:51.602557+0000","last_clean":"2024-03-11T22:21:51.602557+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:21:51.602557+0000","last_undegraded":"2024-03-11T22:21:51.602557+0000","last_fullsized":"2024-03-11T22:21:51.602557+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:21:49.811214+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:21:49.811214+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:16.306128+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":589,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:56.018601+0000","last_change":"2024-03-11T22:21:56.018639+0000","last_active":"2024-03-11T22:21:56.018601+0000","last_peered":"2024-03-11T22:21:56.018601+0000","last_clean":"2024-03-11T22:21:56.018601+0000","last_became_active":"2024-03-11T22:20:29.426224+0000","last_became_peered":"2024-03-11T22:20:29.426224+0000","last_unstale":"2024-03-11T22:21:56.018601+0000","last_undegraded":"2024-03-11T22:21:56.018601+0000","last_fullsized":"2024-03-11T22:21:56.018601+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:56.018519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:21:56.018519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:22.003241+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":456,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:58.510241+0000","last_change":"2024-03-11T22:20:59.636170+0000","last_active":"2024-03-11T22:21:58.510241+0000","last_peered":"2024-03-11T22:21:58.510241+0000","last_clean":"2024-03-11T22:21:58.510241+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:21:58.510241+0000","last_undegraded":"2024-03-11T22:21:58.510241+0000","last_fullsized":"2024-03-11T22:21:58.510241+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:20:59.636041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:59.636041+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:24.576756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":564,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.607509+0000","last_change":"2024-03-11T22:18:50.805816+0000","last_active":"2024-03-11T22:21:51.607509+0000","last_peered":"2024-03-11T22:21:51.607509+0000","last_clean":"2024-03-11T22:21:51.607509+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:21:51.607509+0000","last_undegraded":"2024-03-11T22:21:51.607509+0000","last_fullsized":"2024-03-11T22:21:51.607509+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:50.805698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:18:50.805698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":884,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:57.313507+0000","last_change":"2024-03-11T22:21:57.313533+0000","last_active":"2024-03-11T22:21:57.313507+0000","last_peered":"2024-03-11T22:21:57.313507+0000","last_clean":"2024-03-11T22:21:57.313507+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:21:57.313507+0000","last_undegraded":"2024-03-11T22:21:57.313507+0000","last_fullsized":"2024-03-11T22:21:57.313507+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:21:57.313449+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:21:57.313449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:21.421483+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"332'13893","reported_seq":21718,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.387303+0000","last_change":"2024-03-11T22:21:37.526283+0000","last_active":"2024-03-11T22:22:01.387303+0000","last_peered":"2024-03-11T22:22:01.387303+0000","last_clean":"2024-03-11T22:22:01.387303+0000","last_became_active":"2024-03-11T22:21:37.500225+0000","last_became_peered":"2024-03-11T22:21:37.500225+0000","last_unstale":"2024-03-11T22:22:01.387303+0000","last_undegraded":"2024-03-11T22:22:01.387303+0000","last_fullsized":"2024-03-11T22:22:01.387303+0000","mapping_epoch":325,"log_start":"314'9300","ondisk_log_start":"314'9300","created":273,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103415832,"num_objects":1579,"num_object_clones":0,"num_object_copies":4737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13893,"num_write_kb":495041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":251592728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":353,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.636052+0000","last_change":"2024-03-11T22:20:29.441577+0000","last_active":"2024-03-11T22:21:51.636052+0000","last_peered":"2024-03-11T22:21:51.636052+0000","last_clean":"2024-03-11T22:21:51.636052+0000","last_became_active":"2024-03-11T22:20:29.438896+0000","last_became_peered":"2024-03-11T22:20:29.438896+0000","last_unstale":"2024-03-11T22:21:51.636052+0000","last_undegraded":"2024-03-11T22:21:51.636052+0000","last_fullsized":"2024-03-11T22:21:51.636052+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.293977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.293977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"273'30","reported_seq":666,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.635759+0000","last_change":"2024-03-11T22:20:29.473203+0000","last_active":"2024-03-11T22:21:51.635759+0000","last_peered":"2024-03-11T22:21:51.635759+0000","last_clean":"2024-03-11T22:21:51.635759+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:21:51.635759+0000","last_undegraded":"2024-03-11T22:21:51.635759+0000","last_fullsized":"2024-03-11T22:21:51.635759+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:15:33.497732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:33.497732+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":589,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.637565+0000","last_change":"2024-03-11T22:20:29.441068+0000","last_active":"2024-03-11T22:21:51.637565+0000","last_peered":"2024-03-11T22:21:51.637565+0000","last_clean":"2024-03-11T22:21:51.637565+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:21:51.637565+0000","last_undegraded":"2024-03-11T22:21:51.637565+0000","last_fullsized":"2024-03-11T22:21:51.637565+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.133631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.133631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":335,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.636135+0000","last_change":"2024-03-11T22:15:12.339167+0000","last_active":"2024-03-11T22:21:51.636135+0000","last_peered":"2024-03-11T22:21:51.636135+0000","last_clean":"2024-03-11T22:21:51.636135+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:21:51.636135+0000","last_undegraded":"2024-03-11T22:21:51.636135+0000","last_fullsized":"2024-03-11T22:21:51.636135+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:12.339050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:15:12.339050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":837,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.603611+0000","last_change":"2024-03-11T22:20:57.831615+0000","last_active":"2024-03-11T22:21:51.603611+0000","last_peered":"2024-03-11T22:21:51.603611+0000","last_clean":"2024-03-11T22:21:51.603611+0000","last_became_active":"2024-03-11T22:18:20.240728+0000","last_became_peered":"2024-03-11T22:18:20.240728+0000","last_unstale":"2024-03-11T22:21:51.603611+0000","last_undegraded":"2024-03-11T22:21:51.603611+0000","last_fullsized":"2024-03-11T22:21:51.603611+0000","mapping_epoch":175,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:57.831334+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:20:57.831334+0000","last_clean_scrub_stamp":"2024-03-11T22:20:57.831334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:02.019183+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.c","version":"332'13786","reported_seq":13915,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.359897+0000","last_change":"2024-03-11T22:18:41.260878+0000","last_active":"2024-03-11T22:22:01.359897+0000","last_peered":"2024-03-11T22:22:01.359897+0000","last_clean":"2024-03-11T22:22:01.359897+0000","last_became_active":"2024-03-11T22:18:41.260611+0000","last_became_peered":"2024-03-11T22:18:41.260611+0000","last_unstale":"2024-03-11T22:22:01.359897+0000","last_undegraded":"2024-03-11T22:22:01.359897+0000","last_fullsized":"2024-03-11T22:22:01.359897+0000","mapping_epoch":273,"log_start":"316'9700","ondisk_log_start":"316'9700","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4086,"log_dups_size":0,"ondisk_log_size":4086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":4938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13786,"num_write_kb":493824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":731,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:53.518954+0000","last_change":"2024-03-11T22:15:31.519229+0000","last_active":"2024-03-11T22:21:53.518954+0000","last_peered":"2024-03-11T22:21:53.518954+0000","last_clean":"2024-03-11T22:21:53.518954+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:21:53.518954+0000","last_undegraded":"2024-03-11T22:21:53.518954+0000","last_fullsized":"2024-03-11T22:21:53.518954+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:15:31.513336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:31.513336+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"273'28","reported_seq":404,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.608225+0000","last_change":"2024-03-11T22:21:50.308999+0000","last_active":"2024-03-11T22:21:51.608225+0000","last_peered":"2024-03-11T22:21:51.608225+0000","last_clean":"2024-03-11T22:21:51.608225+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:21:51.608225+0000","last_undegraded":"2024-03-11T22:21:51.608225+0000","last_fullsized":"2024-03-11T22:21:51.608225+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:21:50.308860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:21:50.308860+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:18.140149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":578,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.605488+0000","last_change":"2024-03-11T22:19:28.428409+0000","last_active":"2024-03-11T22:21:51.605488+0000","last_peered":"2024-03-11T22:21:51.605488+0000","last_clean":"2024-03-11T22:21:51.605488+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:21:51.605488+0000","last_undegraded":"2024-03-11T22:21:51.605488+0000","last_fullsized":"2024-03-11T22:21:51.605488+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:19:28.428283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:28.428283+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":531,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:59.625718+0000","last_change":"2024-03-11T22:21:59.625759+0000","last_active":"2024-03-11T22:21:59.625718+0000","last_peered":"2024-03-11T22:21:59.625718+0000","last_clean":"2024-03-11T22:21:59.625718+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:21:59.625718+0000","last_undegraded":"2024-03-11T22:21:59.625718+0000","last_fullsized":"2024-03-11T22:21:59.625718+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:59.625636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:21:59.625636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:03.539206+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":893,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.636343+0000","last_change":"2024-03-11T22:20:32.293956+0000","last_active":"2024-03-11T22:21:51.636343+0000","last_peered":"2024-03-11T22:21:51.636343+0000","last_clean":"2024-03-11T22:21:51.636343+0000","last_became_active":"2024-03-11T22:20:32.293085+0000","last_became_peered":"2024-03-11T22:20:32.293085+0000","last_unstale":"2024-03-11T22:21:51.636343+0000","last_undegraded":"2024-03-11T22:21:51.636343+0000","last_fullsized":"2024-03-11T22:21:51.636343+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:18.905939+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:18.905939+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"332'14404","reported_seq":14533,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:05.523619+0000","last_change":"2024-03-11T22:18:41.236691+0000","last_active":"2024-03-11T22:22:05.523619+0000","last_peered":"2024-03-11T22:22:05.523619+0000","last_clean":"2024-03-11T22:22:05.523619+0000","last_became_active":"2024-03-11T22:18:41.236462+0000","last_became_peered":"2024-03-11T22:18:41.236462+0000","last_unstale":"2024-03-11T22:22:05.523619+0000","last_undegraded":"2024-03-11T22:22:05.523619+0000","last_fullsized":"2024-03-11T22:22:05.523619+0000","mapping_epoch":273,"log_start":"316'9900","ondisk_log_start":"316'9900","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90439680,"num_objects":1380,"num_object_clones":0,"num_object_copies":4140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14404,"num_write_kb":505088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":739,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:53.523177+0000","last_change":"2024-03-11T22:18:35.815130+0000","last_active":"2024-03-11T22:21:53.523177+0000","last_peered":"2024-03-11T22:21:53.523177+0000","last_clean":"2024-03-11T22:21:53.523177+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:21:53.523177+0000","last_undegraded":"2024-03-11T22:21:53.523177+0000","last_fullsized":"2024-03-11T22:21:53.523177+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:18:35.815031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:35.815031+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"273'20","reported_seq":619,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.603652+0000","last_change":"2024-03-11T22:19:31.503395+0000","last_active":"2024-03-11T22:21:51.603652+0000","last_peered":"2024-03-11T22:21:51.603652+0000","last_clean":"2024-03-11T22:21:51.603652+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:21:51.603652+0000","last_undegraded":"2024-03-11T22:21:51.603652+0000","last_fullsized":"2024-03-11T22:21:51.603652+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:19:31.502993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:31.502993+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":585,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.608437+0000","last_change":"2024-03-11T22:19:34.706343+0000","last_active":"2024-03-11T22:21:51.608437+0000","last_peered":"2024-03-11T22:21:51.608437+0000","last_clean":"2024-03-11T22:21:51.608437+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:21:51.608437+0000","last_undegraded":"2024-03-11T22:21:51.608437+0000","last_fullsized":"2024-03-11T22:21:51.608437+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.706222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.706222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":565,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.604334+0000","last_change":"2024-03-11T22:19:34.477472+0000","last_active":"2024-03-11T22:21:51.604334+0000","last_peered":"2024-03-11T22:21:51.604334+0000","last_clean":"2024-03-11T22:21:51.604334+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:21:51.604334+0000","last_undegraded":"2024-03-11T22:21:51.604334+0000","last_fullsized":"2024-03-11T22:21:51.604334+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.477349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.477349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":817,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602927+0000","last_change":"2024-03-11T22:20:43.803186+0000","last_active":"2024-03-11T22:21:51.602927+0000","last_peered":"2024-03-11T22:21:51.602927+0000","last_clean":"2024-03-11T22:21:51.602927+0000","last_became_active":"2024-03-11T22:20:32.290512+0000","last_became_peered":"2024-03-11T22:20:32.290512+0000","last_unstale":"2024-03-11T22:21:51.602927+0000","last_undegraded":"2024-03-11T22:21:51.602927+0000","last_fullsized":"2024-03-11T22:21:51.602927+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:43.803067+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:20:43.803067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"332'13778","reported_seq":25616,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.365526+0000","last_change":"2024-03-11T22:21:15.469013+0000","last_active":"2024-03-11T22:22:01.365526+0000","last_peered":"2024-03-11T22:22:01.365526+0000","last_clean":"2024-03-11T22:22:01.365526+0000","last_became_active":"2024-03-11T22:21:15.407140+0000","last_became_peered":"2024-03-11T22:21:15.407140+0000","last_unstale":"2024-03-11T22:22:01.365526+0000","last_undegraded":"2024-03-11T22:22:01.365526+0000","last_fullsized":"2024-03-11T22:22:01.365526+0000","mapping_epoch":319,"log_start":"314'9200","ondisk_log_start":"314'9200","created":273,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4578,"log_dups_size":0,"ondisk_log_size":4578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105906176,"num_objects":1616,"num_object_clones":0,"num_object_copies":4848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13778,"num_write_kb":492608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5847,"num_bytes_recovered":383188992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":593,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:00.588685+0000","last_change":"2024-03-11T22:22:00.588718+0000","last_active":"2024-03-11T22:22:00.588685+0000","last_peered":"2024-03-11T22:22:00.588685+0000","last_clean":"2024-03-11T22:22:00.588685+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:22:00.588685+0000","last_undegraded":"2024-03-11T22:22:00.588685+0000","last_fullsized":"2024-03-11T22:22:00.588685+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:00.588603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:22:00.588603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:09.767814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"273'29","reported_seq":624,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.605261+0000","last_change":"2024-03-11T22:19:38.450843+0000","last_active":"2024-03-11T22:21:51.605261+0000","last_peered":"2024-03-11T22:21:51.605261+0000","last_clean":"2024-03-11T22:21:51.605261+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:21:51.605261+0000","last_undegraded":"2024-03-11T22:21:51.605261+0000","last_fullsized":"2024-03-11T22:21:51.605261+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'29","last_scrub_stamp":"2024-03-11T22:19:38.450581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.450581+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":585,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602728+0000","last_change":"2024-03-11T22:18:38.153695+0000","last_active":"2024-03-11T22:21:51.602728+0000","last_peered":"2024-03-11T22:21:51.602728+0000","last_clean":"2024-03-11T22:21:51.602728+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:21:51.602728+0000","last_undegraded":"2024-03-11T22:21:51.602728+0000","last_fullsized":"2024-03-11T22:21:51.602728+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:38.153614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.153614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":573,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601945+0000","last_change":"2024-03-11T22:18:23.998360+0000","last_active":"2024-03-11T22:21:51.601945+0000","last_peered":"2024-03-11T22:21:51.601945+0000","last_clean":"2024-03-11T22:21:51.601945+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:21:51.601945+0000","last_undegraded":"2024-03-11T22:21:51.601945+0000","last_fullsized":"2024-03-11T22:21:51.601945+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:23.998251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:23.998251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":792,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602793+0000","last_change":"2024-03-11T22:20:44.849410+0000","last_active":"2024-03-11T22:21:51.602793+0000","last_peered":"2024-03-11T22:21:51.602793+0000","last_clean":"2024-03-11T22:21:51.602793+0000","last_became_active":"2024-03-11T22:18:17.238247+0000","last_became_peered":"2024-03-11T22:18:17.238247+0000","last_unstale":"2024-03-11T22:21:51.602793+0000","last_undegraded":"2024-03-11T22:21:51.602793+0000","last_fullsized":"2024-03-11T22:21:51.602793+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:44.849279+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:20:44.849279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:04.442646+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"332'13667","reported_seq":13796,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.868253+0000","last_change":"2024-03-11T22:18:41.783592+0000","last_active":"2024-03-11T22:22:01.868253+0000","last_peered":"2024-03-11T22:22:01.868253+0000","last_clean":"2024-03-11T22:22:01.868253+0000","last_became_active":"2024-03-11T22:18:41.783277+0000","last_became_peered":"2024-03-11T22:18:41.783277+0000","last_unstale":"2024-03-11T22:22:01.868253+0000","last_undegraded":"2024-03-11T22:22:01.868253+0000","last_fullsized":"2024-03-11T22:22:01.868253+0000","mapping_epoch":273,"log_start":"313'8400","ondisk_log_start":"313'8400","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5267,"log_dups_size":0,"ondisk_log_size":5267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105971712,"num_objects":1617,"num_object_clones":0,"num_object_copies":4851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13667,"num_write_kb":489088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":569,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.608893+0000","last_change":"2024-03-11T22:18:25.668662+0000","last_active":"2024-03-11T22:21:51.608893+0000","last_peered":"2024-03-11T22:21:51.608893+0000","last_clean":"2024-03-11T22:21:51.608893+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:21:51.608893+0000","last_undegraded":"2024-03-11T22:21:51.608893+0000","last_fullsized":"2024-03-11T22:21:51.608893+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:25.668581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:25.668581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"273'24","reported_seq":617,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.605476+0000","last_change":"2024-03-11T22:20:29.380624+0000","last_active":"2024-03-11T22:21:51.605476+0000","last_peered":"2024-03-11T22:21:51.605476+0000","last_clean":"2024-03-11T22:21:51.605476+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:21:51.605476+0000","last_undegraded":"2024-03-11T22:21:51.605476+0000","last_fullsized":"2024-03-11T22:21:51.605476+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'24","last_scrub_stamp":"2024-03-11T22:20:29.380527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:29.380527+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":1373,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:59.927762+0000","last_change":"2024-03-11T22:18:27.077438+0000","last_active":"2024-03-11T22:21:59.927762+0000","last_peered":"2024-03-11T22:21:59.927762+0000","last_clean":"2024-03-11T22:21:59.927762+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:21:59.927762+0000","last_undegraded":"2024-03-11T22:21:59.927762+0000","last_fullsized":"2024-03-11T22:21:59.927762+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:18:27.077275+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.077275+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":773,"num_read_kb":768,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":585,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:58.650906+0000","last_change":"2024-03-11T22:21:58.650945+0000","last_active":"2024-03-11T22:21:58.650906+0000","last_peered":"2024-03-11T22:21:58.650906+0000","last_clean":"2024-03-11T22:21:58.650906+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:21:58.650906+0000","last_undegraded":"2024-03-11T22:21:58.650906+0000","last_fullsized":"2024-03-11T22:21:58.650906+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:58.650607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:21:58.650607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:28.349205+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":581,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602485+0000","last_change":"2024-03-11T22:18:36.127513+0000","last_active":"2024-03-11T22:21:51.602485+0000","last_peered":"2024-03-11T22:21:51.602485+0000","last_clean":"2024-03-11T22:21:51.602485+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:21:51.602485+0000","last_undegraded":"2024-03-11T22:21:51.602485+0000","last_fullsized":"2024-03-11T22:21:51.602485+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:36.127435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:36.127435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":883,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.603033+0000","last_change":"2024-03-11T22:19:29.435468+0000","last_active":"2024-03-11T22:21:51.603033+0000","last_peered":"2024-03-11T22:21:51.603033+0000","last_clean":"2024-03-11T22:21:51.603033+0000","last_became_active":"2024-03-11T22:14:38.164101+0000","last_became_peered":"2024-03-11T22:14:38.164101+0000","last_unstale":"2024-03-11T22:21:51.603033+0000","last_undegraded":"2024-03-11T22:21:51.603033+0000","last_fullsized":"2024-03-11T22:21:51.603033+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:29.435334+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.435334+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"332'13966","reported_seq":14095,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.457102+0000","last_change":"2024-03-11T22:18:41.240089+0000","last_active":"2024-03-11T22:22:01.457102+0000","last_peered":"2024-03-11T22:22:01.457102+0000","last_clean":"2024-03-11T22:22:01.457102+0000","last_became_active":"2024-03-11T22:18:41.239829+0000","last_became_peered":"2024-03-11T22:18:41.239829+0000","last_unstale":"2024-03-11T22:22:01.457102+0000","last_undegraded":"2024-03-11T22:22:01.457102+0000","last_fullsized":"2024-03-11T22:22:01.457102+0000","mapping_epoch":273,"log_start":"316'9900","ondisk_log_start":"316'9900","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4066,"log_dups_size":0,"ondisk_log_size":4066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106037248,"num_objects":1618,"num_object_clones":0,"num_object_copies":4854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13966,"num_write_kb":498688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":543,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602368+0000","last_change":"2024-03-11T22:20:29.420829+0000","last_active":"2024-03-11T22:21:51.602368+0000","last_peered":"2024-03-11T22:21:51.602368+0000","last_clean":"2024-03-11T22:21:51.602368+0000","last_became_active":"2024-03-11T22:20:29.420497+0000","last_became_peered":"2024-03-11T22:20:29.420497+0000","last_unstale":"2024-03-11T22:21:51.602368+0000","last_undegraded":"2024-03-11T22:21:51.602368+0000","last_fullsized":"2024-03-11T22:21:51.602368+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:24.875284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:19:24.875284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"273'28","reported_seq":800,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.629086+0000","last_change":"2024-03-11T22:22:01.629086+0000","last_active":"2024-03-11T22:22:01.629086+0000","last_peered":"2024-03-11T22:22:01.629086+0000","last_clean":"2024-03-11T22:22:01.629086+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:22:01.629086+0000","last_undegraded":"2024-03-11T22:22:01.629086+0000","last_fullsized":"2024-03-11T22:22:01.629086+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:22:01.628978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:22:01.628978+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:18.259196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":577,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.635792+0000","last_change":"2024-03-11T22:19:24.533661+0000","last_active":"2024-03-11T22:21:51.635792+0000","last_peered":"2024-03-11T22:21:51.635792+0000","last_clean":"2024-03-11T22:21:51.635792+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:21:51.635792+0000","last_undegraded":"2024-03-11T22:21:51.635792+0000","last_fullsized":"2024-03-11T22:21:51.635792+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:24.533532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:19:24.533532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":892,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.604636+0000","last_change":"2024-03-11T22:20:32.294325+0000","last_active":"2024-03-11T22:21:51.604636+0000","last_peered":"2024-03-11T22:21:51.604636+0000","last_clean":"2024-03-11T22:21:51.604636+0000","last_became_active":"2024-03-11T22:20:32.293740+0000","last_became_peered":"2024-03-11T22:20:32.293740+0000","last_unstale":"2024-03-11T22:21:51.604636+0000","last_undegraded":"2024-03-11T22:21:51.604636+0000","last_fullsized":"2024-03-11T22:21:51.604636+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:44.952136+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:18:44.952136+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":565,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.607965+0000","last_change":"2024-03-11T22:18:22.688031+0000","last_active":"2024-03-11T22:21:51.607965+0000","last_peered":"2024-03-11T22:21:51.607965+0000","last_clean":"2024-03-11T22:21:51.607965+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:21:51.607965+0000","last_undegraded":"2024-03-11T22:21:51.607965+0000","last_fullsized":"2024-03-11T22:21:51.607965+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:22.687922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.687922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"332'14366","reported_seq":14495,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:05.490548+0000","last_change":"2024-03-11T22:18:41.260551+0000","last_active":"2024-03-11T22:22:05.490548+0000","last_peered":"2024-03-11T22:22:05.490548+0000","last_clean":"2024-03-11T22:22:05.490548+0000","last_became_active":"2024-03-11T22:18:41.260185+0000","last_became_peered":"2024-03-11T22:18:41.260185+0000","last_unstale":"2024-03-11T22:22:05.490548+0000","last_undegraded":"2024-03-11T22:22:05.490548+0000","last_fullsized":"2024-03-11T22:22:05.490548+0000","mapping_epoch":273,"log_start":"314'9800","ondisk_log_start":"314'9800","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4566,"log_dups_size":0,"ondisk_log_size":4566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89128960,"num_objects":1360,"num_object_clones":0,"num_object_copies":4080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14366,"num_write_kb":503232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":593,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.603149+0000","last_change":"2024-03-11T22:19:37.032204+0000","last_active":"2024-03-11T22:21:51.603149+0000","last_peered":"2024-03-11T22:21:51.603149+0000","last_clean":"2024-03-11T22:21:51.603149+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:21:51.603149+0000","last_undegraded":"2024-03-11T22:21:51.603149+0000","last_fullsized":"2024-03-11T22:21:51.603149+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.032058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.032058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"273'17","reported_seq":622,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602722+0000","last_change":"2024-03-11T22:19:34.486422+0000","last_active":"2024-03-11T22:21:51.602722+0000","last_peered":"2024-03-11T22:21:51.602722+0000","last_clean":"2024-03-11T22:21:51.602722+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:21:51.602722+0000","last_undegraded":"2024-03-11T22:21:51.602722+0000","last_fullsized":"2024-03-11T22:21:51.602722+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'17","last_scrub_stamp":"2024-03-11T22:19:34.486239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.486239+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":618,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601490+0000","last_change":"2024-03-11T22:20:52.869557+0000","last_active":"2024-03-11T22:21:51.601490+0000","last_peered":"2024-03-11T22:21:51.601490+0000","last_clean":"2024-03-11T22:21:51.601490+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:21:51.601490+0000","last_undegraded":"2024-03-11T22:21:51.601490+0000","last_fullsized":"2024-03-11T22:21:51.601490+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:52.869444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:20:52.869444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:57.295297+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":593,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.603981+0000","last_change":"2024-03-11T22:19:46.587019+0000","last_active":"2024-03-11T22:21:51.603981+0000","last_peered":"2024-03-11T22:21:51.603981+0000","last_clean":"2024-03-11T22:21:51.603981+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:21:51.603981+0000","last_undegraded":"2024-03-11T22:21:51.603981+0000","last_fullsized":"2024-03-11T22:21:51.603981+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:19:46.586446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":523,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602617+0000","last_change":"2024-03-11T22:19:37.776868+0000","last_active":"2024-03-11T22:21:51.602617+0000","last_peered":"2024-03-11T22:21:51.602617+0000","last_clean":"2024-03-11T22:21:51.602617+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:21:51.602617+0000","last_undegraded":"2024-03-11T22:21:51.602617+0000","last_fullsized":"2024-03-11T22:21:51.602617+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.776600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.776600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":900,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.604747+0000","last_change":"2024-03-11T22:20:32.295561+0000","last_active":"2024-03-11T22:21:51.604747+0000","last_peered":"2024-03-11T22:21:51.604747+0000","last_clean":"2024-03-11T22:21:51.604747+0000","last_became_active":"2024-03-11T22:20:32.295175+0000","last_became_peered":"2024-03-11T22:20:32.295175+0000","last_unstale":"2024-03-11T22:21:51.604747+0000","last_undegraded":"2024-03-11T22:21:51.604747+0000","last_fullsized":"2024-03-11T22:21:51.604747+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:25.392244+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:25.392244+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"332'13836","reported_seq":21998,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.903473+0000","last_change":"2024-03-11T22:21:36.588889+0000","last_active":"2024-03-11T22:22:01.903473+0000","last_peered":"2024-03-11T22:22:01.903473+0000","last_clean":"2024-03-11T22:22:01.903473+0000","last_became_active":"2024-03-11T22:21:36.509068+0000","last_became_peered":"2024-03-11T22:21:36.509068+0000","last_unstale":"2024-03-11T22:22:01.903473+0000","last_undegraded":"2024-03-11T22:22:01.903473+0000","last_fullsized":"2024-03-11T22:22:01.903473+0000","mapping_epoch":324,"log_start":"313'8500","ondisk_log_start":"313'8500","created":273,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5336,"log_dups_size":0,"ondisk_log_size":5336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":4938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13836,"num_write_kb":495424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4009,"num_bytes_recovered":262668288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":577,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:56.639870+0000","last_change":"2024-03-11T22:21:56.639907+0000","last_active":"2024-03-11T22:21:56.639870+0000","last_peered":"2024-03-11T22:21:56.639870+0000","last_clean":"2024-03-11T22:21:56.639870+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:21:56.639870+0000","last_undegraded":"2024-03-11T22:21:56.639870+0000","last_fullsized":"2024-03-11T22:21:56.639870+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:56.639673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:21:56.639673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:15.634286+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"273'40","reported_seq":817,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:53.519738+0000","last_change":"2024-03-11T22:19:39.417690+0000","last_active":"2024-03-11T22:21:53.519738+0000","last_peered":"2024-03-11T22:21:53.519738+0000","last_clean":"2024-03-11T22:21:53.519738+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:21:53.519738+0000","last_undegraded":"2024-03-11T22:21:53.519738+0000","last_fullsized":"2024-03-11T22:21:53.519738+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'40","last_scrub_stamp":"2024-03-11T22:19:39.417411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:39.417411+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":344,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:53.574023+0000","last_change":"2024-03-11T22:21:53.574059+0000","last_active":"2024-03-11T22:21:53.574023+0000","last_peered":"2024-03-11T22:21:53.574023+0000","last_clean":"2024-03-11T22:21:53.574023+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:21:53.574023+0000","last_undegraded":"2024-03-11T22:21:53.574023+0000","last_fullsized":"2024-03-11T22:21:53.574023+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:53.573950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:21:53.573950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:13.389779+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":589,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.608285+0000","last_change":"2024-03-11T22:18:36.684061+0000","last_active":"2024-03-11T22:21:51.608285+0000","last_peered":"2024-03-11T22:21:51.608285+0000","last_clean":"2024-03-11T22:21:51.608285+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:21:51.608285+0000","last_undegraded":"2024-03-11T22:21:51.608285+0000","last_fullsized":"2024-03-11T22:21:51.608285+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:18:36.683930+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:18:36.683930+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":569,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.637251+0000","last_change":"2024-03-11T22:20:29.434967+0000","last_active":"2024-03-11T22:21:51.637251+0000","last_peered":"2024-03-11T22:21:51.637251+0000","last_clean":"2024-03-11T22:21:51.637251+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:21:51.637251+0000","last_undegraded":"2024-03-11T22:21:51.637251+0000","last_fullsized":"2024-03-11T22:21:51.637251+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:32.518054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:15:32.518054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":874,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.608162+0000","last_change":"2024-03-11T22:18:41.790977+0000","last_active":"2024-03-11T22:21:51.608162+0000","last_peered":"2024-03-11T22:21:51.608162+0000","last_clean":"2024-03-11T22:21:51.608162+0000","last_became_active":"2024-03-11T22:14:38.164859+0000","last_became_peered":"2024-03-11T22:14:38.164859+0000","last_unstale":"2024-03-11T22:21:51.608162+0000","last_undegraded":"2024-03-11T22:21:51.608162+0000","last_fullsized":"2024-03-11T22:21:51.608162+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:41.790896+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:41.790896+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"332'13745","reported_seq":13874,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:05.500486+0000","last_change":"2024-03-11T22:18:41.233217+0000","last_active":"2024-03-11T22:22:05.500486+0000","last_peered":"2024-03-11T22:22:05.500486+0000","last_clean":"2024-03-11T22:22:05.500486+0000","last_became_active":"2024-03-11T22:18:41.232989+0000","last_became_peered":"2024-03-11T22:18:41.232989+0000","last_unstale":"2024-03-11T22:22:05.500486+0000","last_undegraded":"2024-03-11T22:22:05.500486+0000","last_fullsized":"2024-03-11T22:22:05.500486+0000","mapping_epoch":273,"log_start":"314'9200","ondisk_log_start":"314'9200","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4545,"log_dups_size":0,"ondisk_log_size":4545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81199104,"num_objects":1239,"num_object_clones":0,"num_object_copies":3717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13745,"num_write_kb":479488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"273'15","reported_seq":613,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.605541+0000","last_change":"2024-03-11T22:19:32.486126+0000","last_active":"2024-03-11T22:21:51.605541+0000","last_peered":"2024-03-11T22:21:51.605541+0000","last_clean":"2024-03-11T22:21:51.605541+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:21:51.605541+0000","last_undegraded":"2024-03-11T22:21:51.605541+0000","last_fullsized":"2024-03-11T22:21:51.605541+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:19:32.485571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.485571+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":577,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.607628+0000","last_change":"2024-03-11T22:18:40.745227+0000","last_active":"2024-03-11T22:21:51.607628+0000","last_peered":"2024-03-11T22:21:51.607628+0000","last_clean":"2024-03-11T22:21:51.607628+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:21:51.607628+0000","last_undegraded":"2024-03-11T22:21:51.607628+0000","last_fullsized":"2024-03-11T22:21:51.607628+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.745151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.745151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":593,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601941+0000","last_change":"2024-03-11T22:19:42.927329+0000","last_active":"2024-03-11T22:21:51.601941+0000","last_peered":"2024-03-11T22:21:51.601941+0000","last_clean":"2024-03-11T22:21:51.601941+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:21:51.601941+0000","last_undegraded":"2024-03-11T22:21:51.601941+0000","last_fullsized":"2024-03-11T22:21:51.601941+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:42.926178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:19:42.926178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":577,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.605047+0000","last_change":"2024-03-11T22:20:29.418105+0000","last_active":"2024-03-11T22:21:51.605047+0000","last_peered":"2024-03-11T22:21:51.605047+0000","last_clean":"2024-03-11T22:21:51.605047+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:21:51.605047+0000","last_undegraded":"2024-03-11T22:21:51.605047+0000","last_fullsized":"2024-03-11T22:21:51.605047+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:27.721245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.721245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"93'130","reported_seq":727,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.636582+0000","last_change":"2024-03-11T22:20:32.293722+0000","last_active":"2024-03-11T22:21:51.636582+0000","last_peered":"2024-03-11T22:21:51.636582+0000","last_clean":"2024-03-11T22:21:51.636582+0000","last_became_active":"2024-03-11T22:20:32.292676+0000","last_became_peered":"2024-03-11T22:20:32.292676+0000","last_unstale":"2024-03-11T22:21:51.636582+0000","last_undegraded":"2024-03-11T22:21:51.636582+0000","last_fullsized":"2024-03-11T22:21:51.636582+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:22.939363+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.939363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.7","version":"332'14256","reported_seq":27684,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:05.478195+0000","last_change":"2024-03-11T22:20:55.630922+0000","last_active":"2024-03-11T22:22:05.478195+0000","last_peered":"2024-03-11T22:22:05.478195+0000","last_clean":"2024-03-11T22:22:05.478195+0000","last_became_active":"2024-03-11T22:20:55.617058+0000","last_became_peered":"2024-03-11T22:20:55.617058+0000","last_unstale":"2024-03-11T22:22:05.478195+0000","last_undegraded":"2024-03-11T22:22:05.478195+0000","last_fullsized":"2024-03-11T22:22:05.478195+0000","mapping_epoch":315,"log_start":"314'9700","ondisk_log_start":"314'9700","created":273,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88604672,"num_objects":1352,"num_object_clones":0,"num_object_copies":4056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14256,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6642,"num_bytes_recovered":435290112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"273'35","reported_seq":704,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:02.687100+0000","last_change":"2024-03-11T22:22:02.687141+0000","last_active":"2024-03-11T22:22:02.687100+0000","last_peered":"2024-03-11T22:22:02.687100+0000","last_clean":"2024-03-11T22:22:02.687100+0000","last_became_active":"2024-03-11T22:20:29.418041+0000","last_became_peered":"2024-03-11T22:20:29.418041+0000","last_unstale":"2024-03-11T22:22:02.687100+0000","last_undegraded":"2024-03-11T22:22:02.687100+0000","last_fullsized":"2024-03-11T22:22:02.687100+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'35","last_scrub_stamp":"2024-03-11T22:22:02.687037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:22:02.687037+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:30.382198+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":528,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.608575+0000","last_change":"2024-03-11T22:20:29.414029+0000","last_active":"2024-03-11T22:21:51.608575+0000","last_peered":"2024-03-11T22:21:51.608575+0000","last_clean":"2024-03-11T22:21:51.608575+0000","last_became_active":"2024-03-11T22:20:29.413664+0000","last_became_peered":"2024-03-11T22:20:29.413664+0000","last_unstale":"2024-03-11T22:21:51.608575+0000","last_undegraded":"2024-03-11T22:21:51.608575+0000","last_fullsized":"2024-03-11T22:21:51.608575+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:32.682157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:32.682157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":593,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601702+0000","last_change":"2024-03-11T22:20:39.805771+0000","last_active":"2024-03-11T22:21:51.601702+0000","last_peered":"2024-03-11T22:21:51.601702+0000","last_clean":"2024-03-11T22:21:51.601702+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:21:51.601702+0000","last_undegraded":"2024-03-11T22:21:51.601702+0000","last_fullsized":"2024-03-11T22:21:51.601702+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:39.805647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:20:39.805647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":551,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:53.139738+0000","last_change":"2024-03-11T22:21:53.139945+0000","last_active":"2024-03-11T22:21:53.139738+0000","last_peered":"2024-03-11T22:21:53.139738+0000","last_clean":"2024-03-11T22:21:53.139738+0000","last_became_active":"2024-03-11T22:20:29.415565+0000","last_became_peered":"2024-03-11T22:20:29.415565+0000","last_unstale":"2024-03-11T22:21:53.139738+0000","last_undegraded":"2024-03-11T22:21:53.139738+0000","last_fullsized":"2024-03-11T22:21:53.139738+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:53.139082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:20.931122+0000","last_clean_scrub_stamp":"2024-03-11T22:21:53.139082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:07.326264+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"93'130","reported_seq":778,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602826+0000","last_change":"2024-03-11T22:20:36.022151+0000","last_active":"2024-03-11T22:21:51.602826+0000","last_peered":"2024-03-11T22:21:51.602826+0000","last_clean":"2024-03-11T22:21:51.602826+0000","last_became_active":"2024-03-11T22:14:43.195433+0000","last_became_peered":"2024-03-11T22:14:43.195433+0000","last_unstale":"2024-03-11T22:21:51.602826+0000","last_undegraded":"2024-03-11T22:21:51.602826+0000","last_fullsized":"2024-03-11T22:21:51.602826+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:36.022039+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:20:36.022039+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"332'13796","reported_seq":13925,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.360165+0000","last_change":"2024-03-11T22:18:41.237962+0000","last_active":"2024-03-11T22:22:01.360165+0000","last_peered":"2024-03-11T22:22:01.360165+0000","last_clean":"2024-03-11T22:22:01.360165+0000","last_became_active":"2024-03-11T22:18:41.237565+0000","last_became_peered":"2024-03-11T22:18:41.237565+0000","last_unstale":"2024-03-11T22:22:01.360165+0000","last_undegraded":"2024-03-11T22:22:01.360165+0000","last_fullsized":"2024-03-11T22:22:01.360165+0000","mapping_epoch":273,"log_start":"314'9200","ondisk_log_start":"314'9200","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4596,"log_dups_size":0,"ondisk_log_size":4596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106823680,"num_objects":1630,"num_object_clones":0,"num_object_copies":4890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13796,"num_write_kb":493632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":593,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601766+0000","last_change":"2024-03-11T22:20:47.845641+0000","last_active":"2024-03-11T22:21:51.601766+0000","last_peered":"2024-03-11T22:21:51.601766+0000","last_clean":"2024-03-11T22:21:51.601766+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:21:51.601766+0000","last_undegraded":"2024-03-11T22:21:51.601766+0000","last_fullsized":"2024-03-11T22:21:51.601766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:47.845156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:20:47.845156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"273'18","reported_seq":401,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.608532+0000","last_change":"2024-03-11T22:19:38.795869+0000","last_active":"2024-03-11T22:21:51.608532+0000","last_peered":"2024-03-11T22:21:51.608532+0000","last_clean":"2024-03-11T22:21:51.608532+0000","last_became_active":"2024-03-11T22:14:52.247400+0000","last_became_peered":"2024-03-11T22:14:52.247400+0000","last_unstale":"2024-03-11T22:21:51.608532+0000","last_undegraded":"2024-03-11T22:21:51.608532+0000","last_fullsized":"2024-03-11T22:21:51.608532+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:19:38.795650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.795650+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":367,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.603457+0000","last_change":"2024-03-11T22:20:29.417391+0000","last_active":"2024-03-11T22:21:51.603457+0000","last_peered":"2024-03-11T22:21:51.603457+0000","last_clean":"2024-03-11T22:21:51.603457+0000","last_became_active":"2024-03-11T22:20:29.416729+0000","last_became_peered":"2024-03-11T22:20:29.416729+0000","last_unstale":"2024-03-11T22:21:51.603457+0000","last_undegraded":"2024-03-11T22:21:51.603457+0000","last_fullsized":"2024-03-11T22:21:51.603457+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.439756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.439756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":578,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.636332+0000","last_change":"2024-03-11T22:20:29.470787+0000","last_active":"2024-03-11T22:21:51.636332+0000","last_peered":"2024-03-11T22:21:51.636332+0000","last_clean":"2024-03-11T22:21:51.636332+0000","last_became_active":"2024-03-11T22:20:29.438413+0000","last_became_peered":"2024-03-11T22:20:29.438413+0000","last_unstale":"2024-03-11T22:21:51.636332+0000","last_undegraded":"2024-03-11T22:21:51.636332+0000","last_fullsized":"2024-03-11T22:21:51.636332+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:15:40.678629+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:15:40.678629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"93'130","reported_seq":903,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.635800+0000","last_change":"2024-03-11T22:18:21.956432+0000","last_active":"2024-03-11T22:21:51.635800+0000","last_peered":"2024-03-11T22:21:51.635800+0000","last_clean":"2024-03-11T22:21:51.635800+0000","last_became_active":"2024-03-11T22:17:57.246241+0000","last_became_peered":"2024-03-11T22:17:57.246241+0000","last_unstale":"2024-03-11T22:21:51.635800+0000","last_undegraded":"2024-03-11T22:21:51.635800+0000","last_fullsized":"2024-03-11T22:21:51.635800+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:21.956334+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.956334+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"332'14204","reported_seq":23227,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.903124+0000","last_change":"2024-03-11T22:21:51.633264+0000","last_active":"2024-03-11T22:22:01.903124+0000","last_peered":"2024-03-11T22:22:01.903124+0000","last_clean":"2024-03-11T22:22:01.903124+0000","last_became_active":"2024-03-11T22:21:51.609572+0000","last_became_peered":"2024-03-11T22:21:51.609572+0000","last_unstale":"2024-03-11T22:22:01.903124+0000","last_undegraded":"2024-03-11T22:22:01.903124+0000","last_fullsized":"2024-03-11T22:22:01.903124+0000","mapping_epoch":331,"log_start":"313'8900","ondisk_log_start":"313'8900","created":273,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109838336,"num_objects":1676,"num_object_clones":0,"num_object_copies":5028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14204,"num_write_kb":508160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4437,"num_bytes_recovered":290783232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"310'11","reported_seq":716,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:58.510918+0000","last_change":"2024-03-11T22:20:57.698062+0000","last_active":"2024-03-11T22:21:58.510918+0000","last_peered":"2024-03-11T22:21:58.510918+0000","last_clean":"2024-03-11T22:21:58.510918+0000","last_became_active":"2024-03-11T22:20:29.418355+0000","last_became_peered":"2024-03-11T22:20:29.418355+0000","last_unstale":"2024-03-11T22:21:58.510918+0000","last_undegraded":"2024-03-11T22:21:58.510918+0000","last_fullsized":"2024-03-11T22:21:58.510918+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:20:57.697970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:57.697970+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:27.002078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"273'34","reported_seq":675,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.607260+0000","last_change":"2024-03-11T22:20:29.431072+0000","last_active":"2024-03-11T22:21:51.607260+0000","last_peered":"2024-03-11T22:21:51.607260+0000","last_clean":"2024-03-11T22:21:51.607260+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:21:51.607260+0000","last_undegraded":"2024-03-11T22:21:51.607260+0000","last_fullsized":"2024-03-11T22:21:51.607260+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:19:46.888741+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:19:46.888741+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":377,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.605935+0000","last_change":"2024-03-11T22:20:29.440395+0000","last_active":"2024-03-11T22:21:51.605935+0000","last_peered":"2024-03-11T22:21:51.605935+0000","last_clean":"2024-03-11T22:21:51.605935+0000","last_became_active":"2024-03-11T22:20:29.415226+0000","last_became_peered":"2024-03-11T22:20:29.415226+0000","last_unstale":"2024-03-11T22:21:51.605935+0000","last_undegraded":"2024-03-11T22:21:51.605935+0000","last_fullsized":"2024-03-11T22:21:51.605935+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:20:26.335219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:26.335219+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":565,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.608640+0000","last_change":"2024-03-11T22:19:33.760079+0000","last_active":"2024-03-11T22:21:51.608640+0000","last_peered":"2024-03-11T22:21:51.608640+0000","last_clean":"2024-03-11T22:21:51.608640+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:21:51.608640+0000","last_undegraded":"2024-03-11T22:21:51.608640+0000","last_fullsized":"2024-03-11T22:21:51.608640+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:33.759955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:33.759955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"93'130","reported_seq":920,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602875+0000","last_change":"2024-03-11T22:20:32.300771+0000","last_active":"2024-03-11T22:21:51.602875+0000","last_peered":"2024-03-11T22:21:51.602875+0000","last_clean":"2024-03-11T22:21:51.602875+0000","last_became_active":"2024-03-11T22:20:32.295606+0000","last_became_peered":"2024-03-11T22:20:32.295606+0000","last_unstale":"2024-03-11T22:21:51.602875+0000","last_undegraded":"2024-03-11T22:21:51.602875+0000","last_fullsized":"2024-03-11T22:21:51.602875+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:41.158328+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:41.158328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"273'12","reported_seq":610,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.637660+0000","last_change":"2024-03-11T22:20:29.483060+0000","last_active":"2024-03-11T22:21:51.637660+0000","last_peered":"2024-03-11T22:21:51.637660+0000","last_clean":"2024-03-11T22:21:51.637660+0000","last_became_active":"2024-03-11T22:20:29.441599+0000","last_became_peered":"2024-03-11T22:20:29.441599+0000","last_unstale":"2024-03-11T22:21:51.637660+0000","last_undegraded":"2024-03-11T22:21:51.637660+0000","last_fullsized":"2024-03-11T22:21:51.637660+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:15:27.448002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:27.448002+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":592,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601834+0000","last_change":"2024-03-11T22:20:42.845142+0000","last_active":"2024-03-11T22:21:51.601834+0000","last_peered":"2024-03-11T22:21:51.601834+0000","last_clean":"2024-03-11T22:21:51.601834+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:21:51.601834+0000","last_undegraded":"2024-03-11T22:21:51.601834+0000","last_fullsized":"2024-03-11T22:21:51.601834+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:42.844876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:20:42.844876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:08.076350+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":576,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.607996+0000","last_change":"2024-03-11T22:20:54.635534+0000","last_active":"2024-03-11T22:21:51.607996+0000","last_peered":"2024-03-11T22:21:51.607996+0000","last_clean":"2024-03-11T22:21:51.607996+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:21:51.607996+0000","last_undegraded":"2024-03-11T22:21:51.607996+0000","last_fullsized":"2024-03-11T22:21:51.607996+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:20:54.635309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:15.693951+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":522,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602708+0000","last_change":"2024-03-11T22:20:32.993441+0000","last_active":"2024-03-11T22:21:51.602708+0000","last_peered":"2024-03-11T22:21:51.602708+0000","last_clean":"2024-03-11T22:21:51.602708+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:21:51.602708+0000","last_undegraded":"2024-03-11T22:21:51.602708+0000","last_fullsized":"2024-03-11T22:21:51.602708+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:32.993301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:20:32.993301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:01.295107+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"93'130","reported_seq":883,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.607909+0000","last_change":"2024-03-11T22:18:45.781426+0000","last_active":"2024-03-11T22:21:51.607909+0000","last_peered":"2024-03-11T22:21:51.607909+0000","last_clean":"2024-03-11T22:21:51.607909+0000","last_became_active":"2024-03-11T22:14:43.194413+0000","last_became_peered":"2024-03-11T22:14:43.194413+0000","last_unstale":"2024-03-11T22:21:51.607909+0000","last_undegraded":"2024-03-11T22:21:51.607909+0000","last_fullsized":"2024-03-11T22:21:51.607909+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:45.781296+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:45.781296+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"93'130","reported_seq":890,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602590+0000","last_change":"2024-03-11T22:19:27.413677+0000","last_active":"2024-03-11T22:21:51.602590+0000","last_peered":"2024-03-11T22:21:51.602590+0000","last_clean":"2024-03-11T22:21:51.602590+0000","last_became_active":"2024-03-11T22:14:43.198216+0000","last_became_peered":"2024-03-11T22:14:43.198216+0000","last_unstale":"2024-03-11T22:21:51.602590+0000","last_undegraded":"2024-03-11T22:21:51.602590+0000","last_fullsized":"2024-03-11T22:21:51.602590+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:27.413560+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:27.413560+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"273'19","reported_seq":633,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.605602+0000","last_change":"2024-03-11T22:20:29.451236+0000","last_active":"2024-03-11T22:21:51.605602+0000","last_peered":"2024-03-11T22:21:51.605602+0000","last_clean":"2024-03-11T22:21:51.605602+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:21:51.605602+0000","last_undegraded":"2024-03-11T22:21:51.605602+0000","last_fullsized":"2024-03-11T22:21:51.605602+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'13","last_scrub_stamp":"2024-03-11T22:18:22.657089+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.657089+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":355,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601997+0000","last_change":"2024-03-11T22:20:46.807212+0000","last_active":"2024-03-11T22:21:51.601997+0000","last_peered":"2024-03-11T22:21:51.601997+0000","last_clean":"2024-03-11T22:21:51.601997+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:21:51.601997+0000","last_undegraded":"2024-03-11T22:21:51.601997+0000","last_fullsized":"2024-03-11T22:21:51.601997+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:46.807096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:46.807096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":580,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.637629+0000","last_change":"2024-03-11T22:20:29.433304+0000","last_active":"2024-03-11T22:21:51.637629+0000","last_peered":"2024-03-11T22:21:51.637629+0000","last_clean":"2024-03-11T22:21:51.637629+0000","last_became_active":"2024-03-11T22:20:29.433005+0000","last_became_peered":"2024-03-11T22:20:29.433005+0000","last_unstale":"2024-03-11T22:21:51.637629+0000","last_undegraded":"2024-03-11T22:21:51.637629+0000","last_fullsized":"2024-03-11T22:21:51.637629+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:43.544959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:43.544959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":596,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.193632+0000","last_change":"2024-03-11T22:22:01.193655+0000","last_active":"2024-03-11T22:22:01.193632+0000","last_peered":"2024-03-11T22:22:01.193632+0000","last_clean":"2024-03-11T22:22:01.193632+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:22:01.193632+0000","last_undegraded":"2024-03-11T22:22:01.193632+0000","last_fullsized":"2024-03-11T22:22:01.193632+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:01.193579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:22:01.193579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:10.981326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"273'19","reported_seq":609,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.608632+0000","last_change":"2024-03-11T22:19:29.865779+0000","last_active":"2024-03-11T22:21:51.608632+0000","last_peered":"2024-03-11T22:21:51.608632+0000","last_clean":"2024-03-11T22:21:51.608632+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:21:51.608632+0000","last_undegraded":"2024-03-11T22:21:51.608632+0000","last_fullsized":"2024-03-11T22:21:51.608632+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:19:29.865528+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.865528+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":588,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.604576+0000","last_change":"2024-03-11T22:20:29.417404+0000","last_active":"2024-03-11T22:21:51.604576+0000","last_peered":"2024-03-11T22:21:51.604576+0000","last_clean":"2024-03-11T22:21:51.604576+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:21:51.604576+0000","last_undegraded":"2024-03-11T22:21:51.604576+0000","last_fullsized":"2024-03-11T22:21:51.604576+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:21.693554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.693554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'11","reported_seq":773,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:53.519939+0000","last_change":"2024-03-11T22:20:29.445996+0000","last_active":"2024-03-11T22:21:53.519939+0000","last_peered":"2024-03-11T22:21:53.519939+0000","last_clean":"2024-03-11T22:21:53.519939+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:21:53.519939+0000","last_undegraded":"2024-03-11T22:21:53.519939+0000","last_fullsized":"2024-03-11T22:21:53.519939+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'10","last_scrub_stamp":"2024-03-11T22:19:32.461683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.461683+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":358,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601723+0000","last_change":"2024-03-11T22:20:58.956323+0000","last_active":"2024-03-11T22:21:51.601723+0000","last_peered":"2024-03-11T22:21:51.601723+0000","last_clean":"2024-03-11T22:21:51.601723+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:21:51.601723+0000","last_undegraded":"2024-03-11T22:21:51.601723+0000","last_fullsized":"2024-03-11T22:21:51.601723+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:58.956196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:20:58.956196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:23.295311+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"93'130","reported_seq":876,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.607609+0000","last_change":"2024-03-11T22:18:53.739248+0000","last_active":"2024-03-11T22:21:51.607609+0000","last_peered":"2024-03-11T22:21:51.607609+0000","last_clean":"2024-03-11T22:21:51.607609+0000","last_became_active":"2024-03-11T22:14:43.194350+0000","last_became_peered":"2024-03-11T22:14:43.194350+0000","last_unstale":"2024-03-11T22:21:51.607609+0000","last_undegraded":"2024-03-11T22:21:51.607609+0000","last_fullsized":"2024-03-11T22:21:51.607609+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:53.738316+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:18:53.738316+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"273'20","reported_seq":612,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.636848+0000","last_change":"2024-03-11T22:16:16.055622+0000","last_active":"2024-03-11T22:21:51.636848+0000","last_peered":"2024-03-11T22:21:51.636848+0000","last_clean":"2024-03-11T22:21:51.636848+0000","last_became_active":"2024-03-11T22:16:15.815814+0000","last_became_peered":"2024-03-11T22:16:15.815814+0000","last_unstale":"2024-03-11T22:21:51.636848+0000","last_undegraded":"2024-03-11T22:21:51.636848+0000","last_fullsized":"2024-03-11T22:21:51.636848+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:15:29.460297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.460297+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":375,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601671+0000","last_change":"2024-03-11T22:20:31.797304+0000","last_active":"2024-03-11T22:21:51.601671+0000","last_peered":"2024-03-11T22:21:51.601671+0000","last_clean":"2024-03-11T22:21:51.601671+0000","last_became_active":"2024-03-11T22:20:29.415019+0000","last_became_peered":"2024-03-11T22:20:29.415019+0000","last_unstale":"2024-03-11T22:21:51.601671+0000","last_undegraded":"2024-03-11T22:21:51.601671+0000","last_fullsized":"2024-03-11T22:21:51.601671+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:31.796712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:31.796712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":747,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:58.510184+0000","last_change":"2024-03-11T22:19:42.854028+0000","last_active":"2024-03-11T22:21:58.510184+0000","last_peered":"2024-03-11T22:21:58.510184+0000","last_clean":"2024-03-11T22:21:58.510184+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:21:58.510184+0000","last_undegraded":"2024-03-11T22:21:58.510184+0000","last_fullsized":"2024-03-11T22:21:58.510184+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:19:42.853428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:42.853428+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":576,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:22:01.690907+0000","last_change":"2024-03-11T22:22:01.690944+0000","last_active":"2024-03-11T22:22:01.690907+0000","last_peered":"2024-03-11T22:22:01.690907+0000","last_clean":"2024-03-11T22:22:01.690907+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:22:01.690907+0000","last_undegraded":"2024-03-11T22:22:01.690907+0000","last_fullsized":"2024-03-11T22:22:01.690907+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:01.690694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:01.690694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:27.427919+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"93'130","reported_seq":832,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.604875+0000","last_change":"2024-03-11T22:19:29.435235+0000","last_active":"2024-03-11T22:21:51.604875+0000","last_peered":"2024-03-11T22:21:51.604875+0000","last_clean":"2024-03-11T22:21:51.604875+0000","last_became_active":"2024-03-11T22:14:43.191297+0000","last_became_peered":"2024-03-11T22:14:43.191297+0000","last_unstale":"2024-03-11T22:21:51.604875+0000","last_undegraded":"2024-03-11T22:21:51.604875+0000","last_fullsized":"2024-03-11T22:21:51.604875+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:29.434647+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.434647+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"273'19","reported_seq":401,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602748+0000","last_change":"2024-03-11T22:20:50.729986+0000","last_active":"2024-03-11T22:21:51.602748+0000","last_peered":"2024-03-11T22:21:51.602748+0000","last_clean":"2024-03-11T22:21:51.602748+0000","last_became_active":"2024-03-11T22:14:52.239810+0000","last_became_peered":"2024-03-11T22:14:52.239810+0000","last_unstale":"2024-03-11T22:21:51.602748+0000","last_undegraded":"2024-03-11T22:21:51.602748+0000","last_fullsized":"2024-03-11T22:21:51.602748+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:20:50.729848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:50.729848+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:58.488489+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":380,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602579+0000","last_change":"2024-03-11T22:20:47.791284+0000","last_active":"2024-03-11T22:21:51.602579+0000","last_peered":"2024-03-11T22:21:51.602579+0000","last_clean":"2024-03-11T22:21:51.602579+0000","last_became_active":"2024-03-11T22:20:29.420867+0000","last_became_peered":"2024-03-11T22:20:29.420867+0000","last_unstale":"2024-03-11T22:21:51.602579+0000","last_undegraded":"2024-03-11T22:21:51.602579+0000","last_fullsized":"2024-03-11T22:21:51.602579+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:20:47.791128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:47.791128+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:56.876944+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":584,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.817342+0000","last_change":"2024-03-11T22:21:51.817381+0000","last_active":"2024-03-11T22:21:51.817342+0000","last_peered":"2024-03-11T22:21:51.817342+0000","last_clean":"2024-03-11T22:21:51.817342+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:21:51.817342+0000","last_undegraded":"2024-03-11T22:21:51.817342+0000","last_fullsized":"2024-03-11T22:21:51.817342+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:51.817265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:21:51.817265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:08.345413+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":556,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.635883+0000","last_change":"2024-03-11T22:18:29.014246+0000","last_active":"2024-03-11T22:21:51.635883+0000","last_peered":"2024-03-11T22:21:51.635883+0000","last_clean":"2024-03-11T22:21:51.635883+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:21:51.635883+0000","last_undegraded":"2024-03-11T22:21:51.635883+0000","last_fullsized":"2024-03-11T22:21:51.635883+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:29.014149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:29.014149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":617,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:58.841528+0000","last_change":"2024-03-11T22:21:58.841578+0000","last_active":"2024-03-11T22:21:58.841528+0000","last_peered":"2024-03-11T22:21:58.841528+0000","last_clean":"2024-03-11T22:21:58.841528+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:21:58.841528+0000","last_undegraded":"2024-03-11T22:21:58.841528+0000","last_fullsized":"2024-03-11T22:21:58.841528+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:21:58.841451+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:21:58.841451+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:59.100999+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":519,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.636474+0000","last_change":"2024-03-11T22:20:29.423108+0000","last_active":"2024-03-11T22:21:51.636474+0000","last_peered":"2024-03-11T22:21:51.636474+0000","last_clean":"2024-03-11T22:21:51.636474+0000","last_became_active":"2024-03-11T22:20:29.422747+0000","last_became_peered":"2024-03-11T22:20:29.422747+0000","last_unstale":"2024-03-11T22:21:51.636474+0000","last_undegraded":"2024-03-11T22:21:51.636474+0000","last_fullsized":"2024-03-11T22:21:51.636474+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:44.563614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:44.563614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":568,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.636091+0000","last_change":"2024-03-11T22:19:39.445467+0000","last_active":"2024-03-11T22:21:51.636091+0000","last_peered":"2024-03-11T22:21:51.636091+0000","last_clean":"2024-03-11T22:21:51.636091+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:21:51.636091+0000","last_undegraded":"2024-03-11T22:21:51.636091+0000","last_fullsized":"2024-03-11T22:21:51.636091+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:39.445349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:19:39.445349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":564,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.604336+0000","last_change":"2024-03-11T22:19:26.459086+0000","last_active":"2024-03-11T22:21:51.604336+0000","last_peered":"2024-03-11T22:21:51.604336+0000","last_clean":"2024-03-11T22:21:51.604336+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:21:51.604336+0000","last_undegraded":"2024-03-11T22:21:51.604336+0000","last_fullsized":"2024-03-11T22:21:51.604336+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:26.458937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:26.458937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"93'130","reported_seq":893,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601717+0000","last_change":"2024-03-11T22:20:49.901428+0000","last_active":"2024-03-11T22:21:51.601717+0000","last_peered":"2024-03-11T22:21:51.601717+0000","last_clean":"2024-03-11T22:21:51.601717+0000","last_became_active":"2024-03-11T22:14:43.194955+0000","last_became_peered":"2024-03-11T22:14:43.194955+0000","last_unstale":"2024-03-11T22:21:51.601717+0000","last_undegraded":"2024-03-11T22:21:51.601717+0000","last_fullsized":"2024-03-11T22:21:51.601717+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:49.901270+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:18:43.722467+0000","last_clean_scrub_stamp":"2024-03-11T22:20:49.901270+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:57.047219+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"274'20","reported_seq":632,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.636077+0000","last_change":"2024-03-11T22:20:29.487125+0000","last_active":"2024-03-11T22:21:51.636077+0000","last_peered":"2024-03-11T22:21:51.636077+0000","last_clean":"2024-03-11T22:21:51.636077+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:21:51.636077+0000","last_undegraded":"2024-03-11T22:21:51.636077+0000","last_fullsized":"2024-03-11T22:21:51.636077+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'20","last_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:18:42.135888+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":596,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:56.391622+0000","last_change":"2024-03-11T22:21:56.391664+0000","last_active":"2024-03-11T22:21:56.391622+0000","last_peered":"2024-03-11T22:21:56.391622+0000","last_clean":"2024-03-11T22:21:56.391622+0000","last_became_active":"2024-03-11T22:20:29.416792+0000","last_became_peered":"2024-03-11T22:20:29.416792+0000","last_unstale":"2024-03-11T22:21:56.391622+0000","last_undegraded":"2024-03-11T22:21:56.391622+0000","last_fullsized":"2024-03-11T22:21:56.391622+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:56.391548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:21:56.391548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:14.928545+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":354,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601828+0000","last_change":"2024-03-11T22:20:40.885587+0000","last_active":"2024-03-11T22:21:51.601828+0000","last_peered":"2024-03-11T22:21:51.601828+0000","last_clean":"2024-03-11T22:21:51.601828+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:21:51.601828+0000","last_undegraded":"2024-03-11T22:21:51.601828+0000","last_fullsized":"2024-03-11T22:21:51.601828+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:40.885360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:20:40.885360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":564,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:55.069930+0000","last_change":"2024-03-11T22:21:55.069970+0000","last_active":"2024-03-11T22:21:55.069930+0000","last_peered":"2024-03-11T22:21:55.069930+0000","last_clean":"2024-03-11T22:21:55.069930+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:21:55.069930+0000","last_undegraded":"2024-03-11T22:21:55.069930+0000","last_fullsized":"2024-03-11T22:21:55.069930+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:55.069595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:21:55.069595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:56.389048+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"273'8","reported_seq":596,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.635665+0000","last_change":"2024-03-11T22:18:53.157176+0000","last_active":"2024-03-11T22:21:51.635665+0000","last_peered":"2024-03-11T22:21:51.635665+0000","last_clean":"2024-03-11T22:21:51.635665+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:21:51.635665+0000","last_undegraded":"2024-03-11T22:21:51.635665+0000","last_fullsized":"2024-03-11T22:21:51.635665+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'8","last_scrub_stamp":"2024-03-11T22:18:53.157080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:53.157080+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":576,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.605938+0000","last_change":"2024-03-11T22:19:45.501704+0000","last_active":"2024-03-11T22:21:51.605938+0000","last_peered":"2024-03-11T22:21:51.605938+0000","last_clean":"2024-03-11T22:21:51.605938+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:21:51.605938+0000","last_undegraded":"2024-03-11T22:21:51.605938+0000","last_fullsized":"2024-03-11T22:21:51.605938+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.501593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.501593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":162,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601845+0000","last_change":"2024-03-11T22:20:30.047479+0000","last_active":"2024-03-11T22:21:51.601845+0000","last_peered":"2024-03-11T22:21:51.601845+0000","last_clean":"2024-03-11T22:21:51.601845+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:21:51.601845+0000","last_undegraded":"2024-03-11T22:21:51.601845+0000","last_fullsized":"2024-03-11T22:21:51.601845+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:30.047368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:30.047368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":530,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602157+0000","last_change":"2024-03-11T22:19:32.989760+0000","last_active":"2024-03-11T22:21:51.602157+0000","last_peered":"2024-03-11T22:21:51.602157+0000","last_clean":"2024-03-11T22:21:51.602157+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:21:51.602157+0000","last_undegraded":"2024-03-11T22:21:51.602157+0000","last_fullsized":"2024-03-11T22:21:51.602157+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:32.989348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.989348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"273'19","reported_seq":620,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602995+0000","last_change":"2024-03-11T22:20:51.762478+0000","last_active":"2024-03-11T22:21:51.602995+0000","last_peered":"2024-03-11T22:21:51.602995+0000","last_clean":"2024-03-11T22:21:51.602995+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:21:51.602995+0000","last_undegraded":"2024-03-11T22:21:51.602995+0000","last_fullsized":"2024-03-11T22:21:51.602995+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:20:51.762318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:51.762318+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:54.676538+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":576,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.635572+0000","last_change":"2024-03-11T22:19:25.669161+0000","last_active":"2024-03-11T22:21:51.635572+0000","last_peered":"2024-03-11T22:21:51.635572+0000","last_clean":"2024-03-11T22:21:51.635572+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:21:51.635572+0000","last_undegraded":"2024-03-11T22:21:51.635572+0000","last_fullsized":"2024-03-11T22:21:51.635572+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:25.668878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:25.668878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":577,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.605728+0000","last_change":"2024-03-11T22:20:27.333574+0000","last_active":"2024-03-11T22:21:51.605728+0000","last_peered":"2024-03-11T22:21:51.605728+0000","last_clean":"2024-03-11T22:21:51.605728+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:21:51.605728+0000","last_undegraded":"2024-03-11T22:21:51.605728+0000","last_fullsized":"2024-03-11T22:21:51.605728+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:27.333450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:27.333450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:54.425323+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":580,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.603411+0000","last_change":"2024-03-11T22:19:45.686859+0000","last_active":"2024-03-11T22:21:51.603411+0000","last_peered":"2024-03-11T22:21:51.603411+0000","last_clean":"2024-03-11T22:21:51.603411+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:21:51.603411+0000","last_undegraded":"2024-03-11T22:21:51.603411+0000","last_fullsized":"2024-03-11T22:21:51.603411+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.686738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.686738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"273'20","reported_seq":639,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.605589+0000","last_change":"2024-03-11T22:20:29.460876+0000","last_active":"2024-03-11T22:21:51.605589+0000","last_peered":"2024-03-11T22:21:51.605589+0000","last_clean":"2024-03-11T22:21:51.605589+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:21:51.605589+0000","last_undegraded":"2024-03-11T22:21:51.605589+0000","last_fullsized":"2024-03-11T22:21:51.605589+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:19:36.469333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.469333+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":586,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602121+0000","last_change":"2024-03-11T22:17:36.687785+0000","last_active":"2024-03-11T22:21:51.602121+0000","last_peered":"2024-03-11T22:21:51.602121+0000","last_clean":"2024-03-11T22:21:51.602121+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:21:51.602121+0000","last_undegraded":"2024-03-11T22:21:51.602121+0000","last_fullsized":"2024-03-11T22:21:51.602121+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:17:36.687660+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'11","reported_seq":789,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:58.510906+0000","last_change":"2024-03-11T22:20:54.741772+0000","last_active":"2024-03-11T22:21:58.510906+0000","last_peered":"2024-03-11T22:21:58.510906+0000","last_clean":"2024-03-11T22:21:58.510906+0000","last_became_active":"2024-03-11T22:20:29.415781+0000","last_became_peered":"2024-03-11T22:20:29.415781+0000","last_unstale":"2024-03-11T22:21:58.510906+0000","last_undegraded":"2024-03-11T22:21:58.510906+0000","last_fullsized":"2024-03-11T22:21:58.510906+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_deep_scrub":"315'11","last_deep_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_clean_scrub_stamp":"2024-03-11T22:20:54.741633+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:10.626413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":525,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.607210+0000","last_change":"2024-03-11T22:19:27.754842+0000","last_active":"2024-03-11T22:21:51.607210+0000","last_peered":"2024-03-11T22:21:51.607210+0000","last_clean":"2024-03-11T22:21:51.607210+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:21:51.607210+0000","last_undegraded":"2024-03-11T22:21:51.607210+0000","last_fullsized":"2024-03-11T22:21:51.607210+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:19:27.754721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:27.754721+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"273'18","reported_seq":604,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:52.914621+0000","last_change":"2024-03-11T22:21:52.914678+0000","last_active":"2024-03-11T22:21:52.914621+0000","last_peered":"2024-03-11T22:21:52.914621+0000","last_clean":"2024-03-11T22:21:52.914621+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:21:52.914621+0000","last_undegraded":"2024-03-11T22:21:52.914621+0000","last_fullsized":"2024-03-11T22:21:52.914621+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:21:52.914535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:21:52.914535+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:04.434254+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":351,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.603390+0000","last_change":"2024-03-11T22:19:36.471241+0000","last_active":"2024-03-11T22:21:51.603390+0000","last_peered":"2024-03-11T22:21:51.603390+0000","last_clean":"2024-03-11T22:21:51.603390+0000","last_became_active":"2024-03-11T22:14:52.245199+0000","last_became_peered":"2024-03-11T22:14:52.245199+0000","last_unstale":"2024-03-11T22:21:51.603390+0000","last_undegraded":"2024-03-11T22:21:51.603390+0000","last_fullsized":"2024-03-11T22:21:51.603390+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.471133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.471133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":334,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.637315+0000","last_change":"2024-03-11T22:21:49.873269+0000","last_active":"2024-03-11T22:21:51.637315+0000","last_peered":"2024-03-11T22:21:51.637315+0000","last_clean":"2024-03-11T22:21:51.637315+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:21:51.637315+0000","last_undegraded":"2024-03-11T22:21:51.637315+0000","last_fullsized":"2024-03-11T22:21:51.637315+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:49.872966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:21:49.872966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:07.291040+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":557,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.637385+0000","last_change":"2024-03-11T22:18:27.116987+0000","last_active":"2024-03-11T22:21:51.637385+0000","last_peered":"2024-03-11T22:21:51.637385+0000","last_clean":"2024-03-11T22:21:51.637385+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:21:51.637385+0000","last_undegraded":"2024-03-11T22:21:51.637385+0000","last_fullsized":"2024-03-11T22:21:51.637385+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:27.116858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.116858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"273'27","reported_seq":647,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601929+0000","last_change":"2024-03-11T22:20:55.875480+0000","last_active":"2024-03-11T22:21:51.601929+0000","last_peered":"2024-03-11T22:21:51.601929+0000","last_clean":"2024-03-11T22:21:51.601929+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:21:51.601929+0000","last_undegraded":"2024-03-11T22:21:51.601929+0000","last_fullsized":"2024-03-11T22:21:51.601929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:20:55.875312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:55.875312+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:21:57.302857+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":588,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.605241+0000","last_change":"2024-03-11T22:20:29.414748+0000","last_active":"2024-03-11T22:21:51.605241+0000","last_peered":"2024-03-11T22:21:51.605241+0000","last_clean":"2024-03-11T22:21:51.605241+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:21:51.605241+0000","last_undegraded":"2024-03-11T22:21:51.605241+0000","last_fullsized":"2024-03-11T22:21:51.605241+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:18:43.931797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":576,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.603239+0000","last_change":"2024-03-11T22:18:31.120211+0000","last_active":"2024-03-11T22:21:51.603239+0000","last_peered":"2024-03-11T22:21:51.603239+0000","last_clean":"2024-03-11T22:21:51.603239+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:21:51.603239+0000","last_undegraded":"2024-03-11T22:21:51.603239+0000","last_fullsized":"2024-03-11T22:21:51.603239+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:31.120131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:31.120131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":351,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602783+0000","last_change":"2024-03-11T22:20:45.819718+0000","last_active":"2024-03-11T22:21:51.602783+0000","last_peered":"2024-03-11T22:21:51.602783+0000","last_clean":"2024-03-11T22:21:51.602783+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:21:51.602783+0000","last_undegraded":"2024-03-11T22:21:51.602783+0000","last_fullsized":"2024-03-11T22:21:51.602783+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:45.819593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:20:45.819593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"273'9","reported_seq":411,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.635882+0000","last_change":"2024-03-11T22:18:51.098736+0000","last_active":"2024-03-11T22:21:51.635882+0000","last_peered":"2024-03-11T22:21:51.635882+0000","last_clean":"2024-03-11T22:21:51.635882+0000","last_became_active":"2024-03-11T22:14:52.238411+0000","last_became_peered":"2024-03-11T22:14:52.238411+0000","last_unstale":"2024-03-11T22:21:51.635882+0000","last_undegraded":"2024-03-11T22:21:51.635882+0000","last_fullsized":"2024-03-11T22:21:51.635882+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9","last_scrub_stamp":"2024-03-11T22:18:51.098646+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:18:51.098646+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":584,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:58.276703+0000","last_change":"2024-03-11T22:21:58.276736+0000","last_active":"2024-03-11T22:21:58.276703+0000","last_peered":"2024-03-11T22:21:58.276703+0000","last_clean":"2024-03-11T22:21:58.276703+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:21:58.276703+0000","last_undegraded":"2024-03-11T22:21:58.276703+0000","last_fullsized":"2024-03-11T22:21:58.276703+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:58.276626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:21:58.276626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:10.861978+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":345,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.603219+0000","last_change":"2024-03-11T22:17:27.798877+0000","last_active":"2024-03-11T22:21:51.603219+0000","last_peered":"2024-03-11T22:21:51.603219+0000","last_clean":"2024-03-11T22:21:51.603219+0000","last_became_active":"2024-03-11T22:14:52.239391+0000","last_became_peered":"2024-03-11T22:14:52.239391+0000","last_unstale":"2024-03-11T22:21:51.603219+0000","last_undegraded":"2024-03-11T22:21:51.603219+0000","last_fullsized":"2024-03-11T22:21:51.603219+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:17:27.798763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_clean_scrub_stamp":"2024-03-11T22:17:27.798763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":564,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.609050+0000","last_change":"2024-03-11T22:18:38.736070+0000","last_active":"2024-03-11T22:21:51.609050+0000","last_peered":"2024-03-11T22:21:51.609050+0000","last_clean":"2024-03-11T22:21:51.609050+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:21:51.609050+0000","last_undegraded":"2024-03-11T22:21:51.609050+0000","last_fullsized":"2024-03-11T22:21:51.609050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:38.735937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.735937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"273'15","reported_seq":378,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.635976+0000","last_change":"2024-03-11T22:15:18.401675+0000","last_active":"2024-03-11T22:21:51.635976+0000","last_peered":"2024-03-11T22:21:51.635976+0000","last_clean":"2024-03-11T22:21:51.635976+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:21:51.635976+0000","last_undegraded":"2024-03-11T22:21:51.635976+0000","last_fullsized":"2024-03-11T22:21:51.635976+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-11T22:15:18.401549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:18.401549+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":584,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.608187+0000","last_change":"2024-03-11T22:19:40.864768+0000","last_active":"2024-03-11T22:21:51.608187+0000","last_peered":"2024-03-11T22:21:51.608187+0000","last_clean":"2024-03-11T22:21:51.608187+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:21:51.608187+0000","last_undegraded":"2024-03-11T22:21:51.608187+0000","last_fullsized":"2024-03-11T22:21:51.608187+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:40.864689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:40.864689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":580,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.607188+0000","last_change":"2024-03-11T22:20:29.420120+0000","last_active":"2024-03-11T22:21:51.607188+0000","last_peered":"2024-03-11T22:21:51.607188+0000","last_clean":"2024-03-11T22:21:51.607188+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:21:51.607188+0000","last_undegraded":"2024-03-11T22:21:51.607188+0000","last_fullsized":"2024-03-11T22:21:51.607188+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:28.713390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:18:28.713390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":564,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.604867+0000","last_change":"2024-03-11T22:20:30.493247+0000","last_active":"2024-03-11T22:21:51.604867+0000","last_peered":"2024-03-11T22:21:51.604867+0000","last_clean":"2024-03-11T22:21:51.604867+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:21:51.604867+0000","last_undegraded":"2024-03-11T22:21:51.604867+0000","last_fullsized":"2024-03-11T22:21:51.604867+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:30.493146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:20:30.493146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"273'27","reported_seq":607,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.637408+0000","last_change":"2024-03-11T22:18:37.066095+0000","last_active":"2024-03-11T22:21:51.637408+0000","last_peered":"2024-03-11T22:21:51.637408+0000","last_clean":"2024-03-11T22:21:51.637408+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:21:51.637408+0000","last_undegraded":"2024-03-11T22:21:51.637408+0000","last_fullsized":"2024-03-11T22:21:51.637408+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:18:37.065988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.065988+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":583,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.636998+0000","last_change":"2024-03-11T22:18:38.095521+0000","last_active":"2024-03-11T22:21:51.636998+0000","last_peered":"2024-03-11T22:21:51.636998+0000","last_clean":"2024-03-11T22:21:51.636998+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:21:51.636998+0000","last_undegraded":"2024-03-11T22:21:51.636998+0000","last_fullsized":"2024-03-11T22:21:51.636998+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:18:38.095367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.095367+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":350,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601872+0000","last_change":"2024-03-11T22:20:26.125601+0000","last_active":"2024-03-11T22:21:51.601872+0000","last_peered":"2024-03-11T22:21:51.601872+0000","last_clean":"2024-03-11T22:21:51.601872+0000","last_became_active":"2024-03-11T22:14:52.247689+0000","last_became_peered":"2024-03-11T22:14:52.247689+0000","last_unstale":"2024-03-11T22:21:51.601872+0000","last_undegraded":"2024-03-11T22:21:51.601872+0000","last_fullsized":"2024-03-11T22:21:51.601872+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:26.125517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:26.125517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":568,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.636234+0000","last_change":"2024-03-11T22:20:29.436389+0000","last_active":"2024-03-11T22:21:51.636234+0000","last_peered":"2024-03-11T22:21:51.636234+0000","last_clean":"2024-03-11T22:21:51.636234+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:21:51.636234+0000","last_undegraded":"2024-03-11T22:21:51.636234+0000","last_fullsized":"2024-03-11T22:21:51.636234+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:24.023596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:24.023596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"273'19","reported_seq":398,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.602103+0000","last_change":"2024-03-11T22:21:50.537463+0000","last_active":"2024-03-11T22:21:51.602103+0000","last_peered":"2024-03-11T22:21:51.602103+0000","last_clean":"2024-03-11T22:21:51.602103+0000","last_became_active":"2024-03-11T22:14:52.247411+0000","last_became_peered":"2024-03-11T22:14:52.247411+0000","last_unstale":"2024-03-11T22:21:51.602103+0000","last_undegraded":"2024-03-11T22:21:51.602103+0000","last_fullsized":"2024-03-11T22:21:51.602103+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:21:50.537367+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:21:50.537367+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:16.692296+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":577,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.605005+0000","last_change":"2024-03-11T22:18:24.613444+0000","last_active":"2024-03-11T22:21:51.605005+0000","last_peered":"2024-03-11T22:21:51.605005+0000","last_clean":"2024-03-11T22:21:51.605005+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:21:51.605005+0000","last_undegraded":"2024-03-11T22:21:51.605005+0000","last_fullsized":"2024-03-11T22:21:51.605005+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:24.613372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:24.613372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":291,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.601909+0000","last_change":"2024-03-11T22:20:29.420332+0000","last_active":"2024-03-11T22:21:51.601909+0000","last_peered":"2024-03-11T22:21:51.601909+0000","last_clean":"2024-03-11T22:21:51.601909+0000","last_became_active":"2024-03-11T22:20:29.418227+0000","last_became_peered":"2024-03-11T22:20:29.418227+0000","last_unstale":"2024-03-11T22:21:51.601909+0000","last_undegraded":"2024-03-11T22:21:51.601909+0000","last_fullsized":"2024-03-11T22:21:51.601909+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.629853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.629853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":342,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T22:21:51.608370+0000","last_change":"2024-03-11T22:19:26.746519+0000","last_active":"2024-03-11T22:21:51.608370+0000","last_peered":"2024-03-11T22:21:51.608370+0000","last_clean":"2024-03-11T22:21:51.608370+0000","last_became_active":"2024-03-11T22:14:52.247273+0000","last_became_peered":"2024-03-11T22:14:52.247273+0000","last_unstale":"2024-03-11T22:21:51.608370+0000","last_undegraded":"2024-03-11T22:21:51.608370+0000","last_fullsized":"2024-03-11T22:21:51.608370+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:26.746435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:26.746435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1636499480,"num_objects":24972,"num_object_clones":0,"num_object_copies":74916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24972,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":223264,"num_write_kb":7943489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34657,"num_bytes_recovered":2270887960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4954599424,"data_stored":4954587208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72964,"ondisk_log_size":72964,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":773,"num_read_kb":768,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":1430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":717,"ondisk_log_size":717,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005223,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1346924,"kb_used_data":824228,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92398228,"statfs":{"total":95995035648,"available":94615785472,"internally_reserved":0,"allocated":844009472,"data_stored":843680927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":535165387},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332268,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":606420,"kb_used_data":268428,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93138732,"statfs":{"total":95995035648,"available":95374061568,"internally_reserved":0,"allocated":274870272,"data_stored":274566089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346094022},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1459400,"kb_used_data":963968,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":92285752,"statfs":{"total":95995035648,"available":94500610048,"internally_reserved":0,"allocated":987103232,"data_stored":986746021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":506722757},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738571,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":853348,"kb_used_data":425116,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92891804,"statfs":{"total":95995035648,"available":95121207296,"internally_reserved":0,"allocated":435318784,"data_stored":434997739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411521,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":587988,"kb_used_data":249996,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93157164,"statfs":{"total":95995035648,"available":95392935936,"internally_reserved":0,"allocated":255995904,"data_stored":255692091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149882,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1531260,"kb_used_data":1030836,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92213892,"statfs":{"total":95995035648,"available":94427025408,"internally_reserved":0,"allocated":1055576064,"data_stored":1055231971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":512424391},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920951,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":612076,"kb_used_data":269924,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93133076,"statfs":{"total":95995035648,"available":95368269824,"internally_reserved":0,"allocated":276402176,"data_stored":276073148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659312,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1486496,"kb_used_data":941656,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92258656,"statfs":{"total":95995035648,"available":94472863744,"internally_reserved":0,"allocated":964255744,"data_stored":963939114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":428474368,"data_stored":428474368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":967118848,"data_stored":967114776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":246743040,"data_stored":246743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1037041664,"data_stored":1037041664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":263716864,"data_stored":263716864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":946540544,"data_stored":946536472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":263913472,"data_stored":263913472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":801050624,"data_stored":801046552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:22:11.287 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T22:22:11.287 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:22:11.287 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_2 2024-03-11T22:22:11.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:22:11.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:11.428 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:11.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:11.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:11.529 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:11.559 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:11.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:11.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:11.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:11.719 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:11.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:11.820 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:11.848 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:11.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:11.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:11.950 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:11.980 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:11.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:12.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:11 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:22:12.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:11 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:22:12.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:11 smithi045 ceph-mon[39715]: pgmap v637: 168 pgs: 168 active+clean; 1.2 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 984 op/s 2024-03-11T22:22:12.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:11 smithi045 ceph-mon[39715]: osdmap e333: 8 total, 8 up, 8 in 2024-03-11T22:22:12.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:11 smithi045 ceph-mon[39715]: 6.5 scrub starts 2024-03-11T22:22:12.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:11 smithi045 ceph-mon[39715]: 6.5 scrub ok 2024-03-11T22:22:12.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:11 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:22:12.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:11 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:22:12.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:11 smithi045 ceph-mon[30178]: pgmap v637: 168 pgs: 168 active+clean; 1.2 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 984 op/s 2024-03-11T22:22:12.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:11 smithi045 ceph-mon[30178]: osdmap e333: 8 total, 8 up, 8 in 2024-03-11T22:22:12.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:11 smithi045 ceph-mon[30178]: 6.5 scrub starts 2024-03-11T22:22:12.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:11 smithi045 ceph-mon[30178]: 6.5 scrub ok 2024-03-11T22:22:12.082 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:12.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:11 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:22:12.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:11 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:22:12.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:11 smithi144 ceph-mon[33984]: pgmap v637: 168 pgs: 168 active+clean; 1.2 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 984 op/s 2024-03-11T22:22:12.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:11 smithi144 ceph-mon[33984]: osdmap e333: 8 total, 8 up, 8 in 2024-03-11T22:22:12.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:11 smithi144 ceph-mon[33984]: 6.5 scrub starts 2024-03-11T22:22:12.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:11 smithi144 ceph-mon[33984]: 6.5 scrub ok 2024-03-11T22:22:12.115 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:12.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:12.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:12.254 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:12.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:12.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:12.355 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:12.411 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:12.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:12.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:12.513 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:12.547 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:12.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:12.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:12.649 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:12.682 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:12.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:12.782 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:22:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:22:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:22:12.785 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:12.786 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:22:12.811 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:12.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:12.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:12.913 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:12.945 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:12.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:12.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:13.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:12 smithi045 ceph-mon[39715]: 9.5 scrub starts 2024-03-11T22:22:13.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:12 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4119318592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:22:13.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:12 smithi045 ceph-mon[30178]: 9.5 scrub starts 2024-03-11T22:22:13.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:12 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4119318592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:22:13.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:13.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:12 smithi144 ceph-mon[33984]: 9.5 scrub starts 2024-03-11T22:22:13.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:12 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4119318592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:22:13.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:13.098 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:13.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:13.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:13.242 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:13.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:13.344 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:13.377 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:13.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:13.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:13.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:13.522 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:13.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:13.603 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:13.623 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:13.654 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:13.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:13.756 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:13.792 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:13.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:13.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[30178]: 9.5 scrub ok 2024-03-11T22:22:13.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[30178]: pgmap v639: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 853 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-11T22:22:13.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[30178]: 5.3 scrub starts 2024-03-11T22:22:13.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[30178]: 5.3 scrub ok 2024-03-11T22:22:13.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:22:13.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:22:13.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4119318592' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:22:13.871 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[30178]: osdmap e334: 8 total, 8 up, 8 in 2024-03-11T22:22:13.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[39715]: 9.5 scrub ok 2024-03-11T22:22:13.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[39715]: pgmap v639: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 853 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-11T22:22:13.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[39715]: 5.3 scrub starts 2024-03-11T22:22:13.872 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[39715]: 5.3 scrub ok 2024-03-11T22:22:13.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:22:13.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:22:13.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4119318592' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:22:13.873 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:13 smithi045 ceph-mon[39715]: osdmap e334: 8 total, 8 up, 8 in 2024-03-11T22:22:13.893 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:13.945 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:13.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:14.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:14.089 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:14.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:14.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:13 smithi144 ceph-mon[33984]: 9.5 scrub ok 2024-03-11T22:22:14.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:13 smithi144 ceph-mon[33984]: pgmap v639: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 853 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-11T22:22:14.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:13 smithi144 ceph-mon[33984]: 5.3 scrub starts 2024-03-11T22:22:14.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:13 smithi144 ceph-mon[33984]: 5.3 scrub ok 2024-03-11T22:22:14.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:13 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:22:14.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:13 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:22:14.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:13 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4119318592' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:22:14.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:13 smithi144 ceph-mon[33984]: osdmap e334: 8 total, 8 up, 8 in 2024-03-11T22:22:14.191 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:14.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:14.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:14.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:14.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:14.373 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:14.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:14.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:14.476 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:14.508 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:14.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:14.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:14.610 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:14.644 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:14.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:14.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:14.745 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:14.778 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:14.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:14.880 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:14.911 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:14.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:14.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:14.936 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:22:14.937 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:22:14.937 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:22:14.937 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:22:14.937 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:22:14.937 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:22:14.937 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:22:14.937 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:22:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:22:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:22:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:22:14.937 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:22:14.938 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:22:14.938 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:22:14.938 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:22:14.938 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:22:15.012 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:15.045 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:15.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:15.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:15.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:14 smithi144 ceph-mon[33984]: 1.12 scrub starts 2024-03-11T22:22:15.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:14 smithi144 ceph-mon[33984]: 1.12 scrub ok 2024-03-11T22:22:15.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:14 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1987882303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:22:15.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:15.188 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:15.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:15.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:15.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:14 smithi045 ceph-mon[39715]: 1.12 scrub starts 2024-03-11T22:22:15.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:14 smithi045 ceph-mon[39715]: 1.12 scrub ok 2024-03-11T22:22:15.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:14 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1987882303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:22:15.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:14 smithi045 ceph-mon[30178]: 1.12 scrub starts 2024-03-11T22:22:15.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:14 smithi045 ceph-mon[30178]: 1.12 scrub ok 2024-03-11T22:22:15.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:14 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1987882303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:22:15.290 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:15.316 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:15.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:15.418 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:15.453 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:15.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:15.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:15.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:15.592 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:15.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:15.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:15.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:22:15.694 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:15.727 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:15.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:15.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:15.829 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:15.865 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:15.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:15.946 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:15.967 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:15.996 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:15.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:16.097 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:16.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:15 smithi144 ceph-mon[33984]: 5.d scrub starts 2024-03-11T22:22:16.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:15 smithi144 ceph-mon[33984]: 5.d scrub ok 2024-03-11T22:22:16.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:15 smithi144 ceph-mon[33984]: pgmap v641: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 853 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-03-11T22:22:16.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:15 smithi144 ceph-mon[33984]: 1.b deep-scrub starts 2024-03-11T22:22:16.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:15 smithi144 ceph-mon[33984]: 1.b deep-scrub ok 2024-03-11T22:22:16.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:15 smithi144 ceph-mon[33984]: 3.1f scrub starts 2024-03-11T22:22:16.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:15 smithi144 ceph-mon[33984]: 3.1f scrub ok 2024-03-11T22:22:16.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:15 smithi144 ceph-mon[33984]: 6.12 scrub starts 2024-03-11T22:22:16.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:22:16.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:22:16.107 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[39715]: 5.d scrub starts 2024-03-11T22:22:16.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[39715]: 5.d scrub ok 2024-03-11T22:22:16.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[39715]: pgmap v641: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 853 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-03-11T22:22:16.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[39715]: 1.b deep-scrub starts 2024-03-11T22:22:16.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[39715]: 1.b deep-scrub ok 2024-03-11T22:22:16.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[39715]: 3.1f scrub starts 2024-03-11T22:22:16.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[39715]: 3.1f scrub ok 2024-03-11T22:22:16.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[39715]: 6.12 scrub starts 2024-03-11T22:22:16.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:22:16.108 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:22:16.108 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[30178]: 5.d scrub starts 2024-03-11T22:22:16.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[30178]: 5.d scrub ok 2024-03-11T22:22:16.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[30178]: pgmap v641: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 853 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-03-11T22:22:16.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[30178]: 1.b deep-scrub starts 2024-03-11T22:22:16.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[30178]: 1.b deep-scrub ok 2024-03-11T22:22:16.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[30178]: 3.1f scrub starts 2024-03-11T22:22:16.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[30178]: 3.1f scrub ok 2024-03-11T22:22:16.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[30178]: 6.12 scrub starts 2024-03-11T22:22:16.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:22:16.109 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:22:16.128 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:16.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:16.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:16.272 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:16.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:16.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:16.374 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:16.404 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:16.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:16.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:16.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:16.542 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:16.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:16.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:16.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:16.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:16.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:16.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:16.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:22:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:22:16.817 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:16.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:16 smithi045 ceph-mon[30178]: 6.12 scrub ok 2024-03-11T22:22:16.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:16 smithi045 ceph-mon[30178]: 5.19 scrub starts 2024-03-11T22:22:16.834 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:16 smithi045 ceph-mon[30178]: 5.19 scrub ok 2024-03-11T22:22:16.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:16 smithi045 ceph-mon[39715]: 6.12 scrub ok 2024-03-11T22:22:16.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:16 smithi045 ceph-mon[39715]: 5.19 scrub starts 2024-03-11T22:22:16.834 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:16 smithi045 ceph-mon[39715]: 5.19 scrub ok 2024-03-11T22:22:16.846 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:16.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:16.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:16.948 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:16.975 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:16.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:16.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:17.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:17.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:16 smithi144 ceph-mon[33984]: 6.12 scrub ok 2024-03-11T22:22:17.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:16 smithi144 ceph-mon[33984]: 5.19 scrub starts 2024-03-11T22:22:17.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:16 smithi144 ceph-mon[33984]: 5.19 scrub ok 2024-03-11T22:22:17.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:17.128 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:17.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:17.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:17.276 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:17.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:17.377 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:17.413 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:17.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:17.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:17.514 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:17.548 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:17.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:17.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:17.574 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:17.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:17.693 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:17.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:17.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:17.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:17.819 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:17.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:17.921 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:17.946 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:22:17.946 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:22:17.962 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:17.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:17.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:22:18.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[39715]: 5.a scrub starts 2024-03-11T22:22:18.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[39715]: 5.a scrub ok 2024-03-11T22:22:18.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[39715]: 5.1b scrub starts 2024-03-11T22:22:18.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[39715]: 5.1b scrub ok 2024-03-11T22:22:18.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[39715]: pgmap v642: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 748 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T22:22:18.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:18.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[39715]: 4.17 scrub starts 2024-03-11T22:22:18.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[39715]: 4.17 scrub ok 2024-03-11T22:22:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[30178]: 5.a scrub starts 2024-03-11T22:22:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[30178]: 5.a scrub ok 2024-03-11T22:22:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[30178]: 5.1b scrub starts 2024-03-11T22:22:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[30178]: 5.1b scrub ok 2024-03-11T22:22:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[30178]: pgmap v642: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 748 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T22:22:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[30178]: 4.17 scrub starts 2024-03-11T22:22:18.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:17 smithi045 ceph-mon[30178]: 4.17 scrub ok 2024-03-11T22:22:18.064 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:18.096 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:18.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:18.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:18.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:17 smithi144 ceph-mon[33984]: 5.a scrub starts 2024-03-11T22:22:18.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:17 smithi144 ceph-mon[33984]: 5.a scrub ok 2024-03-11T22:22:18.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:17 smithi144 ceph-mon[33984]: 5.1b scrub starts 2024-03-11T22:22:18.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:17 smithi144 ceph-mon[33984]: 5.1b scrub ok 2024-03-11T22:22:18.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:17 smithi144 ceph-mon[33984]: pgmap v642: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 748 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T22:22:18.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:18.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:17 smithi144 ceph-mon[33984]: 4.17 scrub starts 2024-03-11T22:22:18.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:17 smithi144 ceph-mon[33984]: 4.17 scrub ok 2024-03-11T22:22:18.198 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:18.228 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:18.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:18.330 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:18.374 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:18.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:18.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:18.476 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:18.512 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:18.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:18.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:18.614 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:18.647 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:18.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:18.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:18.749 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:18.783 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:18.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:18.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:18.926 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:18.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:18.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:19.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:19.069 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:19.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:19.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:19.171 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:19.213 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:19.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:19.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:19.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:19.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:19.353 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:19.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:19.423 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T22:22:19.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:19.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:19.512 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:19.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:19.613 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:19.638 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:19.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:19.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:19.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:19.891 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:19.925 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:19.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:20.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:20.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:19 smithi045 ceph-mon[39715]: pgmap v643: 168 pgs: 168 active+clean; 441 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-03-11T22:22:20.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:19 smithi045 ceph-mon[30178]: pgmap v643: 168 pgs: 168 active+clean; 441 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-03-11T22:22:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:20.071 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:20.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:20.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:19 smithi144 ceph-mon[33984]: pgmap v643: 168 pgs: 168 active+clean; 441 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-03-11T22:22:20.172 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:20.209 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:20.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:20.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:20.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:20.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:20.351 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:20.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:20.452 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:20.488 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:20.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:20.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:20.590 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:20.623 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:20.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:20.724 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:20.753 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:20.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:20.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:20.879 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:20.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:20.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:21.028 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:21.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:21.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:20 smithi144 ceph-mon[33984]: 2.0 scrub starts 2024-03-11T22:22:21.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:20 smithi144 ceph-mon[33984]: 2.0 scrub ok 2024-03-11T22:22:21.116 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:20 smithi045 ceph-mon[39715]: 2.0 scrub starts 2024-03-11T22:22:21.117 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:20 smithi045 ceph-mon[39715]: 2.0 scrub ok 2024-03-11T22:22:21.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:20 smithi045 ceph-mon[30178]: 2.0 scrub starts 2024-03-11T22:22:21.117 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:20 smithi045 ceph-mon[30178]: 2.0 scrub ok 2024-03-11T22:22:21.130 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:21.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:21.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:21.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:21.280 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:21.306 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:21.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:21.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:21.463 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:21.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:21.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:21.611 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:21.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:21.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:21.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:21.755 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:21.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:21.771 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:22:21.776 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:22:21.857 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:21.858 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Removed 124117 objects 2024-03-11T22:22:21.859 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Clean up completed and total clean up time :118.359 2024-03-11T22:22:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:21.881 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:21.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:21.982 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:22.005 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:22.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:22.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[39715]: 9.f scrub starts 2024-03-11T22:22:22.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[39715]: 9.f scrub ok 2024-03-11T22:22:22.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[39715]: pgmap v644: 168 pgs: 168 active+clean; 441 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-11T22:22:22.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[39715]: 1.0 scrub starts 2024-03-11T22:22:22.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[39715]: 1.0 scrub ok 2024-03-11T22:22:22.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[39715]: 4.e scrub starts 2024-03-11T22:22:22.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[39715]: 4.e scrub ok 2024-03-11T22:22:22.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[30178]: 9.f scrub starts 2024-03-11T22:22:22.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[30178]: 9.f scrub ok 2024-03-11T22:22:22.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[30178]: pgmap v644: 168 pgs: 168 active+clean; 441 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-11T22:22:22.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[30178]: 1.0 scrub starts 2024-03-11T22:22:22.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[30178]: 1.0 scrub ok 2024-03-11T22:22:22.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[30178]: 4.e scrub starts 2024-03-11T22:22:22.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:21 smithi045 ceph-mon[30178]: 4.e scrub ok 2024-03-11T22:22:22.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:21 smithi144 ceph-mon[33984]: 9.f scrub starts 2024-03-11T22:22:22.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:21 smithi144 ceph-mon[33984]: 9.f scrub ok 2024-03-11T22:22:22.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:21 smithi144 ceph-mon[33984]: pgmap v644: 168 pgs: 168 active+clean; 441 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-11T22:22:22.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:21 smithi144 ceph-mon[33984]: 1.0 scrub starts 2024-03-11T22:22:22.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:21 smithi144 ceph-mon[33984]: 1.0 scrub ok 2024-03-11T22:22:22.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:21 smithi144 ceph-mon[33984]: 4.e scrub starts 2024-03-11T22:22:22.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:21 smithi144 ceph-mon[33984]: 4.e scrub ok 2024-03-11T22:22:22.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:22.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:22.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:22.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:22.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:22.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:22.339 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:22.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:22.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:22.463 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:22.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:22.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:22.589 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:22.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:22.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:22.714 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:22.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:22.784 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:22:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:22:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:22:22.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:22.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:22.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:22.946 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:22.966 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:22:22.974 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:22.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:23.079 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:23.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:22 smithi144 ceph-mon[33984]: 6.7 scrub starts 2024-03-11T22:22:23.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:22 smithi144 ceph-mon[33984]: 6.7 scrub ok 2024-03-11T22:22:23.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:22 smithi144 ceph-mon[33984]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:22:23.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:22 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1706612086' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:22:23.106 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:23.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:23.208 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:23.235 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:23.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:23.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:23.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:22 smithi045 ceph-mon[39715]: 6.7 scrub starts 2024-03-11T22:22:23.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:22 smithi045 ceph-mon[39715]: 6.7 scrub ok 2024-03-11T22:22:23.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:22 smithi045 ceph-mon[39715]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:22:23.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:22 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1706612086' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:22:23.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:22 smithi045 ceph-mon[30178]: 6.7 scrub starts 2024-03-11T22:22:23.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:22 smithi045 ceph-mon[30178]: 6.7 scrub ok 2024-03-11T22:22:23.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:22 smithi045 ceph-mon[30178]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:22:23.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:22 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1706612086' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:22:23.336 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:23.362 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:23.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:23.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:23.492 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:23.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:23.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:23.594 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:23.622 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:23.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:23.723 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:23.751 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:23.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:23.854 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:23.881 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:23.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:23.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:23.983 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:24.009 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:24.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:24.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:24.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:23 smithi144 ceph-mon[33984]: 5.6 scrub starts 2024-03-11T22:22:24.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:23 smithi144 ceph-mon[33984]: 5.6 scrub ok 2024-03-11T22:22:24.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:23 smithi144 ceph-mon[33984]: pgmap v645: 168 pgs: 168 active+clean; 39 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-11T22:22:24.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:23 smithi144 ceph-mon[33984]: 6.4 deep-scrub starts 2024-03-11T22:22:24.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:23 smithi144 ceph-mon[33984]: 6.4 deep-scrub ok 2024-03-11T22:22:24.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:23 smithi144 ceph-mon[33984]: 3.1b scrub starts 2024-03-11T22:22:24.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:23 smithi144 ceph-mon[33984]: 3.1b scrub ok 2024-03-11T22:22:24.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:23 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:22:24.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:23 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1706612086' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:22:24.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:23 smithi144 ceph-mon[33984]: osdmap e335: 8 total, 8 up, 8 in 2024-03-11T22:22:24.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:23 smithi144 ceph-mon[33984]: 5.18 scrub starts 2024-03-11T22:22:24.112 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:24.139 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:24.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:24.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:24.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:24.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:24.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:24.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[39715]: 5.6 scrub starts 2024-03-11T22:22:24.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[39715]: 5.6 scrub ok 2024-03-11T22:22:24.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[39715]: pgmap v645: 168 pgs: 168 active+clean; 39 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-11T22:22:24.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[39715]: 6.4 deep-scrub starts 2024-03-11T22:22:24.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[39715]: 6.4 deep-scrub ok 2024-03-11T22:22:24.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[39715]: 3.1b scrub starts 2024-03-11T22:22:24.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[39715]: 3.1b scrub ok 2024-03-11T22:22:24.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:22:24.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1706612086' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:22:24.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[39715]: osdmap e335: 8 total, 8 up, 8 in 2024-03-11T22:22:24.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[39715]: 5.18 scrub starts 2024-03-11T22:22:24.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[30178]: 5.6 scrub starts 2024-03-11T22:22:24.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[30178]: 5.6 scrub ok 2024-03-11T22:22:24.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[30178]: pgmap v645: 168 pgs: 168 active+clean; 39 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-11T22:22:24.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[30178]: 6.4 deep-scrub starts 2024-03-11T22:22:24.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[30178]: 6.4 deep-scrub ok 2024-03-11T22:22:24.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[30178]: 3.1b scrub starts 2024-03-11T22:22:24.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[30178]: 3.1b scrub ok 2024-03-11T22:22:24.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:22:24.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1706612086' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:22:24.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[30178]: osdmap e335: 8 total, 8 up, 8 in 2024-03-11T22:22:24.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:23 smithi045 ceph-mon[30178]: 5.18 scrub starts 2024-03-11T22:22:24.370 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:24.398 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:24.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:24.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:24.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:24.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:24.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:24.630 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:24.657 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:24.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:24.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:24.787 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:24.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:24.889 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:24.917 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:24.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:25.019 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:25.047 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:25.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:25.148 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:25.176 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:25.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:25.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:25.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:25.306 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:25.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:25.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:25.497 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:25.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:25.599 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:25.627 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:25.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:25.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:25.756 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:25.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:25.858 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:25.886 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:25.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:25.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:25.892 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":644,"stamp":"2024-03-11T22:22:20.339277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":462559604,"num_objects":7248,"num_object_clones":0,"num_object_copies":21744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7248,"num_whiteouts":0,"num_read":1913,"num_read_kb":1665,"num_write":241908,"num_write_kb":7943534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34765,"num_bytes_recovered":2270890396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74927,"ondisk_log_size":74927,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":506,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5020620,"kb_used_data":1510088,"kb_used_omap":12,"kb_used_meta":3510451,"kb_avail":744940596,"statfs":{"total":767960285184,"available":762819170304,"internally_reserved":0,"allocated":1546330112,"data_stored":1543659964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":3594702387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-752812032,"num_objects":-11487,"num_object_clones":0,"num_object_copies":-34461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11487,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11487,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.576617"},"pg_stats":[{"pgid":"9.1","version":"334'15170","reported_seq":27107,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:16.447300+0000","last_change":"2024-03-11T22:21:14.515399+0000","last_active":"2024-03-11T22:22:16.447300+0000","last_peered":"2024-03-11T22:22:16.447300+0000","last_clean":"2024-03-11T22:22:16.447300+0000","last_became_active":"2024-03-11T22:21:14.450983+0000","last_became_peered":"2024-03-11T22:21:14.450983+0000","last_unstale":"2024-03-11T22:22:16.447300+0000","last_undegraded":"2024-03-11T22:22:16.447300+0000","last_fullsized":"2024-03-11T22:22:16.447300+0000","mapping_epoch":318,"log_start":"320'11100","ondisk_log_start":"320'11100","created":273,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4070,"log_dups_size":0,"ondisk_log_size":4070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28704768,"num_objects":438,"num_object_clones":0,"num_object_copies":1314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15170,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5892,"num_bytes_recovered":386007040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"294'25","reported_seq":393,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.807784+0000","last_change":"2024-03-11T22:19:41.508884+0000","last_active":"2024-03-11T22:22:12.807784+0000","last_peered":"2024-03-11T22:22:12.807784+0000","last_clean":"2024-03-11T22:22:12.807784+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:22:12.807784+0000","last_undegraded":"2024-03-11T22:22:12.807784+0000","last_fullsized":"2024-03-11T22:22:12.807784+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:19:41.508161+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":581,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.800466+0000","last_change":"2024-03-11T22:19:28.586829+0000","last_active":"2024-03-11T22:22:12.800466+0000","last_peered":"2024-03-11T22:22:12.800466+0000","last_clean":"2024-03-11T22:22:12.800466+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:22:12.800466+0000","last_undegraded":"2024-03-11T22:22:12.800466+0000","last_fullsized":"2024-03-11T22:22:12.800466+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:28.586614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:28.586614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":779,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:14.189514+0000","last_change":"2024-03-11T22:22:14.189565+0000","last_active":"2024-03-11T22:22:14.189514+0000","last_peered":"2024-03-11T22:22:14.189514+0000","last_clean":"2024-03-11T22:22:14.189514+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:22:14.189514+0000","last_undegraded":"2024-03-11T22:22:14.189514+0000","last_fullsized":"2024-03-11T22:22:14.189514+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:22:14.189406+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:19:40.008325+0000","last_clean_scrub_stamp":"2024-03-11T22:22:14.189406+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:23.528958+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":347,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.782815+0000","last_change":"2024-03-11T22:18:37.116172+0000","last_active":"2024-03-11T22:22:12.782815+0000","last_peered":"2024-03-11T22:22:12.782815+0000","last_clean":"2024-03-11T22:22:12.782815+0000","last_became_active":"2024-03-11T22:14:52.243690+0000","last_became_peered":"2024-03-11T22:14:52.243690+0000","last_unstale":"2024-03-11T22:22:12.782815+0000","last_undegraded":"2024-03-11T22:22:12.782815+0000","last_fullsized":"2024-03-11T22:22:12.782815+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:37.116093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.116093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":904,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.798144+0000","last_change":"2024-03-11T22:20:32.294412+0000","last_active":"2024-03-11T22:22:12.798144+0000","last_peered":"2024-03-11T22:22:12.798144+0000","last_clean":"2024-03-11T22:22:12.798144+0000","last_became_active":"2024-03-11T22:20:32.293356+0000","last_became_peered":"2024-03-11T22:20:32.293356+0000","last_unstale":"2024-03-11T22:22:12.798144+0000","last_undegraded":"2024-03-11T22:22:12.798144+0000","last_fullsized":"2024-03-11T22:22:12.798144+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:20.907743+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:20.907743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"334'15146","reported_seq":15279,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:16.426912+0000","last_change":"2024-03-11T22:18:41.783594+0000","last_active":"2024-03-11T22:22:16.426912+0000","last_peered":"2024-03-11T22:22:16.426912+0000","last_clean":"2024-03-11T22:22:16.426912+0000","last_became_active":"2024-03-11T22:18:41.783248+0000","last_became_peered":"2024-03-11T22:18:41.783248+0000","last_unstale":"2024-03-11T22:22:16.426912+0000","last_undegraded":"2024-03-11T22:22:16.426912+0000","last_fullsized":"2024-03-11T22:22:16.426912+0000","mapping_epoch":273,"log_start":"317'10600","ondisk_log_start":"317'10600","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4546,"log_dups_size":0,"ondisk_log_size":4546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27131904,"num_objects":414,"num_object_clones":0,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15146,"num_write_kb":497920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"273'31","reported_seq":431,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784027+0000","last_change":"2024-03-11T22:20:40.790500+0000","last_active":"2024-03-11T22:22:12.784027+0000","last_peered":"2024-03-11T22:22:12.784027+0000","last_clean":"2024-03-11T22:22:12.784027+0000","last_became_active":"2024-03-11T22:14:52.239649+0000","last_became_peered":"2024-03-11T22:14:52.239649+0000","last_unstale":"2024-03-11T22:22:12.784027+0000","last_undegraded":"2024-03-11T22:22:12.784027+0000","last_fullsized":"2024-03-11T22:22:12.784027+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'31","last_scrub_stamp":"2024-03-11T22:20:40.790339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:40.790339+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":605,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799210+0000","last_change":"2024-03-11T22:20:52.712468+0000","last_active":"2024-03-11T22:22:12.799210+0000","last_peered":"2024-03-11T22:22:12.799210+0000","last_clean":"2024-03-11T22:22:12.799210+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:22:12.799210+0000","last_undegraded":"2024-03-11T22:22:12.799210+0000","last_fullsized":"2024-03-11T22:22:12.799210+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:52.712329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:20:52.712329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":581,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.786550+0000","last_change":"2024-03-11T22:18:44.130366+0000","last_active":"2024-03-11T22:22:12.786550+0000","last_peered":"2024-03-11T22:22:12.786550+0000","last_clean":"2024-03-11T22:22:12.786550+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:22:12.786550+0000","last_undegraded":"2024-03-11T22:22:12.786550+0000","last_fullsized":"2024-03-11T22:22:12.786550+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:44.130252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:18:44.130252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":593,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.788187+0000","last_change":"2024-03-11T22:22:00.325341+0000","last_active":"2024-03-11T22:22:12.788187+0000","last_peered":"2024-03-11T22:22:12.788187+0000","last_clean":"2024-03-11T22:22:12.788187+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:22:12.788187+0000","last_undegraded":"2024-03-11T22:22:12.788187+0000","last_fullsized":"2024-03-11T22:22:12.788187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:00.325095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:00.325095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:06.335725+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":880,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783565+0000","last_change":"2024-03-11T22:18:21.646464+0000","last_active":"2024-03-11T22:22:12.783565+0000","last_peered":"2024-03-11T22:22:12.783565+0000","last_clean":"2024-03-11T22:22:12.783565+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:22:12.783565+0000","last_undegraded":"2024-03-11T22:22:12.783565+0000","last_fullsized":"2024-03-11T22:22:12.783565+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:21.646305+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.646305+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"334'15272","reported_seq":23410,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:16.446689+0000","last_change":"2024-03-11T22:21:38.531313+0000","last_active":"2024-03-11T22:22:16.446689+0000","last_peered":"2024-03-11T22:22:16.446689+0000","last_clean":"2024-03-11T22:22:16.446689+0000","last_became_active":"2024-03-11T22:21:38.510998+0000","last_became_peered":"2024-03-11T22:21:38.510998+0000","last_unstale":"2024-03-11T22:22:16.446689+0000","last_undegraded":"2024-03-11T22:22:16.446689+0000","last_fullsized":"2024-03-11T22:22:16.446689+0000","mapping_epoch":326,"log_start":"320'11200","ondisk_log_start":"320'11200","created":273,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4072,"log_dups_size":0,"ondisk_log_size":4072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27787264,"num_objects":424,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15272,"num_write_kb":502272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3990,"num_bytes_recovered":261357568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"273'19","reported_seq":405,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799438+0000","last_change":"2024-03-11T22:20:27.089007+0000","last_active":"2024-03-11T22:22:12.799438+0000","last_peered":"2024-03-11T22:22:12.799438+0000","last_clean":"2024-03-11T22:22:12.799438+0000","last_became_active":"2024-03-11T22:14:52.240437+0000","last_became_peered":"2024-03-11T22:14:52.240437+0000","last_unstale":"2024-03-11T22:22:12.799438+0000","last_undegraded":"2024-03-11T22:22:12.799438+0000","last_fullsized":"2024-03-11T22:22:12.799438+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:20:27.088856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:27.088856+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":364,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.782029+0000","last_change":"2024-03-11T22:20:45.895055+0000","last_active":"2024-03-11T22:22:12.782029+0000","last_peered":"2024-03-11T22:22:12.782029+0000","last_clean":"2024-03-11T22:22:12.782029+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:22:12.782029+0000","last_undegraded":"2024-03-11T22:22:12.782029+0000","last_fullsized":"2024-03-11T22:22:12.782029+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:45.894940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:45.894940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":573,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.788049+0000","last_change":"2024-03-11T22:18:37.705396+0000","last_active":"2024-03-11T22:22:12.788049+0000","last_peered":"2024-03-11T22:22:12.788049+0000","last_clean":"2024-03-11T22:22:12.788049+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:22:12.788049+0000","last_undegraded":"2024-03-11T22:22:12.788049+0000","last_fullsized":"2024-03-11T22:22:12.788049+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:37.705305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.705305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":561,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.806340+0000","last_change":"2024-03-11T22:18:48.086934+0000","last_active":"2024-03-11T22:22:12.806340+0000","last_peered":"2024-03-11T22:22:12.806340+0000","last_clean":"2024-03-11T22:22:12.806340+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:22:12.806340+0000","last_undegraded":"2024-03-11T22:22:12.806340+0000","last_fullsized":"2024-03-11T22:22:12.806340+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:48.086856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:48.086856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"93'130","reported_seq":916,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:14.766875+0000","last_change":"2024-03-11T22:22:14.766914+0000","last_active":"2024-03-11T22:22:14.766875+0000","last_peered":"2024-03-11T22:22:14.766875+0000","last_clean":"2024-03-11T22:22:14.766875+0000","last_became_active":"2024-03-11T22:18:00.271764+0000","last_became_peered":"2024-03-11T22:18:00.271764+0000","last_unstale":"2024-03-11T22:22:14.766875+0000","last_undegraded":"2024-03-11T22:22:14.766875+0000","last_fullsized":"2024-03-11T22:22:14.766875+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:22:14.766775+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:22:14.766775+0000","last_clean_scrub_stamp":"2024-03-11T22:22:14.766775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:34.909952+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.2","version":"334'14884","reported_seq":15016,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:16.476285+0000","last_change":"2024-03-11T22:18:41.238608+0000","last_active":"2024-03-11T22:22:16.476285+0000","last_peered":"2024-03-11T22:22:16.476285+0000","last_clean":"2024-03-11T22:22:16.476285+0000","last_became_active":"2024-03-11T22:18:41.238269+0000","last_became_peered":"2024-03-11T22:18:41.238269+0000","last_unstale":"2024-03-11T22:22:16.476285+0000","last_undegraded":"2024-03-11T22:22:16.476285+0000","last_fullsized":"2024-03-11T22:22:16.476285+0000","mapping_epoch":273,"log_start":"318'10800","ondisk_log_start":"318'10800","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4084,"log_dups_size":0,"ondisk_log_size":4084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28311552,"num_objects":432,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14884,"num_write_kb":490112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"273'34","reported_seq":651,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783353+0000","last_change":"2024-03-11T22:21:49.811476+0000","last_active":"2024-03-11T22:22:12.783353+0000","last_peered":"2024-03-11T22:22:12.783353+0000","last_clean":"2024-03-11T22:22:12.783353+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:22:12.783353+0000","last_undegraded":"2024-03-11T22:22:12.783353+0000","last_fullsized":"2024-03-11T22:22:12.783353+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:21:49.811214+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:21:49.811214+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:16.306128+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":593,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799848+0000","last_change":"2024-03-11T22:21:56.018639+0000","last_active":"2024-03-11T22:22:12.799848+0000","last_peered":"2024-03-11T22:22:12.799848+0000","last_clean":"2024-03-11T22:22:12.799848+0000","last_became_active":"2024-03-11T22:20:29.426224+0000","last_became_peered":"2024-03-11T22:20:29.426224+0000","last_unstale":"2024-03-11T22:22:12.799848+0000","last_undegraded":"2024-03-11T22:22:12.799848+0000","last_fullsized":"2024-03-11T22:22:12.799848+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:56.018519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:21:56.018519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:22.003241+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":462,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784325+0000","last_change":"2024-03-11T22:20:59.636170+0000","last_active":"2024-03-11T22:22:12.784325+0000","last_peered":"2024-03-11T22:22:12.784325+0000","last_clean":"2024-03-11T22:22:12.784325+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:22:12.784325+0000","last_undegraded":"2024-03-11T22:22:12.784325+0000","last_fullsized":"2024-03-11T22:22:12.784325+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:20:59.636041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:59.636041+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:24.576756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":569,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784026+0000","last_change":"2024-03-11T22:18:50.805816+0000","last_active":"2024-03-11T22:22:12.784026+0000","last_peered":"2024-03-11T22:22:12.784026+0000","last_clean":"2024-03-11T22:22:12.784026+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:22:12.784026+0000","last_undegraded":"2024-03-11T22:22:12.784026+0000","last_fullsized":"2024-03-11T22:22:12.784026+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:50.805698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:18:50.805698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":889,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784520+0000","last_change":"2024-03-11T22:21:57.313533+0000","last_active":"2024-03-11T22:22:12.784520+0000","last_peered":"2024-03-11T22:22:12.784520+0000","last_clean":"2024-03-11T22:22:12.784520+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:22:12.784520+0000","last_undegraded":"2024-03-11T22:22:12.784520+0000","last_fullsized":"2024-03-11T22:22:12.784520+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:21:57.313449+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:21:57.313449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:21.421483+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"334'15069","reported_seq":22898,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:16.426812+0000","last_change":"2024-03-11T22:21:37.526283+0000","last_active":"2024-03-11T22:22:16.426812+0000","last_peered":"2024-03-11T22:22:16.426812+0000","last_clean":"2024-03-11T22:22:16.426812+0000","last_became_active":"2024-03-11T22:21:37.500225+0000","last_became_peered":"2024-03-11T22:21:37.500225+0000","last_unstale":"2024-03-11T22:22:16.426812+0000","last_undegraded":"2024-03-11T22:22:16.426812+0000","last_fullsized":"2024-03-11T22:22:16.426812+0000","mapping_epoch":325,"log_start":"317'10500","ondisk_log_start":"317'10500","created":273,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4569,"log_dups_size":0,"ondisk_log_size":4569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26345496,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15069,"num_write_kb":495041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3840,"num_bytes_recovered":251592728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":357,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.798535+0000","last_change":"2024-03-11T22:20:29.441577+0000","last_active":"2024-03-11T22:22:12.798535+0000","last_peered":"2024-03-11T22:22:12.798535+0000","last_clean":"2024-03-11T22:22:12.798535+0000","last_became_active":"2024-03-11T22:20:29.438896+0000","last_became_peered":"2024-03-11T22:20:29.438896+0000","last_unstale":"2024-03-11T22:22:12.798535+0000","last_undegraded":"2024-03-11T22:22:12.798535+0000","last_fullsized":"2024-03-11T22:22:12.798535+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:14.293977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:14.293977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"273'30","reported_seq":670,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799930+0000","last_change":"2024-03-11T22:20:29.473203+0000","last_active":"2024-03-11T22:22:12.799930+0000","last_peered":"2024-03-11T22:22:12.799930+0000","last_clean":"2024-03-11T22:22:12.799930+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:22:12.799930+0000","last_undegraded":"2024-03-11T22:22:12.799930+0000","last_fullsized":"2024-03-11T22:22:12.799930+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-11T22:15:33.497732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:33.497732+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":593,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.806078+0000","last_change":"2024-03-11T22:20:29.441068+0000","last_active":"2024-03-11T22:22:12.806078+0000","last_peered":"2024-03-11T22:22:12.806078+0000","last_clean":"2024-03-11T22:22:12.806078+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:22:12.806078+0000","last_undegraded":"2024-03-11T22:22:12.806078+0000","last_fullsized":"2024-03-11T22:22:12.806078+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.133631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.133631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":347,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799996+0000","last_change":"2024-03-11T22:22:04.830996+0000","last_active":"2024-03-11T22:22:12.799996+0000","last_peered":"2024-03-11T22:22:12.799996+0000","last_clean":"2024-03-11T22:22:12.799996+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:22:12.799996+0000","last_undegraded":"2024-03-11T22:22:12.799996+0000","last_fullsized":"2024-03-11T22:22:12.799996+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:04.830875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:04.830875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:33.793634+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":841,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784357+0000","last_change":"2024-03-11T22:20:57.831615+0000","last_active":"2024-03-11T22:22:12.784357+0000","last_peered":"2024-03-11T22:22:12.784357+0000","last_clean":"2024-03-11T22:22:12.784357+0000","last_became_active":"2024-03-11T22:18:20.240728+0000","last_became_peered":"2024-03-11T22:18:20.240728+0000","last_unstale":"2024-03-11T22:22:12.784357+0000","last_undegraded":"2024-03-11T22:22:12.784357+0000","last_fullsized":"2024-03-11T22:22:12.784357+0000","mapping_epoch":175,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:57.831334+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:20:57.831334+0000","last_clean_scrub_stamp":"2024-03-11T22:20:57.831334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.c","version":"334'15010","reported_seq":15143,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:16.454206+0000","last_change":"2024-03-11T22:18:41.260878+0000","last_active":"2024-03-11T22:22:16.454206+0000","last_peered":"2024-03-11T22:22:16.454206+0000","last_clean":"2024-03-11T22:22:16.454206+0000","last_became_active":"2024-03-11T22:18:41.260611+0000","last_became_peered":"2024-03-11T22:18:41.260611+0000","last_unstale":"2024-03-11T22:22:16.454206+0000","last_undegraded":"2024-03-11T22:22:16.454206+0000","last_fullsized":"2024-03-11T22:22:16.454206+0000","mapping_epoch":273,"log_start":"320'10900","ondisk_log_start":"320'10900","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27656192,"num_objects":422,"num_object_clones":0,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15010,"num_write_kb":493824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":738,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.800076+0000","last_change":"2024-03-11T22:15:31.519229+0000","last_active":"2024-03-11T22:22:12.800076+0000","last_peered":"2024-03-11T22:22:12.800076+0000","last_clean":"2024-03-11T22:22:12.800076+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:22:12.800076+0000","last_undegraded":"2024-03-11T22:22:12.800076+0000","last_fullsized":"2024-03-11T22:22:12.800076+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:15:31.513336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:15:31.513336+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"273'28","reported_seq":408,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784897+0000","last_change":"2024-03-11T22:21:50.308999+0000","last_active":"2024-03-11T22:22:12.784897+0000","last_peered":"2024-03-11T22:22:12.784897+0000","last_clean":"2024-03-11T22:22:12.784897+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:22:12.784897+0000","last_undegraded":"2024-03-11T22:22:12.784897+0000","last_fullsized":"2024-03-11T22:22:12.784897+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:21:50.308860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:21:50.308860+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:18.140149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":582,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795314+0000","last_change":"2024-03-11T22:19:28.428409+0000","last_active":"2024-03-11T22:22:12.795314+0000","last_peered":"2024-03-11T22:22:12.795314+0000","last_clean":"2024-03-11T22:22:12.795314+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:22:12.795314+0000","last_undegraded":"2024-03-11T22:22:12.795314+0000","last_fullsized":"2024-03-11T22:22:12.795314+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:19:28.428283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:28.428283+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":535,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799317+0000","last_change":"2024-03-11T22:21:59.625759+0000","last_active":"2024-03-11T22:22:12.799317+0000","last_peered":"2024-03-11T22:22:12.799317+0000","last_clean":"2024-03-11T22:22:12.799317+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:22:12.799317+0000","last_undegraded":"2024-03-11T22:22:12.799317+0000","last_fullsized":"2024-03-11T22:22:12.799317+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:59.625636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:21:59.625636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:03.539206+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":897,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.806423+0000","last_change":"2024-03-11T22:20:32.293956+0000","last_active":"2024-03-11T22:22:12.806423+0000","last_peered":"2024-03-11T22:22:12.806423+0000","last_clean":"2024-03-11T22:22:12.806423+0000","last_became_active":"2024-03-11T22:20:32.293085+0000","last_became_peered":"2024-03-11T22:20:32.293085+0000","last_unstale":"2024-03-11T22:22:12.806423+0000","last_undegraded":"2024-03-11T22:22:12.806423+0000","last_fullsized":"2024-03-11T22:22:12.806423+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:18.905939+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:18.905939+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"334'15219","reported_seq":15352,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:15.516322+0000","last_change":"2024-03-11T22:18:41.236691+0000","last_active":"2024-03-11T22:22:15.516322+0000","last_peered":"2024-03-11T22:22:15.516322+0000","last_clean":"2024-03-11T22:22:15.516322+0000","last_became_active":"2024-03-11T22:18:41.236462+0000","last_became_peered":"2024-03-11T22:18:41.236462+0000","last_unstale":"2024-03-11T22:22:15.516322+0000","last_undegraded":"2024-03-11T22:22:15.516322+0000","last_fullsized":"2024-03-11T22:22:15.516322+0000","mapping_epoch":273,"log_start":"317'10700","ondisk_log_start":"317'10700","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4519,"log_dups_size":0,"ondisk_log_size":4519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37027840,"num_objects":565,"num_object_clones":0,"num_object_copies":1695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15219,"num_write_kb":505088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":754,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.796840+0000","last_change":"2024-03-11T22:22:12.728619+0000","last_active":"2024-03-11T22:22:12.796840+0000","last_peered":"2024-03-11T22:22:12.796840+0000","last_clean":"2024-03-11T22:22:12.796840+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:22:12.796840+0000","last_undegraded":"2024-03-11T22:22:12.796840+0000","last_fullsized":"2024-03-11T22:22:12.796840+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:22:12.728399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:22:12.728399+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:40.541122+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"273'20","reported_seq":623,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.785809+0000","last_change":"2024-03-11T22:19:31.503395+0000","last_active":"2024-03-11T22:22:12.785809+0000","last_peered":"2024-03-11T22:22:12.785809+0000","last_clean":"2024-03-11T22:22:12.785809+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:22:12.785809+0000","last_undegraded":"2024-03-11T22:22:12.785809+0000","last_fullsized":"2024-03-11T22:22:12.785809+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:19:31.502993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:31.502993+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":589,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784328+0000","last_change":"2024-03-11T22:19:34.706343+0000","last_active":"2024-03-11T22:22:12.784328+0000","last_peered":"2024-03-11T22:22:12.784328+0000","last_clean":"2024-03-11T22:22:12.784328+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:22:12.784328+0000","last_undegraded":"2024-03-11T22:22:12.784328+0000","last_fullsized":"2024-03-11T22:22:12.784328+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.706222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.706222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":569,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795118+0000","last_change":"2024-03-11T22:19:34.477472+0000","last_active":"2024-03-11T22:22:12.795118+0000","last_peered":"2024-03-11T22:22:12.795118+0000","last_clean":"2024-03-11T22:22:12.795118+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:22:12.795118+0000","last_undegraded":"2024-03-11T22:22:12.795118+0000","last_fullsized":"2024-03-11T22:22:12.795118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:34.477349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.477349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":822,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799614+0000","last_change":"2024-03-11T22:20:43.803186+0000","last_active":"2024-03-11T22:22:12.799614+0000","last_peered":"2024-03-11T22:22:12.799614+0000","last_clean":"2024-03-11T22:22:12.799614+0000","last_became_active":"2024-03-11T22:20:32.290512+0000","last_became_peered":"2024-03-11T22:20:32.290512+0000","last_unstale":"2024-03-11T22:22:12.799614+0000","last_undegraded":"2024-03-11T22:22:12.799614+0000","last_fullsized":"2024-03-11T22:22:12.799614+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:43.803067+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:20:43.803067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"334'14979","reported_seq":26819,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:16.448441+0000","last_change":"2024-03-11T22:21:15.469013+0000","last_active":"2024-03-11T22:22:16.448441+0000","last_peered":"2024-03-11T22:22:16.448441+0000","last_clean":"2024-03-11T22:22:16.448441+0000","last_became_active":"2024-03-11T22:21:15.407140+0000","last_became_peered":"2024-03-11T22:21:15.407140+0000","last_unstale":"2024-03-11T22:22:16.448441+0000","last_undegraded":"2024-03-11T22:22:16.448441+0000","last_fullsized":"2024-03-11T22:22:16.448441+0000","mapping_epoch":319,"log_start":"317'10400","ondisk_log_start":"317'10400","created":273,"last_epoch_clean":320,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27197440,"num_objects":415,"num_object_clones":0,"num_object_copies":1245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14979,"num_write_kb":492608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5847,"num_bytes_recovered":383188992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":597,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799074+0000","last_change":"2024-03-11T22:22:00.588718+0000","last_active":"2024-03-11T22:22:12.799074+0000","last_peered":"2024-03-11T22:22:12.799074+0000","last_clean":"2024-03-11T22:22:12.799074+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:22:12.799074+0000","last_undegraded":"2024-03-11T22:22:12.799074+0000","last_fullsized":"2024-03-11T22:22:12.799074+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:00.588603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:22:00.588603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:09.767814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"273'29","reported_seq":628,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.796460+0000","last_change":"2024-03-11T22:19:38.450843+0000","last_active":"2024-03-11T22:22:12.796460+0000","last_peered":"2024-03-11T22:22:12.796460+0000","last_clean":"2024-03-11T22:22:12.796460+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:22:12.796460+0000","last_undegraded":"2024-03-11T22:22:12.796460+0000","last_fullsized":"2024-03-11T22:22:12.796460+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'29","last_scrub_stamp":"2024-03-11T22:19:38.450581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.450581+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":589,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.786334+0000","last_change":"2024-03-11T22:18:38.153695+0000","last_active":"2024-03-11T22:22:12.786334+0000","last_peered":"2024-03-11T22:22:12.786334+0000","last_clean":"2024-03-11T22:22:12.786334+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:22:12.786334+0000","last_undegraded":"2024-03-11T22:22:12.786334+0000","last_fullsized":"2024-03-11T22:22:12.786334+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:38.153614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.153614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":577,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.782047+0000","last_change":"2024-03-11T22:18:23.998360+0000","last_active":"2024-03-11T22:22:12.782047+0000","last_peered":"2024-03-11T22:22:12.782047+0000","last_clean":"2024-03-11T22:22:12.782047+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:22:12.782047+0000","last_undegraded":"2024-03-11T22:22:12.782047+0000","last_fullsized":"2024-03-11T22:22:12.782047+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:23.998251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:23.998251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":797,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799764+0000","last_change":"2024-03-11T22:20:44.849410+0000","last_active":"2024-03-11T22:22:12.799764+0000","last_peered":"2024-03-11T22:22:12.799764+0000","last_clean":"2024-03-11T22:22:12.799764+0000","last_became_active":"2024-03-11T22:18:17.238247+0000","last_became_peered":"2024-03-11T22:18:17.238247+0000","last_unstale":"2024-03-11T22:22:12.799764+0000","last_undegraded":"2024-03-11T22:22:12.799764+0000","last_fullsized":"2024-03-11T22:22:12.799764+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:44.849279+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:20:44.849279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"334'14882","reported_seq":15015,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:16.900394+0000","last_change":"2024-03-11T22:18:41.783592+0000","last_active":"2024-03-11T22:22:16.900394+0000","last_peered":"2024-03-11T22:22:16.900394+0000","last_clean":"2024-03-11T22:22:16.900394+0000","last_became_active":"2024-03-11T22:18:41.783277+0000","last_became_peered":"2024-03-11T22:18:41.783277+0000","last_unstale":"2024-03-11T22:22:16.900394+0000","last_undegraded":"2024-03-11T22:22:16.900394+0000","last_fullsized":"2024-03-11T22:22:16.900394+0000","mapping_epoch":273,"log_start":"316'9600","ondisk_log_start":"316'9600","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26345472,"num_objects":402,"num_object_clones":0,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14882,"num_write_kb":489088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":581,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.788077+0000","last_change":"2024-03-11T22:22:09.265906+0000","last_active":"2024-03-11T22:22:12.788077+0000","last_peered":"2024-03-11T22:22:12.788077+0000","last_clean":"2024-03-11T22:22:12.788077+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:22:12.788077+0000","last_undegraded":"2024-03-11T22:22:12.788077+0000","last_fullsized":"2024-03-11T22:22:12.788077+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:09.265780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:22:09.265780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:09.959660+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"273'24","reported_seq":621,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795292+0000","last_change":"2024-03-11T22:20:29.380624+0000","last_active":"2024-03-11T22:22:12.795292+0000","last_peered":"2024-03-11T22:22:12.795292+0000","last_clean":"2024-03-11T22:22:12.795292+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:22:12.795292+0000","last_undegraded":"2024-03-11T22:22:12.795292+0000","last_fullsized":"2024-03-11T22:22:12.795292+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'24","last_scrub_stamp":"2024-03-11T22:20:29.380527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:29.380527+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":1392,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:14.977167+0000","last_change":"2024-03-11T22:18:27.077438+0000","last_active":"2024-03-11T22:22:14.977167+0000","last_peered":"2024-03-11T22:22:14.977167+0000","last_clean":"2024-03-11T22:22:14.977167+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:22:14.977167+0000","last_undegraded":"2024-03-11T22:22:14.977167+0000","last_fullsized":"2024-03-11T22:22:14.977167+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:18:27.077275+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.077275+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":783,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":589,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795406+0000","last_change":"2024-03-11T22:21:58.650945+0000","last_active":"2024-03-11T22:22:12.795406+0000","last_peered":"2024-03-11T22:22:12.795406+0000","last_clean":"2024-03-11T22:22:12.795406+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:22:12.795406+0000","last_undegraded":"2024-03-11T22:22:12.795406+0000","last_fullsized":"2024-03-11T22:22:12.795406+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:58.650607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:21:58.650607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:28.349205+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":585,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.786142+0000","last_change":"2024-03-11T22:18:36.127513+0000","last_active":"2024-03-11T22:22:12.786142+0000","last_peered":"2024-03-11T22:22:12.786142+0000","last_clean":"2024-03-11T22:22:12.786142+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:22:12.786142+0000","last_undegraded":"2024-03-11T22:22:12.786142+0000","last_fullsized":"2024-03-11T22:22:12.786142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:36.127435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:36.127435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":887,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783843+0000","last_change":"2024-03-11T22:19:29.435468+0000","last_active":"2024-03-11T22:22:12.783843+0000","last_peered":"2024-03-11T22:22:12.783843+0000","last_clean":"2024-03-11T22:22:12.783843+0000","last_became_active":"2024-03-11T22:14:38.164101+0000","last_became_peered":"2024-03-11T22:14:38.164101+0000","last_unstale":"2024-03-11T22:22:12.783843+0000","last_undegraded":"2024-03-11T22:22:12.783843+0000","last_fullsized":"2024-03-11T22:22:12.783843+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:29.435334+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.435334+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"334'15182","reported_seq":15315,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:16.480360+0000","last_change":"2024-03-11T22:18:41.240089+0000","last_active":"2024-03-11T22:22:16.480360+0000","last_peered":"2024-03-11T22:22:16.480360+0000","last_clean":"2024-03-11T22:22:16.480360+0000","last_became_active":"2024-03-11T22:18:41.239829+0000","last_became_peered":"2024-03-11T22:18:41.239829+0000","last_unstale":"2024-03-11T22:22:16.480360+0000","last_undegraded":"2024-03-11T22:22:16.480360+0000","last_fullsized":"2024-03-11T22:22:16.480360+0000","mapping_epoch":273,"log_start":"320'11100","ondisk_log_start":"320'11100","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4082,"log_dups_size":0,"ondisk_log_size":4082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26345472,"num_objects":402,"num_object_clones":0,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15182,"num_write_kb":498688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":555,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783539+0000","last_change":"2024-03-11T22:22:06.343563+0000","last_active":"2024-03-11T22:22:12.783539+0000","last_peered":"2024-03-11T22:22:12.783539+0000","last_clean":"2024-03-11T22:22:12.783539+0000","last_became_active":"2024-03-11T22:20:29.420497+0000","last_became_peered":"2024-03-11T22:20:29.420497+0000","last_unstale":"2024-03-11T22:22:12.783539+0000","last_undegraded":"2024-03-11T22:22:12.783539+0000","last_fullsized":"2024-03-11T22:22:12.783539+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:06.343277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:22:06.343277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:32.730443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"273'28","reported_seq":807,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:13.515894+0000","last_change":"2024-03-11T22:22:01.629086+0000","last_active":"2024-03-11T22:22:13.515894+0000","last_peered":"2024-03-11T22:22:13.515894+0000","last_clean":"2024-03-11T22:22:13.515894+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:22:13.515894+0000","last_undegraded":"2024-03-11T22:22:13.515894+0000","last_fullsized":"2024-03-11T22:22:13.515894+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:22:01.628978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:22:01.628978+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:18.259196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":581,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799586+0000","last_change":"2024-03-11T22:19:24.533661+0000","last_active":"2024-03-11T22:22:12.799586+0000","last_peered":"2024-03-11T22:22:12.799586+0000","last_clean":"2024-03-11T22:22:12.799586+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:22:12.799586+0000","last_undegraded":"2024-03-11T22:22:12.799586+0000","last_fullsized":"2024-03-11T22:22:12.799586+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:24.533532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:19:24.533532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":896,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795290+0000","last_change":"2024-03-11T22:20:32.294325+0000","last_active":"2024-03-11T22:22:12.795290+0000","last_peered":"2024-03-11T22:22:12.795290+0000","last_clean":"2024-03-11T22:22:12.795290+0000","last_became_active":"2024-03-11T22:20:32.293740+0000","last_became_peered":"2024-03-11T22:20:32.293740+0000","last_unstale":"2024-03-11T22:22:12.795290+0000","last_undegraded":"2024-03-11T22:22:12.795290+0000","last_fullsized":"2024-03-11T22:22:12.795290+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:44.952136+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:18:44.952136+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":569,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.787911+0000","last_change":"2024-03-11T22:18:22.688031+0000","last_active":"2024-03-11T22:22:12.787911+0000","last_peered":"2024-03-11T22:22:12.787911+0000","last_clean":"2024-03-11T22:22:12.787911+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:22:12.787911+0000","last_undegraded":"2024-03-11T22:22:12.787911+0000","last_fullsized":"2024-03-11T22:22:12.787911+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:22.687922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.687922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"334'15238","reported_seq":15368,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:15.513385+0000","last_change":"2024-03-11T22:18:41.260551+0000","last_active":"2024-03-11T22:22:15.513385+0000","last_peered":"2024-03-11T22:22:15.513385+0000","last_clean":"2024-03-11T22:22:15.513385+0000","last_became_active":"2024-03-11T22:18:41.260185+0000","last_became_peered":"2024-03-11T22:18:41.260185+0000","last_unstale":"2024-03-11T22:22:15.513385+0000","last_undegraded":"2024-03-11T22:22:15.513385+0000","last_fullsized":"2024-03-11T22:22:15.513385+0000","mapping_epoch":273,"log_start":"317'10700","ondisk_log_start":"317'10700","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4538,"log_dups_size":0,"ondisk_log_size":4538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31981568,"num_objects":488,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15238,"num_write_kb":503232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":597,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783094+0000","last_change":"2024-03-11T22:19:37.032204+0000","last_active":"2024-03-11T22:22:12.783094+0000","last_peered":"2024-03-11T22:22:12.783094+0000","last_clean":"2024-03-11T22:22:12.783094+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:22:12.783094+0000","last_undegraded":"2024-03-11T22:22:12.783094+0000","last_fullsized":"2024-03-11T22:22:12.783094+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.032058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.032058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"273'17","reported_seq":626,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783502+0000","last_change":"2024-03-11T22:19:34.486422+0000","last_active":"2024-03-11T22:22:12.783502+0000","last_peered":"2024-03-11T22:22:12.783502+0000","last_clean":"2024-03-11T22:22:12.783502+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:22:12.783502+0000","last_undegraded":"2024-03-11T22:22:12.783502+0000","last_fullsized":"2024-03-11T22:22:12.783502+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'17","last_scrub_stamp":"2024-03-11T22:19:34.486239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:34.486239+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":622,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.782125+0000","last_change":"2024-03-11T22:20:52.869557+0000","last_active":"2024-03-11T22:22:12.782125+0000","last_peered":"2024-03-11T22:22:12.782125+0000","last_clean":"2024-03-11T22:22:12.782125+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:22:12.782125+0000","last_undegraded":"2024-03-11T22:22:12.782125+0000","last_fullsized":"2024-03-11T22:22:12.782125+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:52.869444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:20:52.869444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":597,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783346+0000","last_change":"2024-03-11T22:19:46.587019+0000","last_active":"2024-03-11T22:22:12.783346+0000","last_peered":"2024-03-11T22:22:12.783346+0000","last_clean":"2024-03-11T22:22:12.783346+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:22:12.783346+0000","last_undegraded":"2024-03-11T22:22:12.783346+0000","last_fullsized":"2024-03-11T22:22:12.783346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:19:46.586446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":535,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799593+0000","last_change":"2024-03-11T22:22:10.824891+0000","last_active":"2024-03-11T22:22:12.799593+0000","last_peered":"2024-03-11T22:22:12.799593+0000","last_clean":"2024-03-11T22:22:12.799593+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:22:12.799593+0000","last_undegraded":"2024-03-11T22:22:12.799593+0000","last_fullsized":"2024-03-11T22:22:12.799593+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:10.824769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:22:10.824769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:16.273508+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":904,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795230+0000","last_change":"2024-03-11T22:20:32.295561+0000","last_active":"2024-03-11T22:22:12.795230+0000","last_peered":"2024-03-11T22:22:12.795230+0000","last_clean":"2024-03-11T22:22:12.795230+0000","last_became_active":"2024-03-11T22:20:32.295175+0000","last_became_peered":"2024-03-11T22:20:32.295175+0000","last_unstale":"2024-03-11T22:22:12.795230+0000","last_undegraded":"2024-03-11T22:22:12.795230+0000","last_fullsized":"2024-03-11T22:22:12.795230+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:25.392244+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:25.392244+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"334'15055","reported_seq":23221,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:16.865568+0000","last_change":"2024-03-11T22:21:36.588889+0000","last_active":"2024-03-11T22:22:16.865568+0000","last_peered":"2024-03-11T22:22:16.865568+0000","last_clean":"2024-03-11T22:22:16.865568+0000","last_became_active":"2024-03-11T22:21:36.509068+0000","last_became_peered":"2024-03-11T22:21:36.509068+0000","last_unstale":"2024-03-11T22:22:16.865568+0000","last_undegraded":"2024-03-11T22:22:16.865568+0000","last_fullsized":"2024-03-11T22:22:16.865568+0000","mapping_epoch":324,"log_start":"314'9700","ondisk_log_start":"314'9700","created":273,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":5355,"log_dups_size":0,"ondisk_log_size":5355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27983872,"num_objects":427,"num_object_clones":0,"num_object_copies":1281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15055,"num_write_kb":495424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4009,"num_bytes_recovered":262668288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":581,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795517+0000","last_change":"2024-03-11T22:21:56.639907+0000","last_active":"2024-03-11T22:22:12.795517+0000","last_peered":"2024-03-11T22:22:12.795517+0000","last_clean":"2024-03-11T22:22:12.795517+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:22:12.795517+0000","last_undegraded":"2024-03-11T22:22:12.795517+0000","last_fullsized":"2024-03-11T22:22:12.795517+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:56.639673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:21:56.639673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:15.634286+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"273'40","reported_seq":824,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.786455+0000","last_change":"2024-03-11T22:19:39.417690+0000","last_active":"2024-03-11T22:22:12.786455+0000","last_peered":"2024-03-11T22:22:12.786455+0000","last_clean":"2024-03-11T22:22:12.786455+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:22:12.786455+0000","last_undegraded":"2024-03-11T22:22:12.786455+0000","last_fullsized":"2024-03-11T22:22:12.786455+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'40","last_scrub_stamp":"2024-03-11T22:19:39.417411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:39.417411+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":348,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.796371+0000","last_change":"2024-03-11T22:21:53.574059+0000","last_active":"2024-03-11T22:22:12.796371+0000","last_peered":"2024-03-11T22:22:12.796371+0000","last_clean":"2024-03-11T22:22:12.796371+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:22:12.796371+0000","last_undegraded":"2024-03-11T22:22:12.796371+0000","last_fullsized":"2024-03-11T22:22:12.796371+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:53.573950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:21:53.573950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:13.389779+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":593,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784913+0000","last_change":"2024-03-11T22:18:36.684061+0000","last_active":"2024-03-11T22:22:12.784913+0000","last_peered":"2024-03-11T22:22:12.784913+0000","last_clean":"2024-03-11T22:22:12.784913+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:22:12.784913+0000","last_undegraded":"2024-03-11T22:22:12.784913+0000","last_fullsized":"2024-03-11T22:22:12.784913+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:18:36.683930+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:18:36.683930+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":573,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.797544+0000","last_change":"2024-03-11T22:20:29.434967+0000","last_active":"2024-03-11T22:22:12.797544+0000","last_peered":"2024-03-11T22:22:12.797544+0000","last_clean":"2024-03-11T22:22:12.797544+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:22:12.797544+0000","last_undegraded":"2024-03-11T22:22:12.797544+0000","last_fullsized":"2024-03-11T22:22:12.797544+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:15:32.518054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:15:32.518054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":886,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784803+0000","last_change":"2024-03-11T22:22:07.240481+0000","last_active":"2024-03-11T22:22:12.784803+0000","last_peered":"2024-03-11T22:22:12.784803+0000","last_clean":"2024-03-11T22:22:12.784803+0000","last_became_active":"2024-03-11T22:14:38.164859+0000","last_became_peered":"2024-03-11T22:14:38.164859+0000","last_unstale":"2024-03-11T22:22:12.784803+0000","last_undegraded":"2024-03-11T22:22:12.784803+0000","last_fullsized":"2024-03-11T22:22:12.784803+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:22:07.240358+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:22:07.240358+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:36.193354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"334'14524","reported_seq":14653,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:15.526459+0000","last_change":"2024-03-11T22:18:41.233217+0000","last_active":"2024-03-11T22:22:15.526459+0000","last_peered":"2024-03-11T22:22:15.526459+0000","last_clean":"2024-03-11T22:22:15.526459+0000","last_became_active":"2024-03-11T22:18:41.232989+0000","last_became_peered":"2024-03-11T22:18:41.232989+0000","last_unstale":"2024-03-11T22:22:15.526459+0000","last_undegraded":"2024-03-11T22:22:15.526459+0000","last_fullsized":"2024-03-11T22:22:15.526459+0000","mapping_epoch":273,"log_start":"317'10000","ondisk_log_start":"317'10000","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4524,"log_dups_size":0,"ondisk_log_size":4524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":460,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14524,"num_write_kb":479488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"273'15","reported_seq":618,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.796707+0000","last_change":"2024-03-11T22:19:32.486126+0000","last_active":"2024-03-11T22:22:12.796707+0000","last_peered":"2024-03-11T22:22:12.796707+0000","last_clean":"2024-03-11T22:22:12.796707+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:22:12.796707+0000","last_undegraded":"2024-03-11T22:22:12.796707+0000","last_fullsized":"2024-03-11T22:22:12.796707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:19:32.485571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.485571+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":581,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784165+0000","last_change":"2024-03-11T22:18:40.745227+0000","last_active":"2024-03-11T22:22:12.784165+0000","last_peered":"2024-03-11T22:22:12.784165+0000","last_clean":"2024-03-11T22:22:12.784165+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:22:12.784165+0000","last_undegraded":"2024-03-11T22:22:12.784165+0000","last_fullsized":"2024-03-11T22:22:12.784165+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.745151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.745151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":597,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.782448+0000","last_change":"2024-03-11T22:19:42.927329+0000","last_active":"2024-03-11T22:22:12.782448+0000","last_peered":"2024-03-11T22:22:12.782448+0000","last_clean":"2024-03-11T22:22:12.782448+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:22:12.782448+0000","last_undegraded":"2024-03-11T22:22:12.782448+0000","last_fullsized":"2024-03-11T22:22:12.782448+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:42.926178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:19:42.926178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":589,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799507+0000","last_change":"2024-03-11T22:22:05.802923+0000","last_active":"2024-03-11T22:22:12.799507+0000","last_peered":"2024-03-11T22:22:12.799507+0000","last_clean":"2024-03-11T22:22:12.799507+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:22:12.799507+0000","last_undegraded":"2024-03-11T22:22:12.799507+0000","last_fullsized":"2024-03-11T22:22:12.799507+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:05.802256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:05.802256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:29.491415+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"93'130","reported_seq":731,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.797158+0000","last_change":"2024-03-11T22:20:32.293722+0000","last_active":"2024-03-11T22:22:12.797158+0000","last_peered":"2024-03-11T22:22:12.797158+0000","last_clean":"2024-03-11T22:22:12.797158+0000","last_became_active":"2024-03-11T22:20:32.292676+0000","last_became_peered":"2024-03-11T22:20:32.292676+0000","last_unstale":"2024-03-11T22:22:12.797158+0000","last_undegraded":"2024-03-11T22:22:12.797158+0000","last_fullsized":"2024-03-11T22:22:12.797158+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:22.939363+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:22.939363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.7","version":"334'15097","reported_seq":28529,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:15.483511+0000","last_change":"2024-03-11T22:20:55.630922+0000","last_active":"2024-03-11T22:22:15.483511+0000","last_peered":"2024-03-11T22:22:15.483511+0000","last_clean":"2024-03-11T22:22:15.483511+0000","last_became_active":"2024-03-11T22:20:55.617058+0000","last_became_peered":"2024-03-11T22:20:55.617058+0000","last_unstale":"2024-03-11T22:22:15.483511+0000","last_undegraded":"2024-03-11T22:22:15.483511+0000","last_fullsized":"2024-03-11T22:22:15.483511+0000","mapping_epoch":315,"log_start":"317'10600","ondisk_log_start":"317'10600","created":273,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":511,"num_object_clones":0,"num_object_copies":1533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15097,"num_write_kb":499456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6642,"num_bytes_recovered":435290112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"273'35","reported_seq":708,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799073+0000","last_change":"2024-03-11T22:22:02.687141+0000","last_active":"2024-03-11T22:22:12.799073+0000","last_peered":"2024-03-11T22:22:12.799073+0000","last_clean":"2024-03-11T22:22:12.799073+0000","last_became_active":"2024-03-11T22:20:29.418041+0000","last_became_peered":"2024-03-11T22:20:29.418041+0000","last_unstale":"2024-03-11T22:22:12.799073+0000","last_undegraded":"2024-03-11T22:22:12.799073+0000","last_fullsized":"2024-03-11T22:22:12.799073+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'35","last_scrub_stamp":"2024-03-11T22:22:02.687037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:22:02.687037+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:30.382198+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":532,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784530+0000","last_change":"2024-03-11T22:20:29.414029+0000","last_active":"2024-03-11T22:22:12.784530+0000","last_peered":"2024-03-11T22:22:12.784530+0000","last_clean":"2024-03-11T22:22:12.784530+0000","last_became_active":"2024-03-11T22:20:29.413664+0000","last_became_peered":"2024-03-11T22:20:29.413664+0000","last_unstale":"2024-03-11T22:22:12.784530+0000","last_undegraded":"2024-03-11T22:22:12.784530+0000","last_fullsized":"2024-03-11T22:22:12.784530+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:32.682157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:32.682157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":597,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.782811+0000","last_change":"2024-03-11T22:20:39.805771+0000","last_active":"2024-03-11T22:22:12.782811+0000","last_peered":"2024-03-11T22:22:12.782811+0000","last_clean":"2024-03-11T22:22:12.782811+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:22:12.782811+0000","last_undegraded":"2024-03-11T22:22:12.782811+0000","last_fullsized":"2024-03-11T22:22:12.782811+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:39.805647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:20:39.805647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":555,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.782213+0000","last_change":"2024-03-11T22:21:53.139945+0000","last_active":"2024-03-11T22:22:12.782213+0000","last_peered":"2024-03-11T22:22:12.782213+0000","last_clean":"2024-03-11T22:22:12.782213+0000","last_became_active":"2024-03-11T22:20:29.415565+0000","last_became_peered":"2024-03-11T22:20:29.415565+0000","last_unstale":"2024-03-11T22:22:12.782213+0000","last_undegraded":"2024-03-11T22:22:12.782213+0000","last_fullsized":"2024-03-11T22:22:12.782213+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:53.139082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:20.931122+0000","last_clean_scrub_stamp":"2024-03-11T22:21:53.139082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:07.326264+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"93'130","reported_seq":782,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799628+0000","last_change":"2024-03-11T22:20:36.022151+0000","last_active":"2024-03-11T22:22:12.799628+0000","last_peered":"2024-03-11T22:22:12.799628+0000","last_clean":"2024-03-11T22:22:12.799628+0000","last_became_active":"2024-03-11T22:14:43.195433+0000","last_became_peered":"2024-03-11T22:14:43.195433+0000","last_unstale":"2024-03-11T22:22:12.799628+0000","last_undegraded":"2024-03-11T22:22:12.799628+0000","last_fullsized":"2024-03-11T22:22:12.799628+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:36.022039+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:20:36.022039+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"334'14986","reported_seq":15119,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:16.418012+0000","last_change":"2024-03-11T22:18:41.237962+0000","last_active":"2024-03-11T22:22:16.418012+0000","last_peered":"2024-03-11T22:22:16.418012+0000","last_clean":"2024-03-11T22:22:16.418012+0000","last_became_active":"2024-03-11T22:18:41.237565+0000","last_became_peered":"2024-03-11T22:18:41.237565+0000","last_unstale":"2024-03-11T22:22:16.418012+0000","last_undegraded":"2024-03-11T22:22:16.418012+0000","last_fullsized":"2024-03-11T22:22:16.418012+0000","mapping_epoch":273,"log_start":"317'10400","ondisk_log_start":"317'10400","created":273,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:18:40.379495+0000","objects_scrubbed":0,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28835840,"num_objects":440,"num_object_clones":0,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14986,"num_write_kb":493632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":597,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.781930+0000","last_change":"2024-03-11T22:20:47.845641+0000","last_active":"2024-03-11T22:22:12.781930+0000","last_peered":"2024-03-11T22:22:12.781930+0000","last_clean":"2024-03-11T22:22:12.781930+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:22:12.781930+0000","last_undegraded":"2024-03-11T22:22:12.781930+0000","last_fullsized":"2024-03-11T22:22:12.781930+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:47.845156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:20:47.845156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"273'18","reported_seq":405,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.785143+0000","last_change":"2024-03-11T22:19:38.795869+0000","last_active":"2024-03-11T22:22:12.785143+0000","last_peered":"2024-03-11T22:22:12.785143+0000","last_clean":"2024-03-11T22:22:12.785143+0000","last_became_active":"2024-03-11T22:14:52.247400+0000","last_became_peered":"2024-03-11T22:14:52.247400+0000","last_unstale":"2024-03-11T22:22:12.785143+0000","last_undegraded":"2024-03-11T22:22:12.785143+0000","last_fullsized":"2024-03-11T22:22:12.785143+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:19:38.795650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:38.795650+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":371,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.782991+0000","last_change":"2024-03-11T22:20:29.417391+0000","last_active":"2024-03-11T22:22:12.782991+0000","last_peered":"2024-03-11T22:22:12.782991+0000","last_clean":"2024-03-11T22:22:12.782991+0000","last_became_active":"2024-03-11T22:20:29.416729+0000","last_became_peered":"2024-03-11T22:20:29.416729+0000","last_unstale":"2024-03-11T22:22:12.782991+0000","last_undegraded":"2024-03-11T22:22:12.782991+0000","last_fullsized":"2024-03-11T22:22:12.782991+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:37.439756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:37.439756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":583,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.800658+0000","last_change":"2024-03-11T22:20:29.470787+0000","last_active":"2024-03-11T22:22:12.800658+0000","last_peered":"2024-03-11T22:22:12.800658+0000","last_clean":"2024-03-11T22:22:12.800658+0000","last_became_active":"2024-03-11T22:20:29.438413+0000","last_became_peered":"2024-03-11T22:20:29.438413+0000","last_unstale":"2024-03-11T22:22:12.800658+0000","last_undegraded":"2024-03-11T22:22:12.800658+0000","last_fullsized":"2024-03-11T22:22:12.800658+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:15:40.678629+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:15:40.678629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"93'130","reported_seq":907,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.800089+0000","last_change":"2024-03-11T22:18:21.956432+0000","last_active":"2024-03-11T22:22:12.800089+0000","last_peered":"2024-03-11T22:22:12.800089+0000","last_clean":"2024-03-11T22:22:12.800089+0000","last_became_active":"2024-03-11T22:17:57.246241+0000","last_became_peered":"2024-03-11T22:17:57.246241+0000","last_unstale":"2024-03-11T22:22:12.800089+0000","last_undegraded":"2024-03-11T22:22:12.800089+0000","last_fullsized":"2024-03-11T22:22:12.800089+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:21.956334+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:21.956334+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"334'15473","reported_seq":24518,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:16.900580+0000","last_change":"2024-03-11T22:22:12.091164+0000","last_active":"2024-03-11T22:22:16.900580+0000","last_peered":"2024-03-11T22:22:16.900580+0000","last_clean":"2024-03-11T22:22:16.900580+0000","last_became_active":"2024-03-11T22:21:51.609572+0000","last_became_peered":"2024-03-11T22:21:51.609572+0000","last_unstale":"2024-03-11T22:22:16.900580+0000","last_undegraded":"2024-03-11T22:22:16.900580+0000","last_fullsized":"2024-03-11T22:22:16.900580+0000","mapping_epoch":331,"log_start":"316'10200","ondisk_log_start":"316'10200","created":273,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'15064","last_scrub_stamp":"2024-03-11T22:22:12.090890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:40.379495+0000","last_clean_scrub_stamp":"2024-03-11T22:22:12.090890+0000","objects_scrubbed":829,"log_size":5273,"log_dups_size":0,"ondisk_log_size":5273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:18.107414+0000","scrub_duration":335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26673152,"num_objects":407,"num_object_clones":0,"num_object_copies":1221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15473,"num_write_kb":508160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4437,"num_bytes_recovered":290783232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"310'11","reported_seq":723,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:13.515880+0000","last_change":"2024-03-11T22:20:57.698062+0000","last_active":"2024-03-11T22:22:13.515880+0000","last_peered":"2024-03-11T22:22:13.515880+0000","last_clean":"2024-03-11T22:22:13.515880+0000","last_became_active":"2024-03-11T22:20:29.418355+0000","last_became_peered":"2024-03-11T22:20:29.418355+0000","last_unstale":"2024-03-11T22:22:13.515880+0000","last_undegraded":"2024-03-11T22:22:13.515880+0000","last_fullsized":"2024-03-11T22:22:13.515880+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:20:57.697970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:57.697970+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:27.002078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"273'34","reported_seq":679,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783298+0000","last_change":"2024-03-11T22:20:29.431072+0000","last_active":"2024-03-11T22:22:12.783298+0000","last_peered":"2024-03-11T22:22:12.783298+0000","last_clean":"2024-03-11T22:22:12.783298+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:22:12.783298+0000","last_undegraded":"2024-03-11T22:22:12.783298+0000","last_fullsized":"2024-03-11T22:22:12.783298+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:19:46.888741+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:19:46.888741+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":381,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.797942+0000","last_change":"2024-03-11T22:20:29.440395+0000","last_active":"2024-03-11T22:22:12.797942+0000","last_peered":"2024-03-11T22:22:12.797942+0000","last_clean":"2024-03-11T22:22:12.797942+0000","last_became_active":"2024-03-11T22:20:29.415226+0000","last_became_peered":"2024-03-11T22:20:29.415226+0000","last_unstale":"2024-03-11T22:22:12.797942+0000","last_undegraded":"2024-03-11T22:22:12.797942+0000","last_fullsized":"2024-03-11T22:22:12.797942+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:20:26.335219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:26.335219+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":569,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.785182+0000","last_change":"2024-03-11T22:19:33.760079+0000","last_active":"2024-03-11T22:22:12.785182+0000","last_peered":"2024-03-11T22:22:12.785182+0000","last_clean":"2024-03-11T22:22:12.785182+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:22:12.785182+0000","last_undegraded":"2024-03-11T22:22:12.785182+0000","last_fullsized":"2024-03-11T22:22:12.785182+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:33.759955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:33.759955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"93'130","reported_seq":924,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.791577+0000","last_change":"2024-03-11T22:20:32.300771+0000","last_active":"2024-03-11T22:22:12.791577+0000","last_peered":"2024-03-11T22:22:12.791577+0000","last_clean":"2024-03-11T22:22:12.791577+0000","last_became_active":"2024-03-11T22:20:32.295606+0000","last_became_peered":"2024-03-11T22:20:32.295606+0000","last_unstale":"2024-03-11T22:22:12.791577+0000","last_undegraded":"2024-03-11T22:22:12.791577+0000","last_fullsized":"2024-03-11T22:22:12.791577+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:41.158328+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:18:41.158328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"273'12","reported_seq":614,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.806350+0000","last_change":"2024-03-11T22:20:29.483060+0000","last_active":"2024-03-11T22:22:12.806350+0000","last_peered":"2024-03-11T22:22:12.806350+0000","last_clean":"2024-03-11T22:22:12.806350+0000","last_became_active":"2024-03-11T22:20:29.441599+0000","last_became_peered":"2024-03-11T22:20:29.441599+0000","last_unstale":"2024-03-11T22:22:12.806350+0000","last_undegraded":"2024-03-11T22:22:12.806350+0000","last_fullsized":"2024-03-11T22:22:12.806350+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'6","last_scrub_stamp":"2024-03-11T22:15:27.448002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:27.448002+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":596,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.800780+0000","last_change":"2024-03-11T22:20:42.845142+0000","last_active":"2024-03-11T22:22:12.800780+0000","last_peered":"2024-03-11T22:22:12.800780+0000","last_clean":"2024-03-11T22:22:12.800780+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:22:12.800780+0000","last_undegraded":"2024-03-11T22:22:12.800780+0000","last_fullsized":"2024-03-11T22:22:12.800780+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:42.844876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:20:42.844876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":580,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.785480+0000","last_change":"2024-03-11T22:20:54.635534+0000","last_active":"2024-03-11T22:22:12.785480+0000","last_peered":"2024-03-11T22:22:12.785480+0000","last_clean":"2024-03-11T22:22:12.785480+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:22:12.785480+0000","last_undegraded":"2024-03-11T22:22:12.785480+0000","last_fullsized":"2024-03-11T22:22:12.785480+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:20:54.635309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:15.693951+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":526,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799653+0000","last_change":"2024-03-11T22:20:32.993441+0000","last_active":"2024-03-11T22:22:12.799653+0000","last_peered":"2024-03-11T22:22:12.799653+0000","last_clean":"2024-03-11T22:22:12.799653+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:22:12.799653+0000","last_undegraded":"2024-03-11T22:22:12.799653+0000","last_fullsized":"2024-03-11T22:22:12.799653+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:32.993301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:20:32.993301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"93'130","reported_seq":895,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:13.277562+0000","last_change":"2024-03-11T22:22:13.277611+0000","last_active":"2024-03-11T22:22:13.277562+0000","last_peered":"2024-03-11T22:22:13.277562+0000","last_clean":"2024-03-11T22:22:13.277562+0000","last_became_active":"2024-03-11T22:14:43.194413+0000","last_became_peered":"2024-03-11T22:14:43.194413+0000","last_unstale":"2024-03-11T22:22:13.277562+0000","last_undegraded":"2024-03-11T22:22:13.277562+0000","last_fullsized":"2024-03-11T22:22:13.277562+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:22:13.277465+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:22:13.277465+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:27.867252+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"93'130","reported_seq":894,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783082+0000","last_change":"2024-03-11T22:19:27.413677+0000","last_active":"2024-03-11T22:22:12.783082+0000","last_peered":"2024-03-11T22:22:12.783082+0000","last_clean":"2024-03-11T22:22:12.783082+0000","last_became_active":"2024-03-11T22:14:43.198216+0000","last_became_peered":"2024-03-11T22:14:43.198216+0000","last_unstale":"2024-03-11T22:22:12.783082+0000","last_undegraded":"2024-03-11T22:22:12.783082+0000","last_fullsized":"2024-03-11T22:22:12.783082+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:27.413560+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:27.413560+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"273'19","reported_seq":645,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795438+0000","last_change":"2024-03-11T22:22:07.767024+0000","last_active":"2024-03-11T22:22:12.795438+0000","last_peered":"2024-03-11T22:22:12.795438+0000","last_clean":"2024-03-11T22:22:12.795438+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:22:12.795438+0000","last_undegraded":"2024-03-11T22:22:12.795438+0000","last_fullsized":"2024-03-11T22:22:12.795438+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:22:07.766876+0000","last_deep_scrub":"93'13","last_deep_scrub_stamp":"2024-03-11T22:10:04.123429+0000","last_clean_scrub_stamp":"2024-03-11T22:22:07.766876+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:11.008237+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":359,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.798875+0000","last_change":"2024-03-11T22:20:46.807212+0000","last_active":"2024-03-11T22:22:12.798875+0000","last_peered":"2024-03-11T22:22:12.798875+0000","last_clean":"2024-03-11T22:22:12.798875+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:22:12.798875+0000","last_undegraded":"2024-03-11T22:22:12.798875+0000","last_fullsized":"2024-03-11T22:22:12.798875+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:46.807096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:46.807096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":584,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.806270+0000","last_change":"2024-03-11T22:20:29.433304+0000","last_active":"2024-03-11T22:22:12.806270+0000","last_peered":"2024-03-11T22:22:12.806270+0000","last_clean":"2024-03-11T22:22:12.806270+0000","last_became_active":"2024-03-11T22:20:29.433005+0000","last_became_peered":"2024-03-11T22:20:29.433005+0000","last_unstale":"2024-03-11T22:22:12.806270+0000","last_undegraded":"2024-03-11T22:22:12.806270+0000","last_fullsized":"2024-03-11T22:22:12.806270+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:43.544959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:43.544959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":600,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783445+0000","last_change":"2024-03-11T22:22:01.193655+0000","last_active":"2024-03-11T22:22:12.783445+0000","last_peered":"2024-03-11T22:22:12.783445+0000","last_clean":"2024-03-11T22:22:12.783445+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:22:12.783445+0000","last_undegraded":"2024-03-11T22:22:12.783445+0000","last_fullsized":"2024-03-11T22:22:12.783445+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:01.193579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:22:01.193579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:10.981326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"273'19","reported_seq":613,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.785278+0000","last_change":"2024-03-11T22:19:29.865779+0000","last_active":"2024-03-11T22:22:12.785278+0000","last_peered":"2024-03-11T22:22:12.785278+0000","last_clean":"2024-03-11T22:22:12.785278+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:22:12.785278+0000","last_undegraded":"2024-03-11T22:22:12.785278+0000","last_fullsized":"2024-03-11T22:22:12.785278+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:19:29.865528+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.865528+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":600,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795089+0000","last_change":"2024-03-11T22:22:03.735421+0000","last_active":"2024-03-11T22:22:12.795089+0000","last_peered":"2024-03-11T22:22:12.795089+0000","last_clean":"2024-03-11T22:22:12.795089+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:22:12.795089+0000","last_undegraded":"2024-03-11T22:22:12.795089+0000","last_fullsized":"2024-03-11T22:22:12.795089+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:03.735301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:22:03.735301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:09.195603+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'11","reported_seq":780,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.807710+0000","last_change":"2024-03-11T22:20:29.445996+0000","last_active":"2024-03-11T22:22:12.807710+0000","last_peered":"2024-03-11T22:22:12.807710+0000","last_clean":"2024-03-11T22:22:12.807710+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:22:12.807710+0000","last_undegraded":"2024-03-11T22:22:12.807710+0000","last_fullsized":"2024-03-11T22:22:12.807710+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'10","last_scrub_stamp":"2024-03-11T22:19:32.461683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:32.461683+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":363,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.787977+0000","last_change":"2024-03-11T22:20:58.956323+0000","last_active":"2024-03-11T22:22:12.787977+0000","last_peered":"2024-03-11T22:22:12.787977+0000","last_clean":"2024-03-11T22:22:12.787977+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:22:12.787977+0000","last_undegraded":"2024-03-11T22:22:12.787977+0000","last_fullsized":"2024-03-11T22:22:12.787977+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:58.956196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:20:58.956196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:23.295311+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"93'130","reported_seq":880,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783534+0000","last_change":"2024-03-11T22:18:53.739248+0000","last_active":"2024-03-11T22:22:12.783534+0000","last_peered":"2024-03-11T22:22:12.783534+0000","last_clean":"2024-03-11T22:22:12.783534+0000","last_became_active":"2024-03-11T22:14:43.194350+0000","last_became_peered":"2024-03-11T22:14:43.194350+0000","last_unstale":"2024-03-11T22:22:12.783534+0000","last_undegraded":"2024-03-11T22:22:12.783534+0000","last_fullsized":"2024-03-11T22:22:12.783534+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:18:53.738316+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:18:53.738316+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"273'20","reported_seq":616,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.806730+0000","last_change":"2024-03-11T22:16:16.055622+0000","last_active":"2024-03-11T22:22:12.806730+0000","last_peered":"2024-03-11T22:22:12.806730+0000","last_clean":"2024-03-11T22:22:12.806730+0000","last_became_active":"2024-03-11T22:16:15.815814+0000","last_became_peered":"2024-03-11T22:16:15.815814+0000","last_unstale":"2024-03-11T22:22:12.806730+0000","last_undegraded":"2024-03-11T22:22:12.806730+0000","last_fullsized":"2024-03-11T22:22:12.806730+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-11T22:15:29.460297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:15:29.460297+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":379,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.782721+0000","last_change":"2024-03-11T22:20:31.797304+0000","last_active":"2024-03-11T22:22:12.782721+0000","last_peered":"2024-03-11T22:22:12.782721+0000","last_clean":"2024-03-11T22:22:12.782721+0000","last_became_active":"2024-03-11T22:20:29.415019+0000","last_became_peered":"2024-03-11T22:20:29.415019+0000","last_unstale":"2024-03-11T22:22:12.782721+0000","last_undegraded":"2024-03-11T22:22:12.782721+0000","last_fullsized":"2024-03-11T22:22:12.782721+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:31.796712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:31.796712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":753,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784154+0000","last_change":"2024-03-11T22:19:42.854028+0000","last_active":"2024-03-11T22:22:12.784154+0000","last_peered":"2024-03-11T22:22:12.784154+0000","last_clean":"2024-03-11T22:22:12.784154+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:22:12.784154+0000","last_undegraded":"2024-03-11T22:22:12.784154+0000","last_fullsized":"2024-03-11T22:22:12.784154+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:19:42.853428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:19:42.853428+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":580,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795092+0000","last_change":"2024-03-11T22:22:01.690944+0000","last_active":"2024-03-11T22:22:12.795092+0000","last_peered":"2024-03-11T22:22:12.795092+0000","last_clean":"2024-03-11T22:22:12.795092+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:22:12.795092+0000","last_undegraded":"2024-03-11T22:22:12.795092+0000","last_fullsized":"2024-03-11T22:22:12.795092+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:01.690694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:01.690694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:27.427919+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"93'130","reported_seq":836,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795026+0000","last_change":"2024-03-11T22:19:29.435235+0000","last_active":"2024-03-11T22:22:12.795026+0000","last_peered":"2024-03-11T22:22:12.795026+0000","last_clean":"2024-03-11T22:22:12.795026+0000","last_became_active":"2024-03-11T22:14:43.191297+0000","last_became_peered":"2024-03-11T22:14:43.191297+0000","last_unstale":"2024-03-11T22:22:12.795026+0000","last_undegraded":"2024-03-11T22:22:12.795026+0000","last_fullsized":"2024-03-11T22:22:12.795026+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:19:29.434647+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:19:29.434647+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"273'19","reported_seq":405,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799224+0000","last_change":"2024-03-11T22:20:50.729986+0000","last_active":"2024-03-11T22:22:12.799224+0000","last_peered":"2024-03-11T22:22:12.799224+0000","last_clean":"2024-03-11T22:22:12.799224+0000","last_became_active":"2024-03-11T22:14:52.239810+0000","last_became_peered":"2024-03-11T22:14:52.239810+0000","last_unstale":"2024-03-11T22:22:12.799224+0000","last_undegraded":"2024-03-11T22:22:12.799224+0000","last_fullsized":"2024-03-11T22:22:12.799224+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:20:50.729848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:50.729848+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":384,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799190+0000","last_change":"2024-03-11T22:20:47.791284+0000","last_active":"2024-03-11T22:22:12.799190+0000","last_peered":"2024-03-11T22:22:12.799190+0000","last_clean":"2024-03-11T22:22:12.799190+0000","last_became_active":"2024-03-11T22:20:29.420867+0000","last_became_peered":"2024-03-11T22:20:29.420867+0000","last_unstale":"2024-03-11T22:22:12.799190+0000","last_undegraded":"2024-03-11T22:22:12.799190+0000","last_fullsized":"2024-03-11T22:22:12.799190+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:20:47.791128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:20:47.791128+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":588,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.785719+0000","last_change":"2024-03-11T22:21:51.817381+0000","last_active":"2024-03-11T22:22:12.785719+0000","last_peered":"2024-03-11T22:22:12.785719+0000","last_clean":"2024-03-11T22:22:12.785719+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:22:12.785719+0000","last_undegraded":"2024-03-11T22:22:12.785719+0000","last_fullsized":"2024-03-11T22:22:12.785719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:51.817265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:21:51.817265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:08.345413+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":560,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.798259+0000","last_change":"2024-03-11T22:18:29.014246+0000","last_active":"2024-03-11T22:22:12.798259+0000","last_peered":"2024-03-11T22:22:12.798259+0000","last_clean":"2024-03-11T22:22:12.798259+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:22:12.798259+0000","last_undegraded":"2024-03-11T22:22:12.798259+0000","last_fullsized":"2024-03-11T22:22:12.798259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:29.014149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:29.014149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":621,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.785912+0000","last_change":"2024-03-11T22:21:58.841578+0000","last_active":"2024-03-11T22:22:12.785912+0000","last_peered":"2024-03-11T22:22:12.785912+0000","last_clean":"2024-03-11T22:22:12.785912+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:22:12.785912+0000","last_undegraded":"2024-03-11T22:22:12.785912+0000","last_fullsized":"2024-03-11T22:22:12.785912+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:21:58.841451+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:21:58.841451+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:59.100999+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":523,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.796788+0000","last_change":"2024-03-11T22:20:29.423108+0000","last_active":"2024-03-11T22:22:12.796788+0000","last_peered":"2024-03-11T22:22:12.796788+0000","last_clean":"2024-03-11T22:22:12.796788+0000","last_became_active":"2024-03-11T22:20:29.422747+0000","last_became_peered":"2024-03-11T22:20:29.422747+0000","last_unstale":"2024-03-11T22:22:12.796788+0000","last_undegraded":"2024-03-11T22:22:12.796788+0000","last_fullsized":"2024-03-11T22:22:12.796788+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:44.563614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:44.563614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":572,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.800514+0000","last_change":"2024-03-11T22:19:39.445467+0000","last_active":"2024-03-11T22:22:12.800514+0000","last_peered":"2024-03-11T22:22:12.800514+0000","last_clean":"2024-03-11T22:22:12.800514+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:22:12.800514+0000","last_undegraded":"2024-03-11T22:22:12.800514+0000","last_fullsized":"2024-03-11T22:22:12.800514+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:39.445349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:19:39.445349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":568,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.794994+0000","last_change":"2024-03-11T22:19:26.459086+0000","last_active":"2024-03-11T22:22:12.794994+0000","last_peered":"2024-03-11T22:22:12.794994+0000","last_clean":"2024-03-11T22:22:12.794994+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:22:12.794994+0000","last_undegraded":"2024-03-11T22:22:12.794994+0000","last_fullsized":"2024-03-11T22:22:12.794994+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:26.458937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:26.458937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"93'130","reported_seq":897,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.782935+0000","last_change":"2024-03-11T22:20:49.901428+0000","last_active":"2024-03-11T22:22:12.782935+0000","last_peered":"2024-03-11T22:22:12.782935+0000","last_clean":"2024-03-11T22:22:12.782935+0000","last_became_active":"2024-03-11T22:14:43.194955+0000","last_became_peered":"2024-03-11T22:14:43.194955+0000","last_unstale":"2024-03-11T22:22:12.782935+0000","last_undegraded":"2024-03-11T22:22:12.782935+0000","last_fullsized":"2024-03-11T22:22:12.782935+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:20:49.901270+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:18:43.722467+0000","last_clean_scrub_stamp":"2024-03-11T22:20:49.901270+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"274'20","reported_seq":636,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.798860+0000","last_change":"2024-03-11T22:20:29.487125+0000","last_active":"2024-03-11T22:22:12.798860+0000","last_peered":"2024-03-11T22:22:12.798860+0000","last_clean":"2024-03-11T22:22:12.798860+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:22:12.798860+0000","last_undegraded":"2024-03-11T22:22:12.798860+0000","last_fullsized":"2024-03-11T22:22:12.798860+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'20","last_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:18:42.135888+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":600,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.785388+0000","last_change":"2024-03-11T22:21:56.391664+0000","last_active":"2024-03-11T22:22:12.785388+0000","last_peered":"2024-03-11T22:22:12.785388+0000","last_clean":"2024-03-11T22:22:12.785388+0000","last_became_active":"2024-03-11T22:20:29.416792+0000","last_became_peered":"2024-03-11T22:20:29.416792+0000","last_unstale":"2024-03-11T22:22:12.785388+0000","last_undegraded":"2024-03-11T22:22:12.785388+0000","last_fullsized":"2024-03-11T22:22:12.785388+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:56.391548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:21:56.391548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:14.928545+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":358,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.800773+0000","last_change":"2024-03-11T22:20:40.885587+0000","last_active":"2024-03-11T22:22:12.800773+0000","last_peered":"2024-03-11T22:22:12.800773+0000","last_clean":"2024-03-11T22:22:12.800773+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:22:12.800773+0000","last_undegraded":"2024-03-11T22:22:12.800773+0000","last_fullsized":"2024-03-11T22:22:12.800773+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:40.885360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:20:40.885360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":568,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.800689+0000","last_change":"2024-03-11T22:21:55.069970+0000","last_active":"2024-03-11T22:22:12.800689+0000","last_peered":"2024-03-11T22:22:12.800689+0000","last_clean":"2024-03-11T22:22:12.800689+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:22:12.800689+0000","last_undegraded":"2024-03-11T22:22:12.800689+0000","last_fullsized":"2024-03-11T22:22:12.800689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:55.069595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:21:55.069595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:22:56.389048+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"273'8","reported_seq":600,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799740+0000","last_change":"2024-03-11T22:18:53.157176+0000","last_active":"2024-03-11T22:22:12.799740+0000","last_peered":"2024-03-11T22:22:12.799740+0000","last_clean":"2024-03-11T22:22:12.799740+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:22:12.799740+0000","last_undegraded":"2024-03-11T22:22:12.799740+0000","last_fullsized":"2024-03-11T22:22:12.799740+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'8","last_scrub_stamp":"2024-03-11T22:18:53.157080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:53.157080+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":580,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.798054+0000","last_change":"2024-03-11T22:19:45.501704+0000","last_active":"2024-03-11T22:22:12.798054+0000","last_peered":"2024-03-11T22:22:12.798054+0000","last_clean":"2024-03-11T22:22:12.798054+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:22:12.798054+0000","last_undegraded":"2024-03-11T22:22:12.798054+0000","last_fullsized":"2024-03-11T22:22:12.798054+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.501593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.501593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":166,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.798773+0000","last_change":"2024-03-11T22:20:30.047479+0000","last_active":"2024-03-11T22:22:12.798773+0000","last_peered":"2024-03-11T22:22:12.798773+0000","last_clean":"2024-03-11T22:22:12.798773+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:22:12.798773+0000","last_undegraded":"2024-03-11T22:22:12.798773+0000","last_fullsized":"2024-03-11T22:22:12.798773+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:30.047368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:30.047368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":543,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:15.168944+0000","last_change":"2024-03-11T22:22:15.168944+0000","last_active":"2024-03-11T22:22:15.168944+0000","last_peered":"2024-03-11T22:22:15.168944+0000","last_clean":"2024-03-11T22:22:15.168944+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:22:15.168944+0000","last_undegraded":"2024-03-11T22:22:15.168944+0000","last_fullsized":"2024-03-11T22:22:15.168944+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:15.168450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:15.168450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:33.481666+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"273'19","reported_seq":624,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799666+0000","last_change":"2024-03-11T22:20:51.762478+0000","last_active":"2024-03-11T22:22:12.799666+0000","last_peered":"2024-03-11T22:22:12.799666+0000","last_clean":"2024-03-11T22:22:12.799666+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:22:12.799666+0000","last_undegraded":"2024-03-11T22:22:12.799666+0000","last_fullsized":"2024-03-11T22:22:12.799666+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:20:51.762318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:51.762318+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":580,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799644+0000","last_change":"2024-03-11T22:19:25.669161+0000","last_active":"2024-03-11T22:22:12.799644+0000","last_peered":"2024-03-11T22:22:12.799644+0000","last_clean":"2024-03-11T22:22:12.799644+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:22:12.799644+0000","last_undegraded":"2024-03-11T22:22:12.799644+0000","last_fullsized":"2024-03-11T22:22:12.799644+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:25.668878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:25.668878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":581,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795603+0000","last_change":"2024-03-11T22:20:27.333574+0000","last_active":"2024-03-11T22:22:12.795603+0000","last_peered":"2024-03-11T22:22:12.795603+0000","last_clean":"2024-03-11T22:22:12.795603+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:22:12.795603+0000","last_undegraded":"2024-03-11T22:22:12.795603+0000","last_fullsized":"2024-03-11T22:22:12.795603+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:27.333450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:27.333450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":584,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.792087+0000","last_change":"2024-03-11T22:19:45.686859+0000","last_active":"2024-03-11T22:22:12.792087+0000","last_peered":"2024-03-11T22:22:12.792087+0000","last_clean":"2024-03-11T22:22:12.792087+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:22:12.792087+0000","last_undegraded":"2024-03-11T22:22:12.792087+0000","last_fullsized":"2024-03-11T22:22:12.792087+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:45.686738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:45.686738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"273'20","reported_seq":643,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795780+0000","last_change":"2024-03-11T22:20:29.460876+0000","last_active":"2024-03-11T22:22:12.795780+0000","last_peered":"2024-03-11T22:22:12.795780+0000","last_clean":"2024-03-11T22:22:12.795780+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:22:12.795780+0000","last_undegraded":"2024-03-11T22:22:12.795780+0000","last_fullsized":"2024-03-11T22:22:12.795780+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:19:36.469333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.469333+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":598,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.782251+0000","last_change":"2024-03-11T22:22:06.879148+0000","last_active":"2024-03-11T22:22:12.782251+0000","last_peered":"2024-03-11T22:22:12.782251+0000","last_clean":"2024-03-11T22:22:12.782251+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:22:12.782251+0000","last_undegraded":"2024-03-11T22:22:12.782251+0000","last_fullsized":"2024-03-11T22:22:12.782251+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:22:06.879026+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:22:06.879026+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:15.575587+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'11","reported_seq":796,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:13.515872+0000","last_change":"2024-03-11T22:20:54.741772+0000","last_active":"2024-03-11T22:22:13.515872+0000","last_peered":"2024-03-11T22:22:13.515872+0000","last_clean":"2024-03-11T22:22:13.515872+0000","last_became_active":"2024-03-11T22:20:29.415781+0000","last_became_peered":"2024-03-11T22:20:29.415781+0000","last_unstale":"2024-03-11T22:22:13.515872+0000","last_undegraded":"2024-03-11T22:22:13.515872+0000","last_fullsized":"2024-03-11T22:22:13.515872+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_deep_scrub":"315'11","last_deep_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_clean_scrub_stamp":"2024-03-11T22:20:54.741633+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":529,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783878+0000","last_change":"2024-03-11T22:19:27.754842+0000","last_active":"2024-03-11T22:22:12.783878+0000","last_peered":"2024-03-11T22:22:12.783878+0000","last_clean":"2024-03-11T22:22:12.783878+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:22:12.783878+0000","last_undegraded":"2024-03-11T22:22:12.783878+0000","last_fullsized":"2024-03-11T22:22:12.783878+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:19:27.754721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:27.754721+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"273'18","reported_seq":608,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.797046+0000","last_change":"2024-03-11T22:21:52.914678+0000","last_active":"2024-03-11T22:22:12.797046+0000","last_peered":"2024-03-11T22:22:12.797046+0000","last_clean":"2024-03-11T22:22:12.797046+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:22:12.797046+0000","last_undegraded":"2024-03-11T22:22:12.797046+0000","last_fullsized":"2024-03-11T22:22:12.797046+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:21:52.914535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:21:52.914535+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:04.434254+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":355,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.787668+0000","last_change":"2024-03-11T22:19:36.471241+0000","last_active":"2024-03-11T22:22:12.787668+0000","last_peered":"2024-03-11T22:22:12.787668+0000","last_clean":"2024-03-11T22:22:12.787668+0000","last_became_active":"2024-03-11T22:14:52.245199+0000","last_became_peered":"2024-03-11T22:14:52.245199+0000","last_unstale":"2024-03-11T22:22:12.787668+0000","last_undegraded":"2024-03-11T22:22:12.787668+0000","last_fullsized":"2024-03-11T22:22:12.787668+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:36.471133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:36.471133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":338,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.807632+0000","last_change":"2024-03-11T22:21:49.873269+0000","last_active":"2024-03-11T22:22:12.807632+0000","last_peered":"2024-03-11T22:22:12.807632+0000","last_clean":"2024-03-11T22:22:12.807632+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:22:12.807632+0000","last_undegraded":"2024-03-11T22:22:12.807632+0000","last_fullsized":"2024-03-11T22:22:12.807632+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:49.872966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:21:49.872966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:07.291040+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":561,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.805922+0000","last_change":"2024-03-11T22:18:27.116987+0000","last_active":"2024-03-11T22:22:12.805922+0000","last_peered":"2024-03-11T22:22:12.805922+0000","last_clean":"2024-03-11T22:22:12.805922+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:22:12.805922+0000","last_undegraded":"2024-03-11T22:22:12.805922+0000","last_fullsized":"2024-03-11T22:22:12.805922+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:27.116858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:27.116858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"273'27","reported_seq":651,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783104+0000","last_change":"2024-03-11T22:20:55.875480+0000","last_active":"2024-03-11T22:22:12.783104+0000","last_peered":"2024-03-11T22:22:12.783104+0000","last_clean":"2024-03-11T22:22:12.783104+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:22:12.783104+0000","last_undegraded":"2024-03-11T22:22:12.783104+0000","last_fullsized":"2024-03-11T22:22:12.783104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:20:55.875312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:20:55.875312+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":592,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.796578+0000","last_change":"2024-03-11T22:20:29.414748+0000","last_active":"2024-03-11T22:22:12.796578+0000","last_peered":"2024-03-11T22:22:12.796578+0000","last_clean":"2024-03-11T22:22:12.796578+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:22:12.796578+0000","last_undegraded":"2024-03-11T22:22:12.796578+0000","last_fullsized":"2024-03-11T22:22:12.796578+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:18:43.931797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":580,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.791945+0000","last_change":"2024-03-11T22:18:31.120211+0000","last_active":"2024-03-11T22:22:12.791945+0000","last_peered":"2024-03-11T22:22:12.791945+0000","last_clean":"2024-03-11T22:22:12.791945+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:22:12.791945+0000","last_undegraded":"2024-03-11T22:22:12.791945+0000","last_fullsized":"2024-03-11T22:22:12.791945+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:31.120131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:18:31.120131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":355,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799605+0000","last_change":"2024-03-11T22:20:45.819718+0000","last_active":"2024-03-11T22:22:12.799605+0000","last_peered":"2024-03-11T22:22:12.799605+0000","last_clean":"2024-03-11T22:22:12.799605+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:22:12.799605+0000","last_undegraded":"2024-03-11T22:22:12.799605+0000","last_fullsized":"2024-03-11T22:22:12.799605+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:45.819593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:20:45.819593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"273'9","reported_seq":415,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.800421+0000","last_change":"2024-03-11T22:18:51.098736+0000","last_active":"2024-03-11T22:22:12.800421+0000","last_peered":"2024-03-11T22:22:12.800421+0000","last_clean":"2024-03-11T22:22:12.800421+0000","last_became_active":"2024-03-11T22:14:52.238411+0000","last_became_peered":"2024-03-11T22:14:52.238411+0000","last_unstale":"2024-03-11T22:22:12.800421+0000","last_undegraded":"2024-03-11T22:22:12.800421+0000","last_fullsized":"2024-03-11T22:22:12.800421+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9","last_scrub_stamp":"2024-03-11T22:18:51.098646+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:18:51.098646+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":588,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784073+0000","last_change":"2024-03-11T22:21:58.276736+0000","last_active":"2024-03-11T22:22:12.784073+0000","last_peered":"2024-03-11T22:22:12.784073+0000","last_clean":"2024-03-11T22:22:12.784073+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:22:12.784073+0000","last_undegraded":"2024-03-11T22:22:12.784073+0000","last_fullsized":"2024-03-11T22:22:12.784073+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:58.276626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:21:58.276626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:10.861978+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":357,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783752+0000","last_change":"2024-03-11T22:22:03.796143+0000","last_active":"2024-03-11T22:22:12.783752+0000","last_peered":"2024-03-11T22:22:12.783752+0000","last_clean":"2024-03-11T22:22:12.783752+0000","last_became_active":"2024-03-11T22:14:52.239391+0000","last_became_peered":"2024-03-11T22:14:52.239391+0000","last_unstale":"2024-03-11T22:22:12.783752+0000","last_undegraded":"2024-03-11T22:22:12.783752+0000","last_fullsized":"2024-03-11T22:22:12.783752+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:03.796021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_clean_scrub_stamp":"2024-03-11T22:22:03.796021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:21.201623+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":568,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784986+0000","last_change":"2024-03-11T22:18:38.736070+0000","last_active":"2024-03-11T22:22:12.784986+0000","last_peered":"2024-03-11T22:22:12.784986+0000","last_clean":"2024-03-11T22:22:12.784986+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:22:12.784986+0000","last_undegraded":"2024-03-11T22:22:12.784986+0000","last_fullsized":"2024-03-11T22:22:12.784986+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:38.735937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.735937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"273'15","reported_seq":390,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.799797+0000","last_change":"2024-03-11T22:22:06.785816+0000","last_active":"2024-03-11T22:22:12.799797+0000","last_peered":"2024-03-11T22:22:12.799797+0000","last_clean":"2024-03-11T22:22:12.799797+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:22:12.799797+0000","last_undegraded":"2024-03-11T22:22:12.799797+0000","last_fullsized":"2024-03-11T22:22:12.799797+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:22:06.785363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:22:06.785363+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:06.916846+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":588,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784074+0000","last_change":"2024-03-11T22:19:40.864768+0000","last_active":"2024-03-11T22:22:12.784074+0000","last_peered":"2024-03-11T22:22:12.784074+0000","last_clean":"2024-03-11T22:22:12.784074+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:22:12.784074+0000","last_undegraded":"2024-03-11T22:22:12.784074+0000","last_fullsized":"2024-03-11T22:22:12.784074+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:40.864689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:19:40.864689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":584,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.783947+0000","last_change":"2024-03-11T22:20:29.420120+0000","last_active":"2024-03-11T22:22:12.783947+0000","last_peered":"2024-03-11T22:22:12.783947+0000","last_clean":"2024-03-11T22:22:12.783947+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:22:12.783947+0000","last_undegraded":"2024-03-11T22:22:12.783947+0000","last_fullsized":"2024-03-11T22:22:12.783947+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:28.713390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:18:28.713390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":568,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.795543+0000","last_change":"2024-03-11T22:20:30.493247+0000","last_active":"2024-03-11T22:22:12.795543+0000","last_peered":"2024-03-11T22:22:12.795543+0000","last_clean":"2024-03-11T22:22:12.795543+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:22:12.795543+0000","last_undegraded":"2024-03-11T22:22:12.795543+0000","last_fullsized":"2024-03-11T22:22:12.795543+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:30.493146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:20:30.493146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"273'27","reported_seq":611,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.798968+0000","last_change":"2024-03-11T22:18:37.066095+0000","last_active":"2024-03-11T22:22:12.798968+0000","last_peered":"2024-03-11T22:22:12.798968+0000","last_clean":"2024-03-11T22:22:12.798968+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:22:12.798968+0000","last_undegraded":"2024-03-11T22:22:12.798968+0000","last_fullsized":"2024-03-11T22:22:12.798968+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-11T22:18:37.065988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:18:37.065988+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":587,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.797308+0000","last_change":"2024-03-11T22:18:38.095521+0000","last_active":"2024-03-11T22:22:12.797308+0000","last_peered":"2024-03-11T22:22:12.797308+0000","last_clean":"2024-03-11T22:22:12.797308+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:22:12.797308+0000","last_undegraded":"2024-03-11T22:22:12.797308+0000","last_fullsized":"2024-03-11T22:22:12.797308+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:18:38.095367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:18:38.095367+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":355,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.800915+0000","last_change":"2024-03-11T22:20:26.125601+0000","last_active":"2024-03-11T22:22:12.800915+0000","last_peered":"2024-03-11T22:22:12.800915+0000","last_clean":"2024-03-11T22:22:12.800915+0000","last_became_active":"2024-03-11T22:14:52.247689+0000","last_became_peered":"2024-03-11T22:14:52.247689+0000","last_unstale":"2024-03-11T22:22:12.800915+0000","last_undegraded":"2024-03-11T22:22:12.800915+0000","last_fullsized":"2024-03-11T22:22:12.800915+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:20:26.125517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:20:26.125517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":572,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.800590+0000","last_change":"2024-03-11T22:20:29.436389+0000","last_active":"2024-03-11T22:22:12.800590+0000","last_peered":"2024-03-11T22:22:12.800590+0000","last_clean":"2024-03-11T22:22:12.800590+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:22:12.800590+0000","last_undegraded":"2024-03-11T22:22:12.800590+0000","last_fullsized":"2024-03-11T22:22:12.800590+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:18:24.023596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:18:24.023596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"273'19","reported_seq":403,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.801009+0000","last_change":"2024-03-11T22:21:50.537463+0000","last_active":"2024-03-11T22:22:12.801009+0000","last_peered":"2024-03-11T22:22:12.801009+0000","last_clean":"2024-03-11T22:22:12.801009+0000","last_became_active":"2024-03-11T22:14:52.247411+0000","last_became_peered":"2024-03-11T22:14:52.247411+0000","last_unstale":"2024-03-11T22:22:12.801009+0000","last_undegraded":"2024-03-11T22:22:12.801009+0000","last_fullsized":"2024-03-11T22:22:12.801009+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:21:50.537367+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:21:50.537367+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:16.692296+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":589,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:14.783222+0000","last_change":"2024-03-11T22:22:14.783254+0000","last_active":"2024-03-11T22:22:14.783222+0000","last_peered":"2024-03-11T22:22:14.783222+0000","last_clean":"2024-03-11T22:22:14.783222+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:22:14.783222+0000","last_undegraded":"2024-03-11T22:22:14.783222+0000","last_fullsized":"2024-03-11T22:22:14.783222+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:14.783140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:22:14.783140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:15.236625+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":303,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:15.935561+0000","last_change":"2024-03-11T22:22:15.935582+0000","last_active":"2024-03-11T22:22:15.935561+0000","last_peered":"2024-03-11T22:22:15.935561+0000","last_clean":"2024-03-11T22:22:15.935561+0000","last_became_active":"2024-03-11T22:20:29.418227+0000","last_became_peered":"2024-03-11T22:20:29.418227+0000","last_unstale":"2024-03-11T22:22:15.935561+0000","last_undegraded":"2024-03-11T22:22:15.935561+0000","last_fullsized":"2024-03-11T22:22:15.935561+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:15.935496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:22:15.935496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:23:26.425457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":346,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-11T22:22:12.784647+0000","last_change":"2024-03-11T22:19:26.746519+0000","last_active":"2024-03-11T22:22:12.784647+0000","last_peered":"2024-03-11T22:22:12.784647+0000","last_clean":"2024-03-11T22:22:12.784647+0000","last_became_active":"2024-03-11T22:14:52.247273+0000","last_became_peered":"2024-03-11T22:14:52.247273+0000","last_unstale":"2024-03-11T22:22:12.784647+0000","last_undegraded":"2024-03-11T22:22:12.784647+0000","last_fullsized":"2024-03-11T22:22:12.784647+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:19:26.746435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:19:26.746435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":461963288,"num_objects":7050,"num_object_clones":0,"num_object_copies":21150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7050,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":241186,"num_write_kb":7943489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34657,"num_bytes_recovered":2270887960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1377251328,"data_stored":1377239112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72686,"ondisk_log_size":72686,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":788,"num_read_kb":783,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":1430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":717,"ondisk_log_size":717,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005226,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":630788,"kb_used_data":116476,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93114364,"statfs":{"total":95995035648,"available":95349108736,"internally_reserved":0,"allocated":119271424,"data_stored":118934679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":526645707},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332271,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365804,"kb_used_data":31972,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93379348,"statfs":{"total":95995035648,"available":95620452352,"internally_reserved":0,"allocated":32739328,"data_stored":32426945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":341834182},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575048,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":778336,"kb_used_data":277912,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92966816,"statfs":{"total":95995035648,"available":95198019584,"internally_reserved":0,"allocated":284581888,"data_stored":284216477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738574,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":534780,"kb_used_data":106548,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93210372,"statfs":{"total":95995035648,"available":95447420928,"internally_reserved":0,"allocated":109105152,"data_stored":108775907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411523,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":451248,"kb_used_data":100772,"kb_used_omap":1,"kb_used_meta":350462,"kb_avail":93293904,"statfs":{"total":95995035648,"available":95532957696,"internally_reserved":0,"allocated":103190528,"data_stored":102878515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":358873541},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149885,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888596,"kb_used_data":388172,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92856556,"statfs":{"total":95995035648,"available":95085113344,"internally_reserved":0,"allocated":397488128,"data_stored":397135835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":512424391},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920954,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":380228,"kb_used_data":38076,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93364924,"statfs":{"total":95995035648,"available":95605682176,"internally_reserved":0,"allocated":38989824,"data_stored":38652596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659314,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":990840,"kb_used_data":450160,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92754312,"statfs":{"total":95995035648,"available":94980415488,"internally_reserved":0,"allocated":460963840,"data_stored":460639010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":553646534},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":105709568,"data_stored":105709568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":250548224,"data_stored":250544152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89128960,"data_stored":89128960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":365363200,"data_stored":365363200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25362432,"data_stored":25362432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":446631936,"data_stored":446627864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":23789568,"data_stored":23789568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":70717440,"data_stored":70713368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:22:25.898 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-11T22:22:25.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool set unique_pool_2 pg_num 26 2024-03-11T22:22:25.988 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:26.015 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:26.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:26.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:25 smithi144 ceph-mon[33984]: pgmap v647: 168 pgs: 168 active+clean; 39 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-11T22:22:26.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:25 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:22:26.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:25 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:22:26.117 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:26.145 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:26.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:26.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:26.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:26.273 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:26.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:26.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:26.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:25 smithi045 ceph-mon[39715]: pgmap v647: 168 pgs: 168 active+clean; 39 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-11T22:22:26.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:25 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:22:26.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:25 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:22:26.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:25 smithi045 ceph-mon[30178]: pgmap v647: 168 pgs: 168 active+clean; 39 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s 2024-03-11T22:22:26.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:25 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:22:26.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:25 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:22:26.376 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:26.403 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:26.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:26.505 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:26.534 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:26.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:26.635 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:26.663 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:26.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:26.764 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:26.792 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:26.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:26.831 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:22:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:22:26.894 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:26.915 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:26.922 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:26.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:27.025 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:27.052 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:27.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:27.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:26 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:22:27.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:26 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:22:27.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:26 smithi144 ceph-mon[33984]: osdmap e336: 8 total, 8 up, 8 in 2024-03-11T22:22:27.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:27.179 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:27.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:27.186 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:26 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:22:27.186 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:26 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:22:27.186 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:26 smithi045 ceph-mon[30178]: osdmap e336: 8 total, 8 up, 8 in 2024-03-11T22:22:27.186 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:26 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:22:27.186 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:26 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:22:27.186 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:26 smithi045 ceph-mon[39715]: osdmap e336: 8 total, 8 up, 8 in 2024-03-11T22:22:27.281 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:27.305 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:27.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:27.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:27.407 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:27.435 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:27.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:27.537 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:27.563 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:27.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:27.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:27.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:27.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:27.691 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:27.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:27.793 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:27.820 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:27.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:27.824 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:22:27.846 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:22:27.865 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:22:27.885 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:22:27.905 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:22:27.921 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:27.926 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:22:27.945 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:27.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:27.947 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:22:27.967 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:22:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:22:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:22:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:22:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:22:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:22:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:22:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:22:27.969 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:22:28.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:28.071 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:28.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:28.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:27 smithi144 ceph-mon[33984]: pgmap v649: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 15 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 852 op/s 2024-03-11T22:22:28.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:28.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:27 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/458153297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:22:28.172 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:28.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:28.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:28.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:28.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:27 smithi045 ceph-mon[39715]: pgmap v649: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 15 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 852 op/s 2024-03-11T22:22:28.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:28.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:27 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/458153297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:22:28.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:27 smithi045 ceph-mon[30178]: pgmap v649: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 15 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 852 op/s 2024-03-11T22:22:28.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:28.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:27 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/458153297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:22:28.297 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:28.322 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:28.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:28.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:28.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:28.449 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:28.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:28.550 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:28.578 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:28.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:28.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:28.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:28.704 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:28.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:28.805 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:28.833 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:28.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:28.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:28.941 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:22:28.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:28.960 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:28.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:29.061 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:29.090 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:29.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:28 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:22:29.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:28 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:22:29.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:29.216 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:29.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:29.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:28 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:22:29.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:28 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:22:29.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:28 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:22:29.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:28 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:22:29.317 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:29.347 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:29.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:29.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:29.448 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:29.475 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:29.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:29.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:29.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:29.601 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:29.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:29.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:29.702 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:29.731 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:29.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:29.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:29.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:29.860 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:29.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:29.962 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:29.989 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:29.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:29.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:30.090 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:30.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:29 smithi144 ceph-mon[33984]: pgmap v650: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 882 op/s 2024-03-11T22:22:30.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:29 smithi144 ceph-mon[33984]: osdmap e337: 8 total, 8 up, 8 in 2024-03-11T22:22:30.119 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:30.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:30.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:30.248 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:30.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:30.283 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:29 smithi045 ceph-mon[39715]: pgmap v650: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 882 op/s 2024-03-11T22:22:30.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:29 smithi045 ceph-mon[39715]: osdmap e337: 8 total, 8 up, 8 in 2024-03-11T22:22:30.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:29 smithi045 ceph-mon[30178]: pgmap v650: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 882 op/s 2024-03-11T22:22:30.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:29 smithi045 ceph-mon[30178]: osdmap e337: 8 total, 8 up, 8 in 2024-03-11T22:22:30.351 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:30.379 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:30.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:30.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:30.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:30.508 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:30.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:30.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:30.610 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:30.638 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:30.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:30.740 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:30.768 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:30.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:30.869 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:30.897 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:30.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:30.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:30.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:31.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:31.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:31.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:31.128 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:31.155 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:31.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:31.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:31.285 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:31.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:31.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:31.387 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:31.414 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:31.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:31.421 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:31.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:31.541 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:31.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:31.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:31.643 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:31.671 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:31.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:31.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:31.796 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:31.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:31.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:31.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:31.922 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:31.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:31.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:32.024 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:32.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:31 smithi045 ceph-mon[39715]: pgmap v652: 168 pgs: 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 680 B/s rd, 0 B/s wr, 82 op/s 2024-03-11T22:22:32.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:22:32.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:22:32.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:31 smithi045 ceph-mon[30178]: pgmap v652: 168 pgs: 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 680 B/s rd, 0 B/s wr, 82 op/s 2024-03-11T22:22:32.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:22:32.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:22:32.052 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:32.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:32.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:32.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:31 smithi144 ceph-mon[33984]: pgmap v652: 168 pgs: 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 680 B/s rd, 0 B/s wr, 82 op/s 2024-03-11T22:22:32.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:22:32.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:22:32.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:32.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:32.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:32.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:32.282 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:32.306 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:32.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:32.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:32.407 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:32.430 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:32.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:32.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:32.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:32.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:32.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:32.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:32.656 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:32.680 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:32.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:32.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:32.769 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:32.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:32.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:32.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:32.892 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:22:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:22:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:22:32.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:32 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:22:32.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:32 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:22:32.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:32 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:32.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:32 smithi045 ceph-mon[30178]: osdmap e338: 8 total, 8 up, 8 in 2024-03-11T22:22:32.893 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:32 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:22:32.893 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:32 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:22:32.893 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:32 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:32.893 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:32 smithi045 ceph-mon[39715]: osdmap e338: 8 total, 8 up, 8 in 2024-03-11T22:22:32.909 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:32.933 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:32.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:33.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:33.059 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:33.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:33.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:33.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:32 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:22:33.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:32 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:22:33.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:32 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:22:33.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:32 smithi144 ceph-mon[33984]: osdmap e338: 8 total, 8 up, 8 in 2024-03-11T22:22:33.161 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:33.185 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:33.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:33.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:33.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:33.310 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:33.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:33.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:33.412 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:33.435 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:33.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:33.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:33.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:22:33.537 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:33.560 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:33.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:33.662 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:33.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:33.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:33.787 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:33.810 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:33.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:33.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:33 smithi045 ceph-mon[39715]: pgmap v654: 168 pgs: 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 37 op/s 2024-03-11T22:22:33.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:33 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:33.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:33 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:33.898 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:33 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:22:33.899 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:33 smithi045 ceph-mon[30178]: pgmap v654: 168 pgs: 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 37 op/s 2024-03-11T22:22:33.899 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:33 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:33.899 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:33 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:33.899 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:33 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:22:33.911 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:33.935 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:33.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:33.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:34.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:34.044 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:34.062 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:34.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:34.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:34.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:33 smithi144 ceph-mon[33984]: pgmap v654: 168 pgs: 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 37 op/s 2024-03-11T22:22:34.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:33 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:22:34.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:33 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:22:34.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:33 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:22:34.164 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:34.188 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:34.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:34.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:34.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:34.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:34.416 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:34.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:34.441 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:34.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:34.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:34.565 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:34.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:34.667 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:34.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:34.692 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:34.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:34.772 INFO:teuthology.orchestra.run.smithi045.stderr:set pool 9 pg_num to 26 2024-03-11T22:22:34.793 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:34.817 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:34.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:34.919 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:34.943 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:34.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:34.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:34 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/231222682' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T22:22:35.033 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:34 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T22:22:35.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:34 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/231222682' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T22:22:35.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:34 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T22:22:35.044 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:35.067 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:35.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:35.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:35.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:34 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/231222682' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T22:22:35.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:34 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T22:22:35.169 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:35.193 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:35.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:35.294 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:35.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:35.320 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:35.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:35.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:35.445 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:35.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:35.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:35.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:35.572 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:35.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:35.618 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:22:35.618 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:22:35.618 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-03-11T22:22:35.619 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-03-11T22:22:35.619 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:22:35.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:35.697 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:35.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:35.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:35.823 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:35.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:35.924 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:35.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:35 smithi045 ceph-mon[39715]: pgmap v655: 168 pgs: 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 30 op/s 2024-03-11T22:22:35.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:35 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-03-11T22:22:35.947 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:35 smithi045 ceph-mon[39715]: osdmap e339: 8 total, 8 up, 8 in 2024-03-11T22:22:35.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:35 smithi045 ceph-mon[30178]: pgmap v655: 168 pgs: 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 30 op/s 2024-03-11T22:22:35.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:35 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-03-11T22:22:35.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:35 smithi045 ceph-mon[30178]: osdmap e339: 8 total, 8 up, 8 in 2024-03-11T22:22:35.952 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:35.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:36.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:36.077 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:36.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:36.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:36.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:35 smithi144 ceph-mon[33984]: pgmap v655: 168 pgs: 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 30 op/s 2024-03-11T22:22:36.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:35 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-03-11T22:22:36.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:35 smithi144 ceph-mon[33984]: osdmap e339: 8 total, 8 up, 8 in 2024-03-11T22:22:36.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:36.203 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:36.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:36.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:36.332 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:36.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:36.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T22:22:36.433 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:36.461 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:36.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:36.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:36.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:36.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:36.589 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:36.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:36.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:36.714 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:36.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:36.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:36.816 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:36.844 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:36.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:36.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:36.847 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:22:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:22:36.901 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-11T22:22:36.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-11T22:22:36.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:36.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:36.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:37.072 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:37.102 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:37.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:37.203 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:37.230 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:37.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:37.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:37.332 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:37.358 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:37.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:37.460 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:37.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:37.531 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:37.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:37.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:37 smithi144 ceph-mon[33984]: pgmap v657: 168 pgs: 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-11T22:22:37.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:37 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T22:22:37.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:37.632 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:37.655 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:37.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:37.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:37.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:37 smithi045 ceph-mon[30178]: pgmap v657: 168 pgs: 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-11T22:22:37.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:37 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T22:22:37.681 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:37.682 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:37 smithi045 ceph-mon[39715]: pgmap v657: 168 pgs: 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 681 B/s rd, 0 op/s 2024-03-11T22:22:37.682 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:37 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T22:22:37.682 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:37.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:37.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:37.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:37.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:37.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:37.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:38.006 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:38.030 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:38.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:38.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:38.077 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:38.082 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:38.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:38.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:38.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:38.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:38.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:38.281 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:38.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:38.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:38.423 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:38 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T22:22:38.424 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:38 smithi045 ceph-mon[30178]: osdmap e340: 8 total, 8 up, 8 in 2024-03-11T22:22:38.424 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:38 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T22:22:38.424 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:38 smithi045 ceph-mon[39715]: osdmap e340: 8 total, 8 up, 8 in 2024-03-11T22:22:38.425 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:38.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:38.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:38.527 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:38.551 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:38.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:38.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:38.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:38 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T22:22:38.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:38 smithi144 ceph-mon[33984]: osdmap e340: 8 total, 8 up, 8 in 2024-03-11T22:22:38.653 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:38.676 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:38.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:38.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:38.778 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:38.801 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:38.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:38.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:38.903 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:38.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:38.928 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:38.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:39.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:39.053 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:39.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:39.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:39.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:39.179 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:39.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:39.280 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:39.304 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:39.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:39.332 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:22:39.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:39.430 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:39.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:39.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:39.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:39.556 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:39.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:39.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:39.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:39 smithi045 ceph-mon[30178]: osdmap e341: 8 total, 8 up, 8 in 2024-03-11T22:22:39.574 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:39 smithi045 ceph-mon[30178]: pgmap v660: 178 pgs: 10 unknown, 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:22:39.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:39 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T22:22:39.575 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:39 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4200425226' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:22:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:39 smithi045 ceph-mon[39715]: osdmap e341: 8 total, 8 up, 8 in 2024-03-11T22:22:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:39 smithi045 ceph-mon[39715]: pgmap v660: 178 pgs: 10 unknown, 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:22:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:39 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T22:22:39.575 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:39 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4200425226' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:22:39.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:39 smithi144 ceph-mon[33984]: osdmap e341: 8 total, 8 up, 8 in 2024-03-11T22:22:39.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:39 smithi144 ceph-mon[33984]: pgmap v660: 178 pgs: 10 unknown, 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:22:39.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:39 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T22:22:39.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:39 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4200425226' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:22:39.658 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:39.683 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:39.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:39.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:39.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:39.808 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:39.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:39.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:39.934 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:39.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:40.036 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:40.063 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:40.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:40.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:40.164 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:40.191 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:40.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:40.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:40.359 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:40.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:40.461 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:40.489 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:40.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:40.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:40.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:40 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:22:40.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:40 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-11T22:22:40.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:40 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4200425226' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:22:40.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:40 smithi045 ceph-mon[39715]: osdmap e342: 8 total, 8 up, 8 in 2024-03-11T22:22:40.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:40 smithi045 ceph-mon[39715]: osdmap e343: 8 total, 8 up, 8 in 2024-03-11T22:22:40.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:40 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:22:40.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:40 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-11T22:22:40.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:40 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4200425226' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:22:40.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:40 smithi045 ceph-mon[30178]: osdmap e342: 8 total, 8 up, 8 in 2024-03-11T22:22:40.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:40 smithi045 ceph-mon[30178]: osdmap e343: 8 total, 8 up, 8 in 2024-03-11T22:22:40.591 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:40.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:40 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:22:40.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:40 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-11T22:22:40.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:40 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4200425226' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:22:40.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:40 smithi144 ceph-mon[33984]: osdmap e342: 8 total, 8 up, 8 in 2024-03-11T22:22:40.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:40 smithi144 ceph-mon[33984]: osdmap e343: 8 total, 8 up, 8 in 2024-03-11T22:22:40.618 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:40.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:40.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:40.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:40.743 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:40.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:40.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:40.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:40.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:40.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:40.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:40.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:40.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:40.996 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:40.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:41.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:41.121 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:41.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:41.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:41.246 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:41.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:41.348 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:41.388 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:41.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:41.437 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:41 smithi045 ceph-mon[30178]: pgmap v663: 178 pgs: 10 unknown, 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:22:41.437 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:41 smithi045 ceph-mon[30178]: osdmap e344: 8 total, 8 up, 8 in 2024-03-11T22:22:41.437 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:41 smithi045 ceph-mon[39715]: pgmap v663: 178 pgs: 10 unknown, 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:22:41.437 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:41 smithi045 ceph-mon[39715]: osdmap e344: 8 total, 8 up, 8 in 2024-03-11T22:22:41.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:41.513 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:41.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:41.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:41 smithi144 ceph-mon[33984]: pgmap v663: 178 pgs: 10 unknown, 168 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:22:41.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:41 smithi144 ceph-mon[33984]: osdmap e344: 8 total, 8 up, 8 in 2024-03-11T22:22:41.615 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:41.639 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:41.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:41.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:41.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:41.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:41.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:41.889 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:41.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:41.902 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:22:41.902 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:22:41.902 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-11T22:22:41.902 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T22:22:41.902 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T22:22:41.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-11T22:22:41.990 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:42.015 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:42.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:42.116 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:42.143 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:42.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:42.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:42.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:42.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:42.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:42.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:22:42.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:42.394 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:42.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:42.496 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:42.523 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:42.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:42.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.19 starting backfill to osd.3 from (305'6900,323'12186] MIN to 334'15482 2024-03-11T22:22:42.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.15 starting backfill to osd.1 from (0'0,0'0] MAX to 334'15880 2024-03-11T22:22:42.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 334'15584 2024-03-11T22:22:42.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 334'15316 2024-03-11T22:22:42.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.16 starting backfill to osd.4 from (0'0,0'0] MAX to 334'14984 2024-03-11T22:22:42.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.10 starting backfill to osd.0 from (0'0,0'0] MAX to 334'15560 2024-03-11T22:22:42.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.13 starting backfill to osd.5 from (308'8200,325'12316] MIN to 334'15696 2024-03-11T22:22:42.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.17 starting backfill to osd.0 from (0'0,0'0] MAX to 334'15608 2024-03-11T22:22:42.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 334'15584 2024-03-11T22:22:42.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.17 starting backfill to osd.1 from (0'0,0'0] MAX to 334'15608 2024-03-11T22:22:42.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 334'15482 2024-03-11T22:22:42.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.15 starting backfill to osd.3 from (0'0,0'0] MAX to 334'15880 2024-03-11T22:22:42.599 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 334'15316 2024-03-11T22:22:42.599 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 334'15696 2024-03-11T22:22:42.599 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.10 starting backfill to osd.4 from (0'0,0'0] MAX to 334'15560 2024-03-11T22:22:42.599 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:42 smithi144 ceph-mon[33984]: 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 334'15560 2024-03-11T22:22:42.625 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:42.653 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:42.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:42.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:42.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:42.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:42.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:42.783 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:22:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:22:42.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.19 starting backfill to osd.3 from (305'6900,323'12186] MIN to 334'15482 2024-03-11T22:22:42.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.15 starting backfill to osd.1 from (0'0,0'0] MAX to 334'15880 2024-03-11T22:22:42.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 334'15584 2024-03-11T22:22:42.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 334'15316 2024-03-11T22:22:42.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.16 starting backfill to osd.4 from (0'0,0'0] MAX to 334'14984 2024-03-11T22:22:42.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.10 starting backfill to osd.0 from (0'0,0'0] MAX to 334'15560 2024-03-11T22:22:42.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.13 starting backfill to osd.5 from (308'8200,325'12316] MIN to 334'15696 2024-03-11T22:22:42.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.17 starting backfill to osd.0 from (0'0,0'0] MAX to 334'15608 2024-03-11T22:22:42.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 334'15584 2024-03-11T22:22:42.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.17 starting backfill to osd.1 from (0'0,0'0] MAX to 334'15608 2024-03-11T22:22:42.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 334'15482 2024-03-11T22:22:42.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.15 starting backfill to osd.3 from (0'0,0'0] MAX to 334'15880 2024-03-11T22:22:42.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 334'15316 2024-03-11T22:22:42.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 334'15696 2024-03-11T22:22:42.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.10 starting backfill to osd.4 from (0'0,0'0] MAX to 334'15560 2024-03-11T22:22:42.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[39715]: 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 334'15560 2024-03-11T22:22:42.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.19 starting backfill to osd.3 from (305'6900,323'12186] MIN to 334'15482 2024-03-11T22:22:42.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.15 starting backfill to osd.1 from (0'0,0'0] MAX to 334'15880 2024-03-11T22:22:42.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.18 starting backfill to osd.6 from (0'0,0'0] MAX to 334'15584 2024-03-11T22:22:42.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.12 starting backfill to osd.0 from (0'0,0'0] MAX to 334'15316 2024-03-11T22:22:42.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.16 starting backfill to osd.4 from (0'0,0'0] MAX to 334'14984 2024-03-11T22:22:42.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.10 starting backfill to osd.0 from (0'0,0'0] MAX to 334'15560 2024-03-11T22:22:42.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.13 starting backfill to osd.5 from (308'8200,325'12316] MIN to 334'15696 2024-03-11T22:22:42.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.17 starting backfill to osd.0 from (0'0,0'0] MAX to 334'15608 2024-03-11T22:22:42.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.18 starting backfill to osd.7 from (0'0,0'0] MAX to 334'15584 2024-03-11T22:22:42.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.17 starting backfill to osd.1 from (0'0,0'0] MAX to 334'15608 2024-03-11T22:22:42.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.19 starting backfill to osd.6 from (0'0,0'0] MAX to 334'15482 2024-03-11T22:22:42.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.15 starting backfill to osd.3 from (0'0,0'0] MAX to 334'15880 2024-03-11T22:22:42.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.12 starting backfill to osd.5 from (0'0,0'0] MAX to 334'15316 2024-03-11T22:22:42.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.13 starting backfill to osd.6 from (0'0,0'0] MAX to 334'15696 2024-03-11T22:22:42.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.10 starting backfill to osd.4 from (0'0,0'0] MAX to 334'15560 2024-03-11T22:22:42.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:42 smithi045 ceph-mon[30178]: 9.10 starting backfill to osd.7 from (0'0,0'0] MAX to 334'15560 2024-03-11T22:22:42.881 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:42.909 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:42.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:42.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:43.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:43.038 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:43.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:43.100 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:22:43.122 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:22:43.139 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:43.143 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:22:43.163 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:22:43.167 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:43.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:43.183 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:22:43.204 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:22:43.227 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:22:43.247 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:22:43.248 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:22:43.248 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:22:43.248 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:22:43.249 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:22:43.249 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:22:43.249 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:22:43.249 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:22:43.249 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:22:43.269 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:43.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:43.294 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:43.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:43.375 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'unique_pool_2' removed 2024-03-11T22:22:43.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:43.439 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:43.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:43.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:43.570 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:43.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:43.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:43.695 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:43.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:43.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:43.699 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:43 smithi045 ceph-mon[39715]: pgmap v665: 178 pgs: 2 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 5 active+clean+remapped, 170 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:22:43.700 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:43 smithi045 ceph-mon[39715]: osdmap e345: 8 total, 8 up, 8 in 2024-03-11T22:22:43.700 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:43 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1317169447' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:22:43.700 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:43 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3500572233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:22:43.700 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:43 smithi045 ceph-mon[39715]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:22:43.700 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:43 smithi045 ceph-mon[30178]: pgmap v665: 178 pgs: 2 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 5 active+clean+remapped, 170 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:22:43.700 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:43 smithi045 ceph-mon[30178]: osdmap e345: 8 total, 8 up, 8 in 2024-03-11T22:22:43.700 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:43 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1317169447' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:22:43.700 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:43 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3500572233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:22:43.700 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:43 smithi045 ceph-mon[30178]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:22:43.797 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:43.821 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:43.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:43.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:43 smithi144 ceph-mon[33984]: pgmap v665: 178 pgs: 2 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 5 active+clean+remapped, 170 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:22:43.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:43 smithi144 ceph-mon[33984]: osdmap e345: 8 total, 8 up, 8 in 2024-03-11T22:22:43.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:43 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1317169447' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:22:43.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:43 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3500572233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:22:43.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:43 smithi144 ceph-mon[33984]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:22:43.923 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:43.946 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:43.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:44.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:44.090 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:44.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:44.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:44.192 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:44.216 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:44.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:44.318 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:44.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:44.342 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:44.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:44.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:44.467 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:44.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:44.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:44.489 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:22:44.489 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:22:44.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:44.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:44.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:44.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:44.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:44 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1317169447' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:22:44.598 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:44 smithi045 ceph-mon[39715]: osdmap e346: 8 total, 8 up, 8 in 2024-03-11T22:22:44.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:44 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1317169447' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:22:44.598 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:44 smithi045 ceph-mon[30178]: osdmap e346: 8 total, 8 up, 8 in 2024-03-11T22:22:44.695 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:44.719 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:44.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:44.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:44.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:44.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:44.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:44.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:44 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1317169447' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:22:44.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:44 smithi144 ceph-mon[33984]: osdmap e346: 8 total, 8 up, 8 in 2024-03-11T22:22:44.905 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T22:22:44.906 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T22:22:44.906 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-11T22:22:44.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-11T22:22:44.947 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:44.971 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:44.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:44.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:45.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:45.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:45.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:45.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:45.162 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:45.198 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:45.223 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:45.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:45.323 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:45.347 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:45.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:45.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:45.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:45.472 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:45.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:45.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:45.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:45 smithi045 ceph-mon[30178]: pgmap v668: 152 pgs: 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T22:22:45.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:45 smithi045 ceph-mon[30178]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-11T22:22:45.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:45 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:22:45.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:45 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:22:45.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:45 smithi045 ceph-mon[39715]: pgmap v668: 152 pgs: 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T22:22:45.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:45 smithi045 ceph-mon[39715]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-11T22:22:45.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:45 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:22:45.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:45 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:22:45.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:45.599 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:45.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:45.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:45.608 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:45.621 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T22:22:45.631 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:45.644 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:45 smithi144 ceph-mon[33984]: pgmap v668: 152 pgs: 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T22:22:45.644 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:45 smithi144 ceph-mon[33984]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-11T22:22:45.644 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:45 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:22:45.644 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:45 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:22:45.701 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:45.725 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:45.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:45.827 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:45.852 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:45.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:45.864 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:22:45.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:45.978 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:45.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:45.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:46.026 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:22:46.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:46.104 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:46.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:46.205 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:46.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:46.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:46.332 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:46.356 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:46.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:46.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:46.443 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:46 smithi045 ceph-mon[39715]: osdmap e347: 8 total, 8 up, 8 in 2024-03-11T22:22:46.443 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:22:46.443 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:46 smithi045 ceph-mon[30178]: osdmap e347: 8 total, 8 up, 8 in 2024-03-11T22:22:46.443 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:22:46.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:46.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:46.483 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:46.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:46.584 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:46 smithi144 ceph-mon[33984]: osdmap e347: 8 total, 8 up, 8 in 2024-03-11T22:22:46.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:22:46.598 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:22:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:22:46.608 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:46.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:46.709 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:46.733 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:46.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:46.835 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:46.858 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:46.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:46.960 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:46.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:46.985 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:46.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:47.086 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:47.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:47.111 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:47.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:47.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:47.238 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:47.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:47.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:47.339 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:47.363 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:47.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:47.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:47.419 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:47 smithi144 ceph-mon[33984]: pgmap v670: 152 pgs: 152 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 815 B/s rd, 0 op/s 2024-03-11T22:22:47.419 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:47.465 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:47.491 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:47.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:47.592 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:47.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:47.618 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:47.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:47.676 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:22:47.676 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:22:47.719 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:47.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:47.745 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:47.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:47 smithi045 ceph-mon[39715]: pgmap v670: 152 pgs: 152 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 815 B/s rd, 0 op/s 2024-03-11T22:22:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:47.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:47 smithi045 ceph-mon[30178]: pgmap v670: 152 pgs: 152 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 815 B/s rd, 0 op/s 2024-03-11T22:22:47.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:47.845 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:47.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:47.971 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:47.994 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:47.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:47.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:48.096 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:48.119 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:48.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:48.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:48.221 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:48.245 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:48.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:48.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:48.304 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T22:22:48.346 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:48.370 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:48.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:48.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:48.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:48.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:48.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:48.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:48.547 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:22:48.598 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:48.622 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:48.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:48.722 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:48.746 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:48.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:48.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:48.872 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:48.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:48.973 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:48.997 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:48.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:48.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:49.098 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:49.122 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:49.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:49.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:49.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:49.247 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:49.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:49.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:49.351 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:49.375 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:49.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:49.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:49.476 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:49.500 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:49.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:49.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:49.525 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:49 smithi045 ceph-mon[39715]: pgmap v671: 152 pgs: 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T22:22:49.525 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:49 smithi045 ceph-mon[30178]: pgmap v671: 152 pgs: 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T22:22:49.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:49.626 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:49.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:49.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:49.751 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:49 smithi144 ceph-mon[33984]: pgmap v671: 152 pgs: 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T22:22:49.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:49.752 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:49.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:49.853 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:49.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:49.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:49.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:50.002 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:50.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:50.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:50.129 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:50.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:50.230 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:50.254 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:50.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:50.272 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:22:50.273 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:22:50.274 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:50.356 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:50.380 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:50.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:50.482 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:50.506 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:50.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:50.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:50.608 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:50.631 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:50.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:50.733 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:50.758 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:50.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:50.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:50.860 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:50.884 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:50.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:50.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:50.926 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T22:22:50.986 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:51.010 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:51.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:51.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:51.111 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:51.136 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:51.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:51.210 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:22:51.237 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:51.260 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:51.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:51.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:51.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:51.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:51.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:51.469 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'unique_pool_3' created 2024-03-11T22:22:51.469 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:22:51.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:51.514 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:51.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:51.523 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:51 smithi045 ceph-mon[30178]: pgmap v672: 152 pgs: 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 512 B/s rd, 0 op/s 2024-03-11T22:22:51.523 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:51 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/504934479' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:22:51.523 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:51 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:22:51.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:51 smithi045 ceph-mon[39715]: pgmap v672: 152 pgs: 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 512 B/s rd, 0 op/s 2024-03-11T22:22:51.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:51 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/504934479' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:22:51.524 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:51 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:22:51.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:51 smithi144 ceph-mon[33984]: pgmap v672: 152 pgs: 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 512 B/s rd, 0 op/s 2024-03-11T22:22:51.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:51 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/504934479' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:22:51.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:51 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:22:51.616 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:51.642 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:51.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:51.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:51.744 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:51.767 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:51.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:51.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:51.894 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:51.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:51.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:51.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:52.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:52.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:52.121 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:52.145 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:52.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:52.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:52.247 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:52.271 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:52.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:52.325 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:22:52.325 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "" 2024-03-11T22:22:52.325 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:22:52.372 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:52.397 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:52.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:52.498 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:52.521 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:52.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:52.623 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:52.647 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:52.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:52.677 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:52 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3958002614' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-11T22:22:52.677 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:52 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:22:52.677 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:52 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:22:52.678 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:52 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3958002614' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-11T22:22:52.678 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:52 smithi144 ceph-mon[33984]: osdmap e348: 8 total, 8 up, 8 in 2024-03-11T22:22:52.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:52.774 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:52.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:52.783 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:22:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:22:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:22:52.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:52 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3958002614' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-11T22:22:52.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:52 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:22:52.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:52 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:22:52.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:52 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3958002614' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-11T22:22:52.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:52 smithi045 ceph-mon[39715]: osdmap e348: 8 total, 8 up, 8 in 2024-03-11T22:22:52.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:52 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3958002614' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-11T22:22:52.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:52 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:22:52.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:52 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:22:52.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:52 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3958002614' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-11T22:22:52.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:52 smithi045 ceph-mon[30178]: osdmap e348: 8 total, 8 up, 8 in 2024-03-11T22:22:52.876 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:52.900 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:52.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:52.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:52.934 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:22:52.934 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:22:53.002 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:53.025 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:53.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:53.127 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:53.151 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:53.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:53.253 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:53.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:53.277 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:53.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:53.379 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:53.403 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:53.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:53.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:53.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:53.529 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:53.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:53.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:53.558 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:53 smithi144 ceph-mon[33984]: pgmap v674: 168 pgs: 4 creating+peering, 12 unknown, 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:22:53.558 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:53 smithi144 ceph-mon[33984]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:22:53.559 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:53 smithi144 ceph-mon[33984]: osdmap e349: 8 total, 8 up, 8 in 2024-03-11T22:22:53.585 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T22:22:53.630 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:53.654 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:53.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:53.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:53.780 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:53.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:53.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:53 smithi045 ceph-mon[39715]: pgmap v674: 168 pgs: 4 creating+peering, 12 unknown, 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:22:53.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:53 smithi045 ceph-mon[39715]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:22:53.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:53 smithi045 ceph-mon[39715]: osdmap e349: 8 total, 8 up, 8 in 2024-03-11T22:22:53.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:53 smithi045 ceph-mon[30178]: pgmap v674: 168 pgs: 4 creating+peering, 12 unknown, 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:22:53.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:53 smithi045 ceph-mon[30178]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:22:53.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:53 smithi045 ceph-mon[30178]: osdmap e349: 8 total, 8 up, 8 in 2024-03-11T22:22:53.826 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:22:53.882 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:53.905 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:53.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:53.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:54.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:54.030 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:54.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:54.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:54.131 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:54.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:54.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:54.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:54.280 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:54.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:54.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:54.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:54.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:54.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:54.507 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:54.533 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:54.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:54.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:54.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:54 smithi045 ceph-mon[39715]: osdmap e350: 8 total, 8 up, 8 in 2024-03-11T22:22:54.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:54 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:22:54.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:54 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:22:54.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:54 smithi045 ceph-mon[30178]: osdmap e350: 8 total, 8 up, 8 in 2024-03-11T22:22:54.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:54 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:22:54.535 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:54 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:22:54.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:54 smithi144 ceph-mon[33984]: osdmap e350: 8 total, 8 up, 8 in 2024-03-11T22:22:54.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:54 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:22:54.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:54 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:22:54.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:54.658 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:54.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:54.760 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:54.784 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:54.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:54.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:54.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:54.908 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:54.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:55.010 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:55.034 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:55.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:55.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-11T22:22:55.135 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:55.158 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:55.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:55.260 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:55.282 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:55.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:55.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:55.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:55.412 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:55.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:55.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:55.514 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:55.540 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:55.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:55.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:55.628 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:55 smithi144 ceph-mon[33984]: 5.1 scrub starts 2024-03-11T22:22:55.628 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:55 smithi144 ceph-mon[33984]: 5.1 scrub ok 2024-03-11T22:22:55.628 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:55 smithi144 ceph-mon[33984]: pgmap v677: 168 pgs: 4 creating+peering, 12 unknown, 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:22:55.628 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:55 smithi144 ceph-mon[33984]: osdmap e351: 8 total, 8 up, 8 in 2024-03-11T22:22:55.642 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:55.653 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:22:55.654 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:22:55.665 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:55.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:55.667 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:55 smithi045 ceph-mon[39715]: 5.1 scrub starts 2024-03-11T22:22:55.667 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:55 smithi045 ceph-mon[39715]: 5.1 scrub ok 2024-03-11T22:22:55.667 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:55 smithi045 ceph-mon[39715]: pgmap v677: 168 pgs: 4 creating+peering, 12 unknown, 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:22:55.667 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:55 smithi045 ceph-mon[39715]: osdmap e351: 8 total, 8 up, 8 in 2024-03-11T22:22:55.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:55 smithi045 ceph-mon[30178]: 5.1 scrub starts 2024-03-11T22:22:55.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:55 smithi045 ceph-mon[30178]: 5.1 scrub ok 2024-03-11T22:22:55.667 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:55 smithi045 ceph-mon[30178]: pgmap v677: 168 pgs: 4 creating+peering, 12 unknown, 152 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:22:55.668 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:55 smithi045 ceph-mon[30178]: osdmap e351: 8 total, 8 up, 8 in 2024-03-11T22:22:55.670 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:22:55.691 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:22:55.711 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:55.720 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:22:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:22:55.766 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:55.768 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:22:55.788 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:22:55.790 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:55.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:55.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:55.808 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:22:55.828 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:22:55.830 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:22:55.830 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:22:55.830 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:22:55.830 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:22:55.830 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:22:55.830 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:22:55.830 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:22:55.830 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:22:55.892 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:55.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:22:55.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:55.916 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:55.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:56.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:56.041 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:56.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:56.143 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:56.166 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:56.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:56.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:56.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:56.292 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:56.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:56.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:56.358 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T22:22:56.394 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:56.417 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:56.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:56.518 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:56.543 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:56.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:56.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:56 smithi144 ceph-mon[33984]: 6.0 scrub starts 2024-03-11T22:22:56.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:56 smithi144 ceph-mon[33984]: 6.0 scrub ok 2024-03-11T22:22:56.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:56 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1827200491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:22:56.598 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:22:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:22:56.608 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:22:56.645 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:56.668 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:56.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:56.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:56.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:56 smithi045 ceph-mon[39715]: 6.0 scrub starts 2024-03-11T22:22:56.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:56 smithi045 ceph-mon[39715]: 6.0 scrub ok 2024-03-11T22:22:56.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:56 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1827200491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:22:56.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:56 smithi045 ceph-mon[30178]: 6.0 scrub starts 2024-03-11T22:22:56.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:56 smithi045 ceph-mon[30178]: 6.0 scrub ok 2024-03-11T22:22:56.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:56 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1827200491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:22:56.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:56.795 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:56.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:56.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:56.920 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:56.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:56.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:57.021 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:57.045 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:57.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:57.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:57.146 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:57.170 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:57.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:57.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:57.272 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:57.295 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:57.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:57.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:57.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:57.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:57.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:57.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:57.523 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:57.547 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:57.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:57.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:57.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:57.673 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:57.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:57.774 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:57.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:57 smithi045 ceph-mon[39715]: 3.6 scrub starts 2024-03-11T22:22:57.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:57 smithi045 ceph-mon[39715]: 3.6 scrub ok 2024-03-11T22:22:57.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:57 smithi045 ceph-mon[39715]: pgmap v679: 168 pgs: 4 creating+peering, 1 unknown, 163 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:22:57.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:57.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:57 smithi045 ceph-mon[30178]: 3.6 scrub starts 2024-03-11T22:22:57.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:57 smithi045 ceph-mon[30178]: 3.6 scrub ok 2024-03-11T22:22:57.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:57 smithi045 ceph-mon[30178]: pgmap v679: 168 pgs: 4 creating+peering, 1 unknown, 163 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:22:57.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:57.798 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:57.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:57.824 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:57 smithi144 ceph-mon[33984]: 3.6 scrub starts 2024-03-11T22:22:57.824 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:57 smithi144 ceph-mon[33984]: 3.6 scrub ok 2024-03-11T22:22:57.824 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:57 smithi144 ceph-mon[33984]: pgmap v679: 168 pgs: 4 creating+peering, 1 unknown, 163 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:22:57.824 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:22:57.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:57.923 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:57.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:58.025 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:22:58.049 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:58.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:58.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:22:58.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:58.174 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:58.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:58.235 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:22:58.276 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:58.299 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:58.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:58.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:58.371 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:22:58.371 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-03-11T22:22:58.371 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-03-11T22:22:58.371 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-03-11T22:22:58.372 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:22:58.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:58.425 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:58.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:58.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:22:58.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:58.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:22:58.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:58.677 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:58.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:58.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:58.779 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:22:58.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:58 smithi045 ceph-mon[30178]: 4.1e scrub starts 2024-03-11T22:22:58.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:58 smithi045 ceph-mon[30178]: 4.1e scrub ok 2024-03-11T22:22:58.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:58 smithi045 ceph-mon[30178]: 1.8 scrub starts 2024-03-11T22:22:58.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:58 smithi045 ceph-mon[30178]: 1.8 scrub ok 2024-03-11T22:22:58.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:58 smithi045 ceph-mon[30178]: 5.18 scrub starts 2024-03-11T22:22:58.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:58 smithi045 ceph-mon[30178]: 5.18 scrub ok 2024-03-11T22:22:58.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:58 smithi045 ceph-mon[39715]: 4.1e scrub starts 2024-03-11T22:22:58.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:58 smithi045 ceph-mon[39715]: 4.1e scrub ok 2024-03-11T22:22:58.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:58 smithi045 ceph-mon[39715]: 1.8 scrub starts 2024-03-11T22:22:58.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:58 smithi045 ceph-mon[39715]: 1.8 scrub ok 2024-03-11T22:22:58.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:58 smithi045 ceph-mon[39715]: 5.18 scrub starts 2024-03-11T22:22:58.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:58 smithi045 ceph-mon[39715]: 5.18 scrub ok 2024-03-11T22:22:58.803 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:58.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:22:58.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:58 smithi144 ceph-mon[33984]: 4.1e scrub starts 2024-03-11T22:22:58.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:58 smithi144 ceph-mon[33984]: 4.1e scrub ok 2024-03-11T22:22:58.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:58 smithi144 ceph-mon[33984]: 1.8 scrub starts 2024-03-11T22:22:58.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:58 smithi144 ceph-mon[33984]: 1.8 scrub ok 2024-03-11T22:22:58.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:58 smithi144 ceph-mon[33984]: 5.18 scrub starts 2024-03-11T22:22:58.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:58 smithi144 ceph-mon[33984]: 5.18 scrub ok 2024-03-11T22:22:58.906 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:58.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:58.930 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:58.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:58.974 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T22:22:59.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:59.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:22:59.054 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:59.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:59.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:59.156 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:59.180 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:59.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:22:59.215 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:22:59.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:22:59.305 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:59.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:22:59.407 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:22:59.430 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:59.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:59.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:22:59.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[30178]: 3.c scrub starts 2024-03-11T22:22:59.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[30178]: 3.c scrub ok 2024-03-11T22:22:59.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[30178]: pgmap v680: 168 pgs: 168 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:22:59.489 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[30178]: 6.1 scrub starts 2024-03-11T22:22:59.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:22:59.556 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:59.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:59.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:22:59.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:59 smithi144 ceph-mon[33984]: 3.c scrub starts 2024-03-11T22:22:59.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:59 smithi144 ceph-mon[33984]: 3.c scrub ok 2024-03-11T22:22:59.597 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:59 smithi144 ceph-mon[33984]: pgmap v680: 168 pgs: 168 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:22:59.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:59 smithi144 ceph-mon[33984]: 6.1 scrub starts 2024-03-11T22:22:59.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:59 smithi144 ceph-mon[33984]: 6.1 scrub ok 2024-03-11T22:22:59.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:59 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1916604260' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:22:59.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:22:59 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:22:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-11T22:22:59.646 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:22:59.646 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:22:59.646 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-03-11T22:22:59.646 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T22:22:59.646 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T22:22:59.647 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-03-11T22:22:59.658 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:59.713 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:59.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:59.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:59.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[30178]: 6.1 scrub ok 2024-03-11T22:22:59.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1916604260' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:22:59.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:22:59.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[39715]: 3.c scrub starts 2024-03-11T22:22:59.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[39715]: 3.c scrub ok 2024-03-11T22:22:59.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[39715]: pgmap v680: 168 pgs: 168 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:22:59.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[39715]: 6.1 scrub starts 2024-03-11T22:22:59.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[39715]: 6.1 scrub ok 2024-03-11T22:22:59.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1916604260' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:22:59.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:22:59 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:22:59.816 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:22:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:59.841 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:22:59.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:22:59.942 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:22:59.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:22:59.968 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:22:59.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:00.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:00.093 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:00.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:00.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:00.195 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:00.218 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:00.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:00.320 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:00.344 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:00.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:00.432 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:00.446 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:00.469 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:00.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:00.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:00.517 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:23:00.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:00.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:00.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:00.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:00.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:00.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:00.721 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:00.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:00.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[30178]: 6.19 scrub starts 2024-03-11T22:23:00.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[30178]: 6.19 scrub ok 2024-03-11T22:23:00.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[30178]: 3.13 scrub starts 2024-03-11T22:23:00.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[30178]: 3.13 scrub ok 2024-03-11T22:23:00.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:23:00.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[30178]: osdmap e352: 8 total, 8 up, 8 in 2024-03-11T22:23:00.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[30178]: 5.11 scrub starts 2024-03-11T22:23:00.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[30178]: 5.11 scrub ok 2024-03-11T22:23:00.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[39715]: 6.19 scrub starts 2024-03-11T22:23:00.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[39715]: 6.19 scrub ok 2024-03-11T22:23:00.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[39715]: 3.13 scrub starts 2024-03-11T22:23:00.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[39715]: 3.13 scrub ok 2024-03-11T22:23:00.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:23:00.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[39715]: osdmap e352: 8 total, 8 up, 8 in 2024-03-11T22:23:00.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[39715]: 5.11 scrub starts 2024-03-11T22:23:00.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:00 smithi045 ceph-mon[39715]: 5.11 scrub ok 2024-03-11T22:23:00.785 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:00 smithi144 ceph-mon[33984]: 6.19 scrub starts 2024-03-11T22:23:00.785 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:00 smithi144 ceph-mon[33984]: 6.19 scrub ok 2024-03-11T22:23:00.785 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:00 smithi144 ceph-mon[33984]: 3.13 scrub starts 2024-03-11T22:23:00.785 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:00 smithi144 ceph-mon[33984]: 3.13 scrub ok 2024-03-11T22:23:00.785 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:00 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:23:00.785 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:00 smithi144 ceph-mon[33984]: osdmap e352: 8 total, 8 up, 8 in 2024-03-11T22:23:00.786 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:00 smithi144 ceph-mon[33984]: 5.11 scrub starts 2024-03-11T22:23:00.786 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:00 smithi144 ceph-mon[33984]: 5.11 scrub ok 2024-03-11T22:23:00.822 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:00.845 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:00.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:00.947 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:00.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:00.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:00.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:01.072 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:01.096 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:01.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:01.197 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:01.221 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:01.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:01.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:01.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:01.347 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:01.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:01.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:01.405 DEBUG:teuthology.orchestra.run.smithi045:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-11T22:23:01.407 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T22:23:01.407 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T22:23:01.427 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:23:01.427 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-03-11T22:23:01.427 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-03-11T22:23:01.427 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:23:01.427 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.427 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110613 8.d 8:bf12733a:::benchmark_data_smithi045_115206_object110612:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.427 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.224050+0000", 2024-03-11T22:23:01.427 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 402.13664771800001, 2024-03-11T22:23:01.427 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.007938512, 2024-03-11T22:23:01.427 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.427 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.427 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.427 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110613 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.224050+0000", 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.224050+0000", 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.428 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.429 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.429 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.224052+0000", 2024-03-11T22:23:01.429 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.297e-06 2024-03-11T22:23:01.429 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.429 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.429 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.429 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.224083+0000", 2024-03-11T22:23:01.429 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1565e-05 2024-03-11T22:23:01.429 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.429 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.429 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.429 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.224099+0000", 2024-03-11T22:23:01.429 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5764e-05 2024-03-11T22:23:01.430 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.430 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.430 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.430 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.224108+0000", 2024-03-11T22:23:01.430 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.0899999999999994e-06 2024-03-11T22:23:01.430 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.430 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.430 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.430 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.224169+0000", 2024-03-11T22:23:01.430 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1382000000000003e-05 2024-03-11T22:23:01.430 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.430 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.430 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.224180+0000", 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0976e-05 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.229036+0000", 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.004855829 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.229165+0000", 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012908 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.431 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.229276+0000", 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000111202 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.231796+0000", 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0025200560000000001 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.231864+0000", 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7812999999999996e-05 2024-03-11T22:23:01.432 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.231881+0000", 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6750000000000001e-05 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.231940+0000", 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9314000000000002e-05 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.231989+0000", 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8393999999999998e-05 2024-03-11T22:23:01.433 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.434 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.434 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.434 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.434 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.434 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110615 8.d 8:b12f2657:::benchmark_data_smithi045_115206_object110614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.434 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.225041+0000", 2024-03-11T22:23:01.434 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 402.13565702800003, 2024-03-11T22:23:01.434 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.014589119, 2024-03-11T22:23:01.434 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.434 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.434 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.434 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110615 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.225041+0000", 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.225041+0000", 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.435 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.225042+0000", 2024-03-11T22:23:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2200000000000003e-07 2024-03-11T22:23:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.225134+0000", 2024-03-11T22:23:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.2313999999999997e-05 2024-03-11T22:23:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.225143+0000", 2024-03-11T22:23:01.436 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6049999999999998e-06 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.225149+0000", 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8889999999999996e-06 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.225203+0000", 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3745000000000001e-05 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.225211+0000", 2024-03-11T22:23:01.437 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.875e-06 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.232010+0000", 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.006799228 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.232123+0000", 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000112414 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.438 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.232220+0000", 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7623999999999999e-05 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.239364+0000", 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0071438129999999997 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.239465+0000", 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000100869 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.439 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.239514+0000", 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9277000000000003e-05 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.239581+0000", 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6818999999999998e-05 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.239630+0000", 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8925000000000003e-05 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.440 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110624 8.d 8:beb8c653:::benchmark_data_smithi045_115206_object110623:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.235390+0000", 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 402.12530780100002, 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0107894710000001, 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110624 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.441 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.235390+0000", 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.235390+0000", 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.235392+0000", 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7120000000000001e-06 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.442 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.443 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.443 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.235458+0000", 2024-03-11T22:23:01.443 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5968000000000003e-05 2024-03-11T22:23:01.443 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.443 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.443 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.443 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.235469+0000", 2024-03-11T22:23:01.443 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1174e-05 2024-03-11T22:23:01.443 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.443 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.443 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.443 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.235475+0000", 2024-03-11T22:23:01.443 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1720000000000004e-06 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.235541+0000", 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.5350000000000003e-05 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.235551+0000", 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0481e-05 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.234258+0000", 2024-03-11T22:23:01.444 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.99870675399999997 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.234398+0000", 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000140137 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.234577+0000", 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000179178 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.239660+0000", 2024-03-11T22:23:01.445 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0050833320000000003 2024-03-11T22:23:01.446 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.446 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.446 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.446 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.239690+0000", 2024-03-11T22:23:01.446 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9663e-05 2024-03-11T22:23:01.446 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.446 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.446 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.446 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.246093+0000", 2024-03-11T22:23:01.446 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0064025740000000003 2024-03-11T22:23:01.446 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.446 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.446 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.246143+0000", 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0399000000000003e-05 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.246180+0000", 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6576999999999999e-05 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110643 8.d 8:bf6ccc26:::benchmark_data_smithi045_115206_object110642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.447 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.251097+0000", 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 402.109600613, 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.995259738, 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110643 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.251097+0000", 2024-03-11T22:23:01.448 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.251097+0000", 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.251098+0000", 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8999999999999996e-07 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.449 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.251154+0000", 2024-03-11T22:23:01.450 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5974000000000003e-05 2024-03-11T22:23:01.450 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.450 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.450 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.450 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.251161+0000", 2024-03-11T22:23:01.450 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8970000000000002e-06 2024-03-11T22:23:01.450 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.450 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.450 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.450 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.251165+0000", 2024-03-11T22:23:01.450 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4909999999999999e-06 2024-03-11T22:23:01.450 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.450 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.251216+0000", 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0683000000000002e-05 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.251225+0000", 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9670000000000007e-06 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.236758+0000", 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98553341299999997 2024-03-11T22:23:01.451 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.236864+0000", 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000106066 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.237001+0000", 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000136577 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.246071+0000", 2024-03-11T22:23:01.452 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0090697859999999998 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.246277+0000", 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00020638300000000001 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.246289+0000", 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1647e-05 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.453 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.246325+0000", 2024-03-11T22:23:01.454 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5833000000000001e-05 2024-03-11T22:23:01.454 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.454 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.454 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.454 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.246357+0000", 2024-03-11T22:23:01.454 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2431000000000002e-05 2024-03-11T22:23:01.454 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.454 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.454 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.454 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.454 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.454 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110662 8.d 8:bcf84f01:::benchmark_data_smithi045_115206_object110661:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.454 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.269551+0000", 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 402.09114688699998, 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98354564799999999, 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110662 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.269551+0000", 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.455 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.456 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.456 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.456 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.269551+0000", 2024-03-11T22:23:01.456 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.456 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.456 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.456 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.456 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.269553+0000", 2024-03-11T22:23:01.456 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.677e-06 2024-03-11T22:23:01.456 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.456 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.456 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.456 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.269587+0000", 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4431999999999998e-05 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.269602+0000", 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.489e-05 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.269612+0000", 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.5799999999999998e-06 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.457 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.269684+0000", 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2768000000000005e-05 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.269693+0000", 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6179999999999995e-06 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.239793+0000", 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.97009987499999994 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.458 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.239994+0000", 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00020129999999999999 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.240166+0000", 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00017148900000000001 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.246227+0000", 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0060612879999999997 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.459 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.246364+0000", 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000137195 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.252967+0000", 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0066027350000000002 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.253028+0000", 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1056999999999995e-05 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.460 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.253097+0000", 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8744000000000005e-05 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110676 8.d 8:b986eafa:::benchmark_data_smithi045_115206_object110675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.284756+0000", 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 402.07594148700002, 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.968512017, 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.461 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110676 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.284756+0000", 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.284756+0000", 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.462 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.463 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.463 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.284758+0000", 2024-03-11T22:23:01.463 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2470000000000001e-06 2024-03-11T22:23:01.463 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.463 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.463 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.463 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.284792+0000", 2024-03-11T22:23:01.463 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4109999999999997e-05 2024-03-11T22:23:01.463 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.463 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.463 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.463 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.284804+0000", 2024-03-11T22:23:01.463 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2634e-05 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.284811+0000", 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2899999999999999e-06 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.284864+0000", 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3434999999999999e-05 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.284872+0000", 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2209999999999997e-06 2024-03-11T22:23:01.464 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.242618+0000", 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.957745391 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.242770+0000", 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015264 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.243241+0000", 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000470483 2024-03-11T22:23:01.465 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.466 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.466 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.466 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.246419+0000", 2024-03-11T22:23:01.466 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0031780129999999999 2024-03-11T22:23:01.466 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.466 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.466 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.466 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.249628+0000", 2024-03-11T22:23:01.466 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0032095420000000001 2024-03-11T22:23:01.466 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.466 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.466 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.466 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.253171+0000", 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0035430040000000002 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.253232+0000", 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0520000000000003e-05 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.253268+0000", 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6486999999999998e-05 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.467 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110681 8.d 8:b5001144:::benchmark_data_smithi045_115206_object110680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.287985+0000", 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 402.07271311300002, 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.97069233700000002, 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110681 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.468 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.287985+0000", 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.287985+0000", 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.287986+0000", 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3039999999999999e-06 2024-03-11T22:23:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.288018+0000", 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1841999999999998e-05 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.288040+0000", 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.2177000000000001e-05 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.288049+0000", 2024-03-11T22:23:01.470 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9830000000000002e-06 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.288107+0000", 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7510000000000003e-05 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.288113+0000", 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9190000000000004e-06 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.246455+0000", 2024-03-11T22:23:01.471 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.95834212200000002 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.247312+0000", 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00085717499999999995 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.247478+0000", 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016621299999999999 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.253113+0000", 2024-03-11T22:23:01.472 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0056346419999999996 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.253322+0000", 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000208829 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.258589+0000", 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0052673110000000002 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.473 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.258641+0000", 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2006000000000003e-05 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.258677+0000", 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6303999999999997e-05 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110686 8.d 8:b85a9e89:::benchmark_data_smithi045_115206_object110685:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.292635+0000", 2024-03-11T22:23:01.474 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 402.06806259299998, 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.98124002200000005, 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110686 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.292635+0000", 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.475 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.292635+0000", 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.292637+0000", 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4580000000000001e-06 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.292682+0000", 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5439000000000003e-05 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.476 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.292697+0000", 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4516e-05 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.292704+0000", 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5619999999999998e-06 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.293113+0000", 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00040891199999999999 2024-03-11T22:23:01.477 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.293125+0000", 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1321000000000001e-05 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.249660+0000", 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.95653569800000005 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.249817+0000", 2024-03-11T22:23:01.478 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015720900000000001 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.249982+0000", 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016446700000000001 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.258553+0000", 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0085710260000000007 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.479 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.267613+0000", 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0090601870000000008 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.273777+0000", 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0061641530000000003 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.273826+0000", 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8925000000000003e-05 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.480 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.273875+0000", 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9149e-05 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110692 8.d 8:bc7192a6:::benchmark_data_smithi045_115206_object110691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.295692+0000", 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 402.06500617500001, 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.97826244799999995, 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.481 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110692 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.295692+0000", 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.295692+0000", 2024-03-11T22:23:01.482 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.295693+0000", 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2470000000000001e-06 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.296707+0000", 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0010139680000000001 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.483 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.296720+0000", 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2943e-05 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.296728+0000", 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6990000000000003e-06 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.297432+0000", 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00070416900000000004 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.484 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.297438+0000", 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2029999999999997e-06 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.253353+0000", 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.95591529600000003 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.253500+0000", 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014685100000000001 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.485 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.486 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.253618+0000", 2024-03-11T22:23:01.486 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000118114 2024-03-11T22:23:01.486 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.486 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.486 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.486 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.261407+0000", 2024-03-11T22:23:01.486 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0077883140000000002 2024-03-11T22:23:01.486 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.486 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.486 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.486 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.267656+0000", 2024-03-11T22:23:01.486 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0062495950000000002 2024-03-11T22:23:01.486 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.273884+0000", 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.006228071 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.273923+0000", 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9097000000000002e-05 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.273954+0000", 2024-03-11T22:23:01.487 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0880999999999998e-05 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110698 8.d 8:be31f04a:::benchmark_data_smithi045_115206_object110697:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.299592+0000", 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 402.06110595899997, 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.97444513099999996, 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110698 2024-03-11T22:23:01.488 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.299592+0000", 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.299592+0000", 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.299593+0000", 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.37e-07 2024-03-11T22:23:01.489 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.299628+0000", 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5210999999999999e-05 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.299635+0000", 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1670000000000002e-06 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.299640+0000", 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5140000000000001e-06 2024-03-11T22:23:01.490 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.299685+0000", 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.4555999999999998e-05 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.299692+0000", 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2439999999999996e-06 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.255948+0000", 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.95625542900000005 2024-03-11T22:23:01.491 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.256079+0000", 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013130699999999999 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.256237+0000", 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000158095 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.267695+0000", 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.011457861 2024-03-11T22:23:01.492 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.273760+0000", 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0060653809999999999 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.273960+0000", 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00020007900000000001 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.273998+0000", 2024-03-11T22:23:01.493 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7225999999999999e-05 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274037+0000", 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9524000000000003e-05 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110730 8.d 8:bc3a5c02:::benchmark_data_smithi045_115206_object110729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.339097+0000", 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 402.02160079399999, 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.93511228099999999, 2024-03-11T22:23:01.494 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110730 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.339097+0000", 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.495 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.339097+0000", 2024-03-11T22:23:01.496 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.496 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.496 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.496 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.496 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.339099+0000", 2024-03-11T22:23:01.496 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.677e-06 2024-03-11T22:23:01.496 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.496 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.496 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.496 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.339144+0000", 2024-03-11T22:23:01.496 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5531999999999997e-05 2024-03-11T22:23:01.496 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.496 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.339155+0000", 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0555e-05 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.339163+0000", 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9959999999999993e-06 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.339206+0000", 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3316000000000003e-05 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.497 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.339214+0000", 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3270000000000003e-06 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.258705+0000", 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.91949191900000005 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.258849+0000", 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000143127 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.498 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.259007+0000", 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000158791 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.267734+0000", 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0087263380000000002 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274086+0000", 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0063519759999999996 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.499 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274134+0000", 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8639e-05 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274178+0000", 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3234e-05 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274209+0000", 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1853999999999997e-05 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.500 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110734 8.d 8:b66c2d4a:::benchmark_data_smithi045_115206_object110733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.343440+0000", 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 402.01725787200002, 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.93092416899999997, 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110734 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.501 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.343440+0000", 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.343440+0000", 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.343441+0000", 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1299999999999998e-07 2024-03-11T22:23:01.502 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.343491+0000", 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9599999999999999e-05 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.343500+0000", 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3870000000000001e-06 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.343505+0000", 2024-03-11T22:23:01.503 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3460000000000001e-06 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.343554+0000", 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8440999999999999e-05 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.343562+0000", 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6410000000000008e-06 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.504 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.261444+0000", 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.91788115199999998 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.261578+0000", 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000134826 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.261741+0000", 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016266300000000001 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.505 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.267772+0000", 2024-03-11T22:23:01.506 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0060308760000000001 2024-03-11T22:23:01.506 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.506 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.506 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.506 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274090+0000", 2024-03-11T22:23:01.506 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0063185560000000003 2024-03-11T22:23:01.506 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.506 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.506 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.506 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274283+0000", 2024-03-11T22:23:01.506 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00019252900000000001 2024-03-11T22:23:01.506 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.506 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274331+0000", 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7534000000000003e-05 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274364+0000", 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3704999999999997e-05 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.507 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110743 8.d 8:bb19c039:::benchmark_data_smithi045_115206_object110742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.355073+0000", 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 402.005624969, 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.91941502900000005, 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110743 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.355073+0000", 2024-03-11T22:23:01.508 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.355073+0000", 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.355079+0000", 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5310000000000001e-06 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.355115+0000", 2024-03-11T22:23:01.509 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6761999999999998e-05 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.355129+0000", 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.345e-05 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.355137+0000", 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4770000000000003e-06 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.355207+0000", 2024-03-11T22:23:01.510 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9459e-05 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.355218+0000", 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0942e-05 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.264322+0000", 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.90910463699999999 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.264506+0000", 2024-03-11T22:23:01.511 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00018338299999999999 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.264775+0000", 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026917400000000001 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.267811+0000", 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0030365700000000002 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274095+0000", 2024-03-11T22:23:01.512 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0062837750000000001 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274404+0000", 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00030874199999999998 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274452+0000", 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8112999999999998e-05 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.513 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274488+0000", 2024-03-11T22:23:01.514 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6013999999999998e-05 2024-03-11T22:23:01.514 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.514 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.514 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.514 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.514 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.514 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110777 8.d 8:b81c60bf:::benchmark_data_smithi045_115206_object110776:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.514 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.408817+0000", 2024-03-11T22:23:01.514 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 401.95188090699997, 2024-03-11T22:23:01.514 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.86579415999999998, 2024-03-11T22:23:01.514 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.514 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.514 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110777 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.408817+0000", 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.408817+0000", 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.515 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.408818+0000", 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2020000000000001e-06 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.408895+0000", 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6748000000000004e-05 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.408914+0000", 2024-03-11T22:23:01.516 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8947e-05 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.408923+0000", 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.2280000000000004e-06 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.408976+0000", 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3282000000000002e-05 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.408984+0000", 2024-03-11T22:23:01.517 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6129999999999998e-06 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.267842+0000", 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.85885798999999996 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.267981+0000", 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000139037 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.518 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.268154+0000", 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000173349 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274075+0000", 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0059206490000000001 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274100+0000", 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4658e-05 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.519 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274529+0000", 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00042882400000000001 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274578+0000", 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9870999999999997e-05 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274611+0000", 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2762000000000002e-05 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.520 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110784 8.d 8:bf80995c:::benchmark_data_smithi045_115206_object110783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.427489+0000", 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 401.93320917800003, 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.84723938700000001, 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110784 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.521 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.427489+0000", 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.427489+0000", 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.427490+0000", 2024-03-11T22:23:01.522 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3e-06 2024-03-11T22:23:01.523 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.523 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.523 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.523 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.427536+0000", 2024-03-11T22:23:01.523 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5668999999999999e-05 2024-03-11T22:23:01.523 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.523 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.523 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.523 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.427548+0000", 2024-03-11T22:23:01.523 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2407e-05 2024-03-11T22:23:01.523 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.523 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.523 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.427556+0000", 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4100000000000002e-06 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.427603+0000", 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7080000000000003e-05 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.427612+0000", 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.6150000000000003e-06 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.524 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.270884+0000", 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.84327199600000002 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.271034+0000", 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015010699999999999 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.271204+0000", 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000169808 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.525 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274104+0000", 2024-03-11T22:23:01.526 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0028998930000000002 2024-03-11T22:23:01.526 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.526 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.526 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.526 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274244+0000", 2024-03-11T22:23:01.526 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013998 2024-03-11T22:23:01.526 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.526 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.526 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.526 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274651+0000", 2024-03-11T22:23:01.526 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00040683599999999999 2024-03-11T22:23:01.526 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.526 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.527 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.527 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274697+0000", 2024-03-11T22:23:01.527 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6415999999999997e-05 2024-03-11T22:23:01.527 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.527 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.527 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.527 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274728+0000", 2024-03-11T22:23:01.527 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0870000000000001e-05 2024-03-11T22:23:01.527 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.527 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.527 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.527 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.527 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.24935.0:110804 8.d 8:b1ce3f28:::benchmark_data_smithi045_115206_object110803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:16:19.512333+0000", 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 401.84836460899999, 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.77049338899999997, 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.24935", 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/1464418996", 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 110804 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.512333+0000", 2024-03-11T22:23:01.528 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.512333+0000", 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.512334+0000", 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7100000000000001e-07 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.512384+0000", 2024-03-11T22:23:01.529 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9653e-05 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.512392+0000", 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9980000000000003e-06 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.512397+0000", 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0710000000000002e-06 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.530 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.512436+0000", 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8868000000000001e-05 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for peered", 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:19.512441+0000", 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7989999999999999e-06 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274759+0000", 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.76231768499999997 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.531 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.274887+0000", 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000127498 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.275057+0000", 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000170839 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.282630+0000", 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0075722029999999996 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.532 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.282647+0000", 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7632999999999999e-05 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.282706+0000", 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8530999999999999e-05 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.282777+0000", 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1094999999999996e-05 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.533 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:16:20.282827+0000", 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9845000000000001e-05 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.25142.0:27627 9.a 9:5a94e6de:::benchmark_data_smithi045_141553_object27626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:19:05.235015+0000", 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 236.12568271500001, 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21289703300000001, 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.534 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.25142", 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 27627 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.235015+0000", 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.235015+0000", 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.535 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.235016+0000", 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3499999999999996e-07 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.235183+0000", 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000166794 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.235192+0000", 2024-03-11T22:23:01.536 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.5869999999999999e-06 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.235201+0000", 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3499999999999997e-06 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.237779+0000", 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002578388 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.237903+0000", 2024-03-11T22:23:01.537 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012363900000000001 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,7", 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.238152+0000", 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00024969899999999998 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.240819+0000", 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026671709999999999 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.241717+0000", 2024-03-11T22:23:01.538 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00089774899999999999 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.447851+0000", 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20613405000000001 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.447887+0000", 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5212000000000001e-05 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.447912+0000", 2024-03-11T22:23:01.539 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5758999999999999e-05 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.25142.0:27628 9.a 9:5bb2d031:::benchmark_data_smithi045_141553_object27627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:19:05.235215+0000", 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 236.125482812, 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21279201, 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.25142", 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 27628 2024-03-11T22:23:01.540 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.235215+0000", 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.235215+0000", 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.541 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.235216+0000", 2024-03-11T22:23:01.542 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6599999999999996e-07 2024-03-11T22:23:01.542 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.542 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.542 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.542 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.235242+0000", 2024-03-11T22:23:01.542 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.6015000000000001e-05 2024-03-11T22:23:01.542 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.542 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.542 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.542 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.235258+0000", 2024-03-11T22:23:01.542 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.6269e-05 2024-03-11T22:23:01.542 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.542 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.235265+0000", 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2350000000000002e-06 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.239493+0000", 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0042273019999999996 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.239608+0000", 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00011595300000000001 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.543 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,7", 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.239743+0000", 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00013442599999999999 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.240975+0000", 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00123161 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.241908+0000", 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00093382600000000001 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.544 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.447943+0000", 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.206035048 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.447983+0000", 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9267999999999997e-05 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:19:05.448007+0000", 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4491999999999999e-05 2024-03-11T22:23:01.545 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.546 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.546 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.546 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.546 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.546 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.25142.0:183801 9.6 9:6eb676d8:::benchmark_data_smithi045_141553_object59681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-03-11T22:23:01.546 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:21:24.370214+0000", 2024-03-11T22:23:01.546 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 96.990483984999997, 2024-03-11T22:23:01.546 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21540403899999999, 2024-03-11T22:23:01.546 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.546 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.546 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.546 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.25142", 2024-03-11T22:23:01.546 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:23:01.547 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 183801 2024-03-11T22:23:01.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.547 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.547 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.547 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.547 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.370214+0000", 2024-03-11T22:23:01.547 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.547 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.547 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.547 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.370214+0000", 2024-03-11T22:23:01.547 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.547 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.370214+0000", 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9100000000000004e-07 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.370217+0000", 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8899999999999999e-06 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.370224+0000", 2024-03-11T22:23:01.548 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6279999999999999e-06 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.370230+0000", 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8030000000000002e-06 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.372214+0000", 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0019846 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.372523+0000", 2024-03-11T22:23:01.549 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000308998 2024-03-11T22:23:01.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.550 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:23:01.550 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.372644+0000", 2024-03-11T22:23:01.550 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012027300000000001 2024-03-11T22:23:01.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.550 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.550 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.377930+0000", 2024-03-11T22:23:01.550 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0052866950000000001 2024-03-11T22:23:01.550 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.550 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.550 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.377939+0000", 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.191e-06 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.585414+0000", 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20747510199999999 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.585468+0000", 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.465e-05 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.551 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.585618+0000", 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014971799999999999 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.25142.0:183804 9.6 9:6f88e686:::benchmark_data_smithi045_141553_object59684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:21:24.372724+0000", 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 96.987973780999994, 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.21304375, 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:01.552 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.25142", 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 183804 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.372724+0000", 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:01.553 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.372724+0000", 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.372725+0000", 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6000000000000002e-07 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.372729+0000", 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.703e-06 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.554 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.372738+0000", 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.5009999999999993e-06 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.372745+0000", 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9229999999999996e-06 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.375209+0000", 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024639470000000002 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.555 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.375323+0000", 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000113558 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.375622+0000", 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00029960300000000001 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.380010+0000", 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.004387274 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.556 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.380069+0000", 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9666000000000001e-05 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.585679+0000", 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.20560929999999999 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.585728+0000", 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9242000000000001e-05 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:01.557 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:01.558 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:01.558 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:21:24.585768+0000", 2024-03-11T22:23:01.558 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0173000000000003e-05 2024-03-11T22:23:01.558 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.558 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.558 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.558 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:01.558 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:01.558 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:23:01.558 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:01.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:01.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:01.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:01.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:01.707 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:01.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:01.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:01.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[39715]: pgmap v682: 168 pgs: 168 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-03-11T22:23:01.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[39715]: 5.8 scrub starts 2024-03-11T22:23:01.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[39715]: 5.8 scrub ok 2024-03-11T22:23:01.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[39715]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:23:01.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[39715]: 4.14 scrub starts 2024-03-11T22:23:01.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[39715]: 4.14 scrub ok 2024-03-11T22:23:01.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:23:01.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[30178]: pgmap v682: 168 pgs: 168 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-03-11T22:23:01.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[30178]: 5.8 scrub starts 2024-03-11T22:23:01.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[30178]: 5.8 scrub ok 2024-03-11T22:23:01.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[30178]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:23:01.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[30178]: 4.14 scrub starts 2024-03-11T22:23:01.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[30178]: 4.14 scrub ok 2024-03-11T22:23:01.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:23:01.809 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:01.814 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:hints = 1 2024-03-11T22:23:01.814 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T22:23:01.814 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Object prefix: benchmark_data_smithi045_169175 2024-03-11T22:23:01.815 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:23:01.815 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 0 0 0 0 0 0 - 0 2024-03-11T22:23:01.843 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:01.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:01.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:01.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:01 smithi144 ceph-mon[33984]: pgmap v682: 168 pgs: 168 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-03-11T22:23:01.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:01 smithi144 ceph-mon[33984]: 5.8 scrub starts 2024-03-11T22:23:01.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:01 smithi144 ceph-mon[33984]: 5.8 scrub ok 2024-03-11T22:23:01.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:01 smithi144 ceph-mon[33984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:23:01.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:01 smithi144 ceph-mon[33984]: 4.14 scrub starts 2024-03-11T22:23:01.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:01 smithi144 ceph-mon[33984]: 4.14 scrub ok 2024-03-11T22:23:01.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:23:01.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:01.984 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:01.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:02.086 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:02.122 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:02.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:02.224 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:02.259 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:02.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:02.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:02.361 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:02.387 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:02.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:02.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:02.484 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T22:23:02.488 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:02.566 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:02.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:02.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:02.668 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:02.708 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:02.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:02.784 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:23:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:23:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:23:02.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:02 smithi045 ceph-mon[39715]: 1.f scrub starts 2024-03-11T22:23:02.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:02 smithi045 ceph-mon[39715]: 1.f scrub ok 2024-03-11T22:23:02.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:02 smithi045 ceph-mon[30178]: 1.f scrub starts 2024-03-11T22:23:02.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:02 smithi045 ceph-mon[30178]: 1.f scrub ok 2024-03-11T22:23:02.802 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:02 smithi144 ceph-mon[33984]: 1.f scrub starts 2024-03-11T22:23:02.802 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:02 smithi144 ceph-mon[33984]: 1.f scrub ok 2024-03-11T22:23:02.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:02.815 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 1 16 3099 3083 192.678 192.688 0.00385359 0.00516991 2024-03-11T22:23:02.851 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:02.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:02.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:02.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:02.993 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:02.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:03.095 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:03.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:03.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:03.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:03.237 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:23:03.239 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:03.293 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:03.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:03.395 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:03.430 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:03.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:03.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:03.521 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:23:03.532 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:03.570 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:03.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:03.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:03.672 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:03.710 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:03.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:03.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:03 smithi045 ceph-mon[39715]: 6.b scrub starts 2024-03-11T22:23:03.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:03 smithi045 ceph-mon[39715]: 6.b scrub ok 2024-03-11T22:23:03.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:03 smithi045 ceph-mon[39715]: pgmap v683: 168 pgs: 168 active+clean; 8.0 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 952 KiB/s wr, 16 op/s 2024-03-11T22:23:03.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:03 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/71221918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:03.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:03 smithi045 ceph-mon[30178]: 6.b scrub starts 2024-03-11T22:23:03.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:03 smithi045 ceph-mon[30178]: 6.b scrub ok 2024-03-11T22:23:03.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:03 smithi045 ceph-mon[30178]: pgmap v683: 168 pgs: 168 active+clean; 8.0 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 952 KiB/s wr, 16 op/s 2024-03-11T22:23:03.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:03 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/71221918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:03.804 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:03 smithi144 ceph-mon[33984]: 6.b scrub starts 2024-03-11T22:23:03.804 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:03 smithi144 ceph-mon[33984]: 6.b scrub ok 2024-03-11T22:23:03.804 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:03 smithi144 ceph-mon[33984]: pgmap v683: 168 pgs: 168 active+clean; 8.0 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 952 KiB/s wr, 16 op/s 2024-03-11T22:23:03.804 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:03 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/71221918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:03.811 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:03.815 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 2 16 6073 6057 189.267 185.875 0.0015848 0.00516939 2024-03-11T22:23:03.834 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:03.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:03.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:03.936 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:03.971 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:03.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:04.073 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:04.111 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:04.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:04.214 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:04.254 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:04.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:04.301 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:23:04.301 INFO:teuthology.orchestra.run.smithi144.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T22:23:04.301 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:23:04.355 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:04.389 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:04.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:04.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:04.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:04.533 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:04.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:04.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:04.635 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:04.670 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:04.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:04.740 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:04 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:23:04.740 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:04 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:23:04.740 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:04 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/71221918' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:23:04.740 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:04 smithi144 ceph-mon[33984]: osdmap e353: 8 total, 8 up, 8 in 2024-03-11T22:23:04.740 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:04 smithi144 ceph-mon[33984]: 6.6 scrub starts 2024-03-11T22:23:04.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:04 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:23:04.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:04 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:23:04.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:04 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/71221918' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:23:04.761 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:04 smithi045 ceph-mon[39715]: osdmap e353: 8 total, 8 up, 8 in 2024-03-11T22:23:04.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:04 smithi045 ceph-mon[39715]: 6.6 scrub starts 2024-03-11T22:23:04.762 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:04 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:23:04.762 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:04 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:23:04.762 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:04 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/71221918' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:23:04.762 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:04 smithi045 ceph-mon[30178]: osdmap e353: 8 total, 8 up, 8 in 2024-03-11T22:23:04.762 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:04 smithi045 ceph-mon[30178]: 6.6 scrub starts 2024-03-11T22:23:04.771 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:04.806 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:04.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:04.815 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 3 16 9148 9132 190.236 192.188 0.00350445 0.00524728 2024-03-11T22:23:04.908 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:04.942 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:04.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:04.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:05.044 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:05.082 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:05.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:05.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:05.141 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:23:05.182 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:23:05.184 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:05.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:05.220 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:05.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:05.222 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:23:05.265 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:23:05.304 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:23:05.320 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:05.344 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:23:05.357 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:05.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:05.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:05.384 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:23:05.429 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:23:05.433 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:23:05.433 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:23:05.433 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:23:05.433 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:23:05.433 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:23:05.433 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:23:05.433 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:23:05.433 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:23:05.459 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:05.491 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:05.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:05.593 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:05.628 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:05.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:05.730 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:05.765 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:05.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:05.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[39715]: 6.6 scrub ok 2024-03-11T22:23:05.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[39715]: 5.f scrub starts 2024-03-11T22:23:05.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[39715]: 5.f scrub ok 2024-03-11T22:23:05.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[39715]: pgmap v685: 168 pgs: 168 active+clean; 8.0 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 952 KiB/s wr, 16 op/s 2024-03-11T22:23:05.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[39715]: 3.4 deep-scrub starts 2024-03-11T22:23:05.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[39715]: 3.4 deep-scrub ok 2024-03-11T22:23:05.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/58032519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:05.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[30178]: 6.6 scrub ok 2024-03-11T22:23:05.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[30178]: 5.f scrub starts 2024-03-11T22:23:05.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[30178]: 5.f scrub ok 2024-03-11T22:23:05.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[30178]: pgmap v685: 168 pgs: 168 active+clean; 8.0 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 952 KiB/s wr, 16 op/s 2024-03-11T22:23:05.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[30178]: 3.4 deep-scrub starts 2024-03-11T22:23:05.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[30178]: 3.4 deep-scrub ok 2024-03-11T22:23:05.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:05 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/58032519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:05.815 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 4 16 12091 12075 188.657 183.938 0.00580681 0.00529263 2024-03-11T22:23:05.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:05 smithi144 ceph-mon[33984]: 6.6 scrub ok 2024-03-11T22:23:05.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:05 smithi144 ceph-mon[33984]: 5.f scrub starts 2024-03-11T22:23:05.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:05 smithi144 ceph-mon[33984]: 5.f scrub ok 2024-03-11T22:23:05.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:05 smithi144 ceph-mon[33984]: pgmap v685: 168 pgs: 168 active+clean; 8.0 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 952 KiB/s wr, 16 op/s 2024-03-11T22:23:05.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:05 smithi144 ceph-mon[33984]: 3.4 deep-scrub starts 2024-03-11T22:23:05.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:05 smithi144 ceph-mon[33984]: 3.4 deep-scrub ok 2024-03-11T22:23:05.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:05 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/58032519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:05.867 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:05.907 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:05.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:06.009 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:06.049 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:06.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:06.090 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:23:06.090 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:23:06.151 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:06.191 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:06.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:06.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:06.330 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:06.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:06.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:06.432 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:06.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:06.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:06.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:06.572 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:06.611 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:06.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:06.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:06.712 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:06.749 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:06.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:06.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:06.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:23:06.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:23:06.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[30178]: 4.d deep-scrub starts 2024-03-11T22:23:06.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[30178]: 4.d deep-scrub ok 2024-03-11T22:23:06.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[30178]: osdmap e354: 8 total, 8 up, 8 in 2024-03-11T22:23:06.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[30178]: 4.0 scrub starts 2024-03-11T22:23:06.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[30178]: 4.0 scrub ok 2024-03-11T22:23:06.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:23:06.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:23:06.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[39715]: 4.d deep-scrub starts 2024-03-11T22:23:06.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[39715]: 4.d deep-scrub ok 2024-03-11T22:23:06.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[39715]: osdmap e354: 8 total, 8 up, 8 in 2024-03-11T22:23:06.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[39715]: 4.0 scrub starts 2024-03-11T22:23:06.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:06 smithi045 ceph-mon[39715]: 4.0 scrub ok 2024-03-11T22:23:06.815 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 5 15 14834 14819 185.223 171.5 0.00356736 0.0053912 2024-03-11T22:23:06.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:23:06.826 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:06 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:23:06.826 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:06 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:23:06.826 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:06 smithi144 ceph-mon[33984]: 4.d deep-scrub starts 2024-03-11T22:23:06.826 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:06 smithi144 ceph-mon[33984]: 4.d deep-scrub ok 2024-03-11T22:23:06.826 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:06 smithi144 ceph-mon[33984]: osdmap e354: 8 total, 8 up, 8 in 2024-03-11T22:23:06.826 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:06 smithi144 ceph-mon[33984]: 4.0 scrub starts 2024-03-11T22:23:06.826 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:06 smithi144 ceph-mon[33984]: 4.0 scrub ok 2024-03-11T22:23:06.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:23:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:23:06.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:06.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:06.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:06.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:07.001 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:07.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:07.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:07.145 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:07.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:07.183 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:07.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:07.260 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:07.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:07.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:07.320 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:07.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:07.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:07.455 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:07.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:07.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:07.557 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:07.593 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:07.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:07.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:07.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:07 smithi144 ceph-mon[33984]: 2.2 scrub starts 2024-03-11T22:23:07.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:07 smithi144 ceph-mon[33984]: 2.2 scrub ok 2024-03-11T22:23:07.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:07 smithi144 ceph-mon[33984]: pgmap v687: 168 pgs: 168 active+clean; 478 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 745 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T22:23:07.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:07 smithi144 ceph-mon[33984]: 3.9 scrub starts 2024-03-11T22:23:07.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:07 smithi144 ceph-mon[33984]: 3.9 scrub ok 2024-03-11T22:23:07.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:07.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:07 smithi144 ceph-mon[33984]: 6.1e deep-scrub starts 2024-03-11T22:23:07.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:07 smithi144 ceph-mon[33984]: 6.1e deep-scrub ok 2024-03-11T22:23:07.695 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:07.723 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:07.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:07.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[30178]: 2.2 scrub starts 2024-03-11T22:23:07.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[30178]: 2.2 scrub ok 2024-03-11T22:23:07.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[30178]: pgmap v687: 168 pgs: 168 active+clean; 478 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 745 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T22:23:07.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[30178]: 3.9 scrub starts 2024-03-11T22:23:07.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[30178]: 3.9 scrub ok 2024-03-11T22:23:07.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:07.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[30178]: 6.1e deep-scrub starts 2024-03-11T22:23:07.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[30178]: 6.1e deep-scrub ok 2024-03-11T22:23:07.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[39715]: 2.2 scrub starts 2024-03-11T22:23:07.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[39715]: 2.2 scrub ok 2024-03-11T22:23:07.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[39715]: pgmap v687: 168 pgs: 168 active+clean; 478 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 745 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-11T22:23:07.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[39715]: 3.9 scrub starts 2024-03-11T22:23:07.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[39715]: 3.9 scrub ok 2024-03-11T22:23:07.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:07.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[39715]: 6.1e deep-scrub starts 2024-03-11T22:23:07.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:07 smithi045 ceph-mon[39715]: 6.1e deep-scrub ok 2024-03-11T22:23:07.815 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 6 16 16893 16877 175.788 128.625 0.00373877 0.00568272 2024-03-11T22:23:07.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:07.861 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:07.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:07.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:08.001 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:08.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:08.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:08.085 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T22:23:08.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:08.139 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:08.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:08.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:23:08.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:08.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:08.290 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:08.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:08.391 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:08.414 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:23:08.414 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:08.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:08.517 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:08.552 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:08.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:08.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:08.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:08.691 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:08.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:08.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:08.815 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 7 16 19118 19102 170.54 139.062 0.00478594 0.00584895 2024-03-11T22:23:08.823 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:08.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:08.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:08 smithi144 ceph-mon[33984]: 6.e scrub starts 2024-03-11T22:23:08.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:08 smithi144 ceph-mon[33984]: 6.e scrub ok 2024-03-11T22:23:08.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:08.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:08.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:08.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:08 smithi045 ceph-mon[39715]: 6.e scrub starts 2024-03-11T22:23:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:08 smithi045 ceph-mon[39715]: 6.e scrub ok 2024-03-11T22:23:09.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:08 smithi045 ceph-mon[30178]: 6.e scrub starts 2024-03-11T22:23:09.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:08 smithi045 ceph-mon[30178]: 6.e scrub ok 2024-03-11T22:23:09.067 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:09.101 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:09.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:09.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:09.236 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:09.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:09.338 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:09.374 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:09.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:09.476 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:09.511 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:09.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:09.613 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:09.651 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:09.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:09.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:09.686 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:09 smithi144 ceph-mon[33984]: 5.0 scrub starts 2024-03-11T22:23:09.686 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:09 smithi144 ceph-mon[33984]: 5.0 scrub ok 2024-03-11T22:23:09.686 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:09 smithi144 ceph-mon[33984]: pgmap v688: 168 pgs: 168 active+clean; 765 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-11T22:23:09.686 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:09 smithi144 ceph-mon[33984]: 1.7 scrub starts 2024-03-11T22:23:09.686 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:09 smithi144 ceph-mon[33984]: 1.7 scrub ok 2024-03-11T22:23:09.753 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:09.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:09.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:09.815 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 8 16 21497 21481 167.807 148.688 0.00495339 0.00595405 2024-03-11T22:23:09.891 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:09 smithi045 ceph-mon[39715]: 5.0 scrub starts 2024-03-11T22:23:09.891 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:09 smithi045 ceph-mon[39715]: 5.0 scrub ok 2024-03-11T22:23:09.891 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:09 smithi045 ceph-mon[39715]: pgmap v688: 168 pgs: 168 active+clean; 765 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-11T22:23:09.891 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:09 smithi045 ceph-mon[39715]: 1.7 scrub starts 2024-03-11T22:23:09.891 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:09 smithi045 ceph-mon[39715]: 1.7 scrub ok 2024-03-11T22:23:09.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:09 smithi045 ceph-mon[30178]: 5.0 scrub starts 2024-03-11T22:23:09.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:09 smithi045 ceph-mon[30178]: 5.0 scrub ok 2024-03-11T22:23:09.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:09 smithi045 ceph-mon[30178]: pgmap v688: 168 pgs: 168 active+clean; 765 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-11T22:23:09.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:09 smithi045 ceph-mon[30178]: 1.7 scrub starts 2024-03-11T22:23:09.892 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:09 smithi045 ceph-mon[30178]: 1.7 scrub ok 2024-03-11T22:23:09.893 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:09.932 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:09.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:10.034 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:10.075 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:10.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:10.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:10.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:10.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:10.218 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:10.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:10.319 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:10.360 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:10.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:10.398 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:23:10.398 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:23:10.460 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:10.499 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:10.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:10.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:10.601 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:10.638 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:10.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:10.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:10.740 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:10.775 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:10.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:10.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:10.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:10 smithi045 ceph-mon[39715]: 6.1b scrub starts 2024-03-11T22:23:10.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:10 smithi045 ceph-mon[39715]: 6.1b scrub ok 2024-03-11T22:23:10.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:10 smithi045 ceph-mon[39715]: 6.10 deep-scrub starts 2024-03-11T22:23:10.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:10 smithi045 ceph-mon[39715]: 6.10 deep-scrub ok 2024-03-11T22:23:10.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:10 smithi045 ceph-mon[30178]: 6.1b scrub starts 2024-03-11T22:23:10.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:10 smithi045 ceph-mon[30178]: 6.1b scrub ok 2024-03-11T22:23:10.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:10 smithi045 ceph-mon[30178]: 6.10 deep-scrub starts 2024-03-11T22:23:10.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:10 smithi045 ceph-mon[30178]: 6.10 deep-scrub ok 2024-03-11T22:23:10.815 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 9 16 23906 23890 165.889 150.562 0.00622925 0.0060229 2024-03-11T22:23:10.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:10 smithi144 ceph-mon[33984]: 6.1b scrub starts 2024-03-11T22:23:10.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:10 smithi144 ceph-mon[33984]: 6.1b scrub ok 2024-03-11T22:23:10.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:10 smithi144 ceph-mon[33984]: 6.10 deep-scrub starts 2024-03-11T22:23:10.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:10 smithi144 ceph-mon[33984]: 6.10 deep-scrub ok 2024-03-11T22:23:10.877 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:10.911 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:10.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:10.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:11.012 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:11.048 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:11.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:11.098 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:11.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:11.186 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:11.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:11.189 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T22:23:11.288 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:11.330 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:11.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:11.431 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:11.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:11.471 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:11.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:11.498 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:23:11.498 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:23:11.498 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-11T22:23:11.498 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T22:23:11.498 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T22:23:11.498 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-11T22:23:11.571 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:11.616 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:23:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:11.618 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:11.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:11.719 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:11.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:11.762 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:11.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:11.816 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 10 16 26130 26114 163.199 139 0.0125937 0.00612176 2024-03-11T22:23:11.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:11 smithi144 ceph-mon[33984]: 1.11 scrub starts 2024-03-11T22:23:11.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:11 smithi144 ceph-mon[33984]: 1.11 scrub ok 2024-03-11T22:23:11.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:11 smithi144 ceph-mon[33984]: pgmap v689: 168 pgs: 168 active+clean; 1.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-11T22:23:11.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:11 smithi144 ceph-mon[33984]: 3.d scrub starts 2024-03-11T22:23:11.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:11 smithi144 ceph-mon[33984]: 3.d scrub ok 2024-03-11T22:23:11.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:11 smithi045 ceph-mon[30178]: 1.11 scrub starts 2024-03-11T22:23:11.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:11 smithi045 ceph-mon[30178]: 1.11 scrub ok 2024-03-11T22:23:11.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:11 smithi045 ceph-mon[30178]: pgmap v689: 168 pgs: 168 active+clean; 1.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-11T22:23:11.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:11 smithi045 ceph-mon[30178]: 3.d scrub starts 2024-03-11T22:23:11.854 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:11 smithi045 ceph-mon[30178]: 3.d scrub ok 2024-03-11T22:23:11.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:11 smithi045 ceph-mon[39715]: 1.11 scrub starts 2024-03-11T22:23:11.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:11 smithi045 ceph-mon[39715]: 1.11 scrub ok 2024-03-11T22:23:11.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:11 smithi045 ceph-mon[39715]: pgmap v689: 168 pgs: 168 active+clean; 1.1 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-11T22:23:11.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:11 smithi045 ceph-mon[39715]: 3.d scrub starts 2024-03-11T22:23:11.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:11 smithi045 ceph-mon[39715]: 3.d scrub ok 2024-03-11T22:23:11.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:11.910 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:11.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:12.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:12.049 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:12.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:12.151 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:12.186 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:12.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:12.288 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:12.325 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:12.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:12.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:12.427 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:12.464 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:12.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:12.566 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:12.601 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:12.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:12.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:12.703 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:12.728 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:12.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:12.816 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 11 16 28215 28199 160.209 130.312 0.00985453 0.00623531 2024-03-11T22:23:12.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:12.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:12 smithi144 ceph-mon[33984]: 1.c scrub starts 2024-03-11T22:23:12.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:12 smithi144 ceph-mon[33984]: 1.c scrub ok 2024-03-11T22:23:12.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:12 smithi144 ceph-mon[33984]: 1.2 scrub starts 2024-03-11T22:23:12.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:12 smithi144 ceph-mon[33984]: 1.2 scrub ok 2024-03-11T22:23:12.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:12 smithi144 ceph-mon[33984]: 5.1d scrub starts 2024-03-11T22:23:12.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:12 smithi144 ceph-mon[33984]: 5.1d scrub ok 2024-03-11T22:23:12.869 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:12.870 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:23:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:23:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:23:12.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:12 smithi045 ceph-mon[39715]: 1.c scrub starts 2024-03-11T22:23:12.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:12 smithi045 ceph-mon[39715]: 1.c scrub ok 2024-03-11T22:23:12.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:12 smithi045 ceph-mon[39715]: 1.2 scrub starts 2024-03-11T22:23:12.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:12 smithi045 ceph-mon[39715]: 1.2 scrub ok 2024-03-11T22:23:12.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:12 smithi045 ceph-mon[39715]: 5.1d scrub starts 2024-03-11T22:23:12.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:12 smithi045 ceph-mon[39715]: 5.1d scrub ok 2024-03-11T22:23:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:12.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:12.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:12 smithi045 ceph-mon[30178]: 1.c scrub starts 2024-03-11T22:23:12.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:12 smithi045 ceph-mon[30178]: 1.c scrub ok 2024-03-11T22:23:12.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:12 smithi045 ceph-mon[30178]: 1.2 scrub starts 2024-03-11T22:23:12.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:12 smithi045 ceph-mon[30178]: 1.2 scrub ok 2024-03-11T22:23:12.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:12 smithi045 ceph-mon[30178]: 5.1d scrub starts 2024-03-11T22:23:12.872 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:12 smithi045 ceph-mon[30178]: 5.1d scrub ok 2024-03-11T22:23:12.973 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:13.006 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:13.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:13.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:13.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:13.133 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:13.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:13.234 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:13.267 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:13.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:13.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:13.368 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:13.397 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:13.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:13.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:13.498 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:13.583 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:23:13.587 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:13.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:13.689 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:13.713 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:13.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:13.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[30178]: 3.17 scrub starts 2024-03-11T22:23:13.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[30178]: 3.17 scrub ok 2024-03-11T22:23:13.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[30178]: pgmap v690: 168 pgs: 168 active+clean; 1.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 169 MiB/s wr, 2.70k op/s 2024-03-11T22:23:13.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[30178]: 1.10 scrub starts 2024-03-11T22:23:13.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[30178]: 1.10 scrub ok 2024-03-11T22:23:13.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1480743412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:23:13.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:23:13.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[39715]: 3.17 scrub starts 2024-03-11T22:23:13.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[39715]: 3.17 scrub ok 2024-03-11T22:23:13.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[39715]: pgmap v690: 168 pgs: 168 active+clean; 1.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 169 MiB/s wr, 2.70k op/s 2024-03-11T22:23:13.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[39715]: 1.10 scrub starts 2024-03-11T22:23:13.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[39715]: 1.10 scrub ok 2024-03-11T22:23:13.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1480743412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:23:13.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:13 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:23:13.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:13.816 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 12 15 29658 29643 154.378 90.25 0.00738619 0.00647172 2024-03-11T22:23:13.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:13 smithi144 ceph-mon[33984]: 3.17 scrub starts 2024-03-11T22:23:13.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:13 smithi144 ceph-mon[33984]: 3.17 scrub ok 2024-03-11T22:23:13.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:13 smithi144 ceph-mon[33984]: pgmap v690: 168 pgs: 168 active+clean; 1.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 169 MiB/s wr, 2.70k op/s 2024-03-11T22:23:13.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:13 smithi144 ceph-mon[33984]: 1.10 scrub starts 2024-03-11T22:23:13.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:13 smithi144 ceph-mon[33984]: 1.10 scrub ok 2024-03-11T22:23:13.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:13 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1480743412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:23:13.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:13 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:23:13.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:13.856 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:13.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:13.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:13.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:13.985 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:13.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:14.086 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:14.118 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:14.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:14.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:14.220 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:14.255 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:14.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:14.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:14.357 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:14.392 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:14.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:14.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:14.493 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:14.529 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:14.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:14.630 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:14.653 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:14.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:14.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:14.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:14.794 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:14.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:14.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:14.816 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 13 16 31468 31452 151.199 113.062 0.0101159 0.00660569 2024-03-11T22:23:14.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:14 smithi144 ceph-mon[33984]: 6.1f scrub starts 2024-03-11T22:23:14.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:14 smithi144 ceph-mon[33984]: 6.1f scrub ok 2024-03-11T22:23:14.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:14 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:23:14.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:14 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:23:14.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:14 smithi144 ceph-mon[33984]: osdmap e355: 8 total, 8 up, 8 in 2024-03-11T22:23:14.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:14.922 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:14.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:14.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:15.024 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:15.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:14 smithi045 ceph-mon[30178]: 6.1f scrub starts 2024-03-11T22:23:15.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:14 smithi045 ceph-mon[30178]: 6.1f scrub ok 2024-03-11T22:23:15.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:14 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:23:15.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:14 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:23:15.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:14 smithi045 ceph-mon[30178]: osdmap e355: 8 total, 8 up, 8 in 2024-03-11T22:23:15.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:14 smithi045 ceph-mon[39715]: 6.1f scrub starts 2024-03-11T22:23:15.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:14 smithi045 ceph-mon[39715]: 6.1f scrub ok 2024-03-11T22:23:15.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:14 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:23:15.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:14 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:23:15.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:14 smithi045 ceph-mon[39715]: osdmap e355: 8 total, 8 up, 8 in 2024-03-11T22:23:15.062 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:15.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:15.164 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:15.200 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:15.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:15.301 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:15.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:15.325 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:15.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:15.426 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:15.449 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:15.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:15.550 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:15.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:15.580 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:15.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:15.681 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:15.704 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:15.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:15.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:15.806 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:15.816 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 14 15 32900 32885 146.796 89.5625 0.0168791 0.00680527 2024-03-11T22:23:15.840 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:15.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:15.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:15.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:15 smithi144 ceph-mon[33984]: pgmap v692: 168 pgs: 168 active+clean; 1.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-11T22:23:15.942 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:15.968 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:15.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:15.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:16.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:15 smithi045 ceph-mon[39715]: pgmap v692: 168 pgs: 168 active+clean; 1.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-11T22:23:16.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:15 smithi045 ceph-mon[30178]: pgmap v692: 168 pgs: 168 active+clean; 1.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-11T22:23:16.070 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:16.107 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:16.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:16.209 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:16.244 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:16.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:16.345 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:16.383 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:16.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:16.485 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:16.519 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:16.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:16.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:16.621 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:16.652 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:16.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:16.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:16.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:16.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:16.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:16.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:16.816 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 15 16 34756 34740 144.738 115.938 0.00718172 0.00690381 2024-03-11T22:23:16.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:23:16.848 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:23:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:23:16.881 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:16.916 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:16.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:16.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:16.932 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:23:16.933 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:23:16.952 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:23:17.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:17.058 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:17.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:17.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:17.160 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:17.165 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:23:17.199 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:17.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:17.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:17.207 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:23:17.250 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:23:17.287 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:23:17.301 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:17.328 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:23:17.344 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:17.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:17.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:17.365 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:23:17.407 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:23:17.446 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:17.447 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:23:17.451 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:23:17.451 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:23:17.451 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:23:17.452 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:23:17.452 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:23:17.452 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:23:17.452 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:23:17.452 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:23:17.478 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:17.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:17.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:17.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:17.619 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:17.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:17.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:17.762 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:17.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:17.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:17.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:17 smithi045 ceph-mon[30178]: pgmap v693: 168 pgs: 168 active+clean; 1.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-11T22:23:17.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:17.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:17 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:23:17.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:17 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:23:17.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:17 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1422433602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:17.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:17 smithi045 ceph-mon[39715]: pgmap v693: 168 pgs: 168 active+clean; 1.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-11T22:23:17.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:17.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:17 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:23:17.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:17 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:23:17.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:17 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1422433602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:17.792 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:17 smithi144 ceph-mon[33984]: pgmap v693: 168 pgs: 168 active+clean; 1.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-11T22:23:17.792 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:17.792 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:17 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:23:17.792 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:17 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:23:17.792 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:17 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1422433602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:17.816 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 16 16 36663 36647 143.141 119.188 0.00955047 0.00698054 2024-03-11T22:23:17.864 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:17.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:17.909 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:17.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:18.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:18.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:18.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:18.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:18.145 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:18.168 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:23:18.168 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-03-11T22:23:18.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-03-11T22:23:18.168 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:23:18.168 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.168 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:204820 7.7 7:ec16411b:::benchmark_data_smithi045_83007_object86349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:23:18.168 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:44.296015+0000", 2024-03-11T22:23:18.168 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 573.70899716400004, 2024-03-11T22:23:18.168 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.129960253, 2024-03-11T22:23:18.169 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.169 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.169 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.169 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:23:18.169 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:23:18.169 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 204820 2024-03-11T22:23:18.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.169 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.169 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.169 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.169 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296015+0000", 2024-03-11T22:23:18.169 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.169 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296015+0000", 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296017+0000", 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5030000000000001e-06 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296029+0000", 2024-03-11T22:23:18.170 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2198000000000001e-05 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296040+0000", 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1009000000000001e-05 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296046+0000", 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.1859999999999997e-06 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.171 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296097+0000", 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0995999999999998e-05 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296183+0000", 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5535e-05 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.296254+0000", 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1216000000000004e-05 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.172 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.298666+0000", 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002411641 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.418974+0000", 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.120308251 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.425855+0000", 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0068807549999999997 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.173 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.425924+0000", 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8912999999999996e-05 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:44.425976+0000", 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2049999999999998e-05 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.174 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:205790 7.d 7:bd5a76ed:::benchmark_data_smithi045_83007_object87319:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:13:45.686540+0000", 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 572.31847204600001, 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.157224592, 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 205790 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.175 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686540+0000", 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686540+0000", 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686542+0000", 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4649999999999999e-06 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.176 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686546+0000", 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5700000000000003e-06 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686560+0000", 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3190000000000001e-05 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686567+0000", 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7360000000000001e-06 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.177 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686609+0000", 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1242000000000003e-05 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686695+0000", 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6576000000000006e-05 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.686764+0000", 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8880999999999993e-05 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.178 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.689199+0000", 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024353690000000002 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.691011+0000", 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0018111799999999999 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.843698+0000", 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.15268752499999999 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.179 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.843732+0000", 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3476000000000003e-05 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:13:45.843765+0000", 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3382e-05 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:228199 7.7 7:ee665074:::benchmark_data_smithi045_83007_object109728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:23:18.180 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:09.462446+0000", 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 548.54256680200001, 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.14675967300000001, 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 228199 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462446+0000", 2024-03-11T22:23:18.181 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462446+0000", 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462447+0000", 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9100000000000002e-07 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.182 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462451+0000", 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2880000000000003e-06 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462466+0000", 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.467e-05 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462473+0000", 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.7500000000000003e-06 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.183 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462524+0000", 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1223000000000001e-05 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462633+0000", 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000108226 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.462729+0000", 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.5805999999999997e-05 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.184 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.465857+0000", 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0031280800000000001 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.590331+0000", 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.12447485599999999 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.609081+0000", 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.018749670999999999 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.185 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.609142+0000", 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0516000000000001e-05 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.609205+0000", 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3695999999999995e-05 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:228205 7.d 7:bc358dc8:::benchmark_data_smithi045_83007_object109734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:09.466216+0000", 2024-03-11T22:23:18.186 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 548.53879691600002, 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13463407299999999, 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 228205 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466216+0000", 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.187 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466216+0000", 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466217+0000", 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7199999999999997e-07 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466221+0000", 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5539999999999999e-06 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.188 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.189 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466230+0000", 2024-03-11T22:23:18.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1810000000000007e-06 2024-03-11T22:23:18.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.189 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.189 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.189 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466236+0000", 2024-03-11T22:23:18.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4970000000000001e-06 2024-03-11T22:23:18.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.189 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.189 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.189 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466283+0000", 2024-03-11T22:23:18.189 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7160999999999998e-05 2024-03-11T22:23:18.189 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466380+0000", 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.7510000000000007e-05 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.466508+0000", 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012758100000000001 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.469514+0000", 2024-03-11T22:23:18.190 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.003005495 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.469937+0000", 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00042370599999999998 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.600748+0000", 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.13081040799999999 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.191 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.600800+0000", 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2818000000000002e-05 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:09.600850+0000", 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9190000000000002e-05 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229663 7.d 7:bdf7b07c:::benchmark_data_smithi045_83007_object111192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:11.125093+0000", 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 546.87991952300001, 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.17061443900000001, 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.192 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229663 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.125093+0000", 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.125093+0000", 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.193 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.125093+0000", 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.84e-07 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.125095+0000", 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.984e-06 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.125102+0000", 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.075e-06 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.194 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.125108+0000", 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1710000000000001e-06 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129918+0000", 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0048100460000000001 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.130040+0000", 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000122862 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.195 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.130170+0000", 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000130001 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.139457+0000", 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0092864469999999998 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.142545+0000", 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0030881459999999999 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.196 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.295632+0000", 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.15308732 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.295669+0000", 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6362000000000002e-05 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.295707+0000", 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8640999999999998e-05 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.197 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229666 7.d 7:b2f2096b:::benchmark_data_smithi045_83007_object111195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:11.129450+0000", 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 546.87556256300002, 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.16635360900000001, 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229666 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.198 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129450+0000", 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129450+0000", 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129451+0000", 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.19e-06 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129456+0000", 2024-03-11T22:23:18.199 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7119999999999999e-06 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129467+0000", 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.1005999999999999e-05 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.129474+0000", 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9550000000000003e-06 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.133139+0000", 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0036649059999999999 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.200 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.133269+0000", 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012985899999999999 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.133398+0000", 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012982900000000001 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.139462+0000", 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0060638630000000001 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.201 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.150243+0000", 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010780919999999999 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.295743+0000", 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.14549944300000001 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.295776+0000", 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.3683999999999998e-05 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.202 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.295804+0000", 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7242000000000001e-05 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229669 7.d 7:bd6d76c8:::benchmark_data_smithi045_83007_object111198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:11.131546+0000", 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 546.87346597199996, 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.164917068, 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.203 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229669 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.131546+0000", 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.131546+0000", 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.204 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.131548+0000", 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.124e-06 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.131553+0000", 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0159999999999999e-06 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.131567+0000", 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4267000000000001e-05 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.131575+0000", 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9060000000000002e-06 2024-03-11T22:23:18.205 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.206 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.206 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.206 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.135734+0000", 2024-03-11T22:23:18.206 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.004159177 2024-03-11T22:23:18.206 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.206 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.206 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.206 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.135856+0000", 2024-03-11T22:23:18.206 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012204100000000001 2024-03-11T22:23:18.206 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.206 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.206 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:23:18.206 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.135975+0000", 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000118817 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.139791+0000", 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0038162249999999999 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.150272+0000", 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.010480487 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296386+0000", 2024-03-11T22:23:18.207 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.14611407200000001 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296428+0000", 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2633999999999999e-05 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296464+0000", 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5302000000000002e-05 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.208 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229681 7.d 7:b04b550f:::benchmark_data_smithi045_83007_object111210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:11.144736+0000", 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 546.86027646699995, 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.151893167, 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229681 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144736+0000", 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.209 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144736+0000", 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144737+0000", 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4360000000000001e-06 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144743+0000", 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6849999999999998e-06 2024-03-11T22:23:18.210 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144756+0000", 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3251000000000001e-05 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144765+0000", 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.2709999999999992e-06 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144814+0000", 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9109e-05 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.211 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144902+0000", 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8319000000000006e-05 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.144975+0000", 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2630000000000001e-05 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.150217+0000", 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0052418409999999997 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.212 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.150302+0000", 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5792999999999997e-05 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296526+0000", 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.14622318300000001 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296586+0000", 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.0689000000000001e-05 2024-03-11T22:23:18.213 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296629+0000", 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2960000000000002e-05 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.14790.0:229683 7.d 7:b3eb5a94:::benchmark_data_smithi045_83007_object111212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:14:11.145873+0000", 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 546.85913964400004, 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.15093811200000001, 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.214 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.14790", 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/690427062", 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 229683 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.145873+0000", 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.145873+0000", 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.215 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.145874+0000", 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2099999999999996e-07 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.145877+0000", 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9480000000000001e-06 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.145888+0000", 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0115000000000001e-05 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.216 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.145896+0000", 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3189999999999995e-06 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.147846+0000", 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0019499140000000001 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.147952+0000", 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000106032 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,1", 2024-03-11T22:23:18.217 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.148030+0000", 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8530999999999997e-05 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.150280+0000", 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.002249858 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.292276+0000", 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.14199621200000001 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.218 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296691+0000", 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0044147809999999996 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296768+0000", 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6314000000000002e-05 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:14:11.296811+0000", 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3367e-05 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.219 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.25142.0:146716 9.1 e313/310 9:88bee207:::benchmark_data_smithi045_141553_object22596:head v 313'9302, mlcod=313'9300)", 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:20:46.128708+0000", 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 151.87630473900001, 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28756806499999998, 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.128708+0000", 2024-03-11T22:23:18.220 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.128708+0000", 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.128709+0000", 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.5900000000000005e-07 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.128719+0000", 2024-03-11T22:23:18.221 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0397999999999999e-05 2024-03-11T22:23:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.128732+0000", 2024-03-11T22:23:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3179999999999999e-05 2024-03-11T22:23:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.128741+0000", 2024-03-11T22:23:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.5499999999999995e-06 2024-03-11T22:23:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.222 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.128800+0000", 2024-03-11T22:23:18.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9317000000000001e-05 2024-03-11T22:23:18.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.223 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.128825+0000", 2024-03-11T22:23:18.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.5253999999999999e-05 2024-03-11T22:23:18.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.223 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.223 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.223 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.416234+0000", 2024-03-11T22:23:18.223 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28740849400000001 2024-03-11T22:23:18.223 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.416276+0000", 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1912999999999997e-05 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.25142.0:146721 9.1 e313/310 9:8c58b4a2:::benchmark_data_smithi045_141553_object22601:head v 313'9303, mlcod=313'9300)", 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:20:46.133315+0000", 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 151.87169769100001, 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28905383899999998, 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.224 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.133315+0000", 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.133315+0000", 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.133316+0000", 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.4099999999999997e-07 2024-03-11T22:23:18.225 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.133326+0000", 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.9049999999999992e-06 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.416206+0000", 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28288085699999999 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.416222+0000", 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5483999999999999e-05 2024-03-11T22:23:18.226 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.416301+0000", 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9466000000000001e-05 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.416326+0000", 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4273e-05 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.422326+0000", 2024-03-11T22:23:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0060001990000000003 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.422369+0000", 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.2713999999999998e-05 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.25142.0:146739 9.b e313/273 9:d4eef4f4:::benchmark_data_smithi045_141553_object22619:head v 313'8950, mlcod=313'8948)", 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:20:46.149114+0000", 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 151.85589877800001, 2024-03-11T22:23:18.228 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.28204157400000002, 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.149114+0000", 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.149114+0000", 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.229 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.149115+0000", 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0020000000000001e-06 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.149122+0000", 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0079999999999996e-06 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.416538+0000", 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.26741604099999999 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.230 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.416544+0000", 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2219999999999999e-06 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.416647+0000", 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010254700000000001 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.416667+0000", 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0208e-05 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.231 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.431025+0000", 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.014358579 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:20:46.431155+0000", 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012996700000000001 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.15540.0:1085 10.a 10:5b8ba7d8:::benchmark_data_smithi045_169175_object1084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:02.103201+0000", 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.901811411000001, 2024-03-11T22:23:18.232 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.104962522, 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.15540", 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 1085 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.103201+0000", 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.233 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.103201+0000", 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.103202+0000", 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3e-07 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.103305+0000", 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000103447 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.234 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.103313+0000", 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.926e-06 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.103319+0000", 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4310000000000002e-06 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.103369+0000", 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.9954000000000003e-05 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.235 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.103455+0000", 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6139999999999999e-05 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,5", 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.103546+0000", 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.1138999999999996e-05 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.103935+0000", 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00038870799999999998 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.236 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104272+0000", 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00033722500000000002 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.208051+0000", 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10377944 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.208109+0000", 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8233000000000003e-05 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.208164+0000", 2024-03-11T22:23:18.237 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4148999999999999e-05 2024-03-11T22:23:18.238 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.238 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.238 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.238 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.238 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.238 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.15540.0:1091 10.7 10:e186383f:::benchmark_data_smithi045_169175_object1090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-11T22:23:18.238 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:02.104784+0000", 2024-03-11T22:23:18.238 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.900228631999999, 2024-03-11T22:23:18.238 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10764424, 2024-03-11T22:23:18.238 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.238 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.238 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.238 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.15540", 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 1091 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104784+0000", 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104784+0000", 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.239 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104785+0000", 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.54e-07 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104848+0000", 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3372000000000002e-05 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104856+0000", 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4300000000000006e-06 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.240 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104862+0000", 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.198e-06 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104901+0000", 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8983000000000003e-05 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104984+0000", 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.3086999999999999e-05 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 3,5", 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.207999+0000", 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103015761 2024-03-11T22:23:18.241 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.210489+0000", 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024896499999999999 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.210592+0000", 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010280000000000001 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.212329+0000", 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001736869 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.242 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.212388+0000", 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9066e-05 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.212428+0000", 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0269999999999999e-05 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.15540.0:1093 10.e 10:7f46781f:::benchmark_data_smithi045_169175_object1092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-11T22:23:18.243 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:02.104873+0000", 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.900139159, 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.106690386, 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.15540", 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 1093 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104873+0000", 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.244 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104873+0000", 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104874+0000", 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7300000000000002e-07 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104901+0000", 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.7424999999999999e-05 2024-03-11T22:23:18.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104906+0000", 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6609999999999999e-06 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104909+0000", 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5549999999999998e-06 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104947+0000", 2024-03-11T22:23:18.246 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7641000000000001e-05 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.207947+0000", 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.102999888 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,4", 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.208089+0000", 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014251699999999999 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.208779+0000", 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000689479 2024-03-11T22:23:18.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.209701+0000", 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00092221000000000002 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.211356+0000", 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001654914 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.211416+0000", 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9817e-05 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.211564+0000", 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000147906 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.15540.0:1094 10.f 10:fb04199a:::benchmark_data_smithi045_169175_object1093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:02.104918+0000", 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.900094628, 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10746818800000001, 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.15540", 2024-03-11T22:23:18.249 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 1094 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104918+0000", 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104918+0000", 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.250 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104918+0000", 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.6899999999999998e-07 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104951+0000", 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2412000000000001e-05 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104955+0000", 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7480000000000001e-06 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104959+0000", 2024-03-11T22:23:18.251 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4929999999999998e-06 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.104994+0000", 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5228000000000002e-05 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.208028+0000", 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.103033711 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 2,3", 2024-03-11T22:23:18.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.208198+0000", 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00016992799999999999 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.209799+0000", 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0016010969999999999 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.210495+0000", 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00069613499999999996 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.253 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.212290+0000", 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001795098 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.212349+0000", 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8446999999999998e-05 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.212386+0000", 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.7521999999999998e-05 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop_reply(client.15540.0:1085 10.a e352/348 ondisk, result = 0)", 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:02.106295+0000", 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 15.89871713, 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.101876491, 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.106295+0000", 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.106295+0000", 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.106296+0000", 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6899999999999998e-07 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.106299+0000", 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.1240000000000001e-06 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.207942+0000", 2024-03-11T22:23:18.256 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.10164334999999999 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.207964+0000", 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1248000000000001e-05 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.208034+0000", 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0803000000000006e-05 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.208044+0000", 2024-03-11T22:23:18.257 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.5650000000000007e-06 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:02.208172+0000", 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00012793200000000001 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.15540.0:6046 10.0 10:00090e9b:::benchmark_data_smithi045_169175_object6045:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:03.767673+0000", 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 14.237339657, 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.084615501999999995, 2024-03-11T22:23:18.258 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.15540", 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 6046 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.767673+0000", 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.767673+0000", 2024-03-11T22:23:18.259 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.767674+0000", 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3900000000000003e-07 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.767733+0000", 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9239e-05 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.260 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.767773+0000", 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9776e-05 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.767781+0000", 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6200000000000005e-06 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.767833+0000", 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.1950000000000002e-05 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.261 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.767953+0000", 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000119837 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 1,4", 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.768160+0000", 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00020708 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.770047+0000", 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001886855 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.771429+0000", 2024-03-11T22:23:18.262 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001381588 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.852179+0000", 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.080750665999999999 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.852238+0000", 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.8777999999999999e-05 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.852288+0000", 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0173999999999998e-05 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.263 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.15540.0:6050 10.d 10:b0fc312e:::benchmark_data_smithi045_169175_object6049:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:03.769706+0000", 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 14.235306752, 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.083659541000000004, 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.15540", 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 6050 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769706+0000", 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769706+0000", 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769707+0000", 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.099e-06 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.265 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769814+0000", 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00010695 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769827+0000", 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3658999999999999e-05 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769836+0000", 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1699999999999997e-06 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.266 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769946+0000", 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000110398 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.770086+0000", 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014040400000000001 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,4", 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.770234+0000", 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014735400000000001 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.772231+0000", 2024-03-11T22:23:18.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0019972129999999999 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.772318+0000", 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.6967999999999998e-05 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.853271+0000", 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.080952698000000003 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.268 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.853325+0000", 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4525999999999999e-05 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.853365+0000", 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0102000000000003e-05 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_op(client.15540.0:6051 10.d 10:b5f8fbd2:::benchmark_data_smithi045_169175_object6050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:03.769856+0000", 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 14.235156699999999, 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.083650659000000002, 2024-03-11T22:23:18.269 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: "client_info": { 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: "client": "client.15540", 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: "tid": 6051 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769856+0000", 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.270 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:18.271 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769856+0000", 2024-03-11T22:23:18.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:18.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.271 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.271 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:18.271 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769856+0000", 2024-03-11T22:23:18.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2900000000000003e-07 2024-03-11T22:23:18.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.271 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.271 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:18.271 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769889+0000", 2024-03-11T22:23:18.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.2767e-05 2024-03-11T22:23:18.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769897+0000", 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1429999999999998e-06 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769904+0000", 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6760000000000001e-06 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.770885+0000", 2024-03-11T22:23:18.272 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00098110800000000007 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.771007+0000", 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000121697 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "waiting for subops from 0,4", 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.771161+0000", 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00015386100000000001 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "op_commit", 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.772239+0000", 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001078531 2024-03-11T22:23:18.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.773365+0000", 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0011253719999999999 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.853404+0000", 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.080039451999999997 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.853471+0000", 2024-03-11T22:23:18.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6832e-05 2024-03-11T22:23:18.275 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:18.275 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:18.275 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:23:18.275 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.853506+0000", 2024-03-11T22:23:18.275 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5590999999999999e-05 2024-03-11T22:23:18.275 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.275 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.275 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.275 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:18.275 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:18.275 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:23:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:18.277 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:18.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:18.377 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:18.410 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:18.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:18.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:18.512 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:18.538 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:18.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:18.639 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:18.673 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:18.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:18.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:18.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:23:18.775 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:18.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:18 smithi045 ceph-mon[30178]: osdmap e356: 8 total, 8 up, 8 in 2024-03-11T22:23:18.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:18 smithi045 ceph-mon[39715]: osdmap e356: 8 total, 8 up, 8 in 2024-03-11T22:23:18.807 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:18.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:18.816 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 17 16 38510 38494 141.509 115.438 0.0136096 0.00706083 2024-03-11T22:23:18.909 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:18.945 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:18.946 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:18 smithi144 ceph-mon[33984]: osdmap e356: 8 total, 8 up, 8 in 2024-03-11T22:23:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:18.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:19.047 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:19.081 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:19.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:19.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:19.210 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:19.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:19.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:19.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:19.332 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:19.369 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:19.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:19.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:19.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:19.504 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:19.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:19.606 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:19.640 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:19.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:19.742 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:19.776 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:19.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:19.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:19.816 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 18 16 40383 40367 140.15 117.062 0.00653749 0.00713026 2024-03-11T22:23:19.847 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:19 smithi144 ceph-mon[33984]: pgmap v695: 168 pgs: 168 active+clean; 2.0 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.96k op/s 2024-03-11T22:23:19.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:19.918 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:19.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:19.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:20.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:20.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:19 smithi045 ceph-mon[30178]: pgmap v695: 168 pgs: 168 active+clean; 2.0 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.96k op/s 2024-03-11T22:23:20.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:19 smithi045 ceph-mon[39715]: pgmap v695: 168 pgs: 168 active+clean; 2.0 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.96k op/s 2024-03-11T22:23:20.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:20.057 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:20.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:20.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:20.162 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T22:23:20.200 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:20.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:20.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:20.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:20.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:20.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:20.427 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:20.458 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:20.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:20.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:20.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:20.598 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:20.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:20.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:20.700 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:20.732 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:20.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:20.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:20.817 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 19 16 42071 42055 138.326 105.5 0.0071634 0.00722428 2024-03-11T22:23:20.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:20.875 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:20.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:20.883 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:23:20.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:21.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:21.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:21.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:21.120 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:21.156 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:21.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:21.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:21.257 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:21.296 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:21.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:21.369 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:23:21.369 INFO:teuthology.orchestra.run.smithi144.stdout: "success": "" 2024-03-11T22:23:21.369 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:23:21.398 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:21.445 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:21.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:21.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:21.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:21.585 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:21.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:21.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:21.727 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:21.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:21.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:21.817 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:23:21.814858+0000 min lat: 0.00118312 max lat: 0.134618 avg lat: 0.00729784 2024-03-11T22:23:21.817 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:23:21.817 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 20 16 43835 43819 136.921 110.25 0.0034723 0.00729784 2024-03-11T22:23:21.829 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:21.873 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:21.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:21.975 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:21.981 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:22.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:22.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:22.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:21 smithi045 ceph-mon[39715]: pgmap v696: 168 pgs: 168 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T22:23:22.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:21 smithi045 ceph-mon[30178]: pgmap v696: 168 pgs: 168 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T22:23:22.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:21 smithi144 ceph-mon[33984]: pgmap v696: 168 pgs: 168 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-11T22:23:22.120 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:22.154 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:22.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:22.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:22.235 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:23:22.256 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:22.303 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:22.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:22.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:22.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:22.448 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:22.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:22.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:22.587 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:22.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:22.663 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:23:22.681 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:23:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:23:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:23:22.681 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:22 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/588193036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:22.682 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:22 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:22.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:22 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/588193036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:22.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:22 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:22.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:22.728 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:22 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/588193036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:22.728 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:22 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:22.729 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:22.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:22.817 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 21 16 45487 45471 135.317 103.25 0.00597125 0.00738475 2024-03-11T22:23:22.831 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:22.869 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:22.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:22.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:23.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:23.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:23.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:23.113 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:23.152 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:23.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:23.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:23.294 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:23.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:23.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:23.396 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:23.436 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:23.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:23.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:23.580 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:23.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:23.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:23.719 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:23.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:23.757 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:23 smithi045 ceph-mon[39715]: pgmap v697: 168 pgs: 168 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 130 MiB/s wr, 2.08k op/s 2024-03-11T22:23:23.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:23 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:23:23.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:23 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:23:23.758 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:23 smithi045 ceph-mon[39715]: osdmap e357: 8 total, 8 up, 8 in 2024-03-11T22:23:23.758 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:23 smithi045 ceph-mon[30178]: pgmap v697: 168 pgs: 168 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 130 MiB/s wr, 2.08k op/s 2024-03-11T22:23:23.758 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:23 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:23:23.758 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:23 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:23:23.758 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:23 smithi045 ceph-mon[30178]: osdmap e357: 8 total, 8 up, 8 in 2024-03-11T22:23:23.810 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:23 smithi144 ceph-mon[33984]: pgmap v697: 168 pgs: 168 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 130 MiB/s wr, 2.08k op/s 2024-03-11T22:23:23.810 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:23 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:23:23.810 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:23 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:23:23.810 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:23 smithi144 ceph-mon[33984]: osdmap e357: 8 total, 8 up, 8 in 2024-03-11T22:23:23.817 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 22 16 47295 47279 134.303 113 0.0110686 0.00744072 2024-03-11T22:23:23.821 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:23.861 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:23.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:23.963 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:23.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:23.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:24.101 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:24.140 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:24.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:24.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:24.241 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:24.282 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:24.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:24.313 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:23:24.313 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:23:24.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.313 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:48101 10.9 e357/348 10:96a5a1cf:::benchmark_data_smithi045_169175_object48100:head v 357'3062, mlcod=357'3057)", 2024-03-11T22:23:24.313 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:24.304421+0000", 2024-03-11T22:23:24.313 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.00169235, 2024-03-11T22:23:24.313 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0017225980000000001, 2024-03-11T22:23:24.313 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:24.313 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:23:24.313 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:24.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.313 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:24.313 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.304421+0000", 2024-03-11T22:23:24.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:24.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.314 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:24.314 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.304421+0000", 2024-03-11T22:23:24.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:24.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.314 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:24.314 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.304422+0000", 2024-03-11T22:23:24.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.9600000000000008e-07 2024-03-11T22:23:24.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.304445+0000", 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.3277000000000001e-05 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.304459+0000", 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3876999999999999e-05 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.304466+0000", 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2030000000000003e-06 2024-03-11T22:23:24.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.304874+0000", 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00040746700000000001 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.304931+0000", 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7080000000000002e-05 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.316 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:48109 10.c e357/348 10:3ed8fafe:::benchmark_data_smithi045_169175_object48108:head v 357'3059, mlcod=357'3055)", 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:24.305573+0000", 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.00054017099999999999, 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00064315599999999998, 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "started", 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.305573+0000", 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.317 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.305573+0000", 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.305574+0000", 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.4e-07 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.305675+0000", 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000100723 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.318 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.305687+0000", 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2582000000000001e-05 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.305694+0000", 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1720000000000001e-06 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.305850+0000", 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000155081 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:23:24.319 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:23:24.320 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:24.305893+0000", 2024-03-11T22:23:24.320 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3288000000000003e-05 2024-03-11T22:23:24.320 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:24.320 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:23:24.320 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:24.320 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:23:24.320 INFO:teuthology.orchestra.run.smithi144.stdout: ], 2024-03-11T22:23:24.320 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 2 2024-03-11T22:23:24.320 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:23:24.385 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:24.422 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:24.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:24.523 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:24.558 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:24.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:24.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:24.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:24.667 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:23:24.701 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:24.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:24.705 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:23:24.706 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:24 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/613991209' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:24.706 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:24 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/613991209' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:24.744 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:23:24.791 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:23:24.803 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:24.817 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 23 16 48947 48931 132.952 103.25 0.0129747 0.0075163 2024-03-11T22:23:24.830 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:23:24.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:24.837 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:24.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:24.871 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:23:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:23:24.937 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:24.954 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:23:24.958 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:23:24.958 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:23:24.958 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:23:24.958 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:23:24.958 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:23:24.958 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:23:24.958 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:23:24.958 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:23:24.973 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:24.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:24.995 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:24 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/613991209' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:25.046 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T22:23:25.075 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:25.134 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:25.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:25.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:25.236 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:25.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:25.276 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:25.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:25.377 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:25.418 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:25.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:25.519 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:25.524 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:23:25.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:25.564 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:25.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:25.665 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:25.701 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:25.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:25.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:25.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:25.818 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 24 16 50607 50591 131.731 103.75 0.0166563 0.00758559 2024-03-11T22:23:25.838 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:25.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:25.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:25.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:25.978 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:25.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:25.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:25.991 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:25 smithi144 ceph-mon[33984]: pgmap v699: 168 pgs: 168 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T22:23:26.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:25 smithi045 ceph-mon[39715]: pgmap v699: 168 pgs: 168 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T22:23:26.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:25 smithi045 ceph-mon[30178]: pgmap v699: 168 pgs: 168 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T22:23:26.053 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:23:26.079 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:26.117 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:26.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:26.218 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:26.252 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:26.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:26.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:26.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:26.408 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:26.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:26.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:26.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:26.554 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:26.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:26.655 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:26.700 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:26.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:26.701 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:26.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:26.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:26.818 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 25 16 52202 52186 130.449 99.6875 0.00851879 0.00766064 2024-03-11T22:23:26.842 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:26.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:26.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:26.848 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:23:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:23:26.944 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:26.979 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:26.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:26.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:27.081 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:27.120 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:27.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:27.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:27.222 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:23:27.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T22:23:27.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:27.271 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:27.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:27.372 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:27.413 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:27.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:27.514 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:27.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:27.558 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:27.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:27.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:27.702 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:27.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:27.724 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:23:27.724 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:23:27.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:23:27.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:27.819 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 26 16 53749 53733 129.147 96.6875 0.0123171 0.00773789 2024-03-11T22:23:27.863 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:27.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:27.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:28.001 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:28.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:28.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:27 smithi045 ceph-mon[39715]: pgmap v700: 168 pgs: 168 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 585 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-03-11T22:23:28.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:28.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:27 smithi045 ceph-mon[30178]: pgmap v700: 168 pgs: 168 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 585 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-03-11T22:23:28.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:28.075 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:27 smithi144 ceph-mon[33984]: pgmap v700: 168 pgs: 168 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 585 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-03-11T22:23:28.075 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:28.104 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:28.140 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:28.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:28.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:28.242 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:28.277 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:28.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:28.291 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:28.379 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:28.427 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:28.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:28.476 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T22:23:28.527 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:28.572 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:28.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:28.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:28.708 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:28.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:28.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:28.810 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:28.819 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 27 16 55248 55232 127.833 93.6875 0.00563159 0.00781717 2024-03-11T22:23:28.855 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:28.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:28.891 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:23:28.957 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:29.001 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:29.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:29.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:29.103 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:29.147 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:29.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:29.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:29.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:29.293 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:29.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:29.395 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:29.431 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:29.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:29.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:29.572 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:29.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:29.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:29.674 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:29.683 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:23:29.693 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:29.705 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:29.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:29.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:29.819 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 28 16 56708 56692 126.526 91.25 0.0102822 0.00789792 2024-03-11T22:23:29.846 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:29.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:29.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:29.948 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:29.988 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:29.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:30.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:29 smithi045 ceph-mon[39715]: pgmap v701: 168 pgs: 168 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 110 MiB/s wr, 1.75k op/s 2024-03-11T22:23:30.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:29 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:23:30.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:29 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:23:30.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:29 smithi045 ceph-mon[39715]: osdmap e358: 8 total, 8 up, 8 in 2024-03-11T22:23:30.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:29 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3758180589' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:23:30.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:29 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:23:30.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:29 smithi045 ceph-mon[30178]: pgmap v701: 168 pgs: 168 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 110 MiB/s wr, 1.75k op/s 2024-03-11T22:23:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:29 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:23:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:29 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:23:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:29 smithi045 ceph-mon[30178]: osdmap e358: 8 total, 8 up, 8 in 2024-03-11T22:23:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:29 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3758180589' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:23:30.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:29 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:23:30.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:30.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:29 smithi144 ceph-mon[33984]: pgmap v701: 168 pgs: 168 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 110 MiB/s wr, 1.75k op/s 2024-03-11T22:23:30.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:29 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:23:30.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:29 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:23:30.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:29 smithi144 ceph-mon[33984]: osdmap e358: 8 total, 8 up, 8 in 2024-03-11T22:23:30.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:29 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3758180589' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:23:30.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:29 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:23:30.113 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:30.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:30.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:30.215 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:30.254 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:30.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:30.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:30.355 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:30.396 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:30.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:30.497 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:30.535 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:30.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:30.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:30.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:30.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:30.677 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:30.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:30.778 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:30.819 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 29 16 58251 58235 125.488 96.4375 0.0106994 0.00796374 2024-03-11T22:23:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:30.825 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:30.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:30.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:30.927 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:23:30.928 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:23:30.928 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:30 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:23:30.928 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:30 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:23:30.928 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:30 smithi144 ceph-mon[33984]: osdmap e359: 8 total, 8 up, 8 in 2024-03-11T22:23:30.928 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:30 smithi144 ceph-mon[33984]: 5.19 scrub starts 2024-03-11T22:23:30.928 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:30 smithi144 ceph-mon[33984]: 5.19 scrub ok 2024-03-11T22:23:30.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:30.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:30.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:30.970 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:30 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:23:30.970 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:30 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:23:30.970 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:30 smithi045 ceph-mon[39715]: osdmap e359: 8 total, 8 up, 8 in 2024-03-11T22:23:30.970 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:30 smithi045 ceph-mon[39715]: 5.19 scrub starts 2024-03-11T22:23:30.971 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:30 smithi045 ceph-mon[39715]: 5.19 scrub ok 2024-03-11T22:23:30.971 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:30 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:23:30.971 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:30 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:23:30.971 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:30 smithi045 ceph-mon[30178]: osdmap e359: 8 total, 8 up, 8 in 2024-03-11T22:23:30.971 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:30 smithi045 ceph-mon[30178]: 5.19 scrub starts 2024-03-11T22:23:30.971 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:30 smithi045 ceph-mon[30178]: 5.19 scrub ok 2024-03-11T22:23:31.068 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:31.104 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:31.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:31.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:31.206 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:31.240 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:31.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:31.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:31.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:31.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:31.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:31.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:31.527 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:31.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:31.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:31.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:31.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T22:23:31.664 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:31.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:31.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:31.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:31.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:31.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:31.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:31.819 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 30 16 59763 59747 124.455 94.5 0.00713724 0.00802953 2024-03-11T22:23:31.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:31 smithi144 ceph-mon[33984]: pgmap v704: 168 pgs: 168 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 663 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-11T22:23:31.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:23:31.908 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:31.944 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:31.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:31.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:32.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:31 smithi045 ceph-mon[30178]: pgmap v704: 168 pgs: 168 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 663 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-11T22:23:32.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:23:32.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:31 smithi045 ceph-mon[39715]: pgmap v704: 168 pgs: 168 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 663 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-11T22:23:32.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:23:32.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:32.084 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:32.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:32.185 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:32.222 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:32.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:32.324 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:32.360 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:32.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:32.461 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:32.498 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:32.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:32.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:32.600 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:32.634 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:32.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:32.736 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:32.770 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:32.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:32.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:32.819 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 31 16 61246 61230 123.43 92.6875 0.00667048 0.00809629 2024-03-11T22:23:32.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:32.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:23:32.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:32.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:32.943 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:23:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:23:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:23:33.014 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:33.045 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:33.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:33.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:33.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:33.198 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:33.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:33.299 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:33.334 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:33.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:33.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:33.436 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:33.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:33.471 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:33.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:33.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:33.622 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:33.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:33.724 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:33.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:33.767 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:33.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:33.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:33 smithi045 ceph-mon[39715]: pgmap v705: 168 pgs: 168 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.02k op/s 2024-03-11T22:23:33.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:33 smithi045 ceph-mon[39715]: 1.e scrub starts 2024-03-11T22:23:33.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:33 smithi045 ceph-mon[39715]: 1.e scrub ok 2024-03-11T22:23:33.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:33 smithi045 ceph-mon[39715]: 6.d scrub starts 2024-03-11T22:23:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:33 smithi045 ceph-mon[30178]: pgmap v705: 168 pgs: 168 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.02k op/s 2024-03-11T22:23:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:33 smithi045 ceph-mon[30178]: 1.e scrub starts 2024-03-11T22:23:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:33 smithi045 ceph-mon[30178]: 1.e scrub ok 2024-03-11T22:23:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:33 smithi045 ceph-mon[30178]: 6.d scrub starts 2024-03-11T22:23:33.819 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 32 16 62649 62633 122.313 87.6875 0.0136406 0.00817165 2024-03-11T22:23:33.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:33.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:34.011 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:34.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:33 smithi144 ceph-mon[33984]: pgmap v705: 168 pgs: 168 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.02k op/s 2024-03-11T22:23:34.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:33 smithi144 ceph-mon[33984]: 1.e scrub starts 2024-03-11T22:23:34.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:33 smithi144 ceph-mon[33984]: 1.e scrub ok 2024-03-11T22:23:34.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:33 smithi144 ceph-mon[33984]: 6.d scrub starts 2024-03-11T22:23:34.099 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:34.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:34.201 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:34.247 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:34.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:34.348 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:34.391 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:34.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:34.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:34.493 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:34.527 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:34.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:34.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:34.601 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:23:34.629 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:34.637 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:23:34.675 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:23:34.680 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:34.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:34.722 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:23:34.762 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:23:34.779 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:34.782 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:34.798 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:23:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:34.807 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:34.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:34.819 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 33 16 63805 63789 120.796 72.25 0.00390085 0.00826203 2024-03-11T22:23:34.824 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:34.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:34 smithi045 ceph-mon[30178]: 6.d scrub ok 2024-03-11T22:23:34.884 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:34 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:23:34.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:34 smithi045 ceph-mon[30178]: 5.10 scrub starts 2024-03-11T22:23:34.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:34 smithi045 ceph-mon[30178]: 5.10 scrub ok 2024-03-11T22:23:34.885 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:34 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1431319509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:34.885 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:34 smithi045 ceph-mon[39715]: 6.d scrub ok 2024-03-11T22:23:34.885 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:34 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:23:34.885 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:34 smithi045 ceph-mon[39715]: 5.10 scrub starts 2024-03-11T22:23:34.885 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:34 smithi045 ceph-mon[39715]: 5.10 scrub ok 2024-03-11T22:23:34.885 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:34 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1431319509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:34.886 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:23:34.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:34.931 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:23:34.943 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:23:34.943 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:23:34.943 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:23:34.943 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:23:34.943 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:23:34.943 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:23:34.943 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:23:34.943 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:23:34.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:34.961 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:34.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:35.062 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:35.094 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:35.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:35.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:35.097 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:34 smithi144 ceph-mon[33984]: 6.d scrub ok 2024-03-11T22:23:35.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:34 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:23:35.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:34 smithi144 ceph-mon[33984]: 5.10 scrub starts 2024-03-11T22:23:35.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:34 smithi144 ceph-mon[33984]: 5.10 scrub ok 2024-03-11T22:23:35.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:34 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1431319509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:35.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:35.222 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:35.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:35.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:35.324 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:35.355 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:35.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:35.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:35.457 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:35.481 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:35.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:35.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:35.583 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:35.618 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:35.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:35.707 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T22:23:35.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:35.757 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:35.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:35.820 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 34 16 64771 64755 119.019 60.375 0.00994185 0.00839674 2024-03-11T22:23:35.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:35.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:35.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:35.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:35.998 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:35.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:35 smithi144 ceph-mon[33984]: pgmap v706: 168 pgs: 168 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-11T22:23:35.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:35 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:23:35.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:35 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:23:35.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:35 smithi144 ceph-mon[33984]: osdmap e360: 8 total, 8 up, 8 in 2024-03-11T22:23:35.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:35 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4001008839' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:23:35.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:35 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:23:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:35 smithi045 ceph-mon[39715]: pgmap v706: 168 pgs: 168 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-11T22:23:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:35 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:23:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:35 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:23:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:35 smithi045 ceph-mon[39715]: osdmap e360: 8 total, 8 up, 8 in 2024-03-11T22:23:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:35 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4001008839' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:23:36.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:35 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:23:36.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:35 smithi045 ceph-mon[30178]: pgmap v706: 168 pgs: 168 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-11T22:23:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:35 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:23:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:35 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:23:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:35 smithi045 ceph-mon[30178]: osdmap e360: 8 total, 8 up, 8 in 2024-03-11T22:23:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:35 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4001008839' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:23:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:35 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:23:36.043 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:36.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:36.145 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:36.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:36.184 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:36.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:36.285 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:36.322 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:36.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:36.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:36.423 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:36.461 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:36.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:36.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:36.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:36.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:36.602 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:36.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:36.702 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:36.731 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:36.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:36.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:36.796 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:36 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:23:36.797 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:36 smithi144 ceph-mon[33984]: osdmap e361: 8 total, 8 up, 8 in 2024-03-11T22:23:36.797 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:36 smithi144 ceph-mon[33984]: 3.11 scrub starts 2024-03-11T22:23:36.797 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:36 smithi144 ceph-mon[33984]: 3.11 scrub ok 2024-03-11T22:23:36.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:23:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:23:36.819 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 35 15 65986 65971 117.79 76 0.0110331 0.00847828 2024-03-11T22:23:36.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:36.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:36.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:36.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:36.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:36 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:23:36.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:36 smithi045 ceph-mon[39715]: osdmap e361: 8 total, 8 up, 8 in 2024-03-11T22:23:36.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:36 smithi045 ceph-mon[39715]: 3.11 scrub starts 2024-03-11T22:23:36.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:36 smithi045 ceph-mon[39715]: 3.11 scrub ok 2024-03-11T22:23:36.940 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:36 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:23:36.940 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:36 smithi045 ceph-mon[30178]: osdmap e361: 8 total, 8 up, 8 in 2024-03-11T22:23:36.940 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:36 smithi045 ceph-mon[30178]: 3.11 scrub starts 2024-03-11T22:23:36.940 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:36 smithi045 ceph-mon[30178]: 3.11 scrub ok 2024-03-11T22:23:36.971 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:37.007 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:37.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:37.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:37.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:37.144 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:37.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:37.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:37.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:37.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:37.284 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:37.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:37.384 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:37.423 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:37.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:37.525 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:37.565 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:37.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:37.666 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:37.721 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:37.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:37.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:37.820 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 36 16 67247 67231 116.705 78.75 0.0227795 0.00856285 2024-03-11T22:23:37.823 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:37.885 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:37.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:37.986 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:38.022 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:38.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:38.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:38.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:37 smithi045 ceph-mon[39715]: 4.b scrub starts 2024-03-11T22:23:38.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:37 smithi045 ceph-mon[39715]: 4.b scrub ok 2024-03-11T22:23:38.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:37 smithi045 ceph-mon[39715]: pgmap v709: 168 pgs: 168 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 765 B/s rd, 115 MiB/s wr, 1.83k op/s 2024-03-11T22:23:38.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:38.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:37 smithi045 ceph-mon[39715]: 6.1a scrub starts 2024-03-11T22:23:38.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:37 smithi045 ceph-mon[39715]: 6.1a scrub ok 2024-03-11T22:23:38.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:37 smithi045 ceph-mon[30178]: 4.b scrub starts 2024-03-11T22:23:38.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:37 smithi045 ceph-mon[30178]: 4.b scrub ok 2024-03-11T22:23:38.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:37 smithi045 ceph-mon[30178]: pgmap v709: 168 pgs: 168 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 765 B/s rd, 115 MiB/s wr, 1.83k op/s 2024-03-11T22:23:38.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:38.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:37 smithi045 ceph-mon[30178]: 6.1a scrub starts 2024-03-11T22:23:38.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:37 smithi045 ceph-mon[30178]: 6.1a scrub ok 2024-03-11T22:23:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:37 smithi144 ceph-mon[33984]: 4.b scrub starts 2024-03-11T22:23:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:37 smithi144 ceph-mon[33984]: 4.b scrub ok 2024-03-11T22:23:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:37 smithi144 ceph-mon[33984]: pgmap v709: 168 pgs: 168 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 765 B/s rd, 115 MiB/s wr, 1.83k op/s 2024-03-11T22:23:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:37 smithi144 ceph-mon[33984]: 6.1a scrub starts 2024-03-11T22:23:38.096 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:37 smithi144 ceph-mon[33984]: 6.1a scrub ok 2024-03-11T22:23:38.124 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:38.158 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:38.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:38.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:38.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:38.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:38.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:38.401 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:38.452 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:38.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:38.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:38.554 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:38.596 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:23:38.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:38.597 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:38.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:38.698 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:38.739 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:38.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:38.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:38.820 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 37 16 68512 68496 115.688 79.0625 0.0124712 0.0086384 2024-03-11T22:23:38.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:38.884 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:38.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:38.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:38.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:38 smithi045 ceph-mon[30178]: 4.3 scrub starts 2024-03-11T22:23:38.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:38 smithi045 ceph-mon[30178]: 4.3 scrub ok 2024-03-11T22:23:38.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:38 smithi045 ceph-mon[30178]: 4.15 scrub starts 2024-03-11T22:23:38.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:38 smithi045 ceph-mon[30178]: 4.15 scrub ok 2024-03-11T22:23:38.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:38 smithi045 ceph-mon[39715]: 4.3 scrub starts 2024-03-11T22:23:38.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:38 smithi045 ceph-mon[39715]: 4.3 scrub ok 2024-03-11T22:23:38.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:38 smithi045 ceph-mon[39715]: 4.15 scrub starts 2024-03-11T22:23:38.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:38 smithi045 ceph-mon[39715]: 4.15 scrub ok 2024-03-11T22:23:38.986 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:39.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:39.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:39.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:38 smithi144 ceph-mon[33984]: 4.3 scrub starts 2024-03-11T22:23:39.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:38 smithi144 ceph-mon[33984]: 4.3 scrub ok 2024-03-11T22:23:39.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:38 smithi144 ceph-mon[33984]: 4.15 scrub starts 2024-03-11T22:23:39.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:38 smithi144 ceph-mon[33984]: 4.15 scrub ok 2024-03-11T22:23:39.128 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:39.180 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:39.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:39.280 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:39.319 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:39.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:39.421 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:39.456 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:39.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:39.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:39.557 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:39.603 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:39.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:39.704 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:39.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:39.743 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:39.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:39.820 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 38 16 69858 69842 114.857 84.125 0.00777339 0.00870169 2024-03-11T22:23:39.844 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:39.867 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:39.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:39.969 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:40.022 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:40.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:40.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:40.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:39 smithi045 ceph-mon[39715]: 3.f scrub starts 2024-03-11T22:23:40.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:39 smithi045 ceph-mon[39715]: 3.f scrub ok 2024-03-11T22:23:40.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:39 smithi045 ceph-mon[39715]: pgmap v710: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-11T22:23:40.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:39 smithi045 ceph-mon[30178]: 3.f scrub starts 2024-03-11T22:23:40.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:39 smithi045 ceph-mon[30178]: 3.f scrub ok 2024-03-11T22:23:40.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:39 smithi045 ceph-mon[30178]: pgmap v710: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-11T22:23:40.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:39 smithi144 ceph-mon[33984]: 3.f scrub starts 2024-03-11T22:23:40.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:39 smithi144 ceph-mon[33984]: 3.f scrub ok 2024-03-11T22:23:40.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:39 smithi144 ceph-mon[33984]: pgmap v710: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-11T22:23:40.124 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:40.166 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:40.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:40.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:40.305 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:40.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:40.407 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:40.451 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:40.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:40.553 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:40.588 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:40.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:40.690 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:40.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:40.729 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:40.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:40.820 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 39 16 71002 70986 113.745 71.5 0.00634269 0.00878628 2024-03-11T22:23:40.829 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:40.866 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:40.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:40.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:41.002 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:41.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:41.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:41.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:40 smithi045 ceph-mon[39715]: 1.4 scrub starts 2024-03-11T22:23:41.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:40 smithi045 ceph-mon[39715]: 1.4 scrub ok 2024-03-11T22:23:41.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:40 smithi045 ceph-mon[30178]: 1.4 scrub starts 2024-03-11T22:23:41.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:40 smithi045 ceph-mon[30178]: 1.4 scrub ok 2024-03-11T22:23:41.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:40 smithi144 ceph-mon[33984]: 1.4 scrub starts 2024-03-11T22:23:41.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:40 smithi144 ceph-mon[33984]: 1.4 scrub ok 2024-03-11T22:23:41.104 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:41.126 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:41.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:41.228 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:41.271 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:41.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:41.372 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:41.409 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:41.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:41.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:41.511 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:41.530 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T22:23:41.550 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:41.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:41.651 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:41.689 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:41.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:41.791 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:41.822 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:41.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:41.824 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:23:41.821069+0000 min lat: 0.00118312 max lat: 0.134618 avg lat: 0.00886595 2024-03-11T22:23:41.824 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:23:41.824 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 40 16 72176 72160 112.727 73.375 0.00561379 0.00886595 2024-03-11T22:23:41.924 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:41.961 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:41.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:41.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:42.009 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:41 smithi045 ceph-mon[30178]: 6.18 scrub starts 2024-03-11T22:23:42.009 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:41 smithi045 ceph-mon[30178]: 6.18 scrub ok 2024-03-11T22:23:42.009 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:41 smithi045 ceph-mon[30178]: pgmap v711: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s 2024-03-11T22:23:42.009 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:41 smithi045 ceph-mon[30178]: 5.b scrub starts 2024-03-11T22:23:42.009 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:41 smithi045 ceph-mon[30178]: 5.b scrub ok 2024-03-11T22:23:42.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:41 smithi045 ceph-mon[39715]: 6.18 scrub starts 2024-03-11T22:23:42.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:41 smithi045 ceph-mon[39715]: 6.18 scrub ok 2024-03-11T22:23:42.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:41 smithi045 ceph-mon[39715]: pgmap v711: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s 2024-03-11T22:23:42.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:41 smithi045 ceph-mon[39715]: 5.b scrub starts 2024-03-11T22:23:42.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:41 smithi045 ceph-mon[39715]: 5.b scrub ok 2024-03-11T22:23:42.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:42.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:41 smithi144 ceph-mon[33984]: 6.18 scrub starts 2024-03-11T22:23:42.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:41 smithi144 ceph-mon[33984]: 6.18 scrub ok 2024-03-11T22:23:42.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:41 smithi144 ceph-mon[33984]: pgmap v711: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s 2024-03-11T22:23:42.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:41 smithi144 ceph-mon[33984]: 5.b scrub starts 2024-03-11T22:23:42.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:41 smithi144 ceph-mon[33984]: 5.b scrub ok 2024-03-11T22:23:42.108 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:42.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:42.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:42.244 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:42.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:42.346 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:42.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:42.386 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:42.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:42.487 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:42.527 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:42.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:42.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:42.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:42.691 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:42.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:42.781 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:23:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:23:42.792 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:42.824 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 41 16 73393 73377 111.831 76.0625 0.020589 0.0089358 2024-03-11T22:23:42.827 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:42.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:42.929 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:42.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:42.969 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:42.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:43.008 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:23:43.008 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:23:43.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.009 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:73573 10.2 e361/348 10:45b9d52a:::benchmark_data_smithi045_169175_object73572:head v 361'4517, mlcod=361'4512)", 2024-03-11T22:23:43.009 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:42.988515+0000", 2024-03-11T22:23:43.009 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.015381186999999999, 2024-03-11T22:23:43.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.015415415999999999, 2024-03-11T22:23:43.009 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:23:43.009 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:23:43.009 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:23:43.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:23:43.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.988515+0000", 2024-03-11T22:23:43.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:23:43.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.988515+0000", 2024-03-11T22:23:43.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:23:43.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.988515+0000", 2024-03-11T22:23:43.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0699999999999998e-07 2024-03-11T22:23:43.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:23:43.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.988537+0000", 2024-03-11T22:23:43.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1727999999999999e-05 2024-03-11T22:23:43.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:23:43.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.988545+0000", 2024-03-11T22:23:43.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7719999999999994e-06 2024-03-11T22:23:43.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:23:43.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.988551+0000", 2024-03-11T22:23:43.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8119999999999996e-06 2024-03-11T22:23:43.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:23:43.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.988661+0000", 2024-03-11T22:23:43.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110707 2024-03-11T22:23:43.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:23:43.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.988690+0000", 2024-03-11T22:23:43.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8051999999999998e-05 2024-03-11T22:23:43.013 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.013 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:23:43.013 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.013 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:73582 10.4 10:2ca5e211:::benchmark_data_smithi045_169175_object73581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-03-11T22:23:43.013 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:42.992931+0000", 2024-03-11T22:23:43.013 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.010965419000000001, 2024-03-11T22:23:43.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.01107502, 2024-03-11T22:23:43.013 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:23:43.013 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:23:43.014 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:23:43.014 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:23:43.014 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:23:43.014 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 73582 2024-03-11T22:23:43.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.014 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:23:43.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:23:43.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.992931+0000", 2024-03-11T22:23:43.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:23:43.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.992931+0000", 2024-03-11T22:23:43.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:23:43.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.992932+0000", 2024-03-11T22:23:43.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8700000000000005e-07 2024-03-11T22:23:43.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:23:43.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.992959+0000", 2024-03-11T22:23:43.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7356e-05 2024-03-11T22:23:43.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:23:43.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.992970+0000", 2024-03-11T22:23:43.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.061e-05 2024-03-11T22:23:43.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:23:43.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.992978+0000", 2024-03-11T22:23:43.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.055e-06 2024-03-11T22:23:43.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:23:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.995635+0000", 2024-03-11T22:23:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026571839999999999 2024-03-11T22:23:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:23:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.995765+0000", 2024-03-11T22:23:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012972999999999999 2024-03-11T22:23:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-03-11T22:23:43.017 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.995940+0000", 2024-03-11T22:23:43.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017595600000000001 2024-03-11T22:23:43.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.018 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:23:43.018 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.999566+0000", 2024-03-11T22:23:43.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0036254690000000001 2024-03-11T22:23:43.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.018 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:23:43.018 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.002521+0000", 2024-03-11T22:23:43.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002955184 2024-03-11T22:23:43.018 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:73583 10.2 e361/348 10:41873f12:::benchmark_data_smithi045_169175_object73582:head v 361'4518, mlcod=361'4515)", 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:42.996829+0000", 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0070669909999999999, 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00726905, 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:23:43.019 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.996829+0000", 2024-03-11T22:23:43.020 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.020 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:23:43.020 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.996829+0000", 2024-03-11T22:23:43.020 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.020 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:23:43.020 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.996830+0000", 2024-03-11T22:23:43.020 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.16e-07 2024-03-11T22:23:43.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.996873+0000", 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2752e-05 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.996885+0000", 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1848000000000001e-05 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.996892+0000", 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3499999999999999e-06 2024-03-11T22:23:43.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.022 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:23:43.022 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.997045+0000", 2024-03-11T22:23:43.022 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015301799999999999 2024-03-11T22:23:43.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.022 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:23:43.022 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.997066+0000", 2024-03-11T22:23:43.022 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1092000000000001e-05 2024-03-11T22:23:43.022 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.022 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:23:43.022 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.023 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:73584 10.2 e361/348 10:44ccf525:::benchmark_data_smithi045_169175_object73583:head v 361'4519, mlcod=361'4515)", 2024-03-11T22:23:43.023 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:42.996915+0000", 2024-03-11T22:23:43.023 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.006981503, 2024-03-11T22:23:43.023 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0073358039999999996, 2024-03-11T22:23:43.023 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:23:43.023 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:23:43.023 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:23:43.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.023 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:23:43.023 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.996915+0000", 2024-03-11T22:23:43.024 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.024 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.024 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:23:43.024 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.996915+0000", 2024-03-11T22:23:43.024 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.025 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:23:43.025 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.996915+0000", 2024-03-11T22:23:43.025 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1899999999999998e-07 2024-03-11T22:23:43.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.025 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:23:43.025 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.996939+0000", 2024-03-11T22:23:43.025 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3413000000000001e-05 2024-03-11T22:23:43.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.025 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:23:43.026 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.996961+0000", 2024-03-11T22:23:43.026 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2090999999999999e-05 2024-03-11T22:23:43.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.026 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:23:43.026 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.996978+0000", 2024-03-11T22:23:43.026 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7493999999999999e-05 2024-03-11T22:23:43.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.026 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:23:43.026 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.002899+0000", 2024-03-11T22:23:43.026 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0059207419999999997 2024-03-11T22:23:43.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.027 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.027 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:23:43.027 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.002920+0000", 2024-03-11T22:23:43.027 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1288999999999999e-05 2024-03-11T22:23:43.027 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.027 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:23:43.027 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.027 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.027 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:73591 10.6 e361/348 10:6bea5b38:::benchmark_data_smithi045_169175_object73590:head v 361'4727, mlcod=361'4725)", 2024-03-11T22:23:43.027 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:42.998737+0000", 2024-03-11T22:23:43.027 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0051593059999999998, 2024-03-11T22:23:43.028 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0055803729999999996, 2024-03-11T22:23:43.028 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:23:43.028 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:23:43.028 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:23:43.028 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.028 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:23:43.028 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.998737+0000", 2024-03-11T22:23:43.028 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.028 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.028 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:23:43.028 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.998737+0000", 2024-03-11T22:23:43.029 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.029 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.029 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.029 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:23:43.029 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.998754+0000", 2024-03-11T22:23:43.029 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6631999999999999e-05 2024-03-11T22:23:43.029 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.029 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.029 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:23:43.029 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.998784+0000", 2024-03-11T22:23:43.029 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0694999999999997e-05 2024-03-11T22:23:43.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.030 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.030 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:23:43.030 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.998795+0000", 2024-03-11T22:23:43.030 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0682999999999999e-05 2024-03-11T22:23:43.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.030 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.030 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:23:43.030 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.998973+0000", 2024-03-11T22:23:43.030 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000178228 2024-03-11T22:23:43.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.030 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.030 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:23:43.031 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.001880+0000", 2024-03-11T22:23:43.031 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002907115 2024-03-11T22:23:43.031 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.031 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.031 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:23:43.031 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.001900+0000", 2024-03-11T22:23:43.031 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9762000000000002e-05 2024-03-11T22:23:43.031 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.031 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:23:43.031 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.031 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.032 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.032 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:73592 10.a e361/348 10:58c54dce:::benchmark_data_smithi045_169175_object73591:head v 361'4626, mlcod=361'4624)", 2024-03-11T22:23:43.032 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:42.999824+0000", 2024-03-11T22:23:43.032 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0040722019999999996, 2024-03-11T22:23:43.032 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0045627890000000003, 2024-03-11T22:23:43.032 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:23:43.032 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:23:43.032 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:23:43.032 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.032 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:23:43.032 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.999824+0000", 2024-03-11T22:23:43.032 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.033 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.033 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:23:43.033 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.999824+0000", 2024-03-11T22:23:43.033 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.033 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.033 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:23:43.033 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.999825+0000", 2024-03-11T22:23:43.033 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6700000000000003e-07 2024-03-11T22:23:43.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.033 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:23:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.999879+0000", 2024-03-11T22:23:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4483e-05 2024-03-11T22:23:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:23:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.999895+0000", 2024-03-11T22:23:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5301000000000001e-05 2024-03-11T22:23:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:23:43.034 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.999901+0000", 2024-03-11T22:23:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7399999999999998e-06 2024-03-11T22:23:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:23:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.000968+0000", 2024-03-11T22:23:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001066254 2024-03-11T22:23:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:23:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.000987+0000", 2024-03-11T22:23:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9077999999999999e-05 2024-03-11T22:23:43.035 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:23:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:73587 10.2 e361/348)", 2024-03-11T22:23:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:42.999965+0000", 2024-03-11T22:23:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0039309369999999998, 2024-03-11T22:23:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044851989999999996, 2024-03-11T22:23:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:23:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:23:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:23:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.036 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:23:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.999965+0000", 2024-03-11T22:23:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:23:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.999965+0000", 2024-03-11T22:23:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:23:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.999966+0000", 2024-03-11T22:23:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5300000000000004e-07 2024-03-11T22:23:43.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:23:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:42.999988+0000", 2024-03-11T22:23:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2529000000000001e-05 2024-03-11T22:23:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:23:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.000000+0000", 2024-03-11T22:23:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1841000000000001e-05 2024-03-11T22:23:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.038 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:23:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.000007+0000", 2024-03-11T22:23:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4609999999999996e-06 2024-03-11T22:23:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:23:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:73595 10.c 10:3b2d5d11:::benchmark_data_smithi045_169175_object73594:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-03-11T22:23:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:43.001563+0000", 2024-03-11T22:23:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0023327840000000001, 2024-03-11T22:23:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002945301, 2024-03-11T22:23:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:23:43.039 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:23:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:23:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:23:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:23:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 73595 2024-03-11T22:23:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:23:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:23:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.001563+0000", 2024-03-11T22:23:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.040 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:23:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.001563+0000", 2024-03-11T22:23:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:23:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.001565+0000", 2024-03-11T22:23:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4100000000000001e-06 2024-03-11T22:23:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:23:43.041 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.001588+0000", 2024-03-11T22:23:43.042 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3059000000000001e-05 2024-03-11T22:23:43.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.042 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:23:43.042 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.001599+0000", 2024-03-11T22:23:43.042 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1199999999999999e-05 2024-03-11T22:23:43.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.042 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:23:43.042 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.001607+0000", 2024-03-11T22:23:43.042 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7049999999999999e-06 2024-03-11T22:23:43.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.043 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:23:43.043 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.001671+0000", 2024-03-11T22:23:43.043 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4344000000000006e-05 2024-03-11T22:23:43.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.043 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:23:43.043 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.001811+0000", 2024-03-11T22:23:43.043 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000139665 2024-03-11T22:23:43.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.043 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,7", 2024-03-11T22:23:43.043 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.001965+0000", 2024-03-11T22:23:43.044 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015446000000000001 2024-03-11T22:23:43.044 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.044 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:23:43.044 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.044 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.044 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:73596 10.b e361/348 10:d4fa4c1c:::benchmark_data_smithi045_169175_object73595:head v 361'4658, mlcod=361'4656)", 2024-03-11T22:23:43.044 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:43.003953+0000", 2024-03-11T22:23:43.044 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4294967295.9999433, 2024-03-11T22:23:43.044 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00064431099999999995, 2024-03-11T22:23:43.044 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:23:43.044 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:23:43.045 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:23:43.045 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.045 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:23:43.045 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.003953+0000", 2024-03-11T22:23:43.045 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.045 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.045 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:23:43.045 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.003953+0000", 2024-03-11T22:23:43.045 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.045 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.045 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:23:43.046 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.003954+0000", 2024-03-11T22:23:43.046 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3900000000000004e-07 2024-03-11T22:23:43.046 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.046 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.046 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:23:43.046 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.003988+0000", 2024-03-11T22:23:43.046 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3995000000000002e-05 2024-03-11T22:23:43.046 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.046 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.046 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:23:43.046 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004002+0000", 2024-03-11T22:23:43.046 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4731e-05 2024-03-11T22:23:43.046 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004010+0000", 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2119999999999997e-06 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004058+0000", 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8381000000000003e-05 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004078+0000", 2024-03-11T22:23:43.047 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0316999999999999e-05 2024-03-11T22:23:43.048 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.048 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:23:43.048 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.048 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.048 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:73597 10.d e361/348 10:ba6f2dfe:::benchmark_data_smithi045_169175_object73596:head v 361'4597, mlcod=361'4595)", 2024-03-11T22:23:43.048 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:43.004029+0000", 2024-03-11T22:23:43.048 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4294967295.9998674, 2024-03-11T22:23:43.048 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00063678699999999999, 2024-03-11T22:23:43.048 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:23:43.048 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:23:43.049 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:23:43.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.049 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:23:43.049 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004029+0000", 2024-03-11T22:23:43.049 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.049 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:23:43.049 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004029+0000", 2024-03-11T22:23:43.049 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.050 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:23:43.050 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004029+0000", 2024-03-11T22:23:43.050 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4999999999999998e-07 2024-03-11T22:23:43.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.050 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:23:43.050 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004058+0000", 2024-03-11T22:23:43.050 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.906e-05 2024-03-11T22:23:43.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.050 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:23:43.050 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004065+0000", 2024-03-11T22:23:43.051 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0899999999999999e-06 2024-03-11T22:23:43.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.051 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.051 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:23:43.051 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004071+0000", 2024-03-11T22:23:43.051 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.835e-06 2024-03-11T22:23:43.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.051 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.051 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:23:43.051 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004122+0000", 2024-03-11T22:23:43.051 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1218999999999999e-05 2024-03-11T22:23:43.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.051 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004141+0000", 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8207999999999999e-05 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:73598 10.a e361/348)", 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:43.004089+0000", 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4294967295.9998078, 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00064137999999999999, 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:23:43.052 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004089+0000", 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004089+0000", 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:23:43.053 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004089+0000", 2024-03-11T22:23:43.054 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.27e-07 2024-03-11T22:23:43.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.054 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.054 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:23:43.054 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004116+0000", 2024-03-11T22:23:43.054 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.686e-05 2024-03-11T22:23:43.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.054 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.054 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:23:43.054 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004123+0000", 2024-03-11T22:23:43.054 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4460000000000004e-06 2024-03-11T22:23:43.055 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:23:43.055 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:23:43.055 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:23:43.055 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:43.004129+0000", 2024-03-11T22:23:43.055 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8279999999999999e-06 2024-03-11T22:23:43.055 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.055 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:23:43.055 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.055 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:23:43.055 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:23:43.055 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 11 2024-03-11T22:23:43.055 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:23:43.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[39715]: 4.6 scrub starts 2024-03-11T22:23:43.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[39715]: 4.6 scrub ok 2024-03-11T22:23:43.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[39715]: 1.14 scrub starts 2024-03-11T22:23:43.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[39715]: 1.14 scrub ok 2024-03-11T22:23:43.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[39715]: 3.1e scrub starts 2024-03-11T22:23:43.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[39715]: 3.1e scrub ok 2024-03-11T22:23:43.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:23:43.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:23:43.056 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[39715]: pgmap v712: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-03-11T22:23:43.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[30178]: 4.6 scrub starts 2024-03-11T22:23:43.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[30178]: 4.6 scrub ok 2024-03-11T22:23:43.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[30178]: 1.14 scrub starts 2024-03-11T22:23:43.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[30178]: 1.14 scrub ok 2024-03-11T22:23:43.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[30178]: 3.1e scrub starts 2024-03-11T22:23:43.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[30178]: 3.1e scrub ok 2024-03-11T22:23:43.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:23:43.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:23:43.057 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:42 smithi045 ceph-mon[30178]: pgmap v712: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-03-11T22:23:43.070 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:43.106 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:43.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:43.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:43.208 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:43.240 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:43.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:43.342 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:43.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:42 smithi144 ceph-mon[33984]: 4.6 scrub starts 2024-03-11T22:23:43.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:42 smithi144 ceph-mon[33984]: 4.6 scrub ok 2024-03-11T22:23:43.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:42 smithi144 ceph-mon[33984]: 1.14 scrub starts 2024-03-11T22:23:43.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:42 smithi144 ceph-mon[33984]: 1.14 scrub ok 2024-03-11T22:23:43.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:42 smithi144 ceph-mon[33984]: 3.1e scrub starts 2024-03-11T22:23:43.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:42 smithi144 ceph-mon[33984]: 3.1e scrub ok 2024-03-11T22:23:43.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:42 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:23:43.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:42 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:23:43.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:42 smithi144 ceph-mon[33984]: pgmap v712: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-03-11T22:23:43.380 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:43.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:43.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:43.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:43.522 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:43.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:43.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:43.624 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:43.657 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:43.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:43.759 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:43.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:43.794 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:43.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:43.824 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 42 16 74699 74683 111.112 81.625 0.0111311 0.00899504 2024-03-11T22:23:43.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:43.934 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:43.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:43.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:44.036 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:44.089 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:44.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:44.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:44.191 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:44.239 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:44.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:44.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:44.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:43 smithi045 ceph-mon[39715]: 4.1f scrub starts 2024-03-11T22:23:44.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:43 smithi045 ceph-mon[39715]: 4.1f scrub ok 2024-03-11T22:23:44.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:43 smithi045 ceph-mon[39715]: 3.5 scrub starts 2024-03-11T22:23:44.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:43 smithi045 ceph-mon[39715]: 3.5 scrub ok 2024-03-11T22:23:44.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:43 smithi045 ceph-mon[30178]: 4.1f scrub starts 2024-03-11T22:23:44.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:43 smithi045 ceph-mon[30178]: 4.1f scrub ok 2024-03-11T22:23:44.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:43 smithi045 ceph-mon[30178]: 3.5 scrub starts 2024-03-11T22:23:44.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:43 smithi045 ceph-mon[30178]: 3.5 scrub ok 2024-03-11T22:23:44.341 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:44.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:43 smithi144 ceph-mon[33984]: 4.1f scrub starts 2024-03-11T22:23:44.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:43 smithi144 ceph-mon[33984]: 4.1f scrub ok 2024-03-11T22:23:44.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:43 smithi144 ceph-mon[33984]: 3.5 scrub starts 2024-03-11T22:23:44.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:43 smithi144 ceph-mon[33984]: 3.5 scrub ok 2024-03-11T22:23:44.377 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:44.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:44.479 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:44.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:44.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:44.629 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:44.681 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:44.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:44.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:44.821 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:44.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:44.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:44.824 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 43 16 76003 75987 110.423 81.5 0.0102543 0.00905137 2024-03-11T22:23:44.924 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:44.961 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:44.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:44.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:45.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T22:23:45.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:45.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:45.148 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:45.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:45.203 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:44 smithi045 ceph-mon[39715]: 1.9 scrub starts 2024-03-11T22:23:45.203 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:44 smithi045 ceph-mon[39715]: 1.9 scrub ok 2024-03-11T22:23:45.203 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:44 smithi045 ceph-mon[39715]: pgmap v713: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:23:45.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:44 smithi045 ceph-mon[30178]: 1.9 scrub starts 2024-03-11T22:23:45.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:44 smithi045 ceph-mon[30178]: 1.9 scrub ok 2024-03-11T22:23:45.204 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:44 smithi045 ceph-mon[30178]: pgmap v713: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:23:45.249 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:45.281 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:45.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:45.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:45.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:44 smithi144 ceph-mon[33984]: 1.9 scrub starts 2024-03-11T22:23:45.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:44 smithi144 ceph-mon[33984]: 1.9 scrub ok 2024-03-11T22:23:45.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:44 smithi144 ceph-mon[33984]: pgmap v713: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:23:45.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:45.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:45.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:45.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:45.537 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:45.569 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:45.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:45.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:45.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:45.736 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:45.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:45.824 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 44 16 77230 77214 109.656 76.6875 0.00854112 0.00911385 2024-03-11T22:23:45.837 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:45.867 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:45.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:45.969 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:45.998 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:45.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:46.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:46.164 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:46.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:46.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[30178]: 1.d scrub starts 2024-03-11T22:23:46.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[30178]: 1.d scrub ok 2024-03-11T22:23:46.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[30178]: 3.14 scrub starts 2024-03-11T22:23:46.208 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[30178]: 3.14 scrub ok 2024-03-11T22:23:46.209 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[30178]: 6.a scrub starts 2024-03-11T22:23:46.209 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[30178]: 6.a scrub ok 2024-03-11T22:23:46.209 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:23:46.209 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[39715]: 1.d scrub starts 2024-03-11T22:23:46.209 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[39715]: 1.d scrub ok 2024-03-11T22:23:46.209 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[39715]: 3.14 scrub starts 2024-03-11T22:23:46.209 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[39715]: 3.14 scrub ok 2024-03-11T22:23:46.209 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[39715]: 6.a scrub starts 2024-03-11T22:23:46.209 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[39715]: 6.a scrub ok 2024-03-11T22:23:46.209 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:45 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:23:46.226 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:46.233 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:46.265 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:46.295 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:46.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:46.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:46.328 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:46.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:45 smithi144 ceph-mon[33984]: 1.d scrub starts 2024-03-11T22:23:46.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:45 smithi144 ceph-mon[33984]: 1.d scrub ok 2024-03-11T22:23:46.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:45 smithi144 ceph-mon[33984]: 3.14 scrub starts 2024-03-11T22:23:46.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:45 smithi144 ceph-mon[33984]: 3.14 scrub ok 2024-03-11T22:23:46.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:45 smithi144 ceph-mon[33984]: 6.a scrub starts 2024-03-11T22:23:46.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:45 smithi144 ceph-mon[33984]: 6.a scrub ok 2024-03-11T22:23:46.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:45 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:23:46.397 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:46.427 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:46.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:46.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:46.528 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:46.563 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:46.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:46.665 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:46.700 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:46.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:46.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:46.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:46.824 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 45 16 78418 78402 108.869 74.25 0.0133486 0.00918069 2024-03-11T22:23:46.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:46.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:46.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:46.848 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:23:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:23:46.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:46.975 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:23:46.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[30178]: 2.1 scrub starts 2024-03-11T22:23:46.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[30178]: 2.1 scrub ok 2024-03-11T22:23:46.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[30178]: 4.13 scrub starts 2024-03-11T22:23:46.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[30178]: 4.13 scrub ok 2024-03-11T22:23:46.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3087333503' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:46.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:46.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[30178]: pgmap v714: 168 pgs: 168 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 959 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-11T22:23:46.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:46.992 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[39715]: 2.1 scrub starts 2024-03-11T22:23:46.992 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[39715]: 2.1 scrub ok 2024-03-11T22:23:46.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[39715]: 4.13 scrub starts 2024-03-11T22:23:46.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[39715]: 4.13 scrub ok 2024-03-11T22:23:46.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3087333503' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:46.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:46.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[39715]: pgmap v714: 168 pgs: 168 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 959 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-11T22:23:46.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:46 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:46.995 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:46.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:46.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:47.097 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:47.132 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:47.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:47.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:47.234 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:47.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:46 smithi144 ceph-mon[33984]: 2.1 scrub starts 2024-03-11T22:23:47.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:46 smithi144 ceph-mon[33984]: 2.1 scrub ok 2024-03-11T22:23:47.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:46 smithi144 ceph-mon[33984]: 4.13 scrub starts 2024-03-11T22:23:47.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:46 smithi144 ceph-mon[33984]: 4.13 scrub ok 2024-03-11T22:23:47.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:46 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3087333503' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:47.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:46 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:23:47.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:46 smithi144 ceph-mon[33984]: pgmap v714: 168 pgs: 168 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 959 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-11T22:23:47.248 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:46 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:47.269 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:47.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:47.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:47.371 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:47.412 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:47.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:47.514 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:47.551 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:47.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:47.652 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:47.686 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:47.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:47.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:47.788 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:47.825 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 46 16 79686 79670 108.225 79.25 0.0149436 0.00923491 2024-03-11T22:23:47.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:47.829 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:47.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:47.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:47.971 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:47.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:47.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:48.020 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[30178]: 5.1e scrub starts 2024-03-11T22:23:48.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[30178]: 5.1e scrub ok 2024-03-11T22:23:48.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[30178]: 4.1c scrub starts 2024-03-11T22:23:48.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[30178]: 4.1c scrub ok 2024-03-11T22:23:48.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[30178]: 6.15 scrub starts 2024-03-11T22:23:48.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[30178]: 6.15 scrub ok 2024-03-11T22:23:48.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[30178]: 4.9 scrub starts 2024-03-11T22:23:48.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[30178]: 4.9 scrub ok 2024-03-11T22:23:48.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:23:48.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:23:48.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:23:48.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[30178]: osdmap e362: 8 total, 8 up, 8 in 2024-03-11T22:23:48.021 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[39715]: 5.1e scrub starts 2024-03-11T22:23:48.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[39715]: 5.1e scrub ok 2024-03-11T22:23:48.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[39715]: 4.1c scrub starts 2024-03-11T22:23:48.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[39715]: 4.1c scrub ok 2024-03-11T22:23:48.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[39715]: 6.15 scrub starts 2024-03-11T22:23:48.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[39715]: 6.15 scrub ok 2024-03-11T22:23:48.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[39715]: 4.9 scrub starts 2024-03-11T22:23:48.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[39715]: 4.9 scrub ok 2024-03-11T22:23:48.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:23:48.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:23:48.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:23:48.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:47 smithi045 ceph-mon[39715]: osdmap e362: 8 total, 8 up, 8 in 2024-03-11T22:23:48.073 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:48.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:48.110 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:48.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:48.205 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:47 smithi144 ceph-mon[33984]: 5.1e scrub starts 2024-03-11T22:23:48.205 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:47 smithi144 ceph-mon[33984]: 5.1e scrub ok 2024-03-11T22:23:48.205 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:47 smithi144 ceph-mon[33984]: 4.1c scrub starts 2024-03-11T22:23:48.205 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:47 smithi144 ceph-mon[33984]: 4.1c scrub ok 2024-03-11T22:23:48.205 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:47 smithi144 ceph-mon[33984]: 6.15 scrub starts 2024-03-11T22:23:48.205 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:47 smithi144 ceph-mon[33984]: 6.15 scrub ok 2024-03-11T22:23:48.206 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:47 smithi144 ceph-mon[33984]: 4.9 scrub starts 2024-03-11T22:23:48.206 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:47 smithi144 ceph-mon[33984]: 4.9 scrub ok 2024-03-11T22:23:48.206 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:47 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:23:48.206 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:47 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:23:48.206 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:47 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:23:48.206 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:47 smithi144 ceph-mon[33984]: osdmap e362: 8 total, 8 up, 8 in 2024-03-11T22:23:48.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:48.252 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:48.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:48.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:48.394 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:48.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:48.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:48.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:48.541 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:48.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:48.643 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:48.682 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:48.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:48.784 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:48.818 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:48.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:48.825 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 47 16 80920 80904 107.563 77.125 0.00875046 0.00929199 2024-03-11T22:23:48.920 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:48.956 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:48.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:49.057 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:49.082 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:49.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:49.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:49.183 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:49.217 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:49.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:49.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[30178]: 1.5 scrub starts 2024-03-11T22:23:49.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[30178]: 1.5 scrub ok 2024-03-11T22:23:49.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[30178]: 4.19 scrub starts 2024-03-11T22:23:49.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[30178]: 4.19 scrub ok 2024-03-11T22:23:49.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[30178]: 1.6 scrub starts 2024-03-11T22:23:49.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[30178]: 1.6 scrub ok 2024-03-11T22:23:49.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[30178]: 4.8 scrub starts 2024-03-11T22:23:49.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[30178]: 4.8 scrub ok 2024-03-11T22:23:49.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[30178]: pgmap v716: 168 pgs: 168 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-11T22:23:49.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[39715]: 1.5 scrub starts 2024-03-11T22:23:49.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[39715]: 1.5 scrub ok 2024-03-11T22:23:49.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[39715]: 4.19 scrub starts 2024-03-11T22:23:49.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[39715]: 4.19 scrub ok 2024-03-11T22:23:49.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[39715]: 1.6 scrub starts 2024-03-11T22:23:49.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[39715]: 1.6 scrub ok 2024-03-11T22:23:49.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[39715]: 4.8 scrub starts 2024-03-11T22:23:49.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[39715]: 4.8 scrub ok 2024-03-11T22:23:49.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:48 smithi045 ceph-mon[39715]: pgmap v716: 168 pgs: 168 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-11T22:23:49.319 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:49.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:48 smithi144 ceph-mon[33984]: 1.5 scrub starts 2024-03-11T22:23:49.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:48 smithi144 ceph-mon[33984]: 1.5 scrub ok 2024-03-11T22:23:49.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:48 smithi144 ceph-mon[33984]: 4.19 scrub starts 2024-03-11T22:23:49.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:48 smithi144 ceph-mon[33984]: 4.19 scrub ok 2024-03-11T22:23:49.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:48 smithi144 ceph-mon[33984]: 1.6 scrub starts 2024-03-11T22:23:49.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:48 smithi144 ceph-mon[33984]: 1.6 scrub ok 2024-03-11T22:23:49.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:48 smithi144 ceph-mon[33984]: 4.8 scrub starts 2024-03-11T22:23:49.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:48 smithi144 ceph-mon[33984]: 4.8 scrub ok 2024-03-11T22:23:49.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:48 smithi144 ceph-mon[33984]: pgmap v716: 168 pgs: 168 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-11T22:23:49.353 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:49.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:49.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:49.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:49.494 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:49.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:49.595 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:49.628 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:49.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:49.730 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:49.768 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:49.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:49.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:49.825 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 48 15 82145 82130 106.918 76.625 0.00916385 0.00934836 2024-03-11T22:23:49.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:49.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:49.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:49.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:50.007 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:50.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:23:50.043 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:50.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:50.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:50.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[30178]: 6.17 deep-scrub starts 2024-03-11T22:23:50.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[30178]: 6.17 deep-scrub ok 2024-03-11T22:23:50.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[30178]: 5.c scrub starts 2024-03-11T22:23:50.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[30178]: 5.c scrub ok 2024-03-11T22:23:50.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[30178]: 6.c scrub starts 2024-03-11T22:23:50.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[30178]: 6.c scrub ok 2024-03-11T22:23:50.063 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[30178]: 3.10 scrub starts 2024-03-11T22:23:50.064 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[30178]: 3.10 scrub ok 2024-03-11T22:23:50.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[39715]: 6.17 deep-scrub starts 2024-03-11T22:23:50.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[39715]: 6.17 deep-scrub ok 2024-03-11T22:23:50.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[39715]: 5.c scrub starts 2024-03-11T22:23:50.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[39715]: 5.c scrub ok 2024-03-11T22:23:50.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[39715]: 6.c scrub starts 2024-03-11T22:23:50.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[39715]: 6.c scrub ok 2024-03-11T22:23:50.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[39715]: 3.10 scrub starts 2024-03-11T22:23:50.064 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:49 smithi045 ceph-mon[39715]: 3.10 scrub ok 2024-03-11T22:23:50.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:50.189 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:50.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:50.290 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:50.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:49 smithi144 ceph-mon[33984]: 6.17 deep-scrub starts 2024-03-11T22:23:50.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:49 smithi144 ceph-mon[33984]: 6.17 deep-scrub ok 2024-03-11T22:23:50.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:49 smithi144 ceph-mon[33984]: 5.c scrub starts 2024-03-11T22:23:50.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:49 smithi144 ceph-mon[33984]: 5.c scrub ok 2024-03-11T22:23:50.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:49 smithi144 ceph-mon[33984]: 6.c scrub starts 2024-03-11T22:23:50.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:49 smithi144 ceph-mon[33984]: 6.c scrub ok 2024-03-11T22:23:50.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:49 smithi144 ceph-mon[33984]: 3.10 scrub starts 2024-03-11T22:23:50.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:49 smithi144 ceph-mon[33984]: 3.10 scrub ok 2024-03-11T22:23:50.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:50.362 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:50.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:50.463 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:50.494 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:50.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:50.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:50.596 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:50.632 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:50.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:50.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:50.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:50.788 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:50.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:50.825 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 49 16 83298 83282 106.206 72 0.0075349 0.00941083 2024-03-11T22:23:50.889 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:50.921 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:50.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:51.023 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:51.059 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:51.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:51.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:51.161 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:51.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:51.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:51.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:50 smithi045 ceph-mon[30178]: 6.f scrub starts 2024-03-11T22:23:51.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:50 smithi045 ceph-mon[30178]: 6.f scrub ok 2024-03-11T22:23:51.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:50 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:23:51.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:50 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:23:51.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:50 smithi045 ceph-mon[30178]: osdmap e363: 8 total, 8 up, 8 in 2024-03-11T22:23:51.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:50 smithi045 ceph-mon[30178]: pgmap v718: 168 pgs: 168 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 775 op/s 2024-03-11T22:23:51.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:50 smithi045 ceph-mon[39715]: 6.f scrub starts 2024-03-11T22:23:51.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:50 smithi045 ceph-mon[39715]: 6.f scrub ok 2024-03-11T22:23:51.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:50 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:23:51.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:50 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:23:51.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:50 smithi045 ceph-mon[39715]: osdmap e363: 8 total, 8 up, 8 in 2024-03-11T22:23:51.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:50 smithi045 ceph-mon[39715]: pgmap v718: 168 pgs: 168 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 775 op/s 2024-03-11T22:23:51.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:51.334 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:51.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:51.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:51.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:50 smithi144 ceph-mon[33984]: 6.f scrub starts 2024-03-11T22:23:51.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:50 smithi144 ceph-mon[33984]: 6.f scrub ok 2024-03-11T22:23:51.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:50 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:23:51.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:50 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:23:51.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:50 smithi144 ceph-mon[33984]: osdmap e363: 8 total, 8 up, 8 in 2024-03-11T22:23:51.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:50 smithi144 ceph-mon[33984]: pgmap v718: 168 pgs: 168 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 775 op/s 2024-03-11T22:23:51.436 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:51.479 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:51.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:51.581 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:51.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:51.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:51.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:51.769 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:51.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:51.825 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 50 15 84532 84517 105.625 77.1875 0.0136203 0.00946242 2024-03-11T22:23:51.871 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:51.903 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:51.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:52.004 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:52.033 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:52.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:52.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[39715]: 3.12 deep-scrub starts 2024-03-11T22:23:52.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[39715]: 3.12 deep-scrub ok 2024-03-11T22:23:52.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[39715]: 6.11 scrub starts 2024-03-11T22:23:52.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[39715]: 6.11 scrub ok 2024-03-11T22:23:52.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[39715]: 5.1f scrub starts 2024-03-11T22:23:52.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[39715]: 5.1f scrub ok 2024-03-11T22:23:52.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[39715]: osdmap e364: 8 total, 8 up, 8 in 2024-03-11T22:23:52.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[30178]: 3.12 deep-scrub starts 2024-03-11T22:23:52.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[30178]: 3.12 deep-scrub ok 2024-03-11T22:23:52.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[30178]: 6.11 scrub starts 2024-03-11T22:23:52.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[30178]: 6.11 scrub ok 2024-03-11T22:23:52.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[30178]: 5.1f scrub starts 2024-03-11T22:23:52.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[30178]: 5.1f scrub ok 2024-03-11T22:23:52.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:51 smithi045 ceph-mon[30178]: osdmap e364: 8 total, 8 up, 8 in 2024-03-11T22:23:52.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:52.168 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:23:52.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:52.178 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:52.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:52.212 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:23:52.253 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:23:52.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:52.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:23:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:52.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:52.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:52.335 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:23:52.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:51 smithi144 ceph-mon[33984]: 3.12 deep-scrub starts 2024-03-11T22:23:52.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:51 smithi144 ceph-mon[33984]: 3.12 deep-scrub ok 2024-03-11T22:23:52.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:51 smithi144 ceph-mon[33984]: 6.11 scrub starts 2024-03-11T22:23:52.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:51 smithi144 ceph-mon[33984]: 6.11 scrub ok 2024-03-11T22:23:52.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:51 smithi144 ceph-mon[33984]: 5.1f scrub starts 2024-03-11T22:23:52.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:51 smithi144 ceph-mon[33984]: 5.1f scrub ok 2024-03-11T22:23:52.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:51 smithi144 ceph-mon[33984]: osdmap e364: 8 total, 8 up, 8 in 2024-03-11T22:23:52.378 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:23:52.421 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:52.425 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:23:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:52.452 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:52.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:52.469 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:23:52.475 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:23:52.475 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:23:52.475 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:23:52.475 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:23:52.475 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:23:52.476 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:23:52.476 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:23:52.476 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:23:52.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:52.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:52.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:52.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:52.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:52.735 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:52.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:52.825 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 51 16 85740 85724 105.033 75.4375 0.0131832 0.00951583 2024-03-11T22:23:52.836 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:52.865 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:23:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:23:52.872 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:52.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:52.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:52.974 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:53.007 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:53.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:53.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:53.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:53.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[39715]: 6.8 scrub starts 2024-03-11T22:23:53.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[39715]: 6.8 scrub ok 2024-03-11T22:23:53.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[39715]: 1.13 deep-scrub starts 2024-03-11T22:23:53.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[39715]: 1.13 deep-scrub ok 2024-03-11T22:23:53.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[39715]: 3.a scrub starts 2024-03-11T22:23:53.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[39715]: 3.a scrub ok 2024-03-11T22:23:53.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[39715]: 6.1c scrub starts 2024-03-11T22:23:53.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[39715]: 6.1c scrub ok 2024-03-11T22:23:53.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2164148988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:53.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[39715]: pgmap v720: 168 pgs: 168 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T22:23:53.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[39715]: 4.11 scrub starts 2024-03-11T22:23:53.140 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[39715]: 4.11 scrub ok 2024-03-11T22:23:53.142 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[30178]: 6.8 scrub starts 2024-03-11T22:23:53.142 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[30178]: 6.8 scrub ok 2024-03-11T22:23:53.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[30178]: 1.13 deep-scrub starts 2024-03-11T22:23:53.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[30178]: 1.13 deep-scrub ok 2024-03-11T22:23:53.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[30178]: 3.a scrub starts 2024-03-11T22:23:53.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[30178]: 3.a scrub ok 2024-03-11T22:23:53.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[30178]: 6.1c scrub starts 2024-03-11T22:23:53.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[30178]: 6.1c scrub ok 2024-03-11T22:23:53.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2164148988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:53.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[30178]: pgmap v720: 168 pgs: 168 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T22:23:53.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[30178]: 4.11 scrub starts 2024-03-11T22:23:53.143 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:52 smithi045 ceph-mon[30178]: 4.11 scrub ok 2024-03-11T22:23:53.160 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:53.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:53.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:53.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:53.303 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:53.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:53.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:53.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:52 smithi144 ceph-mon[33984]: 6.8 scrub starts 2024-03-11T22:23:53.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: 6.8 scrub ok 2024-03-11T22:23:53.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: 1.13 deep-scrub starts 2024-03-11T22:23:53.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: 1.13 deep-scrub ok 2024-03-11T22:23:53.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: 3.a scrub starts 2024-03-11T22:23:53.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: 3.a scrub ok 2024-03-11T22:23:53.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: 6.1c scrub starts 2024-03-11T22:23:53.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: 6.1c scrub ok 2024-03-11T22:23:53.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2164148988' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:23:53.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: pgmap v720: 168 pgs: 168 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T22:23:53.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: 4.11 scrub starts 2024-03-11T22:23:53.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: 4.11 scrub ok 2024-03-11T22:23:53.405 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:53.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:53.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:53.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:53.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:53.592 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:53.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:53.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:53.734 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:53.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:53.826 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 52 16 86973 86957 104.494 77.0625 0.0190903 0.00956454 2024-03-11T22:23:53.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:53.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:53.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:53.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:54.033 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:54.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:54.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:53 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:23:54.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:53 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:23:54.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:53 smithi045 ceph-mon[30178]: osdmap e365: 8 total, 8 up, 8 in 2024-03-11T22:23:54.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:53 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:23:54.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:53 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:23:54.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:53 smithi045 ceph-mon[39715]: osdmap e365: 8 total, 8 up, 8 in 2024-03-11T22:23:54.135 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:54.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:54.169 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:54.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:54.270 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:54.319 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:54.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:54.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:23:54.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:23:54.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:53 smithi144 ceph-mon[33984]: osdmap e365: 8 total, 8 up, 8 in 2024-03-11T22:23:54.355 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:23:54.355 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:23:54.420 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:54.455 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:54.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:54.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:54.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:54.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:54.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:54.737 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:54.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:54.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:54.826 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 53 16 88218 88202 103.991 77.8125 0.0148633 0.00961168 2024-03-11T22:23:54.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:54.876 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:54.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:54.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:54.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:55.018 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:55.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:55.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:55.157 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:55.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:55.260 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:55.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:55 smithi045 ceph-mon[39715]: 5.15 scrub starts 2024-03-11T22:23:55.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:55 smithi045 ceph-mon[39715]: 5.15 scrub ok 2024-03-11T22:23:55.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:55 smithi045 ceph-mon[39715]: 4.16 deep-scrub starts 2024-03-11T22:23:55.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:55 smithi045 ceph-mon[39715]: 4.16 deep-scrub ok 2024-03-11T22:23:55.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:55 smithi045 ceph-mon[39715]: pgmap v722: 168 pgs: 168 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T22:23:55.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:55 smithi045 ceph-mon[30178]: 5.15 scrub starts 2024-03-11T22:23:55.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:55 smithi045 ceph-mon[30178]: 5.15 scrub ok 2024-03-11T22:23:55.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:55 smithi045 ceph-mon[30178]: 4.16 deep-scrub starts 2024-03-11T22:23:55.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:55 smithi045 ceph-mon[30178]: 4.16 deep-scrub ok 2024-03-11T22:23:55.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:55 smithi045 ceph-mon[30178]: pgmap v722: 168 pgs: 168 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T22:23:55.302 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:55.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:55.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:55.337 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T22:23:55.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:55 smithi144 ceph-mon[33984]: 5.15 scrub starts 2024-03-11T22:23:55.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:55 smithi144 ceph-mon[33984]: 5.15 scrub ok 2024-03-11T22:23:55.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:55 smithi144 ceph-mon[33984]: 4.16 deep-scrub starts 2024-03-11T22:23:55.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:55 smithi144 ceph-mon[33984]: 4.16 deep-scrub ok 2024-03-11T22:23:55.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:55 smithi144 ceph-mon[33984]: pgmap v722: 168 pgs: 168 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T22:23:55.403 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:55.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:55.436 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:55.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:55.522 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:23:55.537 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:55.564 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:55.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:55.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:55.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:55.737 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:55.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:55.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:55.826 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 54 16 89369 89353 103.397 71.9375 0.0231504 0.00966615 2024-03-11T22:23:55.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:23:55.880 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:55.888 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:55.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:23:55.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:56.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:56.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:56.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:56.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:56.194 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:56.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:56.272 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:56 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:23:56.272 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:56 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:23:56.272 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:56 smithi045 ceph-mon[30178]: osdmap e366: 8 total, 8 up, 8 in 2024-03-11T22:23:56.273 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:56 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:23:56.273 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:56 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:23:56.273 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:56 smithi045 ceph-mon[39715]: osdmap e366: 8 total, 8 up, 8 in 2024-03-11T22:23:56.295 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:56.324 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:56.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:56.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:56.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:56 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:23:56.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:56 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:23:56.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:56 smithi144 ceph-mon[33984]: osdmap e366: 8 total, 8 up, 8 in 2024-03-11T22:23:56.427 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:56.461 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:56.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:56.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:56.484 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T22:23:56.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:56.603 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:56.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:56.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:56.705 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:56.739 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:56.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:56.826 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 55 16 90466 90450 102.763 68.5625 0.0131207 0.00972616 2024-03-11T22:23:56.840 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:56.848 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:23:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:23:56.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:56.878 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:56.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:56.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:57.023 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:57.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:57.125 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:57.163 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:57.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:57.265 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:57.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3843328295' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:23:57.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:23:57.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[39715]: pgmap v724: 168 pgs: 168 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-11T22:23:57.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:23:57.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[39715]: osdmap e367: 8 total, 8 up, 8 in 2024-03-11T22:23:57.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:57.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[39715]: 3.3 scrub starts 2024-03-11T22:23:57.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[39715]: 3.3 scrub ok 2024-03-11T22:23:57.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3843328295' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:23:57.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:23:57.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[30178]: pgmap v724: 168 pgs: 168 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-11T22:23:57.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:23:57.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[30178]: osdmap e367: 8 total, 8 up, 8 in 2024-03-11T22:23:57.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:57.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[30178]: 3.3 scrub starts 2024-03-11T22:23:57.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:57 smithi045 ceph-mon[30178]: 3.3 scrub ok 2024-03-11T22:23:57.299 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:57.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:57.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:57 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3843328295' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:23:57.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:57 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:23:57.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:57 smithi144 ceph-mon[33984]: pgmap v724: 168 pgs: 168 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-11T22:23:57.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:57 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:23:57.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:57 smithi144 ceph-mon[33984]: osdmap e367: 8 total, 8 up, 8 in 2024-03-11T22:23:57.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:23:57.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:57 smithi144 ceph-mon[33984]: 3.3 scrub starts 2024-03-11T22:23:57.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:57 smithi144 ceph-mon[33984]: 3.3 scrub ok 2024-03-11T22:23:57.401 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:57.426 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:57.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:57.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:57.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:57.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:57.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:57.670 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:57.704 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:57.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:57.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:57.827 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 56 16 91612 91596 102.205 71.625 0.0231981 0.00977957 2024-03-11T22:23:57.846 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:57.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:57.948 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:23:57.985 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:57.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:23:58.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:58.123 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:58.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:58.224 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:23:58.265 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:58.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:23:58.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:58 smithi045 ceph-mon[39715]: 5.9 scrub starts 2024-03-11T22:23:58.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:58 smithi045 ceph-mon[39715]: 5.9 scrub ok 2024-03-11T22:23:58.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:58 smithi045 ceph-mon[39715]: 3.7 scrub starts 2024-03-11T22:23:58.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:58 smithi045 ceph-mon[30178]: 5.9 scrub starts 2024-03-11T22:23:58.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:58 smithi045 ceph-mon[30178]: 5.9 scrub ok 2024-03-11T22:23:58.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:58 smithi045 ceph-mon[30178]: 3.7 scrub starts 2024-03-11T22:23:58.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:58 smithi144 ceph-mon[33984]: 5.9 scrub starts 2024-03-11T22:23:58.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:58 smithi144 ceph-mon[33984]: 5.9 scrub ok 2024-03-11T22:23:58.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:58 smithi144 ceph-mon[33984]: 3.7 scrub starts 2024-03-11T22:23:58.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:58.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:58.421 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:58.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:58.522 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:58.562 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:58.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:58.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:58.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T22:23:58.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:58.725 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:58.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:58.825 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:58.828 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 57 16 92873 92857 101.795 78.8125 0.0151221 0.00981895 2024-03-11T22:23:58.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:58.859 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:58.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:58.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:59.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:59.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:59.063 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:59.072 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:23:59.109 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:59.145 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:59.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:59.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:23:59.292 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:59.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:59.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:23:59.305 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[39715]: 4.4 scrub starts 2024-03-11T22:23:59.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[39715]: 4.4 scrub ok 2024-03-11T22:23:59.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[39715]: 6.14 scrub starts 2024-03-11T22:23:59.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[39715]: 6.14 scrub ok 2024-03-11T22:23:59.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[39715]: 3.7 scrub ok 2024-03-11T22:23:59.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[39715]: pgmap v726: 168 pgs: 168 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T22:23:59.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[39715]: 4.1b scrub starts 2024-03-11T22:23:59.306 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[39715]: 4.1b scrub ok 2024-03-11T22:23:59.306 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[30178]: 4.4 scrub starts 2024-03-11T22:23:59.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[30178]: 4.4 scrub ok 2024-03-11T22:23:59.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[30178]: 6.14 scrub starts 2024-03-11T22:23:59.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[30178]: 6.14 scrub ok 2024-03-11T22:23:59.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[30178]: 3.7 scrub ok 2024-03-11T22:23:59.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[30178]: pgmap v726: 168 pgs: 168 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T22:23:59.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[30178]: 4.1b scrub starts 2024-03-11T22:23:59.307 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:23:59 smithi045 ceph-mon[30178]: 4.1b scrub ok 2024-03-11T22:23:59.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:59 smithi144 ceph-mon[33984]: 4.4 scrub starts 2024-03-11T22:23:59.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:59 smithi144 ceph-mon[33984]: 4.4 scrub ok 2024-03-11T22:23:59.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:59 smithi144 ceph-mon[33984]: 6.14 scrub starts 2024-03-11T22:23:59.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:59 smithi144 ceph-mon[33984]: 6.14 scrub ok 2024-03-11T22:23:59.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:59 smithi144 ceph-mon[33984]: 3.7 scrub ok 2024-03-11T22:23:59.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:59 smithi144 ceph-mon[33984]: pgmap v726: 168 pgs: 168 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-11T22:23:59.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:59 smithi144 ceph-mon[33984]: 4.1b scrub starts 2024-03-11T22:23:59.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:23:59 smithi144 ceph-mon[33984]: 4.1b scrub ok 2024-03-11T22:23:59.394 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:23:59.428 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:59.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:59.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:23:59.530 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:23:59.564 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:59.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:59.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:23:59.666 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:23:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:59.716 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:23:59.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:23:59.818 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:59.828 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 58 16 93938 93922 101.187 66.5625 0.0151207 0.00987812 2024-03-11T22:23:59.849 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:59.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:23:59.950 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:23:59.989 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:23:59.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:23:59.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:00.091 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:00.123 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:00.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:00.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:00.248 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[30178]: 3.0 scrub starts 2024-03-11T22:24:00.248 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[30178]: 3.0 scrub ok 2024-03-11T22:24:00.248 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[30178]: 3.1d scrub starts 2024-03-11T22:24:00.248 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[30178]: 3.1d scrub ok 2024-03-11T22:24:00.248 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[30178]: 6.3 scrub starts 2024-03-11T22:24:00.248 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[30178]: 6.3 scrub ok 2024-03-11T22:24:00.248 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:00.248 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:00.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[39715]: 3.0 scrub starts 2024-03-11T22:24:00.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[39715]: 3.0 scrub ok 2024-03-11T22:24:00.248 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[39715]: 3.1d scrub starts 2024-03-11T22:24:00.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[39715]: 3.1d scrub ok 2024-03-11T22:24:00.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[39715]: 6.3 scrub starts 2024-03-11T22:24:00.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[39715]: 6.3 scrub ok 2024-03-11T22:24:00.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:00.249 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:00.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:00.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:00.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:00 smithi144 ceph-mon[33984]: 3.0 scrub starts 2024-03-11T22:24:00.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:00 smithi144 ceph-mon[33984]: 3.0 scrub ok 2024-03-11T22:24:00.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:00 smithi144 ceph-mon[33984]: 3.1d scrub starts 2024-03-11T22:24:00.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:00 smithi144 ceph-mon[33984]: 3.1d scrub ok 2024-03-11T22:24:00.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:00 smithi144 ceph-mon[33984]: 6.3 scrub starts 2024-03-11T22:24:00.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:00 smithi144 ceph-mon[33984]: 6.3 scrub ok 2024-03-11T22:24:00.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:00.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:00.372 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:00.406 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:00.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:00.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:00.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:00.559 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:00.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:00.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:00.704 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:00.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:00.806 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:00.829 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 59 16 95092 95076 100.692 72.125 0.00904028 0.00992548 2024-03-11T22:24:00.848 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:00.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:00.950 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:00.988 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:00.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:01.090 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:01.128 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:01.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:01.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:01.265 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:01.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:01.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:01.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[30178]: 4.10 scrub starts 2024-03-11T22:24:01.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[30178]: 4.10 scrub ok 2024-03-11T22:24:01.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[30178]: 5.16 scrub starts 2024-03-11T22:24:01.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[30178]: 5.16 scrub ok 2024-03-11T22:24:01.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[30178]: 3.1f scrub starts 2024-03-11T22:24:01.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[30178]: 3.1f scrub ok 2024-03-11T22:24:01.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[30178]: pgmap v727: 168 pgs: 168 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 693 B/s rd, 51 MiB/s wr, 820 op/s 2024-03-11T22:24:01.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:01.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:01.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:01.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:01.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:24:01.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[39715]: 4.10 scrub starts 2024-03-11T22:24:01.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[39715]: 4.10 scrub ok 2024-03-11T22:24:01.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[39715]: 5.16 scrub starts 2024-03-11T22:24:01.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[39715]: 5.16 scrub ok 2024-03-11T22:24:01.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[39715]: 3.1f scrub starts 2024-03-11T22:24:01.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[39715]: 3.1f scrub ok 2024-03-11T22:24:01.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[39715]: pgmap v727: 168 pgs: 168 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 693 B/s rd, 51 MiB/s wr, 820 op/s 2024-03-11T22:24:01.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:01.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:01.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:01.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:01.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:24:01.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:01 smithi144 ceph-mon[33984]: 4.10 scrub starts 2024-03-11T22:24:01.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:01 smithi144 ceph-mon[33984]: 4.10 scrub ok 2024-03-11T22:24:01.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:01 smithi144 ceph-mon[33984]: 5.16 scrub starts 2024-03-11T22:24:01.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:01 smithi144 ceph-mon[33984]: 5.16 scrub ok 2024-03-11T22:24:01.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:01 smithi144 ceph-mon[33984]: 3.1f scrub starts 2024-03-11T22:24:01.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:01 smithi144 ceph-mon[33984]: 3.1f scrub ok 2024-03-11T22:24:01.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:01 smithi144 ceph-mon[33984]: pgmap v727: 168 pgs: 168 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 693 B/s rd, 51 MiB/s wr, 820 op/s 2024-03-11T22:24:01.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:24:01.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:24:01.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:01.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:01.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:24:01.368 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:01.406 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:01.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:01.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:01.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:01.558 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:01.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:01.649 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:24:01.659 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:01.692 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:01.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:01.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:01.794 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:01.827 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:01.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:01.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:01.829 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:24:01.826936+0000 min lat: 0.00118312 max lat: 0.142383 avg lat: 0.00997124 2024-03-11T22:24:01.829 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:24:01.829 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 60 16 96265 96249 100.236 73.3125 0.00901509 0.00997124 2024-03-11T22:24:01.930 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:01.963 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:01.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:02.065 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:02.100 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:02.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:02.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:02.202 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:02.236 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:02.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:02.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:02.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:02.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:02 smithi144 ceph-mon[33984]: 5.a scrub starts 2024-03-11T22:24:02.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:02 smithi144 ceph-mon[33984]: 5.a scrub ok 2024-03-11T22:24:02.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:02 smithi144 ceph-mon[33984]: 5.e scrub starts 2024-03-11T22:24:02.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:02 smithi144 ceph-mon[33984]: 5.e scrub ok 2024-03-11T22:24:02.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:02 smithi144 ceph-mon[33984]: 5.2 scrub starts 2024-03-11T22:24:02.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:02 smithi144 ceph-mon[33984]: 5.2 scrub ok 2024-03-11T22:24:02.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:02 smithi144 ceph-mon[33984]: 5.5 scrub starts 2024-03-11T22:24:02.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:02 smithi144 ceph-mon[33984]: 5.5 scrub ok 2024-03-11T22:24:02.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:02 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3186724887' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:24:02.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:02 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:24:02.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:02 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:24:02.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:02 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:24:02.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:02 smithi144 ceph-mon[33984]: osdmap e368: 8 total, 8 up, 8 in 2024-03-11T22:24:02.376 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:02.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:02.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:02.478 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:02.489 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[39715]: 5.a scrub starts 2024-03-11T22:24:02.489 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[39715]: 5.a scrub ok 2024-03-11T22:24:02.489 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[39715]: 5.e scrub starts 2024-03-11T22:24:02.489 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[39715]: 5.e scrub ok 2024-03-11T22:24:02.489 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[39715]: 5.2 scrub starts 2024-03-11T22:24:02.489 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[39715]: 5.2 scrub ok 2024-03-11T22:24:02.489 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[39715]: 5.5 scrub starts 2024-03-11T22:24:02.489 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[39715]: 5.5 scrub ok 2024-03-11T22:24:02.489 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3186724887' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:24:02.490 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:24:02.490 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:24:02.490 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:24:02.490 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[39715]: osdmap e368: 8 total, 8 up, 8 in 2024-03-11T22:24:02.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[30178]: 5.a scrub starts 2024-03-11T22:24:02.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[30178]: 5.a scrub ok 2024-03-11T22:24:02.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[30178]: 5.e scrub starts 2024-03-11T22:24:02.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[30178]: 5.e scrub ok 2024-03-11T22:24:02.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[30178]: 5.2 scrub starts 2024-03-11T22:24:02.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[30178]: 5.2 scrub ok 2024-03-11T22:24:02.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[30178]: 5.5 scrub starts 2024-03-11T22:24:02.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[30178]: 5.5 scrub ok 2024-03-11T22:24:02.490 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3186724887' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:24:02.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:24:02.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:24:02.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:24:02.491 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-mon[30178]: osdmap e368: 8 total, 8 up, 8 in 2024-03-11T22:24:02.512 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:02.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:02.614 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:02.651 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:02.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:02.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:02.784 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:24:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:24:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:24:02.793 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:02.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:02.829 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 61 16 97557 97541 99.9165 80.75 0.026295 0.0100034 2024-03-11T22:24:02.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:02.930 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:02.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:02.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:03.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:03.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:03.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:03.180 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:03.214 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:03.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:03.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:03.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:03 smithi045 ceph-mon[39715]: 5.d scrub starts 2024-03-11T22:24:03.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:03 smithi045 ceph-mon[39715]: 5.d scrub ok 2024-03-11T22:24:03.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:03 smithi045 ceph-mon[39715]: pgmap v729: 168 pgs: 168 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T22:24:03.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:03 smithi045 ceph-mon[30178]: 5.d scrub starts 2024-03-11T22:24:03.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:03 smithi045 ceph-mon[30178]: 5.d scrub ok 2024-03-11T22:24:03.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:03 smithi045 ceph-mon[30178]: pgmap v729: 168 pgs: 168 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T22:24:03.316 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:03.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:03 smithi144 ceph-mon[33984]: 5.d scrub starts 2024-03-11T22:24:03.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:03 smithi144 ceph-mon[33984]: 5.d scrub ok 2024-03-11T22:24:03.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:03 smithi144 ceph-mon[33984]: pgmap v729: 168 pgs: 168 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T22:24:03.353 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:03.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:03.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:03.455 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:03.490 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:03.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:03.592 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:03.624 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:03.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:03.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:03.769 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:03.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:03.829 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 62 15 98782 98767 99.5408 76.625 0.00716612 0.0100413 2024-03-11T22:24:03.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:03.911 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:03.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:03.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:04.013 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:04.047 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:04.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:04.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:04.104 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:04.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:04.196 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:04.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:04.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:04 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:24:04.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:04 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:24:04.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:04 smithi045 ceph-mon[39715]: osdmap e369: 8 total, 8 up, 8 in 2024-03-11T22:24:04.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:04 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:24:04.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:04 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:24:04.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:04 smithi045 ceph-mon[30178]: osdmap e369: 8 total, 8 up, 8 in 2024-03-11T22:24:04.297 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:04.334 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:04.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:04.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:04 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:24:04.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:04 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:24:04.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:04 smithi144 ceph-mon[33984]: osdmap e369: 8 total, 8 up, 8 in 2024-03-11T22:24:04.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:04.477 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:04.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:04.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:04.579 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:04.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:04.616 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:04.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:04.717 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:04.792 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:04.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:04.829 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 63 16 99968 99952 99.1363 74.0625 0.0141998 0.0100817 2024-03-11T22:24:04.894 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:04.925 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:04.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:04.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:04.963 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:24:04.964 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:24:05.027 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:05.063 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:05.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:05.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:05.212 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:05.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:05.284 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:05 smithi045 ceph-mon[30178]: pgmap v731: 168 pgs: 168 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 60 MiB/s wr, 957 op/s 2024-03-11T22:24:05.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:05 smithi045 ceph-mon[39715]: pgmap v731: 168 pgs: 168 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 60 MiB/s wr, 957 op/s 2024-03-11T22:24:05.314 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:05.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:05 smithi144 ceph-mon[33984]: pgmap v731: 168 pgs: 168 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 60 MiB/s wr, 957 op/s 2024-03-11T22:24:05.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:05.353 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:05.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:05.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:05.489 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:05.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:05.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:05.590 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:05.624 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:05.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:05.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:05.726 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:05.762 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:05.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:05.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:05.829 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 64 15 101129 101114 98.722 72.625 0.00803543 0.0101245 2024-03-11T22:24:05.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:05.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:05.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:06.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:06.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:06.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:06.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:06.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:24:06.146 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:06.177 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:06.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:06.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:06.279 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:06.330 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:06.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:06.431 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:06.473 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:06.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:06.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:06.617 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:06.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:06.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:06.753 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:06.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:06.829 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 65 16 102305 102289 98.333 73.4375 0.00752707 0.0101647 2024-03-11T22:24:06.848 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:24:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:24:06.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:06.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:06.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:06.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:07.040 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:07.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:07.141 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:07.175 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:07.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:07.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:07.276 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:07.311 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:07.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:07.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:07.450 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:07.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:07.552 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:07.590 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:07.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:07.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:07.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:07.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:07.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:07 smithi045 ceph-mon[39715]: pgmap v732: 168 pgs: 168 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T22:24:07.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:07 smithi045 ceph-mon[39715]: osdmap e370: 8 total, 8 up, 8 in 2024-03-11T22:24:07.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:07.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:07 smithi045 ceph-mon[30178]: pgmap v732: 168 pgs: 168 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T22:24:07.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:07 smithi045 ceph-mon[30178]: osdmap e370: 8 total, 8 up, 8 in 2024-03-11T22:24:07.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:07.829 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 66 16 103497 103481 97.9718 74.5 0.0125806 0.0102022 2024-03-11T22:24:07.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:07.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:07 smithi144 ceph-mon[33984]: pgmap v732: 168 pgs: 168 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-11T22:24:07.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:07 smithi144 ceph-mon[33984]: osdmap e370: 8 total, 8 up, 8 in 2024-03-11T22:24:07.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:07.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:07.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:07.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:07.975 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:08.011 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:08.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:08.113 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:08.149 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:08.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:08.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:08.285 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:08.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:08.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:08.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:08.434 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:08.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:08.534 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:08.577 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:08.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:08.679 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:08.714 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:08.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:08.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:08.815 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:08.830 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 67 16 104691 104675 97.6233 74.625 0.00364106 0.0102364 2024-03-11T22:24:08.839 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:08.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:08.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:08.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:08.976 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:08.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:09.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:09.100 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:09.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:09.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:09.202 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:09.237 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:09.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:09.337 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:09.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:09.377 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:09.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:09.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:09.502 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:09.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:09.605 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:09.637 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:09.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:09.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:09.709 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:09 smithi045 ceph-mon[39715]: pgmap v734: 168 pgs: 168 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T22:24:09.709 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:09 smithi045 ceph-mon[30178]: pgmap v734: 168 pgs: 168 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T22:24:09.740 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:09.775 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:09.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:09.830 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 68 16 105760 105744 97.1702 66.8125 0.0144264 0.0102865 2024-03-11T22:24:09.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:09 smithi144 ceph-mon[33984]: pgmap v734: 168 pgs: 168 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T22:24:09.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:09.913 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:09.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:10.014 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:10.050 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:10.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:10.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:10.190 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:10.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:10.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:10.279 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:24:10.279 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:24:10.279 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:24:10.279 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:24:10.279 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:24:10.279 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:24:10.279 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:24:10.279 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:24:10.279 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:24:10.280 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:24:10.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:10.333 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:10.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:10.435 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:10.470 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:10.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:10.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:10 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3104742406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:24:10.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:10 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3104742406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:24:10.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:10.599 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:10.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:10.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:10.700 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:10.738 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:10.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:10.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:10.830 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 69 16 106983 106967 96.8696 76.4375 0.00590658 0.0103186 2024-03-11T22:24:10.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:10.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:10 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3104742406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:24:10.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:10.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:10.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:10.984 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:11.019 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:11.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:11.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:11.121 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:11.155 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:11.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:11.256 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:11.293 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:11.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:11.394 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:11.428 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:11.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:11.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:11.530 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:11.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:11.565 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:11.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:11.666 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:11.705 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:11.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:11.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:11.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:11 smithi045 ceph-mon[39715]: pgmap v735: 168 pgs: 168 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 754 op/s 2024-03-11T22:24:11.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:11 smithi045 ceph-mon[30178]: pgmap v735: 168 pgs: 168 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 754 op/s 2024-03-11T22:24:11.807 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:11.830 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 70 16 108205 108189 96.5768 76.375 0.0141493 0.0103499 2024-03-11T22:24:11.842 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:11.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:11.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:11 smithi144 ceph-mon[33984]: pgmap v735: 168 pgs: 168 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 754 op/s 2024-03-11T22:24:11.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:11.978 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:11.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:11.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:12.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:12.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:12.111 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:12.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:12.212 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:12.249 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:12.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:12.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:12.351 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:12.406 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:12.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:12.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:12.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T22:24:12.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:12.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:12.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:12.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:12.700 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:12.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:12.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:12.784 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:24:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:24:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:24:12.802 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:12.830 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 71 16 109277 109261 96.1602 67 0.00814649 0.0103952 2024-03-11T22:24:12.832 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:12.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:12.934 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:12.970 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:12.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:12.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:13.072 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:13.103 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:13.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:13.205 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:13.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:13.251 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:13.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:13.352 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:13.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:13.386 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:13.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:13.487 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:13.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:13.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:13.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:13.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:13.648 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:13 smithi045 ceph-mon[30178]: pgmap v736: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T22:24:13.648 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:13 smithi045 ceph-mon[39715]: pgmap v736: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T22:24:13.661 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:13.682 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:13.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:13.732 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:13.784 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:13.817 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:13.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:13.830 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 72 16 110447 110431 95.8402 73.125 0.019406 0.0104292 2024-03-11T22:24:13.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:13 smithi144 ceph-mon[33984]: pgmap v736: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-11T22:24:13.919 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:13.955 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:13.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:14.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:14.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:14.101 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:14.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:14.202 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:14.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:14.235 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:14.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:14.336 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:14.368 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:14.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:14.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:14.470 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:14.506 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:14.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:14.521 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T22:24:14.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:14 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/634667396' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:24:14.539 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:14 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:24:14.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:14 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/634667396' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:24:14.540 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:14 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:24:14.608 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:14.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:14.657 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:14.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:14.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:14.797 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:14.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:14.830 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 73 16 111612 111596 95.5247 72.8125 0.00758638 0.010464 2024-03-11T22:24:14.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:14 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/634667396' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:24:14.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:14 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:24:14.899 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:14.937 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:14.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:14.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:15.038 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:15.074 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:15.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:15.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:15.085 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:24:15.176 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:15.211 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:15.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:15.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:15.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:15.359 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:15.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:15.460 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:15.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:15.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:15.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:15.602 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:15.638 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:15.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:15.740 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:15.772 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:15.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:15.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:15 smithi045 ceph-mon[39715]: pgmap v737: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T22:24:15.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:15 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:24:15.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:15 smithi045 ceph-mon[39715]: osdmap e371: 8 total, 8 up, 8 in 2024-03-11T22:24:15.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:15 smithi045 ceph-mon[30178]: pgmap v737: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T22:24:15.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:15 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:24:15.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:15 smithi045 ceph-mon[30178]: osdmap e371: 8 total, 8 up, 8 in 2024-03-11T22:24:15.830 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 74 16 112830 112814 95.2625 76.125 0.00939848 0.0104929 2024-03-11T22:24:15.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:15 smithi144 ceph-mon[33984]: pgmap v737: 168 pgs: 168 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T22:24:15.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:15 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:24:15.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:15 smithi144 ceph-mon[33984]: osdmap e371: 8 total, 8 up, 8 in 2024-03-11T22:24:15.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:15.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:15.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:15.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:16.014 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:16.051 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:16.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:16.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:16.153 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:16.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:16.183 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:16.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:16.284 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:16.318 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:16.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:16.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:16.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:16.450 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:16.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:16.552 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:16.575 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:16.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:16.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:16.677 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:16.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:16.711 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:16.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:16.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T22:24:16.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:16.830 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 75 16 114015 113999 94.9798 74.0625 0.00948685 0.0105237 2024-03-11T22:24:16.848 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:24:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:24:16.873 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:16.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:16.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:16.975 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:17.016 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:17.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:17.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:17.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:24:17.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:16 smithi045 ceph-mon[39715]: pgmap v739: 168 pgs: 168 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T22:24:17.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:16 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:17.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:17.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:24:17.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:16 smithi045 ceph-mon[30178]: pgmap v739: 168 pgs: 168 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T22:24:17.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:16 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:17.119 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:17.164 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:17.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:17.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:17.255 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:24:17.255 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:24:17.255 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:16 smithi144 ceph-mon[33984]: pgmap v739: 168 pgs: 168 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T22:24:17.255 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:16 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:17.266 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:17.302 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:17.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:17.404 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:17.443 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:17.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:17.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:17.545 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:17.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:17.586 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:17.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:17.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:17.728 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:17.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:17.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:17.800 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:17.830 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:17.831 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 76 16 115191 115175 94.6971 73.5 0.0120004 0.0105552 2024-03-11T22:24:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:17.865 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:17.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:17.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:18.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:18.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:18.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:18.157 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:18.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:18.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:18.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:18.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:18.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:18.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:18.438 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:18.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:18.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:18.581 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:18.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:18.682 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:18.712 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:18.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:18.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:18.786 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:24:18.786 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:24:18.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.786 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:116293 10.2 10:428ccbed:::benchmark_data_smithi045_169175_object116292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-11T22:24:18.786 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:18.765936+0000", 2024-03-11T22:24:18.786 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.016028906999999998, 2024-03-11T22:24:18.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.016069993000000001, 2024-03-11T22:24:18.786 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:24:18.787 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:24:18.787 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:24:18.787 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:24:18.787 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:24:18.787 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116293 2024-03-11T22:24:18.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.787 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:24:18.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.787 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:24:18.787 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.765936+0000", 2024-03-11T22:24:18.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:18.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.765936+0000", 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.765937+0000", 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8339999999999999e-06 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.765963+0000", 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5948999999999998e-05 2024-03-11T22:24:18.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.765983+0000", 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9802000000000001e-05 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.765996+0000", 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2967e-05 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.769256+0000", 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032598739999999998 2024-03-11T22:24:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:24:18.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.769428+0000", 2024-03-11T22:24:18.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000171786 2024-03-11T22:24:18.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-03-11T22:24:18.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.769612+0000", 2024-03-11T22:24:18.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000184067 2024-03-11T22:24:18.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:24:18.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.772567+0000", 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029551439999999998 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.776368+0000", 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038011759999999999 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:116297 10.0 e371/348 10:0138b7b5:::benchmark_data_smithi045_169175_object116296:head v 371'7207, mlcod=371'7205)", 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:18.773357+0000", 2024-03-11T22:24:18.791 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0086075920000000007, 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0087439849999999993, 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.773357+0000", 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.773357+0000", 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:18.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.773357+0000", 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3600000000000003e-07 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.773385+0000", 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7029000000000002e-05 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.773395+0000", 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.065e-05 2024-03-11T22:24:18.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:24:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.773402+0000", 2024-03-11T22:24:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8190000000000003e-06 2024-03-11T22:24:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:24:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.773680+0000", 2024-03-11T22:24:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00027761099999999998 2024-03-11T22:24:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:24:18.794 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.773738+0000", 2024-03-11T22:24:18.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8353999999999997e-05 2024-03-11T22:24:18.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:18.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:18.795 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:24:18.795 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:18.782047+0000", 2024-03-11T22:24:18.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0083089389999999996 2024-03-11T22:24:18.795 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:18.795 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:24:18.795 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:18.795 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:18.795 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:24:18.795 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 2 2024-03-11T22:24:18.795 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:24:18.814 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:18.831 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 77 16 116345 116329 94.4039 72.125 0.0298465 0.0105876 2024-03-11T22:24:18.843 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:18.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:18.945 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:18.983 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:18.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:19.084 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:19.118 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:19.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:19.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:19.264 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:19.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:19.366 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:19.402 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:19.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:19.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:19.467 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:24:19.504 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:19.538 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:19.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:19.639 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:19.679 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:19.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:19.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:19.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:19 smithi045 ceph-mon[30178]: pgmap v740: 168 pgs: 168 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T22:24:19.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:19 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/914895835' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:24:19.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:19 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:24:19.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:19 smithi045 ceph-mon[39715]: pgmap v740: 168 pgs: 168 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T22:24:19.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:19 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/914895835' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:24:19.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:19 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:24:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:19.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:19.831 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 78 16 117522 117506 94.1367 73.5625 0.013572 0.0106184 2024-03-11T22:24:19.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:19 smithi144 ceph-mon[33984]: pgmap v740: 168 pgs: 168 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T22:24:19.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:19 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/914895835' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:24:19.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:19 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:24:19.927 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:19.964 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:19.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:20.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:20.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:20.111 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:20.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:20.212 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:20.236 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T22:24:20.247 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:20.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:20.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:20.348 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:20.403 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:20.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:20.504 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:20.537 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:20.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:20.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:20.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:20.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:20.687 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:20.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:20.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:20 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:24:20.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:20 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:24:20.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:20 smithi045 ceph-mon[30178]: osdmap e372: 8 total, 8 up, 8 in 2024-03-11T22:24:20.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:20 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:24:20.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:20 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:24:20.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:20 smithi045 ceph-mon[39715]: osdmap e372: 8 total, 8 up, 8 in 2024-03-11T22:24:20.788 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:20.821 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:20.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:20.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:20.831 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 79 16 118711 118695 93.8857 74.3125 0.0105181 0.0106466 2024-03-11T22:24:20.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:20 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:24:20.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:20 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:24:20.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:20 smithi144 ceph-mon[33984]: osdmap e372: 8 total, 8 up, 8 in 2024-03-11T22:24:20.924 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:20.963 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:20.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:20.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:21.065 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:21.105 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:21.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:21.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:21.244 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:21.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:21.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:21.346 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:21.379 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:21.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:21.481 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:21.521 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:21.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:21.534 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:21 smithi045 ceph-mon[39715]: pgmap v742: 168 pgs: 168 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 723 op/s 2024-03-11T22:24:21.534 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:21 smithi045 ceph-mon[30178]: pgmap v742: 168 pgs: 168 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 723 op/s 2024-03-11T22:24:21.623 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:21.660 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:21.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:21.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:21.762 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:21.797 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:21.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:21.832 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:24:21.829845+0000 min lat: 0.00118312 max lat: 0.142383 avg lat: 0.0106684 2024-03-11T22:24:21.832 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:24:21.832 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 80 16 119967 119951 93.692 78.5 0.0176869 0.0106684 2024-03-11T22:24:21.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:21 smithi144 ceph-mon[33984]: pgmap v742: 168 pgs: 168 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 45 MiB/s wr, 723 op/s 2024-03-11T22:24:21.895 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:21.898 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:21.917 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:21.930 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:21.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:21.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:22.032 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:22.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:22.063 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:22.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:22.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:22.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:22.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:22.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:22.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:22.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:22.347 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:22.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:22.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:22.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:22.490 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:22.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:22.591 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:22.623 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:22.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:22.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:22.725 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:22.766 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:22.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:22.784 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:24:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:24:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:24:22.832 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 81 16 121135 121119 93.4364 73 0.0108925 0.0106979 2024-03-11T22:24:22.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:22.910 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:22.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:22.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:23.012 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:23.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:23.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:23.155 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:23.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:23.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:23.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:23.297 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:23.331 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:23.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:23.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:23.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:23.468 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:23.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:23.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:23.613 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:23.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:23.715 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:23.746 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:23.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:23.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:23.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:23 smithi045 ceph-mon[39715]: pgmap v743: 168 pgs: 168 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-11T22:24:23.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:23 smithi045 ceph-mon[30178]: pgmap v743: 168 pgs: 168 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-11T22:24:23.832 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 82 16 122407 122391 93.2664 79.5 0.0178945 0.0107176 2024-03-11T22:24:23.848 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:23.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:23 smithi144 ceph-mon[33984]: pgmap v743: 168 pgs: 168 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-11T22:24:23.873 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:24:23.880 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:23.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:23.982 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:24.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:24.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:24.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:24.207 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:24.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:24.309 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:24.344 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:24.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:24.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:24.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:24.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:24.551 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T22:24:24.589 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:24.625 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:24.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:24.727 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:24.762 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:24.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:24.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:24.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:24 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:24:24.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:24 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:24:24.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:24 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/477815608' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:24:24.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:24 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:24:24.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:24 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:24:24.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:24 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:24:24.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:24 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/477815608' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:24:24.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:24 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:24:24.832 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 83 16 123504 123488 92.9687 68.5625 0.0109122 0.0107516 2024-03-11T22:24:24.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:24 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:24:24.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:24 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:24:24.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:24 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/477815608' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:24:24.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:24 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:24:24.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:24.909 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:24.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:25.010 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:25.047 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:25.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:25.148 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:25.182 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:25.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:25.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:25.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:25.322 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:25.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:25.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:25.424 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:25.458 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:25.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:25.560 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:25.591 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:25.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:25.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:25.693 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:25.724 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:25.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:25.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:25 smithi045 ceph-mon[39715]: pgmap v744: 168 pgs: 168 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T22:24:25.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:25 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:24:25.755 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:25 smithi045 ceph-mon[39715]: osdmap e373: 8 total, 8 up, 8 in 2024-03-11T22:24:25.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:25 smithi045 ceph-mon[30178]: pgmap v744: 168 pgs: 168 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T22:24:25.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:25 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:24:25.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:25 smithi045 ceph-mon[30178]: osdmap e373: 8 total, 8 up, 8 in 2024-03-11T22:24:25.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:25.832 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 84 16 124732 124716 92.7756 76.75 0.0211591 0.0107743 2024-03-11T22:24:25.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:25 smithi144 ceph-mon[33984]: pgmap v744: 168 pgs: 168 active+clean; 7.3 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T22:24:25.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:25 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:24:25.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:25 smithi144 ceph-mon[33984]: osdmap e373: 8 total, 8 up, 8 in 2024-03-11T22:24:25.865 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:25.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:25.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:25.967 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:26.007 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:26.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:26.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:26.146 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:26.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:26.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:26.248 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:26.281 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:26.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:26.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:26.383 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:26.414 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:26.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:26.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:26.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:26.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:26.656 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:26.691 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:26.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:26.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:26.793 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:26.831 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:26.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:26.833 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 85 16 126015 125999 92.6274 80.1875 0.0171824 0.0107913 2024-03-11T22:24:26.848 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:24:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:24:26.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T22:24:26.933 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:26.968 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:26.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:27.071 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:27.117 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:27.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:27.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:27.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:27.262 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:27.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:27.364 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:27.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:27.400 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:27.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:27.501 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:27.533 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:27.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:27.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:27.651 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:24:27.677 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:27 smithi045 ceph-mon[30178]: pgmap v746: 168 pgs: 168 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T22:24:27.677 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:27.678 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:27 smithi045 ceph-mon[39715]: pgmap v746: 168 pgs: 168 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T22:24:27.678 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:27.690 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:27.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:27.697 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:24:27.737 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:24:27.776 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:24:27.791 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:27.826 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:24:27.827 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:27.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:27.833 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 86 15 127140 127125 92.3682 70.375 0.0178762 0.0108221 2024-03-11T22:24:27.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:27 smithi144 ceph-mon[33984]: pgmap v746: 168 pgs: 168 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T22:24:27.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:27.871 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:24:27.913 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:24:27.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:27.957 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:24:27.965 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:24:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:24:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:24:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:24:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:24:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:24:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:24:27.966 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:24:27.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:27.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:28.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:28.116 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:28.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:28.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:28.218 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:28.249 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:28.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:28.351 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:28.390 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:28.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:28.488 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:24:28.488 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:24:28.491 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:28.528 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:28.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:28.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:28.630 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:28.668 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:28.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:28.715 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:28.770 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:28.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:28 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1173902613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:24:28.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:28 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1173902613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:24:28.803 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:28.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:28.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:28.833 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 87 16 128338 128322 92.1661 74.8125 0.00798271 0.0108454 2024-03-11T22:24:28.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:28 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1173902613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:24:28.904 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:28.939 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:28.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:28.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:29.041 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:29.073 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:29.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:29.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:29.215 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:29.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:29.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:29.361 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:29.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:29.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:29.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:29.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:29.600 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:29.641 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:29.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:29.649 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:29 smithi045 ceph-mon[39715]: pgmap v747: 168 pgs: 168 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T22:24:29.649 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:29 smithi045 ceph-mon[30178]: pgmap v747: 168 pgs: 168 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T22:24:29.743 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:29.789 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:29.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:29.833 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 88 16 129601 129585 92.0157 78.9375 0.00632864 0.0108636 2024-03-11T22:24:29.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:29 smithi144 ceph-mon[33984]: pgmap v747: 168 pgs: 168 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T22:24:29.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:29.933 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:29.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:30.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:30.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:30.071 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:30.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:30.172 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:30.208 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:30.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:30.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:30.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:30.346 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:30.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:30.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:30.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:30.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:30.597 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:30.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:30.628 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:30.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:30.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:30.767 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:30.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:30.834 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 89 16 130844 130828 91.8547 77.6875 0.0126181 0.0108824 2024-03-11T22:24:30.869 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:30.904 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:30.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:30.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:31.006 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:31.041 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:31.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:31.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:31.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:31.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:31.182 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:31.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:31.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:31.322 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:31.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:31.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:31.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:24:31.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:31.494 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:31.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:31.596 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:31.627 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:31.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:31.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:31.776 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:31.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:31.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:31 smithi045 ceph-mon[39715]: pgmap v748: 168 pgs: 168 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T22:24:31.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:24:31.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:31 smithi045 ceph-mon[30178]: pgmap v748: 168 pgs: 168 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T22:24:31.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:24:31.834 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Total time run: 90.0069 2024-03-11T22:24:31.834 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Total writes made: 132024 2024-03-11T22:24:31.834 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Write size: 65536 2024-03-11T22:24:31.834 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Object size: 65536 2024-03-11T22:24:31.834 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Bandwidth (MB/sec): 91.6763 2024-03-11T22:24:31.834 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev Bandwidth: 30.1457 2024-03-11T22:24:31.834 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max bandwidth (MB/sec): 192.688 2024-03-11T22:24:31.834 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min bandwidth (MB/sec): 60.375 2024-03-11T22:24:31.834 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Average IOPS: 1466 2024-03-11T22:24:31.835 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev IOPS: 482.331 2024-03-11T22:24:31.835 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max IOPS: 3083 2024-03-11T22:24:31.835 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min IOPS: 966 2024-03-11T22:24:31.835 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Average Latency(s): 0.0109039 2024-03-11T22:24:31.835 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev Latency(s): 0.00773588 2024-03-11T22:24:31.835 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max latency(s): 0.142383 2024-03-11T22:24:31.835 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min latency(s): 0.00118312 2024-03-11T22:24:31.838 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T22:24:31.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:31 smithi144 ceph-mon[33984]: pgmap v748: 168 pgs: 168 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T22:24:31.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:24:31.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:31.916 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:31.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:32.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:32.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:32.052 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:32.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:32.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:32.191 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:32.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:32.292 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:32.327 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:32.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:32.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:32.429 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:32.462 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:32.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:32.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T22:24:32.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:32.651 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:32.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:32.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:32.753 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:32.784 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:24:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:24:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:24:32.786 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:32.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:32.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:32.887 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:32.921 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:32.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:33.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:33.066 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:33.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:33.168 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:33.202 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:33.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:33.304 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:33.344 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:33.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:33.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:33.446 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:33.480 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:33.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:33.581 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:33.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:33.614 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:33.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:33.715 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:33.752 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:33.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:33.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:33.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:33 smithi045 ceph-mon[39715]: pgmap v749: 168 pgs: 168 active+clean; 8.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-11T22:24:33.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:33 smithi045 ceph-mon[30178]: pgmap v749: 168 pgs: 168 active+clean; 8.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-11T22:24:33.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:33 smithi144 ceph-mon[33984]: pgmap v749: 168 pgs: 168 active+clean; 8.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-11T22:24:33.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:33.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:33.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:33.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:34.000 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:34.033 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:34.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:34.135 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:34.168 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:34.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:34.270 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:34.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:34.309 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:34.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:34.407 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:34.410 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:34.418 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:34.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:34.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:34.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:34.529 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:34 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1439876352' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:24:34.555 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T22:24:34.572 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:34.603 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:34.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:34.705 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:34.737 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:34.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:34.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:34 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1439876352' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:24:34.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:34.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:34 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1439876352' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:24:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:34.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:34.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:34.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:35.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:35.131 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:35.164 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:35.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:35.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:35.301 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:35.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:35.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:35.403 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:35.444 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:35.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:35.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:35.546 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:35.568 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:24:35.590 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:35.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:35.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:35.729 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:35.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:35.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:24:35.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:24:35.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[39715]: pgmap v750: 168 pgs: 168 active+clean; 8.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-11T22:24:35.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1439876352' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:24:35.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[39715]: osdmap e374: 8 total, 8 up, 8 in 2024-03-11T22:24:35.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3561438936' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:24:35.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:24:35.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:24:35.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:24:35.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[30178]: pgmap v750: 168 pgs: 168 active+clean; 8.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-11T22:24:35.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1439876352' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:24:35.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[30178]: osdmap e374: 8 total, 8 up, 8 in 2024-03-11T22:24:35.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3561438936' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:24:35.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:35 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:24:35.831 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:35.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:35 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:24:35.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:35 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:24:35.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:35 smithi144 ceph-mon[33984]: pgmap v750: 168 pgs: 168 active+clean; 8.0 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-11T22:24:35.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:35 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1439876352' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:24:35.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:35 smithi144 ceph-mon[33984]: osdmap e374: 8 total, 8 up, 8 in 2024-03-11T22:24:35.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:35 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3561438936' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:24:35.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:35 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:24:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:35.867 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:35.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:35.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:36.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:36.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:36.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:36.142 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:36.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:36.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:36.244 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:36.281 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:36.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:36.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:36.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:36.429 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:36.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:36.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:36.531 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:36.571 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:36.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:36.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:36.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T22:24:36.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:36.741 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:36.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:36.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:36 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:24:36.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:36 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:24:36.827 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:36 smithi045 ceph-mon[30178]: osdmap e375: 8 total, 8 up, 8 in 2024-03-11T22:24:36.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:36 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:24:36.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:36 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:24:36.827 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:36 smithi045 ceph-mon[39715]: osdmap e375: 8 total, 8 up, 8 in 2024-03-11T22:24:36.844 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:36.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:36 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:24:36.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:36 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:24:36.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:36 smithi144 ceph-mon[33984]: osdmap e375: 8 total, 8 up, 8 in 2024-03-11T22:24:36.848 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:24:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:24:36.878 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:36.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:36.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:36.980 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:37.017 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:37.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:37.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:37.163 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:37.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:37.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:37.302 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:37.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:37.405 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:37.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:37.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:37.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:37.549 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:37.581 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:37.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:37.683 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:37.722 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:37.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:37.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:37.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:37 smithi045 ceph-mon[30178]: pgmap v753: 168 pgs: 168 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 767 B/s rd, 54 MiB/s wr, 1.29k op/s 2024-03-11T22:24:37.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:37.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:37 smithi045 ceph-mon[39715]: pgmap v753: 168 pgs: 168 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 767 B/s rd, 54 MiB/s wr, 1.29k op/s 2024-03-11T22:24:37.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:37.824 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:37.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:37 smithi144 ceph-mon[33984]: pgmap v753: 168 pgs: 168 active+clean; 7.9 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 767 B/s rd, 54 MiB/s wr, 1.29k op/s 2024-03-11T22:24:37.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:37.849 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:37.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:37.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:37.991 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:37.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:38.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:38.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:38.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:38.234 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:38.267 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:38.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:38.369 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:38.406 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:38.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:38.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:38.508 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:38.542 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:38.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:38.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:38.644 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:38.680 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:38.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:38.782 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:38 smithi045 ceph-mon[39715]: 5.1d scrub starts 2024-03-11T22:24:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:38 smithi045 ceph-mon[39715]: 5.1d scrub ok 2024-03-11T22:24:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:38 smithi045 ceph-mon[39715]: 3.b scrub starts 2024-03-11T22:24:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:38 smithi045 ceph-mon[39715]: 3.b scrub ok 2024-03-11T22:24:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:38 smithi045 ceph-mon[39715]: 5.11 scrub starts 2024-03-11T22:24:38.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:38 smithi045 ceph-mon[39715]: 5.11 scrub ok 2024-03-11T22:24:38.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:38 smithi045 ceph-mon[30178]: 5.1d scrub starts 2024-03-11T22:24:38.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:38 smithi045 ceph-mon[30178]: 5.1d scrub ok 2024-03-11T22:24:38.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:38 smithi045 ceph-mon[30178]: 3.b scrub starts 2024-03-11T22:24:38.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:38 smithi045 ceph-mon[30178]: 3.b scrub ok 2024-03-11T22:24:38.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:38 smithi045 ceph-mon[30178]: 5.11 scrub starts 2024-03-11T22:24:38.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:38 smithi045 ceph-mon[30178]: 5.11 scrub ok 2024-03-11T22:24:38.818 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:38.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:38.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:38.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:38 smithi144 ceph-mon[33984]: 5.1d scrub starts 2024-03-11T22:24:38.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:38 smithi144 ceph-mon[33984]: 5.1d scrub ok 2024-03-11T22:24:38.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:38 smithi144 ceph-mon[33984]: 3.b scrub starts 2024-03-11T22:24:38.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:38 smithi144 ceph-mon[33984]: 3.b scrub ok 2024-03-11T22:24:38.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:38 smithi144 ceph-mon[33984]: 5.11 scrub starts 2024-03-11T22:24:38.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:38 smithi144 ceph-mon[33984]: 5.11 scrub ok 2024-03-11T22:24:38.914 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:38.920 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:38.950 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:38.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:38.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:39.052 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:39.084 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:39.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:39.186 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:39.219 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:39.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:39.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:39.321 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:39.357 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:39.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:39.458 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:39.505 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:39.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:39.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:24:39.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:39.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:39.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:39.761 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:39.762 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:39 smithi045 ceph-mon[30178]: pgmap v754: 168 pgs: 168 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 55 MiB/s wr, 1.55k op/s 2024-03-11T22:24:39.762 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:39 smithi045 ceph-mon[30178]: 3.18 scrub starts 2024-03-11T22:24:39.762 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:39 smithi045 ceph-mon[30178]: 3.18 scrub ok 2024-03-11T22:24:39.762 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:39 smithi045 ceph-mon[30178]: 6.5 scrub starts 2024-03-11T22:24:39.762 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:39 smithi045 ceph-mon[30178]: 6.5 scrub ok 2024-03-11T22:24:39.762 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:39 smithi045 ceph-mon[39715]: pgmap v754: 168 pgs: 168 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 55 MiB/s wr, 1.55k op/s 2024-03-11T22:24:39.763 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:39 smithi045 ceph-mon[39715]: 3.18 scrub starts 2024-03-11T22:24:39.763 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:39 smithi045 ceph-mon[39715]: 3.18 scrub ok 2024-03-11T22:24:39.763 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:39 smithi045 ceph-mon[39715]: 6.5 scrub starts 2024-03-11T22:24:39.763 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:39 smithi045 ceph-mon[39715]: 6.5 scrub ok 2024-03-11T22:24:39.793 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:39.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:39.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:39.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:39 smithi144 ceph-mon[33984]: pgmap v754: 168 pgs: 168 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 55 MiB/s wr, 1.55k op/s 2024-03-11T22:24:39.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:39 smithi144 ceph-mon[33984]: 3.18 scrub starts 2024-03-11T22:24:39.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:39 smithi144 ceph-mon[33984]: 3.18 scrub ok 2024-03-11T22:24:39.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:39 smithi144 ceph-mon[33984]: 6.5 scrub starts 2024-03-11T22:24:39.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:39 smithi144 ceph-mon[33984]: 6.5 scrub ok 2024-03-11T22:24:39.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:39.945 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:39.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:40.047 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:40.079 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:40.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:40.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:40.181 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:40.217 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:40.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:40.318 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:40.351 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:40.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:40.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:40.357 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:24:40.357 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:24:40.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:40.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:40.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:40.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:40.597 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:40.636 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:40.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:40.738 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:40.772 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:40.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:40.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:40 smithi144 ceph-mon[33984]: 4.c scrub starts 2024-03-11T22:24:40.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:40 smithi144 ceph-mon[33984]: 4.c scrub ok 2024-03-11T22:24:40.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:40 smithi144 ceph-mon[33984]: 4.18 scrub starts 2024-03-11T22:24:40.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:40 smithi144 ceph-mon[33984]: 4.18 scrub ok 2024-03-11T22:24:40.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:40 smithi144 ceph-mon[33984]: 5.7 scrub starts 2024-03-11T22:24:40.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:40 smithi144 ceph-mon[33984]: 5.7 scrub ok 2024-03-11T22:24:40.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:40.908 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:40.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:41.010 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:41.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:40 smithi045 ceph-mon[39715]: 4.c scrub starts 2024-03-11T22:24:41.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:40 smithi045 ceph-mon[39715]: 4.c scrub ok 2024-03-11T22:24:41.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:40 smithi045 ceph-mon[39715]: 4.18 scrub starts 2024-03-11T22:24:41.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:40 smithi045 ceph-mon[39715]: 4.18 scrub ok 2024-03-11T22:24:41.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:40 smithi045 ceph-mon[39715]: 5.7 scrub starts 2024-03-11T22:24:41.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:40 smithi045 ceph-mon[39715]: 5.7 scrub ok 2024-03-11T22:24:41.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:40 smithi045 ceph-mon[30178]: 4.c scrub starts 2024-03-11T22:24:41.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:40 smithi045 ceph-mon[30178]: 4.c scrub ok 2024-03-11T22:24:41.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:40 smithi045 ceph-mon[30178]: 4.18 scrub starts 2024-03-11T22:24:41.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:40 smithi045 ceph-mon[30178]: 4.18 scrub ok 2024-03-11T22:24:41.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:40 smithi045 ceph-mon[30178]: 5.7 scrub starts 2024-03-11T22:24:41.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:40 smithi045 ceph-mon[30178]: 5.7 scrub ok 2024-03-11T22:24:41.047 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:41.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:41.149 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:41.184 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:41.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:41.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:41.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T22:24:41.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:41.346 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:41.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:41.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:41.511 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:41.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:41.612 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:41.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:41.649 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:41.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:41.746 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:41 smithi045 ceph-mon[39715]: pgmap v755: 168 pgs: 168 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 7.8 MiB/s wr, 782 op/s 2024-03-11T22:24:41.746 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:41 smithi045 ceph-mon[39715]: 3.1c scrub starts 2024-03-11T22:24:41.746 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:41 smithi045 ceph-mon[39715]: 3.1c scrub ok 2024-03-11T22:24:41.747 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:41 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1536951454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:24:41.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:41 smithi045 ceph-mon[30178]: pgmap v755: 168 pgs: 168 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 7.8 MiB/s wr, 782 op/s 2024-03-11T22:24:41.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:41 smithi045 ceph-mon[30178]: 3.1c scrub starts 2024-03-11T22:24:41.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:41 smithi045 ceph-mon[30178]: 3.1c scrub ok 2024-03-11T22:24:41.747 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:41 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1536951454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:24:41.751 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:41.783 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:41.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:41.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:41.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:41 smithi144 ceph-mon[33984]: pgmap v755: 168 pgs: 168 active+clean; 7.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 767 B/s rd, 7.8 MiB/s wr, 782 op/s 2024-03-11T22:24:41.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:41 smithi144 ceph-mon[33984]: 3.1c scrub starts 2024-03-11T22:24:41.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:41 smithi144 ceph-mon[33984]: 3.1c scrub ok 2024-03-11T22:24:41.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:41 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1536951454' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:24:41.885 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:41.916 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:41.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:42.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:42.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:42.053 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:42.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:42.154 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:42.186 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:42.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:42.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:24:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:24:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:24:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:24:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:24:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:24:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:24:42.293 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:24:42.293 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:24:42.293 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:24:42.294 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:24:42.294 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:24:42.294 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:24:42.294 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:24:42.294 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:24:42.294 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:24:42.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:42.329 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:42.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:42.430 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:42.468 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:42.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:42.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:42.569 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:42.602 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:42.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:42.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:42.704 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:42.733 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:42.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:42.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:42.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:42 smithi144 ceph-mon[33984]: 4.7 scrub starts 2024-03-11T22:24:42.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:42 smithi144 ceph-mon[33984]: 4.7 scrub ok 2024-03-11T22:24:42.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:42 smithi144 ceph-mon[33984]: 3.15 scrub starts 2024-03-11T22:24:42.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:42 smithi144 ceph-mon[33984]: 3.15 scrub ok 2024-03-11T22:24:42.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:42 smithi144 ceph-mon[33984]: 3.16 scrub starts 2024-03-11T22:24:42.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:42 smithi144 ceph-mon[33984]: 3.16 scrub ok 2024-03-11T22:24:42.873 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:42.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:42.974 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:43.011 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:43.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:43.034 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:24:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:24:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:24:43.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:42 smithi045 ceph-mon[39715]: 4.7 scrub starts 2024-03-11T22:24:43.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:42 smithi045 ceph-mon[39715]: 4.7 scrub ok 2024-03-11T22:24:43.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:42 smithi045 ceph-mon[39715]: 3.15 scrub starts 2024-03-11T22:24:43.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:42 smithi045 ceph-mon[39715]: 3.15 scrub ok 2024-03-11T22:24:43.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:42 smithi045 ceph-mon[39715]: 3.16 scrub starts 2024-03-11T22:24:43.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:42 smithi045 ceph-mon[39715]: 3.16 scrub ok 2024-03-11T22:24:43.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:42 smithi045 ceph-mon[30178]: 4.7 scrub starts 2024-03-11T22:24:43.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:42 smithi045 ceph-mon[30178]: 4.7 scrub ok 2024-03-11T22:24:43.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:42 smithi045 ceph-mon[30178]: 3.15 scrub starts 2024-03-11T22:24:43.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:42 smithi045 ceph-mon[30178]: 3.15 scrub ok 2024-03-11T22:24:43.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:42 smithi045 ceph-mon[30178]: 3.16 scrub starts 2024-03-11T22:24:43.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:42 smithi045 ceph-mon[30178]: 3.16 scrub ok 2024-03-11T22:24:43.112 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:43.147 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:43.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:43.195 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:43.247 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:43.250 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:43.284 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:43.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:43.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:43.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:43.432 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:43.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:43.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:43.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:43.580 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:43.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:43.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:43.682 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:43.720 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:43.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:43.822 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:43.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:43 smithi144 ceph-mon[33984]: pgmap v756: 168 pgs: 168 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 7.8 MiB/s wr, 1.59k op/s 2024-03-11T22:24:43.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:43 smithi144 ceph-mon[33984]: 4.5 scrub starts 2024-03-11T22:24:43.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:43 smithi144 ceph-mon[33984]: 4.5 scrub ok 2024-03-11T22:24:43.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:43 smithi144 ceph-mon[33984]: 1.0 scrub starts 2024-03-11T22:24:43.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:43 smithi144 ceph-mon[33984]: 1.0 scrub ok 2024-03-11T22:24:43.860 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:43.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:43.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:43.961 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:43.990 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:43.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:44.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:43 smithi045 ceph-mon[30178]: pgmap v756: 168 pgs: 168 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 7.8 MiB/s wr, 1.59k op/s 2024-03-11T22:24:44.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:43 smithi045 ceph-mon[30178]: 4.5 scrub starts 2024-03-11T22:24:44.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:43 smithi045 ceph-mon[30178]: 4.5 scrub ok 2024-03-11T22:24:44.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:43 smithi045 ceph-mon[30178]: 1.0 scrub starts 2024-03-11T22:24:44.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:43 smithi045 ceph-mon[30178]: 1.0 scrub ok 2024-03-11T22:24:44.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:43 smithi045 ceph-mon[39715]: pgmap v756: 168 pgs: 168 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 7.8 MiB/s wr, 1.59k op/s 2024-03-11T22:24:44.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:43 smithi045 ceph-mon[39715]: 4.5 scrub starts 2024-03-11T22:24:44.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:43 smithi045 ceph-mon[39715]: 4.5 scrub ok 2024-03-11T22:24:44.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:43 smithi045 ceph-mon[39715]: 1.0 scrub starts 2024-03-11T22:24:44.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:43 smithi045 ceph-mon[39715]: 1.0 scrub ok 2024-03-11T22:24:44.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:44.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:44.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:44.233 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:44.269 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:44.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:44.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:44.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:44.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:44.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:44.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:44.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:44.643 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:44.680 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:44.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:44.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:44.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:44.815 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:44.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:44.916 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:44.951 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:44.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:45.051 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:45.076 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:45.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:45.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:24:45.178 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:45.212 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:45.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:45.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:45.314 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:45.350 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:45.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:45.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:45.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:45.489 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:45.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:45.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:45.592 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:45.634 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:45.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:45.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:45.773 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:45.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:45.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:45.875 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:45.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:45.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:46.001 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:46.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:45 smithi045 ceph-mon[39715]: pgmap v757: 168 pgs: 168 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.4 MiB/s wr, 1.29k op/s 2024-03-11T22:24:46.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:45 smithi045 ceph-mon[39715]: 4.1 scrub starts 2024-03-11T22:24:46.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:45 smithi045 ceph-mon[39715]: 4.1 scrub ok 2024-03-11T22:24:46.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:45 smithi045 ceph-mon[30178]: pgmap v757: 168 pgs: 168 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.4 MiB/s wr, 1.29k op/s 2024-03-11T22:24:46.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:45 smithi045 ceph-mon[30178]: 4.1 scrub starts 2024-03-11T22:24:46.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:45 smithi045 ceph-mon[30178]: 4.1 scrub ok 2024-03-11T22:24:46.046 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:46.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:46.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:46.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:45 smithi144 ceph-mon[33984]: pgmap v757: 168 pgs: 168 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.4 MiB/s wr, 1.29k op/s 2024-03-11T22:24:46.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:45 smithi144 ceph-mon[33984]: 4.1 scrub starts 2024-03-11T22:24:46.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:45 smithi144 ceph-mon[33984]: 4.1 scrub ok 2024-03-11T22:24:46.147 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:46.184 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:46.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:46.285 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:46.325 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:46.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:46.427 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:46.455 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:46.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:46.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:46.557 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:46.598 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:46.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:46.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:46.699 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:46.736 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:46.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:46.837 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:46.848 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:24:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:24:46.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:46 smithi144 ceph-mon[33984]: 1.3 scrub starts 2024-03-11T22:24:46.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:46 smithi144 ceph-mon[33984]: 1.3 scrub ok 2024-03-11T22:24:46.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:46 smithi144 ceph-mon[33984]: 5.4 scrub starts 2024-03-11T22:24:46.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:46 smithi144 ceph-mon[33984]: 5.4 scrub ok 2024-03-11T22:24:46.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:46 smithi144 ceph-mon[33984]: 6.16 scrub starts 2024-03-11T22:24:46.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:24:46.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:46 smithi144 ceph-mon[33984]: 6.16 scrub ok 2024-03-11T22:24:46.871 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:46.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:46.973 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:46.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[30178]: 1.3 scrub starts 2024-03-11T22:24:46.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[30178]: 1.3 scrub ok 2024-03-11T22:24:46.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[30178]: 5.4 scrub starts 2024-03-11T22:24:46.993 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[30178]: 5.4 scrub ok 2024-03-11T22:24:46.993 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[30178]: 6.16 scrub starts 2024-03-11T22:24:46.993 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:24:46.993 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[30178]: 6.16 scrub ok 2024-03-11T22:24:46.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[39715]: 1.3 scrub starts 2024-03-11T22:24:46.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[39715]: 1.3 scrub ok 2024-03-11T22:24:46.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[39715]: 5.4 scrub starts 2024-03-11T22:24:46.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[39715]: 5.4 scrub ok 2024-03-11T22:24:46.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[39715]: 6.16 scrub starts 2024-03-11T22:24:46.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:24:46.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:46 smithi045 ceph-mon[39715]: 6.16 scrub ok 2024-03-11T22:24:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:47.030 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:47.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:47.131 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:47.171 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:47.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:47.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:47.272 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:47.301 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:47.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:47.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:47.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:47.443 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:47.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:47.544 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:47.577 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:47.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:47.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:47.679 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:47.711 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:47.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:47.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[39715]: pgmap v758: 168 pgs: 168 active+clean; 7.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 944 B/s rd, 1.8 MiB/s wr, 1.33k op/s 2024-03-11T22:24:47.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:47.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[39715]: 6.12 scrub starts 2024-03-11T22:24:47.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[39715]: 4.a scrub starts 2024-03-11T22:24:47.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[39715]: 4.a scrub ok 2024-03-11T22:24:47.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[39715]: 6.12 scrub ok 2024-03-11T22:24:47.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[39715]: 3.1 scrub starts 2024-03-11T22:24:47.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[39715]: 3.1 scrub ok 2024-03-11T22:24:47.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[30178]: pgmap v758: 168 pgs: 168 active+clean; 7.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 944 B/s rd, 1.8 MiB/s wr, 1.33k op/s 2024-03-11T22:24:47.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:47.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[30178]: 6.12 scrub starts 2024-03-11T22:24:47.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[30178]: 4.a scrub starts 2024-03-11T22:24:47.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[30178]: 4.a scrub ok 2024-03-11T22:24:47.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[30178]: 6.12 scrub ok 2024-03-11T22:24:47.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[30178]: 3.1 scrub starts 2024-03-11T22:24:47.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:47 smithi045 ceph-mon[30178]: 3.1 scrub ok 2024-03-11T22:24:47.813 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:47.845 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:47.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:47.945 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:47.981 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:47.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:47.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:48.083 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:48.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:47 smithi144 ceph-mon[33984]: pgmap v758: 168 pgs: 168 active+clean; 7.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 944 B/s rd, 1.8 MiB/s wr, 1.33k op/s 2024-03-11T22:24:48.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:48.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:47 smithi144 ceph-mon[33984]: 6.12 scrub starts 2024-03-11T22:24:48.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:47 smithi144 ceph-mon[33984]: 4.a scrub starts 2024-03-11T22:24:48.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:47 smithi144 ceph-mon[33984]: 4.a scrub ok 2024-03-11T22:24:48.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:47 smithi144 ceph-mon[33984]: 6.12 scrub ok 2024-03-11T22:24:48.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:47 smithi144 ceph-mon[33984]: 3.1 scrub starts 2024-03-11T22:24:48.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:47 smithi144 ceph-mon[33984]: 3.1 scrub ok 2024-03-11T22:24:48.120 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:48.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:48.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:48.222 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:48.255 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:48.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:48.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:48.357 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:48.401 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:48.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:48.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:48.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:48.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:48.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:48.625 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T22:24:48.638 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:48.672 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:48.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:48.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:48.773 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:48.810 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:48.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:48.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:48.954 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:48.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:49.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:48 smithi045 ceph-mon[39715]: 4.2 scrub starts 2024-03-11T22:24:49.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:48 smithi045 ceph-mon[39715]: 4.2 scrub ok 2024-03-11T22:24:49.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:48 smithi045 ceph-mon[39715]: 3.2 scrub starts 2024-03-11T22:24:49.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:48 smithi045 ceph-mon[39715]: 3.2 scrub ok 2024-03-11T22:24:49.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:48 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/388993498' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:24:49.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:48 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:24:49.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:48 smithi045 ceph-mon[30178]: 4.2 scrub starts 2024-03-11T22:24:49.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:48 smithi045 ceph-mon[30178]: 4.2 scrub ok 2024-03-11T22:24:49.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:48 smithi045 ceph-mon[30178]: 3.2 scrub starts 2024-03-11T22:24:49.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:48 smithi045 ceph-mon[30178]: 3.2 scrub ok 2024-03-11T22:24:49.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:48 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/388993498' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:24:49.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:48 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:24:49.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:49.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:49.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:49.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:49.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:48 smithi144 ceph-mon[33984]: 4.2 scrub starts 2024-03-11T22:24:49.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:48 smithi144 ceph-mon[33984]: 4.2 scrub ok 2024-03-11T22:24:49.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:48 smithi144 ceph-mon[33984]: 3.2 scrub starts 2024-03-11T22:24:49.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:48 smithi144 ceph-mon[33984]: 3.2 scrub ok 2024-03-11T22:24:49.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:48 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/388993498' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:24:49.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:48 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:24:49.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:49.235 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:49.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:49.337 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:49.372 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:49.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:49.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:49.474 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:49.508 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:49.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:49.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:49.610 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:49.647 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:49.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:49.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:49.786 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:49.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:49.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:49.927 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:49.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:50.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[39715]: pgmap v759: 168 pgs: 168 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 KiB/s wr, 1.22k op/s 2024-03-11T22:24:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:24:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[39715]: osdmap e376: 8 total, 8 up, 8 in 2024-03-11T22:24:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[39715]: 5.6 scrub starts 2024-03-11T22:24:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[39715]: 5.6 scrub ok 2024-03-11T22:24:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[39715]: 1.a scrub starts 2024-03-11T22:24:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[39715]: 1.a scrub ok 2024-03-11T22:24:50.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[30178]: pgmap v759: 168 pgs: 168 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 KiB/s wr, 1.22k op/s 2024-03-11T22:24:50.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:24:50.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[30178]: osdmap e376: 8 total, 8 up, 8 in 2024-03-11T22:24:50.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[30178]: 5.6 scrub starts 2024-03-11T22:24:50.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[30178]: 5.6 scrub ok 2024-03-11T22:24:50.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[30178]: 1.a scrub starts 2024-03-11T22:24:50.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:49 smithi045 ceph-mon[30178]: 1.a scrub ok 2024-03-11T22:24:50.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:50.075 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:50.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:50.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:49 smithi144 ceph-mon[33984]: pgmap v759: 168 pgs: 168 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 KiB/s wr, 1.22k op/s 2024-03-11T22:24:50.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:49 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:24:50.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:49 smithi144 ceph-mon[33984]: osdmap e376: 8 total, 8 up, 8 in 2024-03-11T22:24:50.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:49 smithi144 ceph-mon[33984]: 5.6 scrub starts 2024-03-11T22:24:50.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:49 smithi144 ceph-mon[33984]: 5.6 scrub ok 2024-03-11T22:24:50.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:49 smithi144 ceph-mon[33984]: 1.a scrub starts 2024-03-11T22:24:50.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:49 smithi144 ceph-mon[33984]: 1.a scrub ok 2024-03-11T22:24:50.176 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:50.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T22:24:50.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:50.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:50.332 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:50.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:50.389 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:50.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:50.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:50.538 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:50.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:50.639 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:50.670 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:50.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:50.772 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:50.803 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:50.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:50.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:50.905 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:50.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:50.937 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:50.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:51.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:50 smithi045 ceph-mon[30178]: 5.13 scrub starts 2024-03-11T22:24:51.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:50 smithi045 ceph-mon[30178]: 5.13 scrub ok 2024-03-11T22:24:51.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:50 smithi045 ceph-mon[30178]: 3.13 scrub starts 2024-03-11T22:24:51.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:50 smithi045 ceph-mon[30178]: 3.13 scrub ok 2024-03-11T22:24:51.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:50 smithi045 ceph-mon[30178]: 5.1b scrub starts 2024-03-11T22:24:51.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:50 smithi045 ceph-mon[30178]: 5.1b scrub ok 2024-03-11T22:24:51.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:50 smithi045 ceph-mon[39715]: 5.13 scrub starts 2024-03-11T22:24:51.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:50 smithi045 ceph-mon[39715]: 5.13 scrub ok 2024-03-11T22:24:51.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:50 smithi045 ceph-mon[39715]: 3.13 scrub starts 2024-03-11T22:24:51.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:50 smithi045 ceph-mon[39715]: 3.13 scrub ok 2024-03-11T22:24:51.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:50 smithi045 ceph-mon[39715]: 5.1b scrub starts 2024-03-11T22:24:51.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:50 smithi045 ceph-mon[39715]: 5.1b scrub ok 2024-03-11T22:24:51.038 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:51.070 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:51.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:51.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:50 smithi144 ceph-mon[33984]: 5.13 scrub starts 2024-03-11T22:24:51.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:50 smithi144 ceph-mon[33984]: 5.13 scrub ok 2024-03-11T22:24:51.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:50 smithi144 ceph-mon[33984]: 3.13 scrub starts 2024-03-11T22:24:51.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:50 smithi144 ceph-mon[33984]: 3.13 scrub ok 2024-03-11T22:24:51.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:50 smithi144 ceph-mon[33984]: 5.1b scrub starts 2024-03-11T22:24:51.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:50 smithi144 ceph-mon[33984]: 5.1b scrub ok 2024-03-11T22:24:51.172 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:51.206 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:51.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:51.307 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:51.337 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:51.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:51.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:51.481 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:51.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:51.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:51.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:51.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:51.657 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:51.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:51.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:51.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:51.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:51 smithi045 ceph-mon[30178]: pgmap v761: 168 pgs: 168 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-03-11T22:24:51.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:51 smithi045 ceph-mon[30178]: 10.3 scrub starts 2024-03-11T22:24:51.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:51 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:24:51.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:51 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:24:51.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:51 smithi045 ceph-mon[39715]: pgmap v761: 168 pgs: 168 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-03-11T22:24:51.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:51 smithi045 ceph-mon[39715]: 10.3 scrub starts 2024-03-11T22:24:51.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:51 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:24:51.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:51 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:24:51.802 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:51.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:51.904 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:51.936 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:51.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:52.037 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:52.071 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:52.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:52.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:51 smithi144 ceph-mon[33984]: pgmap v761: 168 pgs: 168 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-03-11T22:24:52.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:51 smithi144 ceph-mon[33984]: 10.3 scrub starts 2024-03-11T22:24:52.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:51 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:24:52.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:51 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:24:52.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:52.216 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:52.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:52.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:52.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:52.353 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:52.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:52.455 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:52.492 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:52.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:52.594 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:52.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:52.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:52.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:52.660 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:24:52.691 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:24:52.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:52.760 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:24:52.760 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:24:52.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.760 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:158109 10.5 10:a45e67a1:::benchmark_data_smithi045_169175_object26082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-11T22:24:52.760 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:52.734626+0000", 2024-03-11T22:24:52.760 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.019625587999999999, 2024-03-11T22:24:52.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.019659197, 2024-03-11T22:24:52.760 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:24:52.760 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:24:52.760 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:24:52.760 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 158109 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.734626+0000", 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.734626+0000", 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.762 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:24:52.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.734627+0000", 2024-03-11T22:24:52.762 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.463e-06 2024-03-11T22:24:52.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.762 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:24:52.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.734633+0000", 2024-03-11T22:24:52.762 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.502e-06 2024-03-11T22:24:52.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.762 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:24:52.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.734644+0000", 2024-03-11T22:24:52.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0668e-05 2024-03-11T22:24:52.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:24:52.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.734653+0000", 2024-03-11T22:24:52.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3009999999999995e-06 2024-03-11T22:24:52.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:24:52.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.738490+0000", 2024-03-11T22:24:52.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003837399 2024-03-11T22:24:52.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.738614+0000", 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000123666 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.738738+0000", 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000123886 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.747916+0000", 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0091781350000000005 2024-03-11T22:24:52.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.748696+0000", 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00077970000000000003 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:158110 10.5 10:a9f960c4:::benchmark_data_smithi045_169175_object26083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:52.737895+0000", 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.016356823, 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.016487677999999999, 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:24:52.765 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 158110 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.737895+0000", 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:24:52.766 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.737895+0000", 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.737896+0000", 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7010000000000001e-06 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.737902+0000", 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9340000000000003e-06 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:24:52.767 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.737915+0000", 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2431e-05 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.737925+0000", 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.798e-06 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.744386+0000", 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0064617490000000001 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.744503+0000", 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116907 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.744669+0000", 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000166288 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.747992+0000", 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003322036 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:24:52.769 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:158118 10.3 10:c4f12bfc:::benchmark_data_smithi045_169175_object26091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-11T22:24:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:52.748666+0000", 2024-03-11T22:24:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.005585497, 2024-03-11T22:24:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058186999999999996, 2024-03-11T22:24:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:24:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:24:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:24:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:24:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:24:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 158118 2024-03-11T22:24:52.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.748666+0000", 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.748666+0000", 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.748667+0000", 2024-03-11T22:24:52.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.367e-06 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.748672+0000", 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7430000000000002e-06 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.748684+0000", 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1464000000000001e-05 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.748692+0000", 2024-03-11T22:24:52.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1829999999999994e-06 2024-03-11T22:24:52.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.773 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:24:52.773 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.748767+0000", 2024-03-11T22:24:52.773 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5060000000000003e-05 2024-03-11T22:24:52.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.773 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:24:52.773 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.748914+0000", 2024-03-11T22:24:52.773 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000147608 2024-03-11T22:24:52.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.773 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,5", 2024-03-11T22:24:52.774 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.749045+0000", 2024-03-11T22:24:52.774 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013078699999999999 2024-03-11T22:24:52.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.774 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:24:52.774 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.753064+0000", 2024-03-11T22:24:52.774 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040183780000000004 2024-03-11T22:24:52.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.774 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:24:52.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.774 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:158123 10.5 10:a9f43df6:::benchmark_data_smithi045_169175_object26096:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:52.752289+0000", 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0019622770000000001, 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0022778590000000001, 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 158123 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:24:52.775 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752289+0000", 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752289+0000", 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752291+0000", 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4929999999999999e-06 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:24:52.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752296+0000", 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5400000000000003e-06 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752312+0000", 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5631e-05 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752322+0000", 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.012e-05 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752531+0000", 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020860400000000001 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752661+0000", 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013077600000000001 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752784+0000", 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012277999999999999 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.778 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:158120 10.e e378/348 10:7e171a95:::benchmark_data_smithi045_169175_object26093:head v 378'10034, mlcod=377'10028)", 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:52.752322+0000", 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0019297330000000001, 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002319153, 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752322+0000", 2024-03-11T22:24:52.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752322+0000", 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752323+0000", 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.378e-06 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752330+0000", 2024-03-11T22:24:52.780 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1289999999999998e-06 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752343+0000", 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.243e-05 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752350+0000", 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1670000000000002e-06 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752538+0000", 2024-03-11T22:24:52.781 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018782 2024-03-11T22:24:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:24:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752553+0000", 2024-03-11T22:24:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4882000000000001e-05 2024-03-11T22:24:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:24:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.15540.0:158109 10.5 e378/348)", 2024-03-11T22:24:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:52.752389+0000", 2024-03-11T22:24:52.782 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.001862346, 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002308676, 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752389+0000", 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752389+0000", 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.783 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752390+0000", 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0500000000000004e-07 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752393+0000", 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.692e-06 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752403+0000", 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3579999999999992e-06 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752409+0000", 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.28e-06 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.15540.0:158110 10.5 e378/348)", 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:52.752435+0000", 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.001816425, 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023084630000000002, 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:24:52.785 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752435+0000", 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752435+0000", 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752436+0000", 2024-03-11T22:24:52.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4999999999999998e-07 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752439+0000", 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5750000000000001e-06 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752448+0000", 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6489999999999997e-06 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752455+0000", 2024-03-11T22:24:52.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8750000000000002e-06 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.15540.0:158110 10.5 e378/348)", 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:52.752544+0000", 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.00170754, 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0022476530000000001, 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752544+0000", 2024-03-11T22:24:52.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752544+0000", 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752544+0000", 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0100000000000005e-07 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752548+0000", 2024-03-11T22:24:52.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6009999999999999e-06 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752556+0000", 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9549999999999992e-06 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.752562+0000", 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3230000000000003e-06 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:158126 10.5 10.9ebd2de5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:52.754060+0000", 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.000191888, 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00078878100000000001, 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 158126 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.791 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.754060+0000", 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.754060+0000", 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.754061+0000", 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.308e-06 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.754066+0000", 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5110000000000003e-06 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.754084+0000", 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7180000000000002e-05 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:52.754093+0000", 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7569999999999999e-06 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.793 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:24:52.794 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.794 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:24:52.794 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:24:52.794 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 9 2024-03-11T22:24:52.794 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:24:52.794 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:24:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:24:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:24:52.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:52 smithi045 ceph-mon[39715]: osdmap e377: 8 total, 8 up, 8 in 2024-03-11T22:24:52.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:52 smithi045 ceph-mon[39715]: 5.17 scrub starts 2024-03-11T22:24:52.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:52 smithi045 ceph-mon[39715]: 5.17 scrub ok 2024-03-11T22:24:52.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:52 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3690557487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:24:52.795 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:52 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:24:52.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:52 smithi045 ceph-mon[30178]: osdmap e377: 8 total, 8 up, 8 in 2024-03-11T22:24:52.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:52 smithi045 ceph-mon[30178]: 5.17 scrub starts 2024-03-11T22:24:52.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:52 smithi045 ceph-mon[30178]: 5.17 scrub ok 2024-03-11T22:24:52.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:52 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3690557487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:24:52.795 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:52 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:24:52.797 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:52.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:52.899 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:52.931 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:52.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:53.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:53.078 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:53.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:53.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:53.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:52 smithi144 ceph-mon[33984]: osdmap e377: 8 total, 8 up, 8 in 2024-03-11T22:24:53.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:52 smithi144 ceph-mon[33984]: 5.17 scrub starts 2024-03-11T22:24:53.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:52 smithi144 ceph-mon[33984]: 5.17 scrub ok 2024-03-11T22:24:53.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:52 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3690557487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:24:53.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:52 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:24:53.180 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:53.210 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:53.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:53.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:53.311 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:53.347 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:53.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:53.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:53.489 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:53.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:53.590 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:53.632 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:53.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:53.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:53.735 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:53.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:53.772 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:53.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:53.873 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:53.909 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:53.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:53.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:54.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:54.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[30178]: pgmap v763: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-11T22:24:54.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:24:54.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:24:54.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:24:54.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[30178]: osdmap e378: 8 total, 8 up, 8 in 2024-03-11T22:24:54.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[30178]: 10.5 scrub starts 2024-03-11T22:24:54.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[30178]: 1.12 scrub starts 2024-03-11T22:24:54.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[30178]: 1.12 scrub ok 2024-03-11T22:24:54.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[39715]: pgmap v763: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-11T22:24:54.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:24:54.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:24:54.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:24:54.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[39715]: osdmap e378: 8 total, 8 up, 8 in 2024-03-11T22:24:54.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[39715]: 10.5 scrub starts 2024-03-11T22:24:54.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[39715]: 1.12 scrub starts 2024-03-11T22:24:54.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:53 smithi045 ceph-mon[39715]: 1.12 scrub ok 2024-03-11T22:24:54.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:54.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:54.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:53 smithi144 ceph-mon[33984]: pgmap v763: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-03-11T22:24:54.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:53 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:24:54.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:53 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:24:54.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:53 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:24:54.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:53 smithi144 ceph-mon[33984]: osdmap e378: 8 total, 8 up, 8 in 2024-03-11T22:24:54.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:53 smithi144 ceph-mon[33984]: 10.5 scrub starts 2024-03-11T22:24:54.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:53 smithi144 ceph-mon[33984]: 1.12 scrub starts 2024-03-11T22:24:54.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:53 smithi144 ceph-mon[33984]: 1.12 scrub ok 2024-03-11T22:24:54.153 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:54.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:54.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:54.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:54.297 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:54.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:54.333 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:54.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:54.434 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:54.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:54.467 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:54.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:54.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:54.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:54.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:54.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:54.757 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:54.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:54.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:54.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:54.901 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:54.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:54.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:55.003 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:55.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[39715]: 10.3 scrub ok 2024-03-11T22:24:55.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[39715]: 3.e scrub starts 2024-03-11T22:24:55.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[39715]: 3.e scrub ok 2024-03-11T22:24:55.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[39715]: 5.14 scrub starts 2024-03-11T22:24:55.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[39715]: 5.14 scrub ok 2024-03-11T22:24:55.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[39715]: 6.9 scrub starts 2024-03-11T22:24:55.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[39715]: 6.9 scrub ok 2024-03-11T22:24:55.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[30178]: 10.3 scrub ok 2024-03-11T22:24:55.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[30178]: 3.e scrub starts 2024-03-11T22:24:55.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[30178]: 3.e scrub ok 2024-03-11T22:24:55.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[30178]: 5.14 scrub starts 2024-03-11T22:24:55.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[30178]: 5.14 scrub ok 2024-03-11T22:24:55.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[30178]: 6.9 scrub starts 2024-03-11T22:24:55.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:54 smithi045 ceph-mon[30178]: 6.9 scrub ok 2024-03-11T22:24:55.038 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:55.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:55.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:54 smithi144 ceph-mon[33984]: 10.3 scrub ok 2024-03-11T22:24:55.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:54 smithi144 ceph-mon[33984]: 3.e scrub starts 2024-03-11T22:24:55.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:54 smithi144 ceph-mon[33984]: 3.e scrub ok 2024-03-11T22:24:55.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:54 smithi144 ceph-mon[33984]: 5.14 scrub starts 2024-03-11T22:24:55.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:54 smithi144 ceph-mon[33984]: 5.14 scrub ok 2024-03-11T22:24:55.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:54 smithi144 ceph-mon[33984]: 6.9 scrub starts 2024-03-11T22:24:55.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:54 smithi144 ceph-mon[33984]: 6.9 scrub ok 2024-03-11T22:24:55.140 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:55.169 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:55.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:55.271 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:55.310 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:55.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:55.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:55.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:55.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:55.449 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:55.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:55.550 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:55.584 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:55.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:55.685 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:55.725 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:55.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:55.827 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:55.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:55.918 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:55.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:56.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:56.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:55 smithi045 ceph-mon[30178]: pgmap v765: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T22:24:56.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:55 smithi045 ceph-mon[30178]: 5.3 deep-scrub starts 2024-03-11T22:24:56.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:55 smithi045 ceph-mon[30178]: 5.3 deep-scrub ok 2024-03-11T22:24:56.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:55 smithi045 ceph-mon[30178]: 6.7 scrub starts 2024-03-11T22:24:56.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:55 smithi045 ceph-mon[30178]: 6.7 scrub ok 2024-03-11T22:24:56.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:55 smithi045 ceph-mon[39715]: pgmap v765: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T22:24:56.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:55 smithi045 ceph-mon[39715]: 5.3 deep-scrub starts 2024-03-11T22:24:56.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:55 smithi045 ceph-mon[39715]: 5.3 deep-scrub ok 2024-03-11T22:24:56.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:55 smithi045 ceph-mon[39715]: 6.7 scrub starts 2024-03-11T22:24:56.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:55 smithi045 ceph-mon[39715]: 6.7 scrub ok 2024-03-11T22:24:56.074 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:56.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:56.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:56.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:55 smithi144 ceph-mon[33984]: pgmap v765: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-11T22:24:56.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:55 smithi144 ceph-mon[33984]: 5.3 deep-scrub starts 2024-03-11T22:24:56.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:55 smithi144 ceph-mon[33984]: 5.3 deep-scrub ok 2024-03-11T22:24:56.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:55 smithi144 ceph-mon[33984]: 6.7 scrub starts 2024-03-11T22:24:56.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:55 smithi144 ceph-mon[33984]: 6.7 scrub ok 2024-03-11T22:24:56.176 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:56.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:56.212 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:56.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:56.312 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:56.346 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:56.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:56.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:56.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:56.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:56.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:56.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:56.589 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:56.616 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:56.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:56.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:56.717 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:24:56.755 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:56.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:56.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:24:56.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:56.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:56.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:56.957 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:56 smithi045 ceph-mon[30178]: 10.5 scrub ok 2024-03-11T22:24:56.957 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:56 smithi045 ceph-mon[30178]: 3.1b scrub starts 2024-03-11T22:24:56.958 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:56 smithi045 ceph-mon[30178]: 3.1b scrub ok 2024-03-11T22:24:56.958 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:56 smithi045 ceph-mon[39715]: 10.5 scrub ok 2024-03-11T22:24:56.958 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:56 smithi045 ceph-mon[39715]: 3.1b scrub starts 2024-03-11T22:24:56.958 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:56 smithi045 ceph-mon[39715]: 3.1b scrub ok 2024-03-11T22:24:56.997 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:57.031 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:57.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:57.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:56 smithi144 ceph-mon[33984]: 10.5 scrub ok 2024-03-11T22:24:57.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:56 smithi144 ceph-mon[33984]: 3.1b scrub starts 2024-03-11T22:24:57.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:56 smithi144 ceph-mon[33984]: 3.1b scrub ok 2024-03-11T22:24:57.098 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:24:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:24:57.133 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:57.164 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:57.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:57.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:57.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:57.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:57.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:57.415 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:57.443 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:57.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:57.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:57.545 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:57.576 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:57.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:57.677 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:57.722 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:57.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:57.801 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[39715]: pgmap v766: 168 pgs: 168 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T22:24:57.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:57.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[39715]: 5.8 scrub starts 2024-03-11T22:24:57.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[39715]: 5.8 scrub ok 2024-03-11T22:24:57.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:24:57.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:24:57.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[39715]: 1.1 scrub starts 2024-03-11T22:24:57.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[39715]: 1.1 scrub ok 2024-03-11T22:24:57.802 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1275991763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:24:57.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[30178]: pgmap v766: 168 pgs: 168 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T22:24:57.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:57.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[30178]: 5.8 scrub starts 2024-03-11T22:24:57.802 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[30178]: 5.8 scrub ok 2024-03-11T22:24:57.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:24:57.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:24:57.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[30178]: 1.1 scrub starts 2024-03-11T22:24:57.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[30178]: 1.1 scrub ok 2024-03-11T22:24:57.803 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:57 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1275991763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:24:57.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:57.860 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:57.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:57.961 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:57.995 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:57.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:58.097 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:58.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:57 smithi144 ceph-mon[33984]: pgmap v766: 168 pgs: 168 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T22:24:58.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:24:58.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:57 smithi144 ceph-mon[33984]: 5.8 scrub starts 2024-03-11T22:24:58.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:57 smithi144 ceph-mon[33984]: 5.8 scrub ok 2024-03-11T22:24:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:57 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:24:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:57 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:24:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:57 smithi144 ceph-mon[33984]: 1.1 scrub starts 2024-03-11T22:24:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:57 smithi144 ceph-mon[33984]: 1.1 scrub ok 2024-03-11T22:24:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:57 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1275991763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:24:58.130 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:58.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:58.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:58.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:58.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:58.275 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:58.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:58.375 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:24:58.418 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:58.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:58.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:24:58.520 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:58.552 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:58.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:24:58.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:24:58.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:58.710 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:58.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:58.778 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:24:58.778 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:24:58.778 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:24:58.778 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:24:58.778 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:24:58.778 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:24:58.778 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:24:58.778 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:24:58.779 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:24:58.779 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:24:58.779 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:24:58.779 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:24:58.779 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:24:58.779 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:24:58.779 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:24:58.779 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:24:58.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:24:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:58.851 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:58.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:24:58.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:58.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:59.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:58 smithi045 ceph-mon[39715]: 3.9 scrub starts 2024-03-11T22:24:59.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:58 smithi045 ceph-mon[39715]: 3.9 scrub ok 2024-03-11T22:24:59.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:58 smithi045 ceph-mon[39715]: osdmap e379: 8 total, 8 up, 8 in 2024-03-11T22:24:59.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:58 smithi045 ceph-mon[39715]: 4.e scrub starts 2024-03-11T22:24:59.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:58 smithi045 ceph-mon[39715]: 4.e scrub ok 2024-03-11T22:24:59.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:58 smithi045 ceph-mon[30178]: 3.9 scrub starts 2024-03-11T22:24:59.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:58 smithi045 ceph-mon[30178]: 3.9 scrub ok 2024-03-11T22:24:59.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:58 smithi045 ceph-mon[30178]: osdmap e379: 8 total, 8 up, 8 in 2024-03-11T22:24:59.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:58 smithi045 ceph-mon[30178]: 4.e scrub starts 2024-03-11T22:24:59.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:58 smithi045 ceph-mon[30178]: 4.e scrub ok 2024-03-11T22:24:59.090 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:24:59.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:58 smithi144 ceph-mon[33984]: 3.9 scrub starts 2024-03-11T22:24:59.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:58 smithi144 ceph-mon[33984]: 3.9 scrub ok 2024-03-11T22:24:59.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:58 smithi144 ceph-mon[33984]: osdmap e379: 8 total, 8 up, 8 in 2024-03-11T22:24:59.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:58 smithi144 ceph-mon[33984]: 4.e scrub starts 2024-03-11T22:24:59.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:58 smithi144 ceph-mon[33984]: 4.e scrub ok 2024-03-11T22:24:59.117 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:59.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:59.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:24:59.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:59.256 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:59.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:59.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:24:59.389 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:59.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:24:59.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:59.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:59.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:59.629 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:24:59.658 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:59.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:59.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:24:59.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:24:59.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:59.801 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:24:59.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:24:59.902 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:24:59.936 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:24:59.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:24:59.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:00.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[30178]: pgmap v768: 168 pgs: 168 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 758 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T22:25:00.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[30178]: 4.d scrub starts 2024-03-11T22:25:00.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[30178]: 4.d scrub ok 2024-03-11T22:25:00.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:25:00.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:25:00.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[30178]: osdmap e380: 8 total, 8 up, 8 in 2024-03-11T22:25:00.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[30178]: 10.f scrub starts 2024-03-11T22:25:00.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[30178]: 3.c scrub starts 2024-03-11T22:25:00.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[30178]: 3.c scrub ok 2024-03-11T22:25:00.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[39715]: pgmap v768: 168 pgs: 168 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 758 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T22:25:00.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[39715]: 4.d scrub starts 2024-03-11T22:25:00.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[39715]: 4.d scrub ok 2024-03-11T22:25:00.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:25:00.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:25:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[39715]: osdmap e380: 8 total, 8 up, 8 in 2024-03-11T22:25:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[39715]: 10.f scrub starts 2024-03-11T22:25:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[39715]: 3.c scrub starts 2024-03-11T22:25:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:24:59 smithi045 ceph-mon[39715]: 3.c scrub ok 2024-03-11T22:25:00.037 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:00.073 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:00.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:00.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:59 smithi144 ceph-mon[33984]: pgmap v768: 168 pgs: 168 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 758 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T22:25:00.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:59 smithi144 ceph-mon[33984]: 4.d scrub starts 2024-03-11T22:25:00.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:59 smithi144 ceph-mon[33984]: 4.d scrub ok 2024-03-11T22:25:00.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:59 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:25:00.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:59 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:25:00.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:59 smithi144 ceph-mon[33984]: osdmap e380: 8 total, 8 up, 8 in 2024-03-11T22:25:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:59 smithi144 ceph-mon[33984]: 10.f scrub starts 2024-03-11T22:25:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:59 smithi144 ceph-mon[33984]: 3.c scrub starts 2024-03-11T22:25:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:24:59 smithi144 ceph-mon[33984]: 3.c scrub ok 2024-03-11T22:25:00.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:00.210 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:00.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:00.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:00.312 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:00.356 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:00.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:00.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T22:25:00.457 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:00.489 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:00.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:00.591 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:00.623 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:00.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:00.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:00.780 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:00.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:00.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:00 smithi045 ceph-mon[30178]: 6.13 scrub starts 2024-03-11T22:25:00.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:00 smithi045 ceph-mon[30178]: 10.e scrub starts 2024-03-11T22:25:00.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:00 smithi045 ceph-mon[30178]: 5.f scrub starts 2024-03-11T22:25:00.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:00 smithi045 ceph-mon[30178]: 5.f scrub ok 2024-03-11T22:25:00.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:00 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:25:00.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:00 smithi045 ceph-mon[39715]: 6.13 scrub starts 2024-03-11T22:25:00.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:00 smithi045 ceph-mon[39715]: 10.e scrub starts 2024-03-11T22:25:00.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:00 smithi045 ceph-mon[39715]: 5.f scrub starts 2024-03-11T22:25:00.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:00 smithi045 ceph-mon[39715]: 5.f scrub ok 2024-03-11T22:25:00.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:00 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:25:00.882 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:00.917 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:00.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:01.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:01.061 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:01.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:01.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:00 smithi144 ceph-mon[33984]: 6.13 scrub starts 2024-03-11T22:25:01.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:00 smithi144 ceph-mon[33984]: 10.e scrub starts 2024-03-11T22:25:01.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:00 smithi144 ceph-mon[33984]: 5.f scrub starts 2024-03-11T22:25:01.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:00 smithi144 ceph-mon[33984]: 5.f scrub ok 2024-03-11T22:25:01.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:00 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:25:01.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:01.255 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:01.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:01.356 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:01.406 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:01.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:01.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:01.477 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:01.481 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:01.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:01.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:01.600 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:01.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:01.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:01.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[30178]: 6.13 scrub ok 2024-03-11T22:25:01.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[30178]: pgmap v770: 168 pgs: 168 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 661 B/s rd, 0 B/s wr, 737 op/s 2024-03-11T22:25:01.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[30178]: 10.f scrub ok 2024-03-11T22:25:01.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[30178]: 10.9 scrub starts 2024-03-11T22:25:01.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:01.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:25:01.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:25:01.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:25:01.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:01.731 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:01.750 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:01.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:01.852 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:01.888 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:01.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:01.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:02.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:02.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:02.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[39715]: 6.13 scrub ok 2024-03-11T22:25:02.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[39715]: pgmap v770: 168 pgs: 168 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 661 B/s rd, 0 B/s wr, 737 op/s 2024-03-11T22:25:02.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[39715]: 10.f scrub ok 2024-03-11T22:25:02.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[39715]: 10.9 scrub starts 2024-03-11T22:25:02.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:02.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:25:02.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:25:02.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:25:02.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:02.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:02.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:01 smithi144 ceph-mon[33984]: 6.13 scrub ok 2024-03-11T22:25:02.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:01 smithi144 ceph-mon[33984]: pgmap v770: 168 pgs: 168 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 661 B/s rd, 0 B/s wr, 737 op/s 2024-03-11T22:25:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:01 smithi144 ceph-mon[33984]: 10.f scrub ok 2024-03-11T22:25:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:01 smithi144 ceph-mon[33984]: 10.9 scrub starts 2024-03-11T22:25:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:25:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:01 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:25:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:01 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:25:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:02.128 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:02.163 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:02.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:02.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:02.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:02.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:02.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:02.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:02.416 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:02.450 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:02.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:02.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:02.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:02.596 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:02.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:02.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:02.698 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:02.731 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:25:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:25:02.731 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:02.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:02.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:02.871 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:02.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:02.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:25:02.973 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:02.986 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[30178]: 10.e scrub ok 2024-03-11T22:25:02.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[30178]: 4.1d scrub starts 2024-03-11T22:25:02.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[30178]: 10.1 deep-scrub starts 2024-03-11T22:25:02.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[30178]: 4.1d scrub ok 2024-03-11T22:25:02.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[30178]: osdmap e381: 8 total, 8 up, 8 in 2024-03-11T22:25:02.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/817562195' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:25:02.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:25:02.987 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[39715]: 10.e scrub ok 2024-03-11T22:25:02.987 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[39715]: 4.1d scrub starts 2024-03-11T22:25:02.987 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[39715]: 10.1 deep-scrub starts 2024-03-11T22:25:02.987 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[39715]: 4.1d scrub ok 2024-03-11T22:25:02.988 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[39715]: osdmap e381: 8 total, 8 up, 8 in 2024-03-11T22:25:02.988 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/817562195' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:25:02.988 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:02 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:25:02.988 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T22:25:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:03.045 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:03.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:03.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:02 smithi144 ceph-mon[33984]: 10.e scrub ok 2024-03-11T22:25:03.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:02 smithi144 ceph-mon[33984]: 4.1d scrub starts 2024-03-11T22:25:03.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:02 smithi144 ceph-mon[33984]: 10.1 deep-scrub starts 2024-03-11T22:25:03.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:02 smithi144 ceph-mon[33984]: 4.1d scrub ok 2024-03-11T22:25:03.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:02 smithi144 ceph-mon[33984]: osdmap e381: 8 total, 8 up, 8 in 2024-03-11T22:25:03.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:02 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/817562195' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:25:03.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:02 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:25:03.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:03.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:03.186 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:03.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:03.287 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:03.320 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:03.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:03.421 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:03.453 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:03.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:03.555 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:03.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:03.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:03.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:03.730 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:03.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:03.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:03.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:03.871 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:03.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:03.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:03.972 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:04.008 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:04.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:04.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:04.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[39715]: pgmap v772: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 165 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-11T22:25:04.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[39715]: 10.9 scrub ok 2024-03-11T22:25:04.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:25:04.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:25:04.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:04.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:25:04.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[39715]: osdmap e382: 8 total, 8 up, 8 in 2024-03-11T22:25:04.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[39715]: 4.1a scrub starts 2024-03-11T22:25:04.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[39715]: 4.1a scrub ok 2024-03-11T22:25:04.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[30178]: pgmap v772: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 165 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-11T22:25:04.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[30178]: 10.9 scrub ok 2024-03-11T22:25:04.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:25:04.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:25:04.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:04.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:25:04.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[30178]: osdmap e382: 8 total, 8 up, 8 in 2024-03-11T22:25:04.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[30178]: 4.1a scrub starts 2024-03-11T22:25:04.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:03 smithi045 ceph-mon[30178]: 4.1a scrub ok 2024-03-11T22:25:04.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:03 smithi144 ceph-mon[33984]: pgmap v772: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 165 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-11T22:25:04.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:03 smithi144 ceph-mon[33984]: 10.9 scrub ok 2024-03-11T22:25:04.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:03 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:25:04.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:03 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:25:04.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:03 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:04.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:03 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:25:04.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:03 smithi144 ceph-mon[33984]: osdmap e382: 8 total, 8 up, 8 in 2024-03-11T22:25:04.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:03 smithi144 ceph-mon[33984]: 4.1a scrub starts 2024-03-11T22:25:04.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:03 smithi144 ceph-mon[33984]: 4.1a scrub ok 2024-03-11T22:25:04.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:04.148 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:04.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:04.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:04.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:04.288 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:04.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:04.390 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:04.425 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:04.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:04.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:04.527 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:04.562 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:04.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:04.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:04.594 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T22:25:04.665 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:04.689 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:04.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:04.790 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:04.823 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:04.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:04.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:04.925 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:04.966 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:04.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:04.971 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:04 smithi045 ceph-mon[39715]: 10.8 scrub starts 2024-03-11T22:25:04.971 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:04 smithi045 ceph-mon[39715]: 10.a scrub starts 2024-03-11T22:25:04.971 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:04 smithi045 ceph-mon[30178]: 10.8 scrub starts 2024-03-11T22:25:04.971 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:04 smithi045 ceph-mon[30178]: 10.a scrub starts 2024-03-11T22:25:05.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:05.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:04 smithi144 ceph-mon[33984]: 10.8 scrub starts 2024-03-11T22:25:05.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:04 smithi144 ceph-mon[33984]: 10.a scrub starts 2024-03-11T22:25:05.108 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:05.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:05.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:05.210 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:05.243 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:05.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:05.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:05.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:05.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:05.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:05.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:05.525 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:05.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:05.626 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:05.657 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:05.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:05.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:05.758 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:05.798 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:05.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:05.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:05 smithi045 ceph-mon[39715]: 3.1a scrub starts 2024-03-11T22:25:05.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:05 smithi045 ceph-mon[39715]: 3.1a scrub ok 2024-03-11T22:25:05.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:05 smithi045 ceph-mon[39715]: pgmap v774: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 165 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 985 op/s 2024-03-11T22:25:05.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:05 smithi045 ceph-mon[39715]: 5.1c scrub starts 2024-03-11T22:25:05.886 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:05 smithi045 ceph-mon[39715]: 5.1c scrub ok 2024-03-11T22:25:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:05 smithi045 ceph-mon[30178]: 3.1a scrub starts 2024-03-11T22:25:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:05 smithi045 ceph-mon[30178]: 3.1a scrub ok 2024-03-11T22:25:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:05 smithi045 ceph-mon[30178]: pgmap v774: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 165 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 985 op/s 2024-03-11T22:25:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:05 smithi045 ceph-mon[30178]: 5.1c scrub starts 2024-03-11T22:25:05.887 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:05 smithi045 ceph-mon[30178]: 5.1c scrub ok 2024-03-11T22:25:05.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:05.936 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:05.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:06.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:06.074 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:06.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:06.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:06.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:05 smithi144 ceph-mon[33984]: 3.1a scrub starts 2024-03-11T22:25:06.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:05 smithi144 ceph-mon[33984]: 3.1a scrub ok 2024-03-11T22:25:06.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:05 smithi144 ceph-mon[33984]: pgmap v774: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 165 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 985 op/s 2024-03-11T22:25:06.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:05 smithi144 ceph-mon[33984]: 5.1c scrub starts 2024-03-11T22:25:06.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:05 smithi144 ceph-mon[33984]: 5.1c scrub ok 2024-03-11T22:25:06.176 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:06.206 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:06.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:06.308 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:06.347 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:06.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:06.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:06.449 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:06.482 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:06.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:06.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:06.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:06.623 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:06.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:06.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:06.762 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:06.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:06.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:06.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:06.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:06.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:06.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[39715]: 10.a scrub ok 2024-03-11T22:25:06.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[39715]: 6.1d scrub starts 2024-03-11T22:25:06.993 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[39715]: 6.1d scrub ok 2024-03-11T22:25:06.994 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[39715]: 10.8 scrub ok 2024-03-11T22:25:06.994 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[39715]: 1.5 deep-scrub starts 2024-03-11T22:25:06.994 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[39715]: 1.5 deep-scrub ok 2024-03-11T22:25:06.994 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[39715]: 2.2 scrub starts 2024-03-11T22:25:06.994 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[39715]: 2.2 scrub ok 2024-03-11T22:25:06.994 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[39715]: 3.8 scrub starts 2024-03-11T22:25:06.994 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[39715]: 3.8 scrub ok 2024-03-11T22:25:06.994 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[30178]: 10.a scrub ok 2024-03-11T22:25:06.994 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[30178]: 6.1d scrub starts 2024-03-11T22:25:06.995 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[30178]: 6.1d scrub ok 2024-03-11T22:25:06.995 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[30178]: 10.8 scrub ok 2024-03-11T22:25:06.995 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[30178]: 1.5 deep-scrub starts 2024-03-11T22:25:06.995 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[30178]: 1.5 deep-scrub ok 2024-03-11T22:25:06.995 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[30178]: 2.2 scrub starts 2024-03-11T22:25:06.995 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[30178]: 2.2 scrub ok 2024-03-11T22:25:06.995 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[30178]: 3.8 scrub starts 2024-03-11T22:25:06.995 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:06 smithi045 ceph-mon[30178]: 3.8 scrub ok 2024-03-11T22:25:06.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:07.041 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:07.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:07.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:07.098 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:25:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:25:07.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:06 smithi144 ceph-mon[33984]: 10.a scrub ok 2024-03-11T22:25:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:06 smithi144 ceph-mon[33984]: 6.1d scrub starts 2024-03-11T22:25:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:06 smithi144 ceph-mon[33984]: 6.1d scrub ok 2024-03-11T22:25:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:06 smithi144 ceph-mon[33984]: 10.8 scrub ok 2024-03-11T22:25:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:06 smithi144 ceph-mon[33984]: 1.5 deep-scrub starts 2024-03-11T22:25:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:06 smithi144 ceph-mon[33984]: 1.5 deep-scrub ok 2024-03-11T22:25:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:06 smithi144 ceph-mon[33984]: 2.2 scrub starts 2024-03-11T22:25:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:06 smithi144 ceph-mon[33984]: 2.2 scrub ok 2024-03-11T22:25:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:06 smithi144 ceph-mon[33984]: 3.8 scrub starts 2024-03-11T22:25:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:06 smithi144 ceph-mon[33984]: 3.8 scrub ok 2024-03-11T22:25:07.143 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:07.177 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:07.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:07.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:07.315 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:07.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:07.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:07.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:07.465 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:07.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:07.566 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:07.603 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:07.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:07.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:07.705 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:07.739 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:07.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:07.841 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:07.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:07.865 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:07.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:07.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:08.005 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:08.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:07 smithi045 ceph-mon[39715]: pgmap v775: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 665 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:25:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:07 smithi045 ceph-mon[39715]: 10.d scrub starts 2024-03-11T22:25:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:07 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/26120421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:25:08.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:07 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:25:08.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:07 smithi045 ceph-mon[30178]: pgmap v775: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 665 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:25:08.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:08.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:07 smithi045 ceph-mon[30178]: 10.d scrub starts 2024-03-11T22:25:08.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:07 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/26120421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:25:08.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:07 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:25:08.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:07 smithi144 ceph-mon[33984]: pgmap v775: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 665 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:25:08.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:08.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:07 smithi144 ceph-mon[33984]: 10.d scrub starts 2024-03-11T22:25:08.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:07 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/26120421' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:25:08.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:07 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:25:08.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:08.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:08.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:08.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:08.268 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:08.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:08.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:08.405 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:25:08.409 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:08.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:08.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:08.511 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:08.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:08.545 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:08.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:08.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:08.683 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:08.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:08.784 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:08.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:08.827 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:08.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:08.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:08.961 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:08.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:08.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:08 smithi045 ceph-mon[39715]: 10.7 scrub starts 2024-03-11T22:25:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:08 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:25:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:08 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:25:09.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:08 smithi045 ceph-mon[39715]: osdmap e383: 8 total, 8 up, 8 in 2024-03-11T22:25:09.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:08 smithi045 ceph-mon[30178]: 10.7 scrub starts 2024-03-11T22:25:09.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:08 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:25:09.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:08 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:25:09.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:08 smithi045 ceph-mon[30178]: osdmap e383: 8 total, 8 up, 8 in 2024-03-11T22:25:09.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:09.094 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:09.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:09.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:08 smithi144 ceph-mon[33984]: 10.7 scrub starts 2024-03-11T22:25:09.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:08 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:25:09.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:08 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:25:09.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:08 smithi144 ceph-mon[33984]: osdmap e383: 8 total, 8 up, 8 in 2024-03-11T22:25:09.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:09.235 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:09.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:09.337 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:09.379 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:09.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:09.480 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:09.517 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:09.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:09.619 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:09.653 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:09.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:09.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:09.793 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:09.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:09.894 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:09.932 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:09.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:10.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:10.035 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:10.084 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:10.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:10.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:09 smithi144 ceph-mon[33984]: pgmap v777: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 794 B/s rd, 0 B/s wr, 962 op/s 2024-03-11T22:25:10.184 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:09 smithi045 ceph-mon[39715]: pgmap v777: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 794 B/s rd, 0 B/s wr, 962 op/s 2024-03-11T22:25:10.185 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:09 smithi045 ceph-mon[30178]: pgmap v777: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 794 B/s rd, 0 B/s wr, 962 op/s 2024-03-11T22:25:10.186 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:10.219 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:10.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:10.321 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:10.356 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:10.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:10.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:10.458 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:10.482 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:10.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:10.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:10.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:10.619 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:10.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:10.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:10.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:10.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:10.752 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:10.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:10.853 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:10.887 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:10.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:10.988 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:11.020 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:11.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:11.105 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:25:11.106 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:25:11.122 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:11.163 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:11.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:11.264 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:11.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:11.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:11.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:11.432 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:11.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:11.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:11.534 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:11.571 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:11.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:11.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:11.708 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:11.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:11.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:11.840 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:11.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:11.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:11.941 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:11.977 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:11.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:11.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:12.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:11 smithi045 ceph-mon[39715]: pgmap v778: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 608 op/s 2024-03-11T22:25:12.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:11 smithi045 ceph-mon[30178]: pgmap v778: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 608 op/s 2024-03-11T22:25:12.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:12.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:11 smithi144 ceph-mon[33984]: pgmap v778: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 608 op/s 2024-03-11T22:25:12.110 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:12.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:12.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:12.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T22:25:12.212 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:12.252 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:12.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:12.353 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:12.386 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:12.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:12.488 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:12.524 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:12.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:12.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:12.653 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:25:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:25:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:25:12.667 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:12.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:12.768 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:12.802 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:12.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:12.853 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:12.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:12.946 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:12.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:12.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:12.994 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:12 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:25:12.995 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:12 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:25:12.995 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:12 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:25:12.995 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:12 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:25:13.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:13.093 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:13.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:13.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:13.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:12 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:25:13.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:12 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:25:13.195 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:13.231 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:13.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:13.332 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:13.367 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:13.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:13.468 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:13.503 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:13.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:13.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:13.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:13.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:13.741 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:13.773 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:13.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:13.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:13.878 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:13.909 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:13.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:14.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:14.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:13 smithi045 ceph-mon[39715]: pgmap v779: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T22:25:14.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:13 smithi045 ceph-mon[39715]: osdmap e384: 8 total, 8 up, 8 in 2024-03-11T22:25:14.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:13 smithi045 ceph-mon[30178]: pgmap v779: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T22:25:14.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:13 smithi045 ceph-mon[30178]: osdmap e384: 8 total, 8 up, 8 in 2024-03-11T22:25:14.047 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:14.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:14.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:13 smithi144 ceph-mon[33984]: pgmap v779: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T22:25:14.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:13 smithi144 ceph-mon[33984]: osdmap e384: 8 total, 8 up, 8 in 2024-03-11T22:25:14.149 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:14.183 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:14.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:14.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:14.285 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:14.319 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:14.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:14.420 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:14.455 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:14.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:14.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:14.557 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:14.591 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:14.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:14.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:14.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:14.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:14.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:14.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:14.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:14.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:14.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:14.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:25:14.973 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:15.027 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:15.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:15.129 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:15.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:15.165 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:15.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:15.266 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:15.300 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:15.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:15.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:15.402 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:15.438 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:15.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:15.539 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:15.573 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:15.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:15.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:15.675 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:15.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:15.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:15.816 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:15.852 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:15.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:15.949 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:15 smithi045 ceph-mon[39715]: pgmap v781: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 837 op/s 2024-03-11T22:25:15.949 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:15 smithi045 ceph-mon[30178]: pgmap v781: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 837 op/s 2024-03-11T22:25:15.954 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:15.960 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:15.983 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:15.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:16.056 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:25:16.076 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:25:16.085 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:16.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:15 smithi144 ceph-mon[33984]: pgmap v781: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 837 op/s 2024-03-11T22:25:16.104 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:25:16.118 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:16.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:16.143 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:25:16.184 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:25:16.220 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:16.231 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:25:16.267 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:16.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:16.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:16.271 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:25:16.309 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:25:16.320 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:25:16.320 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:25:16.320 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:25:16.320 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:25:16.320 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:25:16.320 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:25:16.320 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:25:16.320 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:25:16.369 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:16.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:16.407 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:16.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:16.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:16.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:16.558 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:16.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:16.659 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:16.693 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:16.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:16.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:16.834 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:16.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:16.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:16.981 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:16.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:17.005 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:25:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:25:17.082 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:17.114 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:17.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:17.197 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:17.216 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:17.250 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:17.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:17.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:17 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:17.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:17 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:25:17.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:17 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3607599721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:25:17.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:17 smithi045 ceph-mon[30178]: pgmap v782: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 592 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T22:25:17.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:17.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:17 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:17.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:17 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:25:17.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:17 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3607599721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:25:17.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:17 smithi045 ceph-mon[39715]: pgmap v782: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 592 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T22:25:17.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:17.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:17 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:25:17.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:17 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:25:17.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:17 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3607599721' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:25:17.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:17 smithi144 ceph-mon[33984]: pgmap v782: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 592 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T22:25:17.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:17.352 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:17.384 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:17.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:17.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:17.516 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:17.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:17.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:17.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:17.657 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:17.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:17.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:17.759 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:17.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:17.795 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:17.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:17.897 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:17.929 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:17.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:17.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:18.030 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T22:25:18.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:18.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:18 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4141569616' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:25:18.032 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:18 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:25:18.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:18 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4141569616' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:25:18.033 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:18 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:25:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:18.082 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:18.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:18.184 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:18.219 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:18.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:18.320 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:18.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:18 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4141569616' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:25:18.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:18 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:25:18.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:18.357 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:18.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:18.458 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:18.491 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:18.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:18.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:18.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:18.626 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:18.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:18.727 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:18.764 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:18.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:18.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:18.866 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:18.915 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:18.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:18.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:19.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:19.058 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:19.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:19.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:19.160 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:19.191 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:19.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:19.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:19.255 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:19 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:25:19.255 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:19 smithi045 ceph-mon[39715]: osdmap e385: 8 total, 8 up, 8 in 2024-03-11T22:25:19.255 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:19 smithi045 ceph-mon[39715]: pgmap v784: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-11T22:25:19.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:19 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:25:19.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:19 smithi045 ceph-mon[30178]: osdmap e385: 8 total, 8 up, 8 in 2024-03-11T22:25:19.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:19 smithi045 ceph-mon[30178]: pgmap v784: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-11T22:25:19.293 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:19.329 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:19.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:19.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:19.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:19 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:25:19.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:19 smithi144 ceph-mon[33984]: osdmap e385: 8 total, 8 up, 8 in 2024-03-11T22:25:19.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:19 smithi144 ceph-mon[33984]: pgmap v784: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-11T22:25:19.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:19.467 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:19.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:19.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:19.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:19.604 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:19.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:19.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:19.706 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:19.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:25:19.820 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:19.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:19.843 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T22:25:19.843 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:25:19.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:19.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:19.969 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:19.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:20.069 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:20.123 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:20.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:20.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:20.273 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:20.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:20.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:20.375 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:20.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:20.417 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:20.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:20.518 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:20.552 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:20.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:20.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:20.654 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:20.690 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:20.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:20.792 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:20.825 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:20.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:20.927 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:20.960 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:20.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:21.062 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:21.100 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:21.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:21.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:21.240 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:21.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:21.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:21.387 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:21.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:21.489 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:21.516 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:25:21.516 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-03-11T22:25:21.516 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-03-11T22:25:21.516 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:25:21.516 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.516 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:81569 10.f e362/348 10:f91ac8d0:::benchmark_data_smithi045_169175_object81568:head v 362'5021, mlcod=362'5019)", 2024-03-11T22:25:21.516 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:49.359820+0000", 2024-03-11T22:25:21.516 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 92.147159654000006, 2024-03-11T22:25:21.516 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13740970299999999, 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.359820+0000", 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.359820+0000", 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.517 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.359820+0000", 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2500000000000005e-07 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.359922+0000", 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000101673 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.359934+0000", 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2197e-05 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.359942+0000", 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7190000000000001e-06 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.360011+0000", 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9391e-05 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.360031+0000", 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9610999999999999e-05 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.497186+0000", 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13715540300000001 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.497229+0000", 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2984e-05 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:81606 10.f e362/348 10:fafd0ba4:::benchmark_data_smithi045_169175_object81605:head v 362'5022, mlcod=362'5019)", 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:49.388065+0000", 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 92.118914142999998, 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.109191074, 2024-03-11T22:25:21.520 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.388065+0000", 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.388065+0000", 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.388066+0000", 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2500000000000005e-07 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.388121+0000", 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4970999999999999e-05 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.388131+0000", 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9359999999999994e-06 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.388137+0000", 2024-03-11T22:25:21.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5479999999999998e-06 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.388197+0000", 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9681e-05 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.388215+0000", 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8051e-05 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.497233+0000", 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.109017634 2024-03-11T22:25:21.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.497256+0000", 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3527999999999999e-05 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:81619 10.f e362/348 10:f0c61dfe:::benchmark_data_smithi045_169175_object81618:head v 362'5023, mlcod=362'5019)", 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:49.396313+0000", 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 92.110666769000005, 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100967504, 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.524 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.396313+0000", 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.396313+0000", 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.396314+0000", 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.08e-06 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.396349+0000", 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5327000000000002e-05 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.396360+0000", 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1114000000000001e-05 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.396368+0000", 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4560000000000003e-06 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.396507+0000", 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013922000000000001 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.396526+0000", 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.929e-05 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.497258+0000", 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100732154 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.497280+0000", 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1863000000000001e-05 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15540.0:81631 10.f e362/348 10:fc455004:::benchmark_data_smithi045_169175_object81630:head v 362'5024, mlcod=362'5019)", 2024-03-11T22:25:21.528 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:23:49.406141+0000", 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 92.100838382999996, 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.091162472999999994, 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.406141+0000", 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.406141+0000", 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.406142+0000", 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8700000000000005e-07 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.406169+0000", 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7178999999999998e-05 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.406179+0000", 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0556e-05 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.406187+0000", 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1389999999999998e-06 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.409486+0000", 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003299443 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.409506+0000", 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0475999999999998e-05 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.497283+0000", 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.087776245000000003 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:23:49.497303+0000", 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0747999999999999e-05 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:108529 10.5 10:abe14c7c:::benchmark_data_smithi045_169175_object108528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:12.085370+0000", 2024-03-11T22:25:21.532 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 69.421609094000004, 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.12978513999999999, 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 108529 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.085370+0000", 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.085370+0000", 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.085372+0000", 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.42e-06 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.085396+0000", 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3801000000000001e-05 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.085408+0000", 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2653999999999999e-05 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.085416+0000", 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9719999999999992e-06 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.085481+0000", 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5234000000000006e-05 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.085688+0000", 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000206827 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.085875+0000", 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000186873 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.092938+0000", 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0070629350000000002 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.092955+0000", 2024-03-11T22:25:21.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.736e-05 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.212249+0000", 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.119294018 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.215101+0000", 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002851405 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.215155+0000", 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4641000000000001e-05 2024-03-11T22:25:21.537 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:108535 10.5 10:a8e1a8b4:::benchmark_data_smithi045_169175_object108534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:12.089277+0000", 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 69.417702661000007, 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13470918500000001, 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 108535 2024-03-11T22:25:21.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.089277+0000", 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.089277+0000", 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.089278+0000", 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6109999999999999e-06 2024-03-11T22:25:21.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.089305+0000", 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6190000000000002e-05 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.089319+0000", 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4205e-05 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.089329+0000", 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9170000000000001e-06 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.089389+0000", 2024-03-11T22:25:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9879e-05 2024-03-11T22:25:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.541 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.541 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:21 smithi045 ceph-mon[39715]: pgmap v785: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 688 op/s 2024-03-11T22:25:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.089571+0000", 2024-03-11T22:25:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018273700000000001 2024-03-11T22:25:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:25:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.089715+0000", 2024-03-11T22:25:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014345199999999999 2024-03-11T22:25:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.542 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.093093+0000", 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033786100000000002 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.093306+0000", 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000213076 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.223822+0000", 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13051558699999999 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.223937+0000", 2024-03-11T22:25:21.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000115176 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:12.223986+0000", 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8745000000000002e-05 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:146872 10.5 10:a9b421e9:::benchmark_data_smithi045_169175_object14845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:43.431062+0000", 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 38.075917236999999, 2024-03-11T22:25:21.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.075088218999999998, 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 146872 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.431062+0000", 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.545 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.431062+0000", 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.431063+0000", 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7000000000000004e-07 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.431067+0000", 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.34e-06 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.546 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.431081+0000", 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3402e-05 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.431088+0000", 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1130000000000003e-06 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.435457+0000", 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0043694989999999998 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.435574+0000", 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011688000000000001 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.435689+0000", 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000115275 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.445357+0000", 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0096677970000000005 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.445460+0000", 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000102612 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.505640+0000", 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.060180451000000003 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.506101+0000", 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00046021400000000002 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.506150+0000", 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9866e-05 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:146873 10.5 10:a74a5726:::benchmark_data_smithi045_169175_object14846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:43.431116+0000", 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 38.075863503999997, 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.081896652, 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.550 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 146873 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.431116+0000", 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.431116+0000", 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.551 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.431116+0000", 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0200000000000002e-07 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.431119+0000", 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7180000000000001e-06 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.431127+0000", 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4819999999999997e-06 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.431133+0000", 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3690000000000003e-06 2024-03-11T22:25:21.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.441051+0000", 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0099178970000000002 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.441176+0000", 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000124964 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.441641+0000", 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00046560699999999999 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.445464+0000", 2024-03-11T22:25:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0038227449999999998 2024-03-11T22:25:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.445924+0000", 2024-03-11T22:25:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00046000199999999999 2024-03-11T22:25:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.554 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.554 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:21 smithi045 ceph-mon[30178]: pgmap v785: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 688 op/s 2024-03-11T22:25:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:21.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:21.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.510344+0000", 2024-03-11T22:25:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.064420127999999993 2024-03-11T22:25:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.512964+0000", 2024-03-11T22:25:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026195089999999999 2024-03-11T22:25:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:43.513013+0000", 2024-03-11T22:25:21.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8729000000000001e-05 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:148435 10.5 10:a3414e87:::benchmark_data_smithi045_169175_object16408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:44.702342+0000", 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 36.804637472000003, 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100101622, 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.557 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 148435 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.702342+0000", 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.702342+0000", 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.558 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.702344+0000", 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7460000000000001e-06 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.702349+0000", 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4179999999999996e-06 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.702362+0000", 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2670000000000001e-05 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.702370+0000", 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3359999999999995e-06 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.702432+0000", 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1508999999999997e-05 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.702545+0000", 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011336300000000001 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.702668+0000", 2024-03-11T22:25:21.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012286 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.711746+0000", 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0090777589999999995 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.712406+0000", 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00066020200000000003 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.802310+0000", 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.089903914000000001 2024-03-11T22:25:21.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.802393+0000", 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3399999999999994e-05 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.802444+0000", 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0445000000000002e-05 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.562 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:148438 10.5 10:acb1e17c:::benchmark_data_smithi045_169175_object16411:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:44.705572+0000", 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 36.801407699999999, 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.097018967999999997, 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 148438 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.705572+0000", 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.705572+0000", 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.705573+0000", 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.341e-06 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.705578+0000", 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4160000000000003e-06 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.564 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.705592+0000", 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3291e-05 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.705601+0000", 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.302e-06 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.706589+0000", 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00098755599999999989 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.565 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.706709+0000", 2024-03-11T22:25:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012029 2024-03-11T22:25:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:25:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.706826+0000", 2024-03-11T22:25:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116811 2024-03-11T22:25:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.711809+0000", 2024-03-11T22:25:21.566 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0049835210000000003 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.712440+0000", 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00063078600000000004 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.802499+0000", 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.090058876999999996 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.802551+0000", 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2491000000000003e-05 2024-03-11T22:25:21.567 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.802591+0000", 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9286000000000003e-05 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:148466 10.5 10:a7a65cca:::benchmark_data_smithi045_169175_object16439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:44.725424+0000", 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 36.781555113000003, 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.077348708000000002, 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.568 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 148466 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.725424+0000", 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.725424+0000", 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.569 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.725425+0000", 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7499999999999999e-07 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.725429+0000", 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4800000000000001e-06 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.725436+0000", 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5859999999999999e-06 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.570 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.725441+0000", 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1889999999999998e-06 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.725495+0000", 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3995000000000001e-05 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.794613+0000", 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.069117257000000001 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.571 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.794757+0000", 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014473900000000001 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.802641+0000", 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0078837319999999992 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.802659+0000", 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7649999999999999e-05 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.572 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.802686+0000", 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7178999999999998e-05 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.802736+0000", 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0182000000000002e-05 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:44.802773+0000", 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6844000000000002e-05 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.573 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:150585 10.5 10:a03883f1:::benchmark_data_smithi045_169175_object18558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:46.590103+0000", 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 34.916876809999998, 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.077432473000000002, 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 150585 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.590103+0000", 2024-03-11T22:25:21.574 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.590103+0000", 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.590104+0000", 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7600000000000001e-06 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.590110+0000", 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.49e-06 2024-03-11T22:25:21.575 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.590125+0000", 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5153999999999999e-05 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.590134+0000", 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3430000000000002e-06 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.590199+0000", 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4311000000000001e-05 2024-03-11T22:25:21.576 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.590325+0000", 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012610400000000001 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.590449+0000", 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012464 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.597738+0000", 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0072887890000000004 2024-03-11T22:25:21.577 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.598274+0000", 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00053627899999999997 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.667431+0000", 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.069156966 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.667490+0000", 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8677000000000001e-05 2024-03-11T22:25:21.578 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.667535+0000", 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4960000000000003e-05 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:150590 10.5 10:a63a2a5b:::benchmark_data_smithi045_169175_object18563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:46.593877+0000", 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 34.913102811000002, 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.073872902000000004, 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.579 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 150590 2024-03-11T22:25:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.580 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.593877+0000", 2024-03-11T22:25:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.593877+0000", 2024-03-11T22:25:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.593878+0000", 2024-03-11T22:25:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.525e-06 2024-03-11T22:25:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.581 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.593883+0000", 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7820000000000001e-06 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.593897+0000", 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3986e-05 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.593907+0000", 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6900000000000004e-06 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.582 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.594792+0000", 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00088524199999999995 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.594950+0000", 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000158338 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.595071+0000", 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000120951 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.583 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.597867+0000", 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027963480000000001 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.598402+0000", 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000534353 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.667605+0000", 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.069203471000000003 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.584 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.667702+0000", 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6302000000000001e-05 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:46.667749+0000", 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7914000000000002e-05 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:161237 10.5 10:a63c0018:::benchmark_data_smithi045_169175_object29210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:55.441878+0000", 2024-03-11T22:25:21.585 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 26.065101650999999, 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102360375, 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 161237 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441878+0000", 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.586 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441878+0000", 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441879+0000", 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.638e-06 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441885+0000", 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9349999999999999e-06 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.587 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441903+0000", 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7414e-05 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441911+0000", 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3010000000000007e-06 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.443955+0000", 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002044054 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.588 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.444078+0000", 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000122698 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.444205+0000", 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000127534 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.451139+0000", 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0069340970000000002 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.589 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.454594+0000", 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0034550420000000002 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.544100+0000", 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.089505827999999996 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.544171+0000", 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0254999999999997e-05 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.590 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.544238+0000", 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7578999999999998e-05 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:161238 10.3 10:c67277ce:::benchmark_data_smithi045_169175_object29211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:55.441931+0000", 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 26.065048132000001, 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.098967078, 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.591 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 161238 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441931+0000", 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441931+0000", 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.592 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441932+0000", 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7899999999999999e-07 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441935+0000", 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7690000000000001e-06 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441950+0000", 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5126000000000001e-05 2024-03-11T22:25:21.593 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441960+0000", 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.048e-05 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.442046+0000", 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5585000000000001e-05 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.442168+0000", 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012252399999999999 2024-03-11T22:25:21.594 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,5", 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.442283+0000", 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000115097 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.445851+0000", 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035677769999999998 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.446035+0000", 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018358599999999999 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.595 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.540717+0000", 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.094682529000000001 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.540824+0000", 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000107199 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.540898+0000", 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3926999999999999e-05 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.596 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:161242 10.5 10:a69f9e81:::benchmark_data_smithi045_169175_object29215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:55.441983+0000", 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 26.064996526000002, 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.113678001, 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 161242 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.597 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441983+0000", 2024-03-11T22:25:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441983+0000", 2024-03-11T22:25:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441984+0000", 2024-03-11T22:25:21.598 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5300000000000004e-07 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441987+0000", 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.529e-06 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.441996+0000", 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8720000000000008e-06 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.442003+0000", 2024-03-11T22:25:21.599 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7989999999999996e-06 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447555+0000", 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0055520709999999996 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447686+0000", 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013114800000000001 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447808+0000", 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012215199999999999 2024-03-11T22:25:21.600 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.454556+0000", 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.006748179 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.456148+0000", 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015915359999999999 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.555539+0000", 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.099391462999999999 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.601 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.555598+0000", 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8854000000000003e-05 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.555661+0000", 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2744999999999996e-05 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:161250 10.3 10:c2465f35:::benchmark_data_smithi045_169175_object29223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:55.447310+0000", 2024-03-11T22:25:21.602 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 26.059669237000001, 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.093727569999999996, 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 161250 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447310+0000", 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.603 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447310+0000", 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447312+0000", 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7910000000000001e-06 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447319+0000", 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9090000000000003e-06 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.604 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447337+0000", 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7847e-05 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447347+0000", 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9680000000000001e-06 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447405+0000", 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8384999999999998e-05 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.605 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447523+0000", 2024-03-11T22:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118145 2024-03-11T22:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,5", 2024-03-11T22:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447634+0000", 2024-03-11T22:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110969 2024-03-11T22:25:21.606 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.450792+0000", 2024-03-11T22:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031574760000000002 2024-03-11T22:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.454921+0000", 2024-03-11T22:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0041289480000000003 2024-03-11T22:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.607 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.540948+0000", 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.086026963999999997 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.540998+0000", 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0631999999999999e-05 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.541038+0000", 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9536000000000002e-05 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.608 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:161251 10.3 10:c29f836c:::benchmark_data_smithi045_169175_object29224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:55.447483+0000", 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 26.059496751000001, 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100786085, 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 161251 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.609 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447483+0000", 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447483+0000", 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447484+0000", 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5000000000000001e-07 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.610 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447488+0000", 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.792e-06 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447499+0000", 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1110999999999999e-05 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.447508+0000", 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6139999999999992e-06 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.611 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.450859+0000", 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033513380000000001 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.450997+0000", 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013794499999999999 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,5", 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.451224+0000", 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00022676599999999999 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.612 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.454980+0000", 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00375545 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.455209+0000", 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000229769 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.548153+0000", 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.092943943000000001 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.613 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.548217+0000", 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3886999999999993e-05 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.548269+0000", 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1520000000000001e-05 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:161252 10.5 10:a5a9493c:::benchmark_data_smithi045_169175_object29225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:55.448268+0000", 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 26.058711253999999, 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.107560079, 2024-03-11T22:25:21.614 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 161252 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.448268+0000", 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.615 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.448268+0000", 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.448269+0000", 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5300000000000003e-07 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.448274+0000", 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7210000000000002e-06 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.448284+0000", 2024-03-11T22:25:21.616 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0771000000000001e-05 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.448292+0000", 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4769999999999997e-06 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.451189+0000", 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002896694 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.451315+0000", 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012625200000000001 2024-03-11T22:25:21.617 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,3", 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.451445+0000", 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000130388 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.454702+0000", 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032562569999999998 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.456298+0000", 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015961899999999999 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.618 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.555715+0000", 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.099417345000000004 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.555777+0000", 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1958000000000005e-05 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.555828+0000", 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1172999999999999e-05 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.619 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15540.0:161270 10.3 10:c24fb56a:::benchmark_data_smithi045_169175_object29243:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:24:55.467606+0000", 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 26.039373225999999, 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.080892717000000003, 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15540", 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/3354878804", 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 161270 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:25:21.620 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.467606+0000", 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.467606+0000", 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.467608+0000", 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0329999999999998e-06 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.621 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.467625+0000", 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6847e-05 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.467640+0000", 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5099000000000001e-05 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.467650+0000", 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5689999999999994e-06 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.622 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.467704+0000", 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4172999999999998e-05 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.467790+0000", 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6389000000000002e-05 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,5", 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.467886+0000", 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5801e-05 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.623 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.470997+0000", 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031111720000000002 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.471055+0000", 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7853999999999999e-05 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.548309+0000", 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.077254319000000002 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.624 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.548458+0000", 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014838300000000001 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:24:55.548499+0000", 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1078000000000002e-05 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:25:21.625 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:25:21.657 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:21.689 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:21.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:21.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:21.791 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:21.826 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:21.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:21.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:21 smithi144 ceph-mon[33984]: pgmap v785: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 688 op/s 2024-03-11T22:25:21.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:21.966 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:21.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:22.068 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:22.102 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:22.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:22.203 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:22.240 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:22.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:22.342 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:22.374 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:22.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:22.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:22.475 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:22.501 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:22.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:22.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:22.647 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:22.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:22.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:22.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:22.785 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:25:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:25:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:25:22.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:22.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:22.878 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:22.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:22.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:22.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:22.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:23.042 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:23.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:23.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:23.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:23.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:23.238 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:23.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:23.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:23.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:23.379 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:23.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:23.480 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:23.514 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:23.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:23.615 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:23.660 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:23.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:23.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:23.761 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:23.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:23 smithi045 ceph-mon[39715]: pgmap v786: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 997 op/s 2024-03-11T22:25:23.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:23 smithi045 ceph-mon[30178]: pgmap v786: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 997 op/s 2024-03-11T22:25:23.795 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:23.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:23.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:23.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:23 smithi144 ceph-mon[33984]: pgmap v786: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 997 op/s 2024-03-11T22:25:23.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T22:25:23.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:23.968 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:23.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:23.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:24.071 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:24.116 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:24.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:24.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:24.218 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:24.284 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:24.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:24.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:24.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:24.425 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:24.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:24.478 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:24.527 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:24.575 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:24.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:24.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:24.677 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:24.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:24.714 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:24.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:24.815 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:24.849 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:24.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:24.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:24.988 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:24.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:25.089 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:25.126 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:25.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:25.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:25.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:25.275 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:25.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:25.376 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:25.414 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:25.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:25.516 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:25.548 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:25.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:25.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:25.615 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:25 smithi045 ceph-mon[30178]: pgmap v787: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-11T22:25:25.616 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:25 smithi045 ceph-mon[39715]: pgmap v787: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-11T22:25:25.650 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:25.680 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:25.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:25.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:25.782 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:25.815 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:25.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:25.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:25.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:25 smithi144 ceph-mon[33984]: pgmap v787: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-11T22:25:25.917 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:25.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:25.953 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:25.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:26.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:26.093 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:26.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:26.195 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:26.235 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:26.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:26.336 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:26.369 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:26.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:26.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:26.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:26.511 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:26.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:26.561 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:25:26.694 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:26.730 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:26.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:26.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:26 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4239997566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:25:26.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:26 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:25:26.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:26 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4239997566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:25:26.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:26 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:25:26.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:26.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:26 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4239997566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:25:26.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:26 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:25:26.849 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:25:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:25:26.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:26.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:26.972 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:27.004 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:27.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:27.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:27.106 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:27.145 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:27.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:27.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:27.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:27.284 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:27.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:27.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:27.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:27.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:27.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:27.523 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:27.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:27.564 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:27.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:27.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:27.702 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:27.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:27.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:27 smithi045 ceph-mon[39715]: pgmap v788: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-11T22:25:27.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:27 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:25:27.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:27 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:25:27.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:27 smithi045 ceph-mon[39715]: osdmap e386: 8 total, 8 up, 8 in 2024-03-11T22:25:27.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:27.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:27 smithi045 ceph-mon[39715]: from='client.25508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:25:27.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:27 smithi045 ceph-mon[30178]: pgmap v788: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-11T22:25:27.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:27 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:25:27.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:27 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:25:27.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:27 smithi045 ceph-mon[30178]: osdmap e386: 8 total, 8 up, 8 in 2024-03-11T22:25:27.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:27.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:27 smithi045 ceph-mon[30178]: from='client.25508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:25:27.804 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:27.841 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:27.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:27.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:27 smithi144 ceph-mon[33984]: pgmap v788: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-11T22:25:27.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:27 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:25:27.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:27 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:25:27.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:27 smithi144 ceph-mon[33984]: osdmap e386: 8 total, 8 up, 8 in 2024-03-11T22:25:27.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:27.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:27 smithi144 ceph-mon[33984]: from='client.25508 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:25:27.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:27.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:27.980 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:27.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:28.081 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:28.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:28.128 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:28.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:28.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:28.236 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:25:28.252 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:25:28.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:28.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:28.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:28.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:28.443 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:28.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:28.545 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:28.579 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:28.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:28.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:28.596 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:28 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:25:28.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:28 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:25:28.597 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:28 smithi045 ceph-mon[30178]: osdmap e387: 8 total, 8 up, 8 in 2024-03-11T22:25:28.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:28 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:25:28.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:28 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:25:28.597 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:28 smithi045 ceph-mon[39715]: osdmap e387: 8 total, 8 up, 8 in 2024-03-11T22:25:28.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:28.726 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:28.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:28.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:28.828 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:28.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:28 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:25:28.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:28 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:25:28.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:28 smithi144 ceph-mon[33984]: osdmap e387: 8 total, 8 up, 8 in 2024-03-11T22:25:28.862 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:28.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:28.964 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:28.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:28.998 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:28.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:29.099 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:29.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:29.134 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:29.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:29.235 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:29.275 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:29.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:29.377 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:29.415 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:29.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:29.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:29.517 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:29.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:29.555 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:29.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:29.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:29.694 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:29.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:29.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:29.792 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":789,"stamp":"2024-03-11T22:25:26.539519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4685633908,"num_objects":71687,"num_object_clones":0,"num_object_copies":215061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71687,"num_whiteouts":0,"num_read":2098,"num_read_kb":1850,"num_write":193283,"num_write_kb":8449582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":2436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83677,"ondisk_log_size":83677,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18505740,"kb_used_data":13941448,"kb_used_omap":12,"kb_used_meta":4564147,"kb_avail":731455476,"statfs":{"total":767960285184,"available":749010407424,"internally_reserved":0,"allocated":14276042752,"data_stored":14273530252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":4673687091},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,10,6,9,7,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-603193344,"num_objects":-9204,"num_object_clones":0,"num_object_copies":-27612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9204,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9204,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.521617"},"pg_stats":[{"pgid":"10.2","version":"385'11749","reported_seq":11834,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:21.576957+0000","last_change":"2024-03-11T22:22:52.799244+0000","last_active":"2024-03-11T22:25:21.576957+0000","last_peered":"2024-03-11T22:25:21.576957+0000","last_clean":"2024-03-11T22:25:21.576957+0000","last_became_active":"2024-03-11T22:22:52.798875+0000","last_became_peered":"2024-03-11T22:22:52.798875+0000","last_unstale":"2024-03-11T22:25:21.576957+0000","last_undegraded":"2024-03-11T22:25:21.576957+0000","last_fullsized":"2024-03-11T22:25:21.576957+0000","mapping_epoch":348,"log_start":"370'6700","ondisk_log_start":"370'6700","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":5049,"log_dups_size":0,"ondisk_log_size":5049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:24.498562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304152576,"num_objects":4641,"num_object_clones":0,"num_object_copies":13923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11749,"num_write_kb":524480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"294'25","reported_seq":504,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.026042+0000","last_change":"2024-03-11T22:24:39.621975+0000","last_active":"2024-03-11T22:25:18.026042+0000","last_peered":"2024-03-11T22:25:18.026042+0000","last_clean":"2024-03-11T22:25:18.026042+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:25:18.026042+0000","last_undegraded":"2024-03-11T22:25:18.026042+0000","last_fullsized":"2024-03-11T22:25:18.026042+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-03-11T22:24:39.621707+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.621707+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.425365+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":692,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.021680+0000","last_change":"2024-03-11T22:24:37.604756+0000","last_active":"2024-03-11T22:25:18.021680+0000","last_peered":"2024-03-11T22:25:18.021680+0000","last_clean":"2024-03-11T22:25:18.021680+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:25:18.021680+0000","last_undegraded":"2024-03-11T22:25:18.021680+0000","last_fullsized":"2024-03-11T22:25:18.021680+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.601240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.601240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:51.956237+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":928,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.551224+0000","last_change":"2024-03-11T22:24:01.449741+0000","last_active":"2024-03-11T22:25:18.551224+0000","last_peered":"2024-03-11T22:25:18.551224+0000","last_clean":"2024-03-11T22:25:18.551224+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:25:18.551224+0000","last_undegraded":"2024-03-11T22:25:18.551224+0000","last_fullsized":"2024-03-11T22:25:18.551224+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:24:01.447108+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:19:40.008325+0000","last_clean_scrub_stamp":"2024-03-11T22:24:01.447108+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:26.730874+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":458,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.041074+0000","last_change":"2024-03-11T22:23:07.686811+0000","last_active":"2024-03-11T22:25:18.041074+0000","last_peered":"2024-03-11T22:25:18.041074+0000","last_clean":"2024-03-11T22:25:18.041074+0000","last_became_active":"2024-03-11T22:14:52.243690+0000","last_became_peered":"2024-03-11T22:14:52.243690+0000","last_unstale":"2024-03-11T22:25:18.041074+0000","last_undegraded":"2024-03-11T22:25:18.041074+0000","last_fullsized":"2024-03-11T22:25:18.041074+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:07.686730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:07.686730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":1015,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.021988+0000","last_change":"2024-03-11T22:23:43.670715+0000","last_active":"2024-03-11T22:25:18.021988+0000","last_peered":"2024-03-11T22:25:18.021988+0000","last_clean":"2024-03-11T22:25:18.021988+0000","last_became_active":"2024-03-11T22:20:32.293356+0000","last_became_peered":"2024-03-11T22:20:32.293356+0000","last_unstale":"2024-03-11T22:25:18.021988+0000","last_undegraded":"2024-03-11T22:25:18.021988+0000","last_fullsized":"2024-03-11T22:25:18.021988+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:43.670585+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:43.670585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.704176+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"385'12156","reported_seq":12317,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:26.032284+0000","last_change":"2024-03-11T22:24:53.352685+0000","last_active":"2024-03-11T22:25:26.032284+0000","last_peered":"2024-03-11T22:25:26.032284+0000","last_clean":"2024-03-11T22:25:26.032284+0000","last_became_active":"2024-03-11T22:22:52.469327+0000","last_became_peered":"2024-03-11T22:22:52.469327+0000","last_unstale":"2024-03-11T22:25:26.032284+0000","last_undegraded":"2024-03-11T22:25:26.032284+0000","last_fullsized":"2024-03-11T22:25:26.032284+0000","mapping_epoch":348,"log_start":"368'6100","ondisk_log_start":"368'6100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9962","last_scrub_stamp":"2024-03-11T22:24:53.352359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.352359+0000","objects_scrubbed":6655,"log_size":6056,"log_dups_size":0,"ondisk_log_size":6056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:58.557123+0000","scrub_duration":2633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285999104,"num_objects":4364,"num_object_clones":0,"num_object_copies":13092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12156,"num_write_kb":528640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"353'39","reported_seq":558,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030795+0000","last_change":"2024-03-11T22:24:58.663588+0000","last_active":"2024-03-11T22:25:18.030795+0000","last_peered":"2024-03-11T22:25:18.030795+0000","last_clean":"2024-03-11T22:25:18.030795+0000","last_became_active":"2024-03-11T22:14:52.239649+0000","last_became_peered":"2024-03-11T22:14:52.239649+0000","last_unstale":"2024-03-11T22:25:18.030795+0000","last_undegraded":"2024-03-11T22:25:18.030795+0000","last_fullsized":"2024-03-11T22:25:18.030795+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'39","last_scrub_stamp":"2024-03-11T22:24:58.663290+0000","last_deep_scrub":"353'39","last_deep_scrub_stamp":"2024-03-11T22:23:05.461729+0000","last_clean_scrub_stamp":"2024-03-11T22:24:58.663290+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:01.646724+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":716,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.024785+0000","last_change":"2024-03-11T22:23:51.794264+0000","last_active":"2024-03-11T22:25:18.024785+0000","last_peered":"2024-03-11T22:25:18.024785+0000","last_clean":"2024-03-11T22:25:18.024785+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:25:18.024785+0000","last_undegraded":"2024-03-11T22:25:18.024785+0000","last_fullsized":"2024-03-11T22:25:18.024785+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:51.794140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.794140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":692,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.041125+0000","last_change":"2024-03-11T22:23:48.515368+0000","last_active":"2024-03-11T22:25:18.041125+0000","last_peered":"2024-03-11T22:25:18.041125+0000","last_clean":"2024-03-11T22:25:18.041125+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:25:18.041125+0000","last_undegraded":"2024-03-11T22:25:18.041125+0000","last_fullsized":"2024-03-11T22:25:18.041125+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.515251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.515251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":704,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.037327+0000","last_change":"2024-03-11T22:23:49.537509+0000","last_active":"2024-03-11T22:25:18.037327+0000","last_peered":"2024-03-11T22:25:18.037327+0000","last_clean":"2024-03-11T22:25:18.037327+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:25:18.037327+0000","last_undegraded":"2024-03-11T22:25:18.037327+0000","last_fullsized":"2024-03-11T22:25:18.037327+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:49.537151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:49.537151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:20.675254+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":991,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029369+0000","last_change":"2024-03-11T22:22:57.117070+0000","last_active":"2024-03-11T22:25:18.029369+0000","last_peered":"2024-03-11T22:25:18.029369+0000","last_clean":"2024-03-11T22:25:18.029369+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:25:18.029369+0000","last_undegraded":"2024-03-11T22:25:18.029369+0000","last_fullsized":"2024-03-11T22:25:18.029369+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:22:57.116971+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:22:57.116971+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"385'12102","reported_seq":12187,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:25.613147+0000","last_change":"2024-03-11T22:22:52.799080+0000","last_active":"2024-03-11T22:25:25.613147+0000","last_peered":"2024-03-11T22:25:25.613147+0000","last_clean":"2024-03-11T22:25:25.613147+0000","last_became_active":"2024-03-11T22:22:52.798328+0000","last_became_peered":"2024-03-11T22:22:52.798328+0000","last_unstale":"2024-03-11T22:25:25.613147+0000","last_undegraded":"2024-03-11T22:25:25.613147+0000","last_fullsized":"2024-03-11T22:25:25.613147+0000","mapping_epoch":348,"log_start":"371'7100","ondisk_log_start":"371'7100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284033024,"num_objects":4334,"num_object_clones":0,"num_object_copies":13002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12102,"num_write_kb":525952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"273'19","reported_seq":524,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.024737+0000","last_change":"2024-03-11T22:24:57.716291+0000","last_active":"2024-03-11T22:25:18.024737+0000","last_peered":"2024-03-11T22:25:18.024737+0000","last_clean":"2024-03-11T22:25:18.024737+0000","last_became_active":"2024-03-11T22:14:52.240437+0000","last_became_peered":"2024-03-11T22:14:52.240437+0000","last_unstale":"2024-03-11T22:25:18.024737+0000","last_undegraded":"2024-03-11T22:25:18.024737+0000","last_fullsized":"2024-03-11T22:25:18.024737+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:24:57.716144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:57.716144+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:16.100178+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":483,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030392+0000","last_change":"2024-03-11T22:24:57.690108+0000","last_active":"2024-03-11T22:25:18.030392+0000","last_peered":"2024-03-11T22:25:18.030392+0000","last_clean":"2024-03-11T22:25:18.030392+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:25:18.030392+0000","last_undegraded":"2024-03-11T22:25:18.030392+0000","last_fullsized":"2024-03-11T22:25:18.030392+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:57.689696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:57.689696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.466412+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":692,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030713+0000","last_change":"2024-03-11T22:24:59.851270+0000","last_active":"2024-03-11T22:25:18.030713+0000","last_peered":"2024-03-11T22:25:18.030713+0000","last_clean":"2024-03-11T22:25:18.030713+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:25:18.030713+0000","last_undegraded":"2024-03-11T22:25:18.030713+0000","last_fullsized":"2024-03-11T22:25:18.030713+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:59.851040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:59.851040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:06.559357+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":672,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.023332+0000","last_change":"2024-03-11T22:23:48.716684+0000","last_active":"2024-03-11T22:25:18.023332+0000","last_peered":"2024-03-11T22:25:18.023332+0000","last_clean":"2024-03-11T22:25:18.023332+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:25:18.023332+0000","last_undegraded":"2024-03-11T22:25:18.023332+0000","last_fullsized":"2024-03-11T22:25:18.023332+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.716557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.716557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.704266+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"93'130","reported_seq":1019,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.043502+0000","last_change":"2024-03-11T22:22:14.766914+0000","last_active":"2024-03-11T22:25:18.043502+0000","last_peered":"2024-03-11T22:25:18.043502+0000","last_clean":"2024-03-11T22:25:18.043502+0000","last_became_active":"2024-03-11T22:18:00.271764+0000","last_became_peered":"2024-03-11T22:18:00.271764+0000","last_unstale":"2024-03-11T22:25:18.043502+0000","last_undegraded":"2024-03-11T22:25:18.043502+0000","last_fullsized":"2024-03-11T22:25:18.043502+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:22:14.766775+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:22:14.766775+0000","last_clean_scrub_stamp":"2024-03-11T22:22:14.766775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"385'15375","reported_seq":15608,"reported_epoch":385,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-11T22:25:22.531838+0000","last_change":"2024-03-11T22:25:01.620534+0000","last_active":"2024-03-11T22:25:22.531838+0000","last_peered":"2024-03-11T22:25:22.531838+0000","last_clean":"2024-03-11T22:25:22.531838+0000","last_became_active":"2024-03-11T22:22:52.797873+0000","last_became_peered":"2024-03-11T22:22:52.797873+0000","last_unstale":"2024-03-11T22:25:22.531838+0000","last_undegraded":"2024-03-11T22:25:22.531838+0000","last_fullsized":"2024-03-11T22:25:22.531838+0000","mapping_epoch":348,"log_start":"381'10601","ondisk_log_start":"381'10601","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":3567,"log_size":4774,"log_dups_size":0,"ondisk_log_size":4774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 20s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306446336,"num_objects":4676,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11808,"num_write_kb":527488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"273'34","reported_seq":754,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.040420+0000","last_change":"2024-03-11T22:21:49.811476+0000","last_active":"2024-03-11T22:25:18.040420+0000","last_peered":"2024-03-11T22:25:18.040420+0000","last_clean":"2024-03-11T22:25:18.040420+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:25:18.040420+0000","last_undegraded":"2024-03-11T22:25:18.040420+0000","last_fullsized":"2024-03-11T22:25:18.040420+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:21:49.811214+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:21:49.811214+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":704,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.021641+0000","last_change":"2024-03-11T22:25:06.617067+0000","last_active":"2024-03-11T22:25:18.021641+0000","last_peered":"2024-03-11T22:25:18.021641+0000","last_clean":"2024-03-11T22:25:18.021641+0000","last_became_active":"2024-03-11T22:20:29.426224+0000","last_became_peered":"2024-03-11T22:20:29.426224+0000","last_unstale":"2024-03-11T22:25:18.021641+0000","last_undegraded":"2024-03-11T22:25:18.021641+0000","last_fullsized":"2024-03-11T22:25:18.021641+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:06.616932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:25:06.616932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:21.802971+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":611,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.552711+0000","last_change":"2024-03-11T22:24:00.708144+0000","last_active":"2024-03-11T22:25:18.552711+0000","last_peered":"2024-03-11T22:25:18.552711+0000","last_clean":"2024-03-11T22:25:18.552711+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:25:18.552711+0000","last_undegraded":"2024-03-11T22:25:18.552711+0000","last_fullsized":"2024-03-11T22:25:18.552711+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:24:00.708018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.708018+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:23.974631+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":680,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030006+0000","last_change":"2024-03-11T22:23:33.067353+0000","last_active":"2024-03-11T22:25:18.030006+0000","last_peered":"2024-03-11T22:25:18.030006+0000","last_clean":"2024-03-11T22:25:18.030006+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:25:18.030006+0000","last_undegraded":"2024-03-11T22:25:18.030006+0000","last_fullsized":"2024-03-11T22:25:18.030006+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:33.067236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:23:33.067236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":1000,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029828+0000","last_change":"2024-03-11T22:24:48.925640+0000","last_active":"2024-03-11T22:25:18.029828+0000","last_peered":"2024-03-11T22:25:18.029828+0000","last_clean":"2024-03-11T22:25:18.029828+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:25:18.029828+0000","last_undegraded":"2024-03-11T22:25:18.029828+0000","last_fullsized":"2024-03-11T22:25:18.029828+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:48.925477+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:48.925477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:02.846482+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"385'12202","reported_seq":12355,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:25.582752+0000","last_change":"2024-03-11T22:25:01.501408+0000","last_active":"2024-03-11T22:25:25.582752+0000","last_peered":"2024-03-11T22:25:25.582752+0000","last_clean":"2024-03-11T22:25:25.582752+0000","last_became_active":"2024-03-11T22:22:52.799804+0000","last_became_peered":"2024-03-11T22:22:52.799804+0000","last_unstale":"2024-03-11T22:25:25.582752+0000","last_undegraded":"2024-03-11T22:25:25.582752+0000","last_fullsized":"2024-03-11T22:25:25.582752+0000","mapping_epoch":348,"log_start":"371'7200","ondisk_log_start":"371'7200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10662","last_scrub_stamp":"2024-03-11T22:25:01.500954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:01.500954+0000","objects_scrubbed":6034,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:26.252693+0000","scrub_duration":1796,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290324480,"num_objects":4430,"num_object_clones":0,"num_object_copies":13290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12202,"num_write_kb":532224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":469,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022587+0000","last_change":"2024-03-11T22:22:53.846450+0000","last_active":"2024-03-11T22:25:18.022587+0000","last_peered":"2024-03-11T22:25:18.022587+0000","last_clean":"2024-03-11T22:25:18.022587+0000","last_became_active":"2024-03-11T22:20:29.438896+0000","last_became_peered":"2024-03-11T22:20:29.438896+0000","last_unstale":"2024-03-11T22:25:18.022587+0000","last_undegraded":"2024-03-11T22:25:18.022587+0000","last_fullsized":"2024-03-11T22:25:18.022587+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:53.846355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:22:53.846355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.704024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"273'30","reported_seq":782,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.021734+0000","last_change":"2024-03-11T22:23:05.635969+0000","last_active":"2024-03-11T22:25:18.021734+0000","last_peered":"2024-03-11T22:25:18.021734+0000","last_clean":"2024-03-11T22:25:18.021734+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:25:18.021734+0000","last_undegraded":"2024-03-11T22:25:18.021734+0000","last_fullsized":"2024-03-11T22:25:18.021734+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'30","last_scrub_stamp":"2024-03-11T22:23:05.635556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:05.635556+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.703997+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":704,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.031184+0000","last_change":"2024-03-11T22:23:57.680282+0000","last_active":"2024-03-11T22:25:18.031184+0000","last_peered":"2024-03-11T22:25:18.031184+0000","last_clean":"2024-03-11T22:25:18.031184+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:25:18.031184+0000","last_undegraded":"2024-03-11T22:25:18.031184+0000","last_fullsized":"2024-03-11T22:25:18.031184+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:57.680156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:57.680156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:24.838705+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":451,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.023052+0000","last_change":"2024-03-11T22:22:04.830996+0000","last_active":"2024-03-11T22:25:18.023052+0000","last_peered":"2024-03-11T22:25:18.023052+0000","last_clean":"2024-03-11T22:25:18.023052+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:25:18.023052+0000","last_undegraded":"2024-03-11T22:25:18.023052+0000","last_fullsized":"2024-03-11T22:25:18.023052+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:04.830875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:04.830875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.703820+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":961,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.031410+0000","last_change":"2024-03-11T22:25:05.774090+0000","last_active":"2024-03-11T22:25:18.031410+0000","last_peered":"2024-03-11T22:25:18.031410+0000","last_clean":"2024-03-11T22:25:18.031410+0000","last_became_active":"2024-03-11T22:18:20.240728+0000","last_became_peered":"2024-03-11T22:18:20.240728+0000","last_unstale":"2024-03-11T22:25:18.031410+0000","last_undegraded":"2024-03-11T22:25:18.031410+0000","last_fullsized":"2024-03-11T22:25:18.031410+0000","mapping_epoch":175,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:25:05.773963+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:25:05.773963+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.773963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:17.950615+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.f","version":"385'11999","reported_seq":12152,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:25.573510+0000","last_change":"2024-03-11T22:25:00.782159+0000","last_active":"2024-03-11T22:25:25.573510+0000","last_peered":"2024-03-11T22:25:25.573510+0000","last_clean":"2024-03-11T22:25:25.573510+0000","last_became_active":"2024-03-11T22:22:52.469064+0000","last_became_peered":"2024-03-11T22:22:52.469064+0000","last_unstale":"2024-03-11T22:25:25.573510+0000","last_undegraded":"2024-03-11T22:25:25.573510+0000","last_fullsized":"2024-03-11T22:25:25.573510+0000","mapping_epoch":348,"log_start":"371'6900","ondisk_log_start":"371'6900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10438","last_scrub_stamp":"2024-03-11T22:25:00.781894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:00.781894+0000","objects_scrubbed":6054,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:20.149816+0000","scrub_duration":2051,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289603584,"num_objects":4419,"num_object_clones":0,"num_object_copies":13257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11999,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":887,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022270+0000","last_change":"2024-03-11T22:23:07.693414+0000","last_active":"2024-03-11T22:25:18.022270+0000","last_peered":"2024-03-11T22:25:18.022270+0000","last_clean":"2024-03-11T22:25:18.022270+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:25:18.022270+0000","last_undegraded":"2024-03-11T22:25:18.022270+0000","last_fullsized":"2024-03-11T22:25:18.022270+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:23:07.693340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:07.693340+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.704055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"273'28","reported_seq":520,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029483+0000","last_change":"2024-03-11T22:24:44.976788+0000","last_active":"2024-03-11T22:25:18.029483+0000","last_peered":"2024-03-11T22:25:18.029483+0000","last_clean":"2024-03-11T22:25:18.029483+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:25:18.029483+0000","last_undegraded":"2024-03-11T22:25:18.029483+0000","last_fullsized":"2024-03-11T22:25:18.029483+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:24:44.976524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:44.976524+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:13.465935+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":693,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.021715+0000","last_change":"2024-03-11T22:22:56.087742+0000","last_active":"2024-03-11T22:25:18.021715+0000","last_peered":"2024-03-11T22:25:18.021715+0000","last_clean":"2024-03-11T22:25:18.021715+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:25:18.021715+0000","last_undegraded":"2024-03-11T22:25:18.021715+0000","last_fullsized":"2024-03-11T22:25:18.021715+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:22:56.087614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:22:56.087614+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":647,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.024344+0000","last_change":"2024-03-11T22:23:59.089317+0000","last_active":"2024-03-11T22:25:18.024344+0000","last_peered":"2024-03-11T22:25:18.024344+0000","last_clean":"2024-03-11T22:25:18.024344+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:25:18.024344+0000","last_undegraded":"2024-03-11T22:25:18.024344+0000","last_fullsized":"2024-03-11T22:25:18.024344+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:59.089227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:23:59.089227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.691932+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":1008,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022313+0000","last_change":"2024-03-11T22:23:39.755855+0000","last_active":"2024-03-11T22:25:18.022313+0000","last_peered":"2024-03-11T22:25:18.022313+0000","last_clean":"2024-03-11T22:25:18.022313+0000","last_became_active":"2024-03-11T22:20:32.293085+0000","last_became_peered":"2024-03-11T22:20:32.293085+0000","last_unstale":"2024-03-11T22:25:18.022313+0000","last_undegraded":"2024-03-11T22:25:18.022313+0000","last_fullsized":"2024-03-11T22:25:18.022313+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:39.755701+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:39.755701+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.704335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"385'11903","reported_seq":11988,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:22.509909+0000","last_change":"2024-03-11T22:22:52.799793+0000","last_active":"2024-03-11T22:25:22.509909+0000","last_peered":"2024-03-11T22:25:22.509909+0000","last_clean":"2024-03-11T22:25:22.509909+0000","last_became_active":"2024-03-11T22:22:52.799425+0000","last_became_peered":"2024-03-11T22:22:52.799425+0000","last_unstale":"2024-03-11T22:25:22.509909+0000","last_undegraded":"2024-03-11T22:25:22.509909+0000","last_fullsized":"2024-03-11T22:25:22.509909+0000","mapping_epoch":348,"log_start":"371'7100","ondisk_log_start":"371'7100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:23.734175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298123264,"num_objects":4549,"num_object_clones":0,"num_object_copies":13647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11903,"num_write_kb":526464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"378'9","reported_seq":904,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.551869+0000","last_change":"2024-03-11T22:24:54.792194+0000","last_active":"2024-03-11T22:25:18.551869+0000","last_peered":"2024-03-11T22:25:18.551869+0000","last_clean":"2024-03-11T22:25:18.551869+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:25:18.551869+0000","last_undegraded":"2024-03-11T22:25:18.551869+0000","last_fullsized":"2024-03-11T22:25:18.551869+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9","last_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_deep_scrub":"378'9","last_deep_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_clean_scrub_stamp":"2024-03-11T22:24:54.792049+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:07.965072+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"273'20","reported_seq":735,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.041534+0000","last_change":"2024-03-11T22:24:47.479305+0000","last_active":"2024-03-11T22:25:18.041534+0000","last_peered":"2024-03-11T22:25:18.041534+0000","last_clean":"2024-03-11T22:25:18.041534+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:25:18.041534+0000","last_undegraded":"2024-03-11T22:25:18.041534+0000","last_fullsized":"2024-03-11T22:25:18.041534+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:24:47.479158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:47.479158+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:09.501559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":701,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030280+0000","last_change":"2024-03-11T22:23:42.943711+0000","last_active":"2024-03-11T22:25:18.030280+0000","last_peered":"2024-03-11T22:25:18.030280+0000","last_clean":"2024-03-11T22:25:18.030280+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:25:18.030280+0000","last_undegraded":"2024-03-11T22:25:18.030280+0000","last_fullsized":"2024-03-11T22:25:18.030280+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:42.943546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:42.943546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":681,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.020795+0000","last_change":"2024-03-11T22:22:55.110611+0000","last_active":"2024-03-11T22:25:18.020795+0000","last_peered":"2024-03-11T22:25:18.020795+0000","last_clean":"2024-03-11T22:25:18.020795+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:25:18.020795+0000","last_undegraded":"2024-03-11T22:25:18.020795+0000","last_fullsized":"2024-03-11T22:25:18.020795+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:55.110461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:55.110461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":933,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.027717+0000","last_change":"2024-03-11T22:23:09.000258+0000","last_active":"2024-03-11T22:25:18.027717+0000","last_peered":"2024-03-11T22:25:18.027717+0000","last_clean":"2024-03-11T22:25:18.027717+0000","last_became_active":"2024-03-11T22:20:32.290512+0000","last_became_peered":"2024-03-11T22:20:32.290512+0000","last_unstale":"2024-03-11T22:25:18.027717+0000","last_undegraded":"2024-03-11T22:25:18.027717+0000","last_fullsized":"2024-03-11T22:25:18.027717+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:09.000091+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:09.000091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.691571+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"385'12190","reported_seq":12311,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:25.612819+0000","last_change":"2024-03-11T22:25:07.810616+0000","last_active":"2024-03-11T22:25:25.612819+0000","last_peered":"2024-03-11T22:25:25.612819+0000","last_clean":"2024-03-11T22:25:25.612819+0000","last_became_active":"2024-03-11T22:22:52.797795+0000","last_became_peered":"2024-03-11T22:22:52.797795+0000","last_unstale":"2024-03-11T22:25:25.612819+0000","last_undegraded":"2024-03-11T22:25:25.612819+0000","last_fullsized":"2024-03-11T22:25:25.612819+0000","mapping_epoch":348,"log_start":"371'7100","ondisk_log_start":"371'7100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":3299,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290652184,"num_objects":4436,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12190,"num_write_kb":531969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":709,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.024675+0000","last_change":"2024-03-11T22:24:00.941070+0000","last_active":"2024-03-11T22:25:18.024675+0000","last_peered":"2024-03-11T22:25:18.024675+0000","last_clean":"2024-03-11T22:25:18.024675+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:25:18.024675+0000","last_undegraded":"2024-03-11T22:25:18.024675+0000","last_fullsized":"2024-03-11T22:25:18.024675+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:00.940803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.940803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:24.733586+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"273'29","reported_seq":740,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.023310+0000","last_change":"2024-03-11T22:23:37.154525+0000","last_active":"2024-03-11T22:25:18.023310+0000","last_peered":"2024-03-11T22:25:18.023310+0000","last_clean":"2024-03-11T22:25:18.023310+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:25:18.023310+0000","last_undegraded":"2024-03-11T22:25:18.023310+0000","last_fullsized":"2024-03-11T22:25:18.023310+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'29","last_scrub_stamp":"2024-03-11T22:23:37.154389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:37.154389+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":701,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.040921+0000","last_change":"2024-03-11T22:23:04.706330+0000","last_active":"2024-03-11T22:25:18.040921+0000","last_peered":"2024-03-11T22:25:18.040921+0000","last_clean":"2024-03-11T22:25:18.040921+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:25:18.040921+0000","last_undegraded":"2024-03-11T22:25:18.040921+0000","last_fullsized":"2024-03-11T22:25:18.040921+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:04.706082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:04.706082+0000","last_clean_scrub_stamp":"2024-03-11T22:23:04.706082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":689,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.038947+0000","last_change":"2024-03-11T22:22:58.736160+0000","last_active":"2024-03-11T22:25:18.038947+0000","last_peered":"2024-03-11T22:25:18.038947+0000","last_clean":"2024-03-11T22:25:18.038947+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:25:18.038947+0000","last_undegraded":"2024-03-11T22:25:18.038947+0000","last_fullsized":"2024-03-11T22:25:18.038947+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:58.736057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:58.736057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:24.498590+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":908,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.028160+0000","last_change":"2024-03-11T22:23:47.783004+0000","last_active":"2024-03-11T22:25:18.028160+0000","last_peered":"2024-03-11T22:25:18.028160+0000","last_clean":"2024-03-11T22:25:18.028160+0000","last_became_active":"2024-03-11T22:18:17.238247+0000","last_became_peered":"2024-03-11T22:18:17.238247+0000","last_unstale":"2024-03-11T22:25:18.028160+0000","last_undegraded":"2024-03-11T22:25:18.028160+0000","last_fullsized":"2024-03-11T22:25:18.028160+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:47.782891+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:47.782891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.691797+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"385'11906","reported_seq":12059,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:22.502575+0000","last_change":"2024-03-11T22:25:05.585418+0000","last_active":"2024-03-11T22:25:22.502575+0000","last_peered":"2024-03-11T22:25:22.502575+0000","last_clean":"2024-03-11T22:25:22.502575+0000","last_became_active":"2024-03-11T22:22:52.496115+0000","last_became_peered":"2024-03-11T22:22:52.496115+0000","last_unstale":"2024-03-11T22:25:22.502575+0000","last_undegraded":"2024-03-11T22:25:22.502575+0000","last_fullsized":"2024-03-11T22:25:22.502575+0000","mapping_epoch":348,"log_start":"371'7100","ondisk_log_start":"371'7100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'10743","last_scrub_stamp":"2024-03-11T22:25:05.585079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.585079+0000","objects_scrubbed":5803,"log_size":4806,"log_dups_size":0,"ondisk_log_size":4806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:06.494058+0000","scrub_duration":1911,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299892736,"num_objects":4576,"num_object_clones":0,"num_object_copies":13728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":527424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":692,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030720+0000","last_change":"2024-03-11T22:24:39.906388+0000","last_active":"2024-03-11T22:25:18.030720+0000","last_peered":"2024-03-11T22:25:18.030720+0000","last_clean":"2024-03-11T22:25:18.030720+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:25:18.030720+0000","last_undegraded":"2024-03-11T22:25:18.030720+0000","last_fullsized":"2024-03-11T22:25:18.030720+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:39.906243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.906243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:00.278120+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"273'24","reported_seq":732,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.021622+0000","last_change":"2024-03-11T22:23:41.208818+0000","last_active":"2024-03-11T22:25:18.021622+0000","last_peered":"2024-03-11T22:25:18.021622+0000","last_clean":"2024-03-11T22:25:18.021622+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:25:18.021622+0000","last_undegraded":"2024-03-11T22:25:18.021622+0000","last_fullsized":"2024-03-11T22:25:18.021622+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'24","last_scrub_stamp":"2024-03-11T22:23:41.208655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:41.208655+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":1689,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:20.351469+0000","last_change":"2024-03-11T22:22:19.742008+0000","last_active":"2024-03-11T22:25:20.351469+0000","last_peered":"2024-03-11T22:25:20.351469+0000","last_clean":"2024-03-11T22:25:20.351469+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:25:20.351469+0000","last_undegraded":"2024-03-11T22:25:20.351469+0000","last_fullsized":"2024-03-11T22:25:20.351469+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:22:19.741884+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:22:19.741884+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:24.498501+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":973,"num_read_kb":968,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":701,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.025065+0000","last_change":"2024-03-11T22:24:46.870653+0000","last_active":"2024-03-11T22:25:18.025065+0000","last_peered":"2024-03-11T22:25:18.025065+0000","last_clean":"2024-03-11T22:25:18.025065+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:25:18.025065+0000","last_undegraded":"2024-03-11T22:25:18.025065+0000","last_fullsized":"2024-03-11T22:25:18.025065+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:46.870527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.870527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:51.216116+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":697,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.039596+0000","last_change":"2024-03-11T22:22:22.750641+0000","last_active":"2024-03-11T22:25:18.039596+0000","last_peered":"2024-03-11T22:25:18.039596+0000","last_clean":"2024-03-11T22:25:18.039596+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:25:18.039596+0000","last_undegraded":"2024-03-11T22:25:18.039596+0000","last_fullsized":"2024-03-11T22:25:18.039596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:22.750560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:22.750560+0000","last_clean_scrub_stamp":"2024-03-11T22:22:22.750560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:24.498454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":998,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.044551+0000","last_change":"2024-03-11T22:24:45.428664+0000","last_active":"2024-03-11T22:25:18.044551+0000","last_peered":"2024-03-11T22:25:18.044551+0000","last_clean":"2024-03-11T22:25:18.044551+0000","last_became_active":"2024-03-11T22:14:38.164101+0000","last_became_peered":"2024-03-11T22:14:38.164101+0000","last_unstale":"2024-03-11T22:25:18.044551+0000","last_undegraded":"2024-03-11T22:25:18.044551+0000","last_fullsized":"2024-03-11T22:25:18.044551+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:45.428590+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:45.428590+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:04.484198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"385'12087","reported_seq":12172,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:25.582211+0000","last_change":"2024-03-11T22:22:52.472336+0000","last_active":"2024-03-11T22:25:25.582211+0000","last_peered":"2024-03-11T22:25:25.582211+0000","last_clean":"2024-03-11T22:25:25.582211+0000","last_became_active":"2024-03-11T22:22:52.472068+0000","last_became_peered":"2024-03-11T22:22:52.472068+0000","last_unstale":"2024-03-11T22:25:25.582211+0000","last_undegraded":"2024-03-11T22:25:25.582211+0000","last_fullsized":"2024-03-11T22:25:25.582211+0000","mapping_epoch":348,"log_start":"371'7000","ondisk_log_start":"371'7000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288555008,"num_objects":4403,"num_object_clones":0,"num_object_copies":13209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":527680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":667,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.031078+0000","last_change":"2024-03-11T22:24:45.777817+0000","last_active":"2024-03-11T22:25:18.031078+0000","last_peered":"2024-03-11T22:25:18.031078+0000","last_clean":"2024-03-11T22:25:18.031078+0000","last_became_active":"2024-03-11T22:20:29.420497+0000","last_became_peered":"2024-03-11T22:20:29.420497+0000","last_unstale":"2024-03-11T22:25:18.031078+0000","last_undegraded":"2024-03-11T22:25:18.031078+0000","last_fullsized":"2024-03-11T22:25:18.031078+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:45.777703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:24:45.777703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:54.742850+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"273'28","reported_seq":956,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:20.611213+0000","last_change":"2024-03-11T22:24:42.729950+0000","last_active":"2024-03-11T22:25:20.611213+0000","last_peered":"2024-03-11T22:25:20.611213+0000","last_clean":"2024-03-11T22:25:20.611213+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:25:20.611213+0000","last_undegraded":"2024-03-11T22:25:20.611213+0000","last_fullsized":"2024-03-11T22:25:20.611213+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:24:42.729799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:42.729799+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.455019+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":693,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022024+0000","last_change":"2024-03-11T22:24:47.653085+0000","last_active":"2024-03-11T22:25:18.022024+0000","last_peered":"2024-03-11T22:25:18.022024+0000","last_clean":"2024-03-11T22:25:18.022024+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:25:18.022024+0000","last_undegraded":"2024-03-11T22:25:18.022024+0000","last_fullsized":"2024-03-11T22:25:18.022024+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:47.652970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:24:47.652970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.393586+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":1016,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.020895+0000","last_change":"2024-03-11T22:24:42.773278+0000","last_active":"2024-03-11T22:25:18.020895+0000","last_peered":"2024-03-11T22:25:18.020895+0000","last_clean":"2024-03-11T22:25:18.020895+0000","last_became_active":"2024-03-11T22:20:32.293740+0000","last_became_peered":"2024-03-11T22:20:32.293740+0000","last_unstale":"2024-03-11T22:25:18.020895+0000","last_undegraded":"2024-03-11T22:25:18.020895+0000","last_fullsized":"2024-03-11T22:25:18.020895+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:42.773024+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:24:42.773024+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:47.036511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":688,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030551+0000","last_change":"2024-03-11T22:24:54.991281+0000","last_active":"2024-03-11T22:25:18.030551+0000","last_peered":"2024-03-11T22:25:18.030551+0000","last_clean":"2024-03-11T22:25:18.030551+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:25:18.030551+0000","last_undegraded":"2024-03-11T22:25:18.030551+0000","last_fullsized":"2024-03-11T22:25:18.030551+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:54.991081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:54.991081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:59.396448+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"385'12261","reported_seq":12415,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:25.649484+0000","last_change":"2024-03-11T22:25:02.575828+0000","last_active":"2024-03-11T22:25:25.649484+0000","last_peered":"2024-03-11T22:25:25.649484+0000","last_clean":"2024-03-11T22:25:25.649484+0000","last_became_active":"2024-03-11T22:22:52.496328+0000","last_became_peered":"2024-03-11T22:22:52.496328+0000","last_unstale":"2024-03-11T22:25:25.649484+0000","last_undegraded":"2024-03-11T22:25:25.649484+0000","last_fullsized":"2024-03-11T22:25:25.649484+0000","mapping_epoch":348,"log_start":"372'7400","ondisk_log_start":"372'7400","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'10701","last_scrub_stamp":"2024-03-11T22:25:02.575600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:02.575600+0000","objects_scrubbed":6024,"log_size":4861,"log_dups_size":0,"ondisk_log_size":4861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:02.718025+0000","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289865728,"num_objects":4423,"num_object_clones":0,"num_object_copies":13269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12261,"num_write_kb":533888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":716,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.031684+0000","last_change":"2024-03-11T22:24:48.727176+0000","last_active":"2024-03-11T22:25:18.031684+0000","last_peered":"2024-03-11T22:25:18.031684+0000","last_clean":"2024-03-11T22:25:18.031684+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:25:18.031684+0000","last_undegraded":"2024-03-11T22:25:18.031684+0000","last_fullsized":"2024-03-11T22:25:18.031684+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:48.727061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:48.727061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:17.884566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"273'17","reported_seq":737,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.040565+0000","last_change":"2024-03-11T22:24:41.396924+0000","last_active":"2024-03-11T22:25:18.040565+0000","last_peered":"2024-03-11T22:25:18.040565+0000","last_clean":"2024-03-11T22:25:18.040565+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:25:18.040565+0000","last_undegraded":"2024-03-11T22:25:18.040565+0000","last_fullsized":"2024-03-11T22:25:18.040565+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'17","last_scrub_stamp":"2024-03-11T22:24:41.396754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.396754+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:02.581690+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":734,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.035246+0000","last_change":"2024-03-11T22:23:45.364864+0000","last_active":"2024-03-11T22:25:18.035246+0000","last_peered":"2024-03-11T22:25:18.035246+0000","last_clean":"2024-03-11T22:25:18.035246+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:25:18.035246+0000","last_undegraded":"2024-03-11T22:25:18.035246+0000","last_fullsized":"2024-03-11T22:25:18.035246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:45.364724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:23:45.364724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:20.675368+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":709,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.040444+0000","last_change":"2024-03-11T22:23:58.411257+0000","last_active":"2024-03-11T22:25:18.040444+0000","last_peered":"2024-03-11T22:25:18.040444+0000","last_clean":"2024-03-11T22:25:18.040444+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:25:18.040444+0000","last_undegraded":"2024-03-11T22:25:18.040444+0000","last_fullsized":"2024-03-11T22:25:18.040444+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:58.411052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.411052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":647,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.025246+0000","last_change":"2024-03-11T22:24:38.834963+0000","last_active":"2024-03-11T22:25:18.025246+0000","last_peered":"2024-03-11T22:25:18.025246+0000","last_clean":"2024-03-11T22:25:18.025246+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:25:18.025246+0000","last_undegraded":"2024-03-11T22:25:18.025246+0000","last_fullsized":"2024-03-11T22:25:18.025246+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:38.834828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:24:38.834828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.525769+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":1015,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.032128+0000","last_change":"2024-03-11T22:23:11.030109+0000","last_active":"2024-03-11T22:25:18.032128+0000","last_peered":"2024-03-11T22:25:18.032128+0000","last_clean":"2024-03-11T22:25:18.032128+0000","last_became_active":"2024-03-11T22:20:32.295175+0000","last_became_peered":"2024-03-11T22:20:32.295175+0000","last_unstale":"2024-03-11T22:25:18.032128+0000","last_undegraded":"2024-03-11T22:25:18.032128+0000","last_fullsized":"2024-03-11T22:25:18.032128+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:11.029981+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:11.029981+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"385'12186","reported_seq":12337,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:25.572337+0000","last_change":"2024-03-11T22:25:05.305867+0000","last_active":"2024-03-11T22:25:25.572337+0000","last_peered":"2024-03-11T22:25:25.572337+0000","last_clean":"2024-03-11T22:25:25.572337+0000","last_became_active":"2024-03-11T22:22:52.467125+0000","last_became_peered":"2024-03-11T22:22:52.467125+0000","last_unstale":"2024-03-11T22:25:25.572337+0000","last_undegraded":"2024-03-11T22:25:25.572337+0000","last_fullsized":"2024-03-11T22:25:25.572337+0000","mapping_epoch":348,"log_start":"371'7100","ondisk_log_start":"371'7100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'10771","last_scrub_stamp":"2024-03-11T22:25:05.305569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.305569+0000","objects_scrubbed":5888,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:07.737385+0000","scrub_duration":1552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290979840,"num_objects":4440,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12186,"num_write_kb":532032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":692,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.028809+0000","last_change":"2024-03-11T22:24:01.031042+0000","last_active":"2024-03-11T22:25:18.028809+0000","last_peered":"2024-03-11T22:25:18.028809+0000","last_clean":"2024-03-11T22:25:18.028809+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:25:18.028809+0000","last_undegraded":"2024-03-11T22:25:18.028809+0000","last_fullsized":"2024-03-11T22:25:18.028809+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:01.030925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:01.030925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.663872+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"273'40","reported_seq":973,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.041067+0000","last_change":"2024-03-11T22:23:57.429937+0000","last_active":"2024-03-11T22:25:18.041067+0000","last_peered":"2024-03-11T22:25:18.041067+0000","last_clean":"2024-03-11T22:25:18.041067+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:25:18.041067+0000","last_undegraded":"2024-03-11T22:25:18.041067+0000","last_fullsized":"2024-03-11T22:25:18.041067+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'40","last_scrub_stamp":"2024-03-11T22:23:57.429825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:57.429825+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:18.655119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":460,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.023195+0000","last_change":"2024-03-11T22:23:56.935550+0000","last_active":"2024-03-11T22:25:18.023195+0000","last_peered":"2024-03-11T22:25:18.023195+0000","last_clean":"2024-03-11T22:25:18.023195+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:25:18.023195+0000","last_undegraded":"2024-03-11T22:25:18.023195+0000","last_fullsized":"2024-03-11T22:25:18.023195+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:56.934997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:56.934997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:23.734509+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":713,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.031979+0000","last_change":"2024-03-11T22:25:05.938232+0000","last_active":"2024-03-11T22:25:18.031979+0000","last_peered":"2024-03-11T22:25:18.031979+0000","last_clean":"2024-03-11T22:25:18.031979+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:25:18.031979+0000","last_undegraded":"2024-03-11T22:25:18.031979+0000","last_fullsized":"2024-03-11T22:25:18.031979+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:25:05.938003+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.938003+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:25.242523+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":684,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030670+0000","last_change":"2024-03-11T22:23:03.679822+0000","last_active":"2024-03-11T22:25:18.030670+0000","last_peered":"2024-03-11T22:25:18.030670+0000","last_clean":"2024-03-11T22:25:18.030670+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:25:18.030670+0000","last_undegraded":"2024-03-11T22:25:18.030670+0000","last_fullsized":"2024-03-11T22:25:18.030670+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:03.679700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:23:03.679700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.703925+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":997,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029318+0000","last_change":"2024-03-11T22:24:56.886620+0000","last_active":"2024-03-11T22:25:18.029318+0000","last_peered":"2024-03-11T22:25:18.029318+0000","last_clean":"2024-03-11T22:25:18.029318+0000","last_became_active":"2024-03-11T22:14:38.164859+0000","last_became_peered":"2024-03-11T22:14:38.164859+0000","last_unstale":"2024-03-11T22:25:18.029318+0000","last_undegraded":"2024-03-11T22:25:18.029318+0000","last_fullsized":"2024-03-11T22:25:18.029318+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:56.886500+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:56.886500+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:58.328021+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"385'12015","reported_seq":12173,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:26.021302+0000","last_change":"2024-03-11T22:24:55.556260+0000","last_active":"2024-03-11T22:25:26.021302+0000","last_peered":"2024-03-11T22:25:26.021302+0000","last_clean":"2024-03-11T22:25:26.021302+0000","last_became_active":"2024-03-11T22:22:52.469442+0000","last_became_peered":"2024-03-11T22:22:52.469442+0000","last_unstale":"2024-03-11T22:25:26.021302+0000","last_undegraded":"2024-03-11T22:25:26.021302+0000","last_fullsized":"2024-03-11T22:25:26.021302+0000","mapping_epoch":348,"log_start":"368'6000","ondisk_log_start":"368'6000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9999","last_scrub_stamp":"2024-03-11T22:24:55.555941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:24:55.555941+0000","objects_scrubbed":6582,"log_size":6015,"log_dups_size":0,"ondisk_log_size":6015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:00.936378+0000","scrub_duration":2782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292749312,"num_objects":4467,"num_object_clones":0,"num_object_copies":13401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12015,"num_write_kb":527424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"273'15","reported_seq":729,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.023572+0000","last_change":"2024-03-11T22:23:36.167248+0000","last_active":"2024-03-11T22:25:18.023572+0000","last_peered":"2024-03-11T22:25:18.023572+0000","last_clean":"2024-03-11T22:25:18.023572+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:25:18.023572+0000","last_undegraded":"2024-03-11T22:25:18.023572+0000","last_fullsized":"2024-03-11T22:25:18.023572+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:23:36.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:36.167121+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":700,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029630+0000","last_change":"2024-03-11T22:24:58.853739+0000","last_active":"2024-03-11T22:25:18.029630+0000","last_peered":"2024-03-11T22:25:18.029630+0000","last_clean":"2024-03-11T22:25:18.029630+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:25:18.029630+0000","last_undegraded":"2024-03-11T22:25:18.029630+0000","last_fullsized":"2024-03-11T22:25:18.029630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:58.853630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:58.853630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:23.082251+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":716,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030201+0000","last_change":"2024-03-11T22:24:00.422907+0000","last_active":"2024-03-11T22:25:18.030201+0000","last_peered":"2024-03-11T22:25:18.030201+0000","last_clean":"2024-03-11T22:25:18.030201+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:25:18.030201+0000","last_undegraded":"2024-03-11T22:25:18.030201+0000","last_fullsized":"2024-03-11T22:25:18.030201+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:00.422761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.422761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.775572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":700,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.025258+0000","last_change":"2024-03-11T22:24:53.948958+0000","last_active":"2024-03-11T22:25:18.025258+0000","last_peered":"2024-03-11T22:25:18.025258+0000","last_clean":"2024-03-11T22:25:18.025258+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:25:18.025258+0000","last_undegraded":"2024-03-11T22:25:18.025258+0000","last_fullsized":"2024-03-11T22:25:18.025258+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:53.948853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.948853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.727979+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"93'130","reported_seq":842,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029619+0000","last_change":"2024-03-11T22:23:32.711840+0000","last_active":"2024-03-11T22:25:18.029619+0000","last_peered":"2024-03-11T22:25:18.029619+0000","last_clean":"2024-03-11T22:25:18.029619+0000","last_became_active":"2024-03-11T22:20:32.292676+0000","last_became_peered":"2024-03-11T22:20:32.292676+0000","last_unstale":"2024-03-11T22:25:18.029619+0000","last_undegraded":"2024-03-11T22:25:18.029619+0000","last_fullsized":"2024-03-11T22:25:18.029619+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:32.711693+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:32.711693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.704298+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"385'11726","reported_seq":11811,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:22.535107+0000","last_change":"2024-03-11T22:22:52.800885+0000","last_active":"2024-03-11T22:25:22.535107+0000","last_peered":"2024-03-11T22:25:22.535107+0000","last_clean":"2024-03-11T22:25:22.535107+0000","last_became_active":"2024-03-11T22:22:52.800526+0000","last_became_peered":"2024-03-11T22:22:52.800526+0000","last_unstale":"2024-03-11T22:25:22.535107+0000","last_undegraded":"2024-03-11T22:25:22.535107+0000","last_fullsized":"2024-03-11T22:25:22.535107+0000","mapping_epoch":348,"log_start":"370'6900","ondisk_log_start":"370'6900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":4826,"log_dups_size":0,"ondisk_log_size":4826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:23.734087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303824896,"num_objects":4636,"num_object_clones":0,"num_object_copies":13908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11726,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"273'35","reported_seq":819,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.024622+0000","last_change":"2024-03-11T22:24:46.752170+0000","last_active":"2024-03-11T22:25:18.024622+0000","last_peered":"2024-03-11T22:25:18.024622+0000","last_clean":"2024-03-11T22:25:18.024622+0000","last_became_active":"2024-03-11T22:20:29.418041+0000","last_became_peered":"2024-03-11T22:20:29.418041+0000","last_unstale":"2024-03-11T22:25:18.024622+0000","last_undegraded":"2024-03-11T22:25:18.024622+0000","last_fullsized":"2024-03-11T22:25:18.024622+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'35","last_scrub_stamp":"2024-03-11T22:24:46.751969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.751969+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:58.638998+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":643,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030367+0000","last_change":"2024-03-11T22:23:10.963581+0000","last_active":"2024-03-11T22:25:18.030367+0000","last_peered":"2024-03-11T22:25:18.030367+0000","last_clean":"2024-03-11T22:25:18.030367+0000","last_became_active":"2024-03-11T22:20:29.413664+0000","last_became_peered":"2024-03-11T22:20:29.413664+0000","last_unstale":"2024-03-11T22:25:18.030367+0000","last_undegraded":"2024-03-11T22:25:18.030367+0000","last_fullsized":"2024-03-11T22:25:18.030367+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:10.963448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:10.963448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":708,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.032280+0000","last_change":"2024-03-11T22:23:40.405746+0000","last_active":"2024-03-11T22:25:18.032280+0000","last_peered":"2024-03-11T22:25:18.032280+0000","last_clean":"2024-03-11T22:25:18.032280+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:25:18.032280+0000","last_undegraded":"2024-03-11T22:25:18.032280+0000","last_fullsized":"2024-03-11T22:25:18.032280+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:40.405626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:23:40.405626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:20.675394+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":666,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.031393+0000","last_change":"2024-03-11T22:23:51.364137+0000","last_active":"2024-03-11T22:25:18.031393+0000","last_peered":"2024-03-11T22:25:18.031393+0000","last_clean":"2024-03-11T22:25:18.031393+0000","last_became_active":"2024-03-11T22:20:29.415565+0000","last_became_peered":"2024-03-11T22:20:29.415565+0000","last_unstale":"2024-03-11T22:25:18.031393+0000","last_undegraded":"2024-03-11T22:25:18.031393+0000","last_fullsized":"2024-03-11T22:25:18.031393+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:51.362666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:20.931122+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.362666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:20.675341+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"93'130","reported_seq":893,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.027469+0000","last_change":"2024-03-11T22:23:02.040555+0000","last_active":"2024-03-11T22:25:18.027469+0000","last_peered":"2024-03-11T22:25:18.027469+0000","last_clean":"2024-03-11T22:25:18.027469+0000","last_became_active":"2024-03-11T22:14:43.195433+0000","last_became_peered":"2024-03-11T22:14:43.195433+0000","last_unstale":"2024-03-11T22:25:18.027469+0000","last_undegraded":"2024-03-11T22:25:18.027469+0000","last_fullsized":"2024-03-11T22:25:18.027469+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:02.040354+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:02.040354+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.691625+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"385'12001","reported_seq":12091,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:25.573405+0000","last_change":"2024-03-11T22:25:07.811931+0000","last_active":"2024-03-11T22:25:25.573405+0000","last_peered":"2024-03-11T22:25:25.573405+0000","last_clean":"2024-03-11T22:25:25.573405+0000","last_became_active":"2024-03-11T22:22:52.465673+0000","last_became_peered":"2024-03-11T22:22:52.465673+0000","last_unstale":"2024-03-11T22:25:25.573405+0000","last_undegraded":"2024-03-11T22:25:25.573405+0000","last_fullsized":"2024-03-11T22:25:25.573405+0000","mapping_epoch":348,"log_start":"371'7000","ondisk_log_start":"371'7000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":199,"log_size":5001,"log_dups_size":0,"ondisk_log_size":5001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288948224,"num_objects":4409,"num_object_clones":0,"num_object_copies":13227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12001,"num_write_kb":525120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":716,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.031120+0000","last_change":"2024-03-11T22:24:56.704048+0000","last_active":"2024-03-11T22:25:18.031120+0000","last_peered":"2024-03-11T22:25:18.031120+0000","last_clean":"2024-03-11T22:25:18.031120+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:25:18.031120+0000","last_undegraded":"2024-03-11T22:25:18.031120+0000","last_fullsized":"2024-03-11T22:25:18.031120+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:56.703933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:24:56.703933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:24.261508+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"273'18","reported_seq":516,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029695+0000","last_change":"2024-03-11T22:23:46.897451+0000","last_active":"2024-03-11T22:25:18.029695+0000","last_peered":"2024-03-11T22:25:18.029695+0000","last_clean":"2024-03-11T22:25:18.029695+0000","last_became_active":"2024-03-11T22:14:52.247400+0000","last_became_peered":"2024-03-11T22:14:52.247400+0000","last_unstale":"2024-03-11T22:25:18.029695+0000","last_undegraded":"2024-03-11T22:25:18.029695+0000","last_fullsized":"2024-03-11T22:25:18.029695+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:23:46.897161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.897161+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":482,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.041253+0000","last_change":"2024-03-11T22:24:53.435632+0000","last_active":"2024-03-11T22:25:18.041253+0000","last_peered":"2024-03-11T22:25:18.041253+0000","last_clean":"2024-03-11T22:25:18.041253+0000","last_became_active":"2024-03-11T22:20:29.416729+0000","last_became_peered":"2024-03-11T22:20:29.416729+0000","last_unstale":"2024-03-11T22:25:18.041253+0000","last_undegraded":"2024-03-11T22:25:18.041253+0000","last_fullsized":"2024-03-11T22:25:18.041253+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:53.435516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.435516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:08.499620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":694,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022754+0000","last_change":"2024-03-11T22:23:01.776787+0000","last_active":"2024-03-11T22:25:18.022754+0000","last_peered":"2024-03-11T22:25:18.022754+0000","last_clean":"2024-03-11T22:25:18.022754+0000","last_became_active":"2024-03-11T22:20:29.438413+0000","last_became_peered":"2024-03-11T22:20:29.438413+0000","last_unstale":"2024-03-11T22:25:18.022754+0000","last_undegraded":"2024-03-11T22:25:18.022754+0000","last_fullsized":"2024-03-11T22:25:18.022754+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:23:01.776660+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:23:01.776660+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.704115+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"93'130","reported_seq":1018,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.021342+0000","last_change":"2024-03-11T22:23:10.731794+0000","last_active":"2024-03-11T22:25:18.021342+0000","last_peered":"2024-03-11T22:25:18.021342+0000","last_clean":"2024-03-11T22:25:18.021342+0000","last_became_active":"2024-03-11T22:17:57.246241+0000","last_became_peered":"2024-03-11T22:17:57.246241+0000","last_unstale":"2024-03-11T22:25:18.021342+0000","last_undegraded":"2024-03-11T22:25:18.021342+0000","last_fullsized":"2024-03-11T22:25:18.021342+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:10.731483+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:10.731483+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.703957+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"385'12270","reported_seq":12355,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:25.628754+0000","last_change":"2024-03-11T22:22:52.798264+0000","last_active":"2024-03-11T22:25:25.628754+0000","last_peered":"2024-03-11T22:25:25.628754+0000","last_clean":"2024-03-11T22:25:25.628754+0000","last_became_active":"2024-03-11T22:22:52.797922+0000","last_became_peered":"2024-03-11T22:22:52.797922+0000","last_unstale":"2024-03-11T22:25:25.628754+0000","last_undegraded":"2024-03-11T22:25:25.628754+0000","last_fullsized":"2024-03-11T22:25:25.628754+0000","mapping_epoch":348,"log_start":"372'7400","ondisk_log_start":"372'7400","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":4870,"log_dups_size":0,"ondisk_log_size":4870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.992856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280887296,"num_objects":4286,"num_object_clones":0,"num_object_copies":12858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12270,"num_write_kb":529792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"310'11","reported_seq":871,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.551654+0000","last_change":"2024-03-11T22:23:56.945205+0000","last_active":"2024-03-11T22:25:18.551654+0000","last_peered":"2024-03-11T22:25:18.551654+0000","last_clean":"2024-03-11T22:25:18.551654+0000","last_became_active":"2024-03-11T22:20:29.418355+0000","last_became_peered":"2024-03-11T22:20:29.418355+0000","last_unstale":"2024-03-11T22:25:18.551654+0000","last_undegraded":"2024-03-11T22:25:18.551654+0000","last_fullsized":"2024-03-11T22:25:18.551654+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:23:56.945075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:56.945075+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:24.733648+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"273'34","reported_seq":790,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029115+0000","last_change":"2024-03-11T22:23:47.917460+0000","last_active":"2024-03-11T22:25:18.029115+0000","last_peered":"2024-03-11T22:25:18.029115+0000","last_clean":"2024-03-11T22:25:18.029115+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:25:18.029115+0000","last_undegraded":"2024-03-11T22:25:18.029115+0000","last_fullsized":"2024-03-11T22:25:18.029115+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:23:47.917308+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:23:47.917308+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":492,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022494+0000","last_change":"2024-03-11T22:23:38.153481+0000","last_active":"2024-03-11T22:25:18.022494+0000","last_peered":"2024-03-11T22:25:18.022494+0000","last_clean":"2024-03-11T22:25:18.022494+0000","last_became_active":"2024-03-11T22:20:29.415226+0000","last_became_peered":"2024-03-11T22:20:29.415226+0000","last_unstale":"2024-03-11T22:25:18.022494+0000","last_undegraded":"2024-03-11T22:25:18.022494+0000","last_fullsized":"2024-03-11T22:25:18.022494+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:23:38.153326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:38.153326+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":680,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.032223+0000","last_change":"2024-03-11T22:23:44.941526+0000","last_active":"2024-03-11T22:25:18.032223+0000","last_peered":"2024-03-11T22:25:18.032223+0000","last_clean":"2024-03-11T22:25:18.032223+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:25:18.032223+0000","last_undegraded":"2024-03-11T22:25:18.032223+0000","last_fullsized":"2024-03-11T22:25:18.032223+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:44.941284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:44.941284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"93'130","reported_seq":1035,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.039838+0000","last_change":"2024-03-11T22:23:44.564522+0000","last_active":"2024-03-11T22:25:18.039838+0000","last_peered":"2024-03-11T22:25:18.039838+0000","last_clean":"2024-03-11T22:25:18.039838+0000","last_became_active":"2024-03-11T22:20:32.295606+0000","last_became_peered":"2024-03-11T22:20:32.295606+0000","last_unstale":"2024-03-11T22:25:18.039838+0000","last_undegraded":"2024-03-11T22:25:18.039838+0000","last_fullsized":"2024-03-11T22:25:18.039838+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:44.564377+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:44.564377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"273'12","reported_seq":725,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.031479+0000","last_change":"2024-03-11T22:22:16.742232+0000","last_active":"2024-03-11T22:25:18.031479+0000","last_peered":"2024-03-11T22:25:18.031479+0000","last_clean":"2024-03-11T22:25:18.031479+0000","last_became_active":"2024-03-11T22:20:29.441599+0000","last_became_peered":"2024-03-11T22:20:29.441599+0000","last_unstale":"2024-03-11T22:25:18.031479+0000","last_undegraded":"2024-03-11T22:25:18.031479+0000","last_fullsized":"2024-03-11T22:25:18.031479+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'12","last_scrub_stamp":"2024-03-11T22:22:16.742100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:22:16.742100+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.703864+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":708,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.024421+0000","last_change":"2024-03-11T22:23:48.806202+0000","last_active":"2024-03-11T22:25:18.024421+0000","last_peered":"2024-03-11T22:25:18.024421+0000","last_clean":"2024-03-11T22:25:18.024421+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:25:18.024421+0000","last_undegraded":"2024-03-11T22:25:18.024421+0000","last_fullsized":"2024-03-11T22:25:18.024421+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.805960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.805960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.691907+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":691,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030135+0000","last_change":"2024-03-11T22:23:59.713751+0000","last_active":"2024-03-11T22:25:18.030135+0000","last_peered":"2024-03-11T22:25:18.030135+0000","last_clean":"2024-03-11T22:25:18.030135+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:25:18.030135+0000","last_undegraded":"2024-03-11T22:25:18.030135+0000","last_fullsized":"2024-03-11T22:25:18.030135+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:59.713626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:23:59.713626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":638,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.025339+0000","last_change":"2024-03-11T22:23:46.876978+0000","last_active":"2024-03-11T22:25:18.025339+0000","last_peered":"2024-03-11T22:25:18.025339+0000","last_clean":"2024-03-11T22:25:18.025339+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:25:18.025339+0000","last_undegraded":"2024-03-11T22:25:18.025339+0000","last_fullsized":"2024-03-11T22:25:18.025339+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:46.876749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.876749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:20.705719+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"93'130","reported_seq":1006,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029710+0000","last_change":"2024-03-11T22:24:52.878462+0000","last_active":"2024-03-11T22:25:18.029710+0000","last_peered":"2024-03-11T22:25:18.029710+0000","last_clean":"2024-03-11T22:25:18.029710+0000","last_became_active":"2024-03-11T22:14:43.194413+0000","last_became_peered":"2024-03-11T22:14:43.194413+0000","last_unstale":"2024-03-11T22:25:18.029710+0000","last_undegraded":"2024-03-11T22:25:18.029710+0000","last_fullsized":"2024-03-11T22:25:18.029710+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:52.878201+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:52.878201+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:04.489712+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"93'130","reported_seq":1005,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.043075+0000","last_change":"2024-03-11T22:23:51.445370+0000","last_active":"2024-03-11T22:25:18.043075+0000","last_peered":"2024-03-11T22:25:18.043075+0000","last_clean":"2024-03-11T22:25:18.043075+0000","last_became_active":"2024-03-11T22:14:43.198216+0000","last_became_peered":"2024-03-11T22:14:43.198216+0000","last_unstale":"2024-03-11T22:25:18.043075+0000","last_undegraded":"2024-03-11T22:25:18.043075+0000","last_fullsized":"2024-03-11T22:25:18.043075+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:51.445038+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:23:51.445038+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.445038+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"273'19","reported_seq":756,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022621+0000","last_change":"2024-03-11T22:23:53.926458+0000","last_active":"2024-03-11T22:25:18.022621+0000","last_peered":"2024-03-11T22:25:18.022621+0000","last_clean":"2024-03-11T22:25:18.022621+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:25:18.022621+0000","last_undegraded":"2024-03-11T22:25:18.022621+0000","last_fullsized":"2024-03-11T22:25:18.022621+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_clean_scrub_stamp":"2024-03-11T22:23:53.926316+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:18.640318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":470,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.023609+0000","last_change":"2024-03-11T22:23:35.792737+0000","last_active":"2024-03-11T22:25:18.023609+0000","last_peered":"2024-03-11T22:25:18.023609+0000","last_clean":"2024-03-11T22:25:18.023609+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:25:18.023609+0000","last_undegraded":"2024-03-11T22:25:18.023609+0000","last_fullsized":"2024-03-11T22:25:18.023609+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:35.792609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:35.792609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.691854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":695,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.031388+0000","last_change":"2024-03-11T22:24:51.687892+0000","last_active":"2024-03-11T22:25:18.031388+0000","last_peered":"2024-03-11T22:25:18.031388+0000","last_clean":"2024-03-11T22:25:18.031388+0000","last_became_active":"2024-03-11T22:20:29.433005+0000","last_became_peered":"2024-03-11T22:20:29.433005+0000","last_unstale":"2024-03-11T22:25:18.031388+0000","last_undegraded":"2024-03-11T22:25:18.031388+0000","last_fullsized":"2024-03-11T22:25:18.031388+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:51.687767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:51.687767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:20.042381+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":712,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030985+0000","last_change":"2024-03-11T22:23:57.498120+0000","last_active":"2024-03-11T22:25:18.030985+0000","last_peered":"2024-03-11T22:25:18.030985+0000","last_clean":"2024-03-11T22:25:18.030985+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:25:18.030985+0000","last_undegraded":"2024-03-11T22:25:18.030985+0000","last_fullsized":"2024-03-11T22:25:18.030985+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:57.498047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:23:57.498047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:20.675482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"273'19","reported_seq":724,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030390+0000","last_change":"2024-03-11T22:23:37.882805+0000","last_active":"2024-03-11T22:25:18.030390+0000","last_peered":"2024-03-11T22:25:18.030390+0000","last_clean":"2024-03-11T22:25:18.030390+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:25:18.030390+0000","last_undegraded":"2024-03-11T22:25:18.030390+0000","last_fullsized":"2024-03-11T22:25:18.030390+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:37.882661+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:23:37.882661+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":712,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.032026+0000","last_change":"2024-03-11T22:23:50.030718+0000","last_active":"2024-03-11T22:25:18.032026+0000","last_peered":"2024-03-11T22:25:18.032026+0000","last_clean":"2024-03-11T22:25:18.032026+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:25:18.032026+0000","last_undegraded":"2024-03-11T22:25:18.032026+0000","last_fullsized":"2024-03-11T22:25:18.032026+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:50.030272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:50.030272+0000","last_clean_scrub_stamp":"2024-03-11T22:23:50.030272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'11","reported_seq":929,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.025663+0000","last_change":"2024-03-11T22:24:53.646061+0000","last_active":"2024-03-11T22:25:18.025663+0000","last_peered":"2024-03-11T22:25:18.025663+0000","last_clean":"2024-03-11T22:25:18.025663+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:25:18.025663+0000","last_undegraded":"2024-03-11T22:25:18.025663+0000","last_fullsized":"2024-03-11T22:25:18.025663+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:24:53.645906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.645906+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:03.406173+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":474,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.036097+0000","last_change":"2024-03-11T22:23:48.490897+0000","last_active":"2024-03-11T22:25:18.036097+0000","last_peered":"2024-03-11T22:25:18.036097+0000","last_clean":"2024-03-11T22:25:18.036097+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:25:18.036097+0000","last_undegraded":"2024-03-11T22:25:18.036097+0000","last_fullsized":"2024-03-11T22:25:18.036097+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.490757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:48.490757+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.490757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:20.675315+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"93'130","reported_seq":991,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029577+0000","last_change":"2024-03-11T22:23:12.969623+0000","last_active":"2024-03-11T22:25:18.029577+0000","last_peered":"2024-03-11T22:25:18.029577+0000","last_clean":"2024-03-11T22:25:18.029577+0000","last_became_active":"2024-03-11T22:14:43.194350+0000","last_became_peered":"2024-03-11T22:14:43.194350+0000","last_unstale":"2024-03-11T22:25:18.029577+0000","last_undegraded":"2024-03-11T22:25:18.029577+0000","last_fullsized":"2024-03-11T22:25:18.029577+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:12.969362+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:23:12.969362+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"273'20","reported_seq":728,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.025146+0000","last_change":"2024-03-11T22:23:00.635612+0000","last_active":"2024-03-11T22:25:18.025146+0000","last_peered":"2024-03-11T22:25:18.025146+0000","last_clean":"2024-03-11T22:25:18.025146+0000","last_became_active":"2024-03-11T22:16:15.815814+0000","last_became_peered":"2024-03-11T22:16:15.815814+0000","last_unstale":"2024-03-11T22:25:18.025146+0000","last_undegraded":"2024-03-11T22:25:18.025146+0000","last_fullsized":"2024-03-11T22:25:18.025146+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:23:00.635505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:00.635505+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.703897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":498,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.032039+0000","last_change":"2024-03-11T22:24:49.777349+0000","last_active":"2024-03-11T22:25:18.032039+0000","last_peered":"2024-03-11T22:25:18.032039+0000","last_clean":"2024-03-11T22:25:18.032039+0000","last_became_active":"2024-03-11T22:20:29.415019+0000","last_became_peered":"2024-03-11T22:20:29.415019+0000","last_unstale":"2024-03-11T22:25:18.032039+0000","last_undegraded":"2024-03-11T22:25:18.032039+0000","last_fullsized":"2024-03-11T22:25:18.032039+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:49.777229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:49.777229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:00.825874+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":902,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.552719+0000","last_change":"2024-03-11T22:23:53.776480+0000","last_active":"2024-03-11T22:25:18.552719+0000","last_peered":"2024-03-11T22:25:18.552719+0000","last_clean":"2024-03-11T22:25:18.552719+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:25:18.552719+0000","last_undegraded":"2024-03-11T22:25:18.552719+0000","last_fullsized":"2024-03-11T22:25:18.552719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:23:53.776342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:53.776342+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:26.968305+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":691,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.021535+0000","last_change":"2024-03-11T22:24:45.964526+0000","last_active":"2024-03-11T22:25:18.021535+0000","last_peered":"2024-03-11T22:25:18.021535+0000","last_clean":"2024-03-11T22:25:18.021535+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:25:18.021535+0000","last_undegraded":"2024-03-11T22:25:18.021535+0000","last_fullsized":"2024-03-11T22:25:18.021535+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:45.964401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:45.964401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:07.337646+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"93'130","reported_seq":947,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.024774+0000","last_change":"2024-03-11T22:23:10.014311+0000","last_active":"2024-03-11T22:25:18.024774+0000","last_peered":"2024-03-11T22:25:18.024774+0000","last_clean":"2024-03-11T22:25:18.024774+0000","last_became_active":"2024-03-11T22:14:43.191297+0000","last_became_peered":"2024-03-11T22:14:43.191297+0000","last_unstale":"2024-03-11T22:25:18.024774+0000","last_undegraded":"2024-03-11T22:25:18.024774+0000","last_fullsized":"2024-03-11T22:25:18.024774+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:10.014235+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:10.014235+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"273'19","reported_seq":516,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.024255+0000","last_change":"2024-03-11T22:23:45.819413+0000","last_active":"2024-03-11T22:25:18.024255+0000","last_peered":"2024-03-11T22:25:18.024255+0000","last_clean":"2024-03-11T22:25:18.024255+0000","last_became_active":"2024-03-11T22:14:52.239810+0000","last_became_peered":"2024-03-11T22:14:52.239810+0000","last_unstale":"2024-03-11T22:25:18.024255+0000","last_undegraded":"2024-03-11T22:25:18.024255+0000","last_fullsized":"2024-03-11T22:25:18.024255+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:45.818986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:45.818986+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.691682+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":495,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.023772+0000","last_change":"2024-03-11T22:23:44.776567+0000","last_active":"2024-03-11T22:25:18.023772+0000","last_peered":"2024-03-11T22:25:18.023772+0000","last_clean":"2024-03-11T22:25:18.023772+0000","last_became_active":"2024-03-11T22:20:29.420867+0000","last_became_peered":"2024-03-11T22:20:29.420867+0000","last_unstale":"2024-03-11T22:25:18.023772+0000","last_undegraded":"2024-03-11T22:25:18.023772+0000","last_fullsized":"2024-03-11T22:25:18.023772+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:23:44.776433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:44.776433+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.691722+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":692,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.041450+0000","last_change":"2024-03-11T22:21:51.817381+0000","last_active":"2024-03-11T22:25:18.041450+0000","last_peered":"2024-03-11T22:25:18.041450+0000","last_clean":"2024-03-11T22:25:18.041450+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:25:18.041450+0000","last_undegraded":"2024-03-11T22:25:18.041450+0000","last_fullsized":"2024-03-11T22:25:18.041450+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:51.817265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:21:51.817265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":672,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022196+0000","last_change":"2024-03-11T22:23:50.723665+0000","last_active":"2024-03-11T22:25:18.022196+0000","last_peered":"2024-03-11T22:25:18.022196+0000","last_clean":"2024-03-11T22:25:18.022196+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:25:18.022196+0000","last_undegraded":"2024-03-11T22:25:18.022196+0000","last_fullsized":"2024-03-11T22:25:18.022196+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:50.723528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:50.723528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.704395+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":725,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.041634+0000","last_change":"2024-03-11T22:21:58.841578+0000","last_active":"2024-03-11T22:25:18.041634+0000","last_peered":"2024-03-11T22:25:18.041634+0000","last_clean":"2024-03-11T22:25:18.041634+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:25:18.041634+0000","last_undegraded":"2024-03-11T22:25:18.041634+0000","last_fullsized":"2024-03-11T22:25:18.041634+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:21:58.841451+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:21:58.841451+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":635,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.028449+0000","last_change":"2024-03-11T22:24:41.612477+0000","last_active":"2024-03-11T22:25:18.028449+0000","last_peered":"2024-03-11T22:25:18.028449+0000","last_clean":"2024-03-11T22:25:18.028449+0000","last_became_active":"2024-03-11T22:20:29.422747+0000","last_became_peered":"2024-03-11T22:20:29.422747+0000","last_unstale":"2024-03-11T22:25:18.028449+0000","last_undegraded":"2024-03-11T22:25:18.028449+0000","last_fullsized":"2024-03-11T22:25:18.028449+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:41.612363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.612363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:56.508483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":684,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022565+0000","last_change":"2024-03-11T22:24:49.679683+0000","last_active":"2024-03-11T22:25:18.022565+0000","last_peered":"2024-03-11T22:25:18.022565+0000","last_clean":"2024-03-11T22:25:18.022565+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:25:18.022565+0000","last_undegraded":"2024-03-11T22:25:18.022565+0000","last_fullsized":"2024-03-11T22:25:18.022565+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:49.679560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:24:49.679560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:13.908754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":680,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.020679+0000","last_change":"2024-03-11T22:23:09.058314+0000","last_active":"2024-03-11T22:25:18.020679+0000","last_peered":"2024-03-11T22:25:18.020679+0000","last_clean":"2024-03-11T22:25:18.020679+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:25:18.020679+0000","last_undegraded":"2024-03-11T22:25:18.020679+0000","last_fullsized":"2024-03-11T22:25:18.020679+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:09.057970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:09.057970+0000","last_clean_scrub_stamp":"2024-03-11T22:23:09.057970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"93'130","reported_seq":1008,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030502+0000","last_change":"2024-03-11T22:23:41.379256+0000","last_active":"2024-03-11T22:25:18.030502+0000","last_peered":"2024-03-11T22:25:18.030502+0000","last_clean":"2024-03-11T22:25:18.030502+0000","last_became_active":"2024-03-11T22:14:43.194955+0000","last_became_peered":"2024-03-11T22:14:43.194955+0000","last_unstale":"2024-03-11T22:25:18.030502+0000","last_undegraded":"2024-03-11T22:25:18.030502+0000","last_fullsized":"2024-03-11T22:25:18.030502+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:41.379110+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:18:43.722467+0000","last_clean_scrub_stamp":"2024-03-11T22:23:41.379110+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:20.675288+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"274'20","reported_seq":748,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022636+0000","last_change":"2024-03-11T22:23:52.653463+0000","last_active":"2024-03-11T22:25:18.022636+0000","last_peered":"2024-03-11T22:25:18.022636+0000","last_clean":"2024-03-11T22:25:18.022636+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:25:18.022636+0000","last_undegraded":"2024-03-11T22:25:18.022636+0000","last_fullsized":"2024-03-11T22:25:18.022636+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'20","last_scrub_stamp":"2024-03-11T22:23:52.653309+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:23:52.653309+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:18.380983+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":712,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030264+0000","last_change":"2024-03-11T22:24:41.895990+0000","last_active":"2024-03-11T22:25:18.030264+0000","last_peered":"2024-03-11T22:25:18.030264+0000","last_clean":"2024-03-11T22:25:18.030264+0000","last_became_active":"2024-03-11T22:20:29.416792+0000","last_became_peered":"2024-03-11T22:20:29.416792+0000","last_unstale":"2024-03-11T22:25:18.030264+0000","last_undegraded":"2024-03-11T22:25:18.030264+0000","last_fullsized":"2024-03-11T22:25:18.030264+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:41.895742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.895742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:45.455063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":469,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.024088+0000","last_change":"2024-03-11T22:23:33.856940+0000","last_active":"2024-03-11T22:25:18.024088+0000","last_peered":"2024-03-11T22:25:18.024088+0000","last_clean":"2024-03-11T22:25:18.024088+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:25:18.024088+0000","last_undegraded":"2024-03-11T22:25:18.024088+0000","last_fullsized":"2024-03-11T22:25:18.024088+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:33.856676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:23:33.856676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.691882+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":680,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022870+0000","last_change":"2024-03-11T22:24:59.829935+0000","last_active":"2024-03-11T22:25:18.022870+0000","last_peered":"2024-03-11T22:25:18.022870+0000","last_clean":"2024-03-11T22:25:18.022870+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:25:18.022870+0000","last_undegraded":"2024-03-11T22:25:18.022870+0000","last_fullsized":"2024-03-11T22:25:18.022870+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:59.829811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:59.829811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:17.495056+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"273'8","reported_seq":712,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.021546+0000","last_change":"2024-03-11T22:23:59.693653+0000","last_active":"2024-03-11T22:25:18.021546+0000","last_peered":"2024-03-11T22:25:18.021546+0000","last_clean":"2024-03-11T22:25:18.021546+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:25:18.021546+0000","last_undegraded":"2024-03-11T22:25:18.021546+0000","last_fullsized":"2024-03-11T22:25:18.021546+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'8","last_scrub_stamp":"2024-03-11T22:23:59.693510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:59.693510+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.704368+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":691,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022290+0000","last_change":"2024-03-11T22:23:12.019597+0000","last_active":"2024-03-11T22:25:18.022290+0000","last_peered":"2024-03-11T22:25:18.022290+0000","last_clean":"2024-03-11T22:25:18.022290+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:25:18.022290+0000","last_undegraded":"2024-03-11T22:25:18.022290+0000","last_fullsized":"2024-03-11T22:25:18.022290+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:12.019468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:12.019468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":286,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.024052+0000","last_change":"2024-03-11T22:24:37.763829+0000","last_active":"2024-03-11T22:25:18.024052+0000","last_peered":"2024-03-11T22:25:18.024052+0000","last_clean":"2024-03-11T22:25:18.024052+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:25:18.024052+0000","last_undegraded":"2024-03-11T22:25:18.024052+0000","last_fullsized":"2024-03-11T22:25:18.024052+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.763709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.763709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.935899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":654,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.031363+0000","last_change":"2024-03-11T22:24:46.776804+0000","last_active":"2024-03-11T22:25:18.031363+0000","last_peered":"2024-03-11T22:25:18.031363+0000","last_clean":"2024-03-11T22:25:18.031363+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:25:18.031363+0000","last_undegraded":"2024-03-11T22:25:18.031363+0000","last_fullsized":"2024-03-11T22:25:18.031363+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:46.776600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.776600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:12.814680+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"273'19","reported_seq":736,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.027701+0000","last_change":"2024-03-11T22:23:42.850339+0000","last_active":"2024-03-11T22:25:18.027701+0000","last_peered":"2024-03-11T22:25:18.027701+0000","last_clean":"2024-03-11T22:25:18.027701+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:25:18.027701+0000","last_undegraded":"2024-03-11T22:25:18.027701+0000","last_fullsized":"2024-03-11T22:25:18.027701+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:42.850214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:42.850214+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.691827+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":691,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.021453+0000","last_change":"2024-03-11T22:24:38.611323+0000","last_active":"2024-03-11T22:25:18.021453+0000","last_peered":"2024-03-11T22:25:18.021453+0000","last_clean":"2024-03-11T22:25:18.021453+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:25:18.021453+0000","last_undegraded":"2024-03-11T22:25:18.021453+0000","last_fullsized":"2024-03-11T22:25:18.021453+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:38.610924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:38.610924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:03.792615+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":693,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022760+0000","last_change":"2024-03-11T22:23:46.048359+0000","last_active":"2024-03-11T22:25:18.022760+0000","last_peered":"2024-03-11T22:25:18.022760+0000","last_clean":"2024-03-11T22:25:18.022760+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:25:18.022760+0000","last_undegraded":"2024-03-11T22:25:18.022760+0000","last_fullsized":"2024-03-11T22:25:18.022760+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:46.048254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.048254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":695,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.040311+0000","last_change":"2024-03-11T22:25:05.513523+0000","last_active":"2024-03-11T22:25:18.040311+0000","last_peered":"2024-03-11T22:25:18.040311+0000","last_clean":"2024-03-11T22:25:18.040311+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:25:18.040311+0000","last_undegraded":"2024-03-11T22:25:18.040311+0000","last_fullsized":"2024-03-11T22:25:18.040311+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:05.513210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.513210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:06.854659+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"273'20","reported_seq":755,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.021806+0000","last_change":"2024-03-11T22:22:57.045033+0000","last_active":"2024-03-11T22:25:18.021806+0000","last_peered":"2024-03-11T22:25:18.021806+0000","last_clean":"2024-03-11T22:25:18.021806+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:25:18.021806+0000","last_undegraded":"2024-03-11T22:25:18.021806+0000","last_fullsized":"2024-03-11T22:25:18.021806+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:22:57.044931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:22:57.044931+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":701,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.039124+0000","last_change":"2024-03-11T22:22:06.879148+0000","last_active":"2024-03-11T22:25:18.039124+0000","last_peered":"2024-03-11T22:25:18.039124+0000","last_clean":"2024-03-11T22:25:18.039124+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:25:18.039124+0000","last_undegraded":"2024-03-11T22:25:18.039124+0000","last_fullsized":"2024-03-11T22:25:18.039124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:22:06.879026+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:22:06.879026+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'11","reported_seq":945,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.552142+0000","last_change":"2024-03-11T22:23:50.837614+0000","last_active":"2024-03-11T22:25:18.552142+0000","last_peered":"2024-03-11T22:25:18.552142+0000","last_clean":"2024-03-11T22:25:18.552142+0000","last_became_active":"2024-03-11T22:20:29.415781+0000","last_became_peered":"2024-03-11T22:20:29.415781+0000","last_unstale":"2024-03-11T22:25:18.552142+0000","last_undegraded":"2024-03-11T22:25:18.552142+0000","last_fullsized":"2024-03-11T22:25:18.552142+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-11T22:23:50.837494+0000","last_deep_scrub":"315'11","last_deep_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_clean_scrub_stamp":"2024-03-11T22:23:50.837494+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:24.733254+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":640,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029336+0000","last_change":"2024-03-11T22:23:51.821013+0000","last_active":"2024-03-11T22:25:18.029336+0000","last_peered":"2024-03-11T22:25:18.029336+0000","last_clean":"2024-03-11T22:25:18.029336+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:25:18.029336+0000","last_undegraded":"2024-03-11T22:25:18.029336+0000","last_fullsized":"2024-03-11T22:25:18.029336+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:23:51.820865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.820865+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"273'18","reported_seq":719,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029572+0000","last_change":"2024-03-11T22:25:01.629462+0000","last_active":"2024-03-11T22:25:18.029572+0000","last_peered":"2024-03-11T22:25:18.029572+0000","last_clean":"2024-03-11T22:25:18.029572+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:25:18.029572+0000","last_undegraded":"2024-03-11T22:25:18.029572+0000","last_fullsized":"2024-03-11T22:25:18.029572+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:25:01.629321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:01.629321+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:29.544545+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":467,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.041314+0000","last_change":"2024-03-11T22:25:04.394810+0000","last_active":"2024-03-11T22:25:18.041314+0000","last_peered":"2024-03-11T22:25:18.041314+0000","last_clean":"2024-03-11T22:25:18.041314+0000","last_became_active":"2024-03-11T22:14:52.245199+0000","last_became_peered":"2024-03-11T22:14:52.245199+0000","last_unstale":"2024-03-11T22:25:18.041314+0000","last_undegraded":"2024-03-11T22:25:18.041314+0000","last_fullsized":"2024-03-11T22:25:18.041314+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:04.394612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:25:04.394612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:30.314020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":450,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.025957+0000","last_change":"2024-03-11T22:25:05.656524+0000","last_active":"2024-03-11T22:25:18.025957+0000","last_peered":"2024-03-11T22:25:18.025957+0000","last_clean":"2024-03-11T22:25:18.025957+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:25:18.025957+0000","last_undegraded":"2024-03-11T22:25:18.025957+0000","last_fullsized":"2024-03-11T22:25:18.025957+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:05.656208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.656208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:14.267362+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":673,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029946+0000","last_change":"2024-03-11T22:23:12.914118+0000","last_active":"2024-03-11T22:25:18.029946+0000","last_peered":"2024-03-11T22:25:18.029946+0000","last_clean":"2024-03-11T22:25:18.029946+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:25:18.029946+0000","last_undegraded":"2024-03-11T22:25:18.029946+0000","last_fullsized":"2024-03-11T22:25:18.029946+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:12.914007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:12.914007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.704144+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"273'27","reported_seq":762,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030696+0000","last_change":"2024-03-11T22:23:46.333974+0000","last_active":"2024-03-11T22:25:18.030696+0000","last_peered":"2024-03-11T22:25:18.030696+0000","last_clean":"2024-03-11T22:25:18.030696+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:25:18.030696+0000","last_undegraded":"2024-03-11T22:25:18.030696+0000","last_fullsized":"2024-03-11T22:25:18.030696+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:23:46.333799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.333799+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:20.675424+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":711,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.023447+0000","last_change":"2024-03-11T22:24:55.760207+0000","last_active":"2024-03-11T22:25:18.023447+0000","last_peered":"2024-03-11T22:25:18.023447+0000","last_clean":"2024-03-11T22:25:18.023447+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:25:18.023447+0000","last_undegraded":"2024-03-11T22:25:18.023447+0000","last_fullsized":"2024-03-11T22:25:18.023447+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:55.760084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:24:55.760084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:21.111674+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":699,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.040166+0000","last_change":"2024-03-11T22:24:37.481790+0000","last_active":"2024-03-11T22:25:18.040166+0000","last_peered":"2024-03-11T22:25:18.040166+0000","last_clean":"2024-03-11T22:25:18.040166+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:25:18.040166+0000","last_undegraded":"2024-03-11T22:25:18.040166+0000","last_fullsized":"2024-03-11T22:25:18.040166+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.481668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.481668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:03.122817+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":467,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.028045+0000","last_change":"2024-03-11T22:23:07.097233+0000","last_active":"2024-03-11T22:25:18.028045+0000","last_peered":"2024-03-11T22:25:18.028045+0000","last_clean":"2024-03-11T22:25:18.028045+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:25:18.028045+0000","last_undegraded":"2024-03-11T22:25:18.028045+0000","last_fullsized":"2024-03-11T22:25:18.028045+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:07.097049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:07.097049+0000","last_clean_scrub_stamp":"2024-03-11T22:23:07.097049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.691596+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"273'9","reported_seq":527,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.021587+0000","last_change":"2024-03-11T22:23:58.648966+0000","last_active":"2024-03-11T22:25:18.021587+0000","last_peered":"2024-03-11T22:25:18.021587+0000","last_clean":"2024-03-11T22:25:18.021587+0000","last_became_active":"2024-03-11T22:14:52.238411+0000","last_became_peered":"2024-03-11T22:14:52.238411+0000","last_unstale":"2024-03-11T22:25:18.021587+0000","last_undegraded":"2024-03-11T22:25:18.021587+0000","last_fullsized":"2024-03-11T22:25:18.021587+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9","last_scrub_stamp":"2024-03-11T22:23:58.648385+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.648385+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:26.592010+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":699,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029549+0000","last_change":"2024-03-11T22:24:40.905435+0000","last_active":"2024-03-11T22:25:18.029549+0000","last_peered":"2024-03-11T22:25:18.029549+0000","last_clean":"2024-03-11T22:25:18.029549+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:25:18.029549+0000","last_undegraded":"2024-03-11T22:25:18.029549+0000","last_fullsized":"2024-03-11T22:25:18.029549+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:40.905314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:24:40.905314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:41.648326+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":461,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.039330+0000","last_change":"2024-03-11T22:22:03.796143+0000","last_active":"2024-03-11T22:25:18.039330+0000","last_peered":"2024-03-11T22:25:18.039330+0000","last_clean":"2024-03-11T22:25:18.039330+0000","last_became_active":"2024-03-11T22:14:52.239391+0000","last_became_peered":"2024-03-11T22:14:52.239391+0000","last_unstale":"2024-03-11T22:25:18.039330+0000","last_undegraded":"2024-03-11T22:25:18.039330+0000","last_fullsized":"2024-03-11T22:25:18.039330+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:03.796021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_clean_scrub_stamp":"2024-03-11T22:22:03.796021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":679,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029537+0000","last_change":"2024-03-11T22:22:59.182771+0000","last_active":"2024-03-11T22:25:18.029537+0000","last_peered":"2024-03-11T22:25:18.029537+0000","last_clean":"2024-03-11T22:25:18.029537+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:25:18.029537+0000","last_undegraded":"2024-03-11T22:25:18.029537+0000","last_fullsized":"2024-03-11T22:25:18.029537+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:59.182621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:59.182621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"273'15","reported_seq":502,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022117+0000","last_change":"2024-03-11T22:25:03.685762+0000","last_active":"2024-03-11T22:25:18.022117+0000","last_peered":"2024-03-11T22:25:18.022117+0000","last_clean":"2024-03-11T22:25:18.022117+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:25:18.022117+0000","last_undegraded":"2024-03-11T22:25:18.022117+0000","last_fullsized":"2024-03-11T22:25:18.022117+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:25:03.685629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:03.685629+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:21.379436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":699,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.030102+0000","last_change":"2024-03-11T22:23:58.700286+0000","last_active":"2024-03-11T22:25:18.030102+0000","last_peered":"2024-03-11T22:25:18.030102+0000","last_clean":"2024-03-11T22:25:18.030102+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:25:18.030102+0000","last_undegraded":"2024-03-11T22:25:18.030102+0000","last_fullsized":"2024-03-11T22:25:18.030102+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:58.700176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.700176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":703,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029256+0000","last_change":"2024-03-11T22:24:49.941039+0000","last_active":"2024-03-11T22:25:18.029256+0000","last_peered":"2024-03-11T22:25:18.029256+0000","last_clean":"2024-03-11T22:25:18.029256+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:25:18.029256+0000","last_undegraded":"2024-03-11T22:25:18.029256+0000","last_fullsized":"2024-03-11T22:25:18.029256+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:49.940903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:24:49.940903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.554477+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":679,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.021034+0000","last_change":"2024-03-11T22:23:40.218666+0000","last_active":"2024-03-11T22:25:18.021034+0000","last_peered":"2024-03-11T22:25:18.021034+0000","last_clean":"2024-03-11T22:25:18.021034+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:25:18.021034+0000","last_undegraded":"2024-03-11T22:25:18.021034+0000","last_fullsized":"2024-03-11T22:25:18.021034+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:40.218559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:40.218559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"273'27","reported_seq":722,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022729+0000","last_change":"2024-03-11T22:23:47.683228+0000","last_active":"2024-03-11T22:25:18.022729+0000","last_peered":"2024-03-11T22:25:18.022729+0000","last_clean":"2024-03-11T22:25:18.022729+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:25:18.022729+0000","last_undegraded":"2024-03-11T22:25:18.022729+0000","last_fullsized":"2024-03-11T22:25:18.022729+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:23:47.683120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:47.683120+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.704207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":699,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029821+0000","last_change":"2024-03-11T22:23:41.703918+0000","last_active":"2024-03-11T22:25:18.029821+0000","last_peered":"2024-03-11T22:25:18.029821+0000","last_clean":"2024-03-11T22:25:18.029821+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:25:18.029821+0000","last_undegraded":"2024-03-11T22:25:18.029821+0000","last_fullsized":"2024-03-11T22:25:18.029821+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:23:41.703753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:41.703753+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.704238+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":469,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.024512+0000","last_change":"2024-03-11T22:22:58.001733+0000","last_active":"2024-03-11T22:25:18.024512+0000","last_peered":"2024-03-11T22:25:18.024512+0000","last_clean":"2024-03-11T22:25:18.024512+0000","last_became_active":"2024-03-11T22:14:52.247689+0000","last_became_peered":"2024-03-11T22:14:52.247689+0000","last_unstale":"2024-03-11T22:25:18.024512+0000","last_undegraded":"2024-03-11T22:25:18.024512+0000","last_fullsized":"2024-03-11T22:25:18.024512+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:58.001650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:22:58.001650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.691513+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":683,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022656+0000","last_change":"2024-03-11T22:23:08.833422+0000","last_active":"2024-03-11T22:25:18.022656+0000","last_peered":"2024-03-11T22:25:18.022656+0000","last_clean":"2024-03-11T22:25:18.022656+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:25:18.022656+0000","last_undegraded":"2024-03-11T22:25:18.022656+0000","last_fullsized":"2024-03-11T22:25:18.022656+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:08.833328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:08.833328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:22.704083+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"273'19","reported_seq":514,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.024720+0000","last_change":"2024-03-11T22:24:39.755038+0000","last_active":"2024-03-11T22:25:18.024720+0000","last_peered":"2024-03-11T22:25:18.024720+0000","last_clean":"2024-03-11T22:25:18.024720+0000","last_became_active":"2024-03-11T22:14:52.247411+0000","last_became_peered":"2024-03-11T22:14:52.247411+0000","last_unstale":"2024-03-11T22:25:18.024720+0000","last_undegraded":"2024-03-11T22:25:18.024720+0000","last_fullsized":"2024-03-11T22:25:18.024720+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:24:39.754894+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.754894+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:44.634751+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":701,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.022040+0000","last_change":"2024-03-11T22:24:00.015418+0000","last_active":"2024-03-11T22:25:18.022040+0000","last_peered":"2024-03-11T22:25:18.022040+0000","last_clean":"2024-03-11T22:25:18.022040+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:25:18.022040+0000","last_undegraded":"2024-03-11T22:25:18.022040+0000","last_fullsized":"2024-03-11T22:25:18.022040+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:00.015301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.015301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:18.640347+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":414,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.023505+0000","last_change":"2024-03-11T22:23:29.877294+0000","last_active":"2024-03-11T22:25:18.023505+0000","last_peered":"2024-03-11T22:25:18.023505+0000","last_clean":"2024-03-11T22:25:18.023505+0000","last_became_active":"2024-03-11T22:20:29.418227+0000","last_became_peered":"2024-03-11T22:20:29.418227+0000","last_unstale":"2024-03-11T22:25:18.023505+0000","last_undegraded":"2024-03-11T22:25:18.023505+0000","last_fullsized":"2024-03-11T22:25:18.023505+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:29.877172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:29.877172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:19.691652+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":458,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-11T22:25:18.029929+0000","last_change":"2024-03-11T22:23:36.916038+0000","last_active":"2024-03-11T22:25:18.029929+0000","last_peered":"2024-03-11T22:25:18.029929+0000","last_clean":"2024-03-11T22:25:18.029929+0000","last_became_active":"2024-03-11T22:14:52.247273+0000","last_became_peered":"2024-03-11T22:14:52.247273+0000","last_unstale":"2024-03-11T22:25:18.029929+0000","last_undegraded":"2024-03-11T22:25:18.029929+0000","last_fullsized":"2024-03-11T22:25:18.029929+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:36.915904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:36.915904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4685037592,"num_objects":71489,"num_object_clones":0,"num_object_copies":214467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71489,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192561,"num_write_kb":8449537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14133178368,"data_stored":14133166152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81427,"ondisk_log_size":81427,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":973,"num_read_kb":968,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":1430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005263,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1753152,"kb_used_data":1136120,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":91992000,"statfs":{"total":95995035648,"available":94199808000,"internally_reserved":0,"allocated":1163386880,"data_stored":1163073913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":631830987},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332308,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2507504,"kb_used_data":2002920,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":91237648,"statfs":{"total":95995035648,"available":93427351552,"internally_reserved":0,"allocated":2050990080,"data_stored":2050693331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":516684230},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575085,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1453212,"kb_used_data":894484,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92291940,"statfs":{"total":95995035648,"available":94506946560,"internally_reserved":0,"allocated":915951616,"data_stored":915609983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738611,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3635904,"kb_used_data":2991032,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":90109248,"statfs":{"total":95995035648,"available":92271869952,"internally_reserved":0,"allocated":3062816768,"data_stored":3062503157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,2,6,7,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411561,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2160044,"kb_used_data":1695716,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":91585108,"statfs":{"total":95995035648,"available":93783150592,"internally_reserved":0,"allocated":1736413184,"data_stored":1736120877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149922,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2132952,"kb_used_data":1520076,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":91612200,"statfs":{"total":95995035648,"available":93810892800,"internally_reserved":0,"allocated":1556557824,"data_stored":1556225237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":627571143},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920991,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2808712,"kb_used_data":2281920,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":90936440,"statfs":{"total":95995035648,"available":93118914560,"internally_reserved":0,"allocated":2336686080,"data_stored":2336364486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":539425222},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659352,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2054260,"kb_used_data":1419180,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":91690892,"statfs":{"total":95995035648,"available":93891473408,"internally_reserved":0,"allocated":1453240320,"data_stored":1452939268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":650312134},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3023835136,"data_stored":3023831064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":913833984,"data_stored":913833984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1730412544,"data_stored":1730412544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1537081344,"data_stored":1537081344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2307854336,"data_stored":2307850264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1438384128,"data_stored":1438384128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2022838272,"data_stored":2022834200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1158938624,"data_stored":1158938624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:25:29.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:25:29.832 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:29.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:29 smithi144 ceph-mon[33984]: pgmap v791: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-11T22:25:29.855 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:29 smithi045 ceph-mon[39715]: pgmap v791: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-11T22:25:29.856 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:29 smithi045 ceph-mon[30178]: pgmap v791: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-11T22:25:29.899 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:29.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:29.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:30.002 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:30.032 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:30.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:30.118 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:25:30.132 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:30.158 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:25:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:30.162 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:30.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:30.200 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:25:30.238 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:25:30.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:30.275 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:25:30.299 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:30.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:30.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:30.313 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:25:30.356 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:25:30.397 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:25:30.400 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:25:30.400 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:25:30.400 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:25:30.400 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:25:30.400 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:25:30.401 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:25:30.401 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:25:30.401 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:25:30.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:30.443 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:30.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:30.451 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:30.545 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:30.583 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:30.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:30.685 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:30.715 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:30.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:30.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:30.783 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:30 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1205612167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:25:30.783 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:30 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1205612167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:25:30.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:30.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:30 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1205612167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:25:30.866 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:30.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:30.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:30.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:31.007 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:31.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:31.108 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:31.140 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:31.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:31.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:31.242 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:31.279 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:31.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:31.380 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:31.405 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:31.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:31.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:31.507 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:31.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:31.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:31.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:31.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:31.686 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:31.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:31.788 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:31.826 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:31.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:31.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:31 smithi144 ceph-mon[33984]: pgmap v792: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 852 op/s 2024-03-11T22:25:31.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:25:31.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:31.969 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:31.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:32.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:31 smithi045 ceph-mon[30178]: pgmap v792: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 852 op/s 2024-03-11T22:25:32.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:25:32.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:31 smithi045 ceph-mon[39715]: pgmap v792: 168 pgs: 168 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 852 op/s 2024-03-11T22:25:32.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:25:32.071 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:32.246 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:32.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:32.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:32.348 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:32.386 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:32.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:32.487 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:32.494 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:25:32.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:32.522 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:32.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:32.622 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:32.646 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:32.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:32.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:32.785 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:25:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:25:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:25:32.791 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:32.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:32.893 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:32.925 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:32.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:32.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:33.027 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:33.060 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:33.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:33.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:33.161 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:33.200 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:33.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:33.302 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:33.339 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:33.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:33.441 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:33.475 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:33.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:33.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:33.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:33.615 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:33.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:33.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:33.717 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:33.750 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:33.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:33.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:33 smithi144 ceph-mon[33984]: pgmap v793: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-11T22:25:33.852 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:33.886 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:33.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:33.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:33.988 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:34.023 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:34.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:34.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:33 smithi045 ceph-mon[39715]: pgmap v793: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-11T22:25:34.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:33 smithi045 ceph-mon[30178]: pgmap v793: 168 pgs: 168 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-11T22:25:34.125 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:34.156 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:34.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:34.198 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:34.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:34.305 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:34.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:34.407 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:34.442 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:34.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:34.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:34.544 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:34.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:34.584 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:34.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:34.685 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:34.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:34.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:34.834 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:34.866 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:34.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:34.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:34.994 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:25:34.994 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:25:35.009 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:35.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:35.111 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:35.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:35.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:35.252 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:35.291 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:35.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:35.392 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:35.434 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:35.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:35.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:35.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:35.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:35.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:35.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:35.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:35.724 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:35.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:35.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:35.784 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:35 smithi045 ceph-mon[39715]: pgmap v794: 168 pgs: 168 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s 2024-03-11T22:25:35.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:35 smithi045 ceph-mon[30178]: pgmap v794: 168 pgs: 168 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s 2024-03-11T22:25:35.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:35.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:35 smithi144 ceph-mon[33984]: pgmap v794: 168 pgs: 168 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s 2024-03-11T22:25:35.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:35.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:35.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:35.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:36.005 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:36.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:36.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:36.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:36.142 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:36.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:36.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:36.156 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:25:36.245 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:36.274 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:36.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:36.376 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:36.414 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:36.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:36.516 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:36.553 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:36.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:36.655 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:36.695 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:36.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:36.797 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:36.818 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:36.828 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:36.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:36.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T22:25:36.929 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:36.962 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:36.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:36.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:37.064 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:37.100 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:37.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:37.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:25:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:25:37.201 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:37.243 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:37.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:37.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:37.345 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:37.369 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:37.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:37.470 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:37.507 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:37.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:37.608 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:37.644 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:37.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:37.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:37.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:37.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:37.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:37.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:37 smithi144 ceph-mon[33984]: pgmap v795: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-11T22:25:37.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:37.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:37.909 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:37.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:37.932 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:37 smithi045 ceph-mon[39715]: pgmap v795: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-11T22:25:37.932 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:37.932 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:37 smithi045 ceph-mon[30178]: pgmap v795: 168 pgs: 168 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-11T22:25:37.932 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:38.011 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:38.047 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:38.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:38.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:38.148 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:38.181 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:38.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:38.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:38.283 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:38.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:38.316 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:38.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:38.417 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:38.453 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:38.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:38.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:38.554 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:38.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:38.592 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:38.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:38.693 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:38.741 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:38.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:38.843 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:38.878 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:38.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:38.979 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:39.013 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:39.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:39.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:39.115 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:39.146 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:39.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:39.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:39.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:39.291 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:39.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:39.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:39.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:39.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:39.537 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:39.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:39 smithi144 ceph-mon[33984]: pgmap v796: 168 pgs: 168 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 941 B/s rd, 0 B/s wr, 1.26k op/s 2024-03-11T22:25:40.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:39 smithi045 ceph-mon[30178]: pgmap v796: 168 pgs: 168 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 941 B/s rd, 0 B/s wr, 1.26k op/s 2024-03-11T22:25:40.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:39 smithi045 ceph-mon[39715]: pgmap v796: 168 pgs: 168 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 941 B/s rd, 0 B/s wr, 1.26k op/s 2024-03-11T22:25:40.099 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:40.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:40.201 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:40.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:40.234 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:40.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:40.335 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:40.358 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:40.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:40.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:40.460 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:40.494 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:40.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:40.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:40.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:40.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:40.667 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:40.761 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:40.798 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:40.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:40.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:40.957 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:40.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:41.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:41.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:41.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:41.198 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:41.230 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:41.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:41.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:41.332 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:41.365 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:41.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:41.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:41.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:41.511 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:41.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:41.613 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:41.650 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:41.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:41.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:41.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:41.794 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:41.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:41.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:41 smithi144 ceph-mon[33984]: pgmap v797: 168 pgs: 168 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T22:25:41.896 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:41.921 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:41 smithi045 ceph-mon[39715]: pgmap v797: 168 pgs: 168 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T22:25:41.922 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:41 smithi045 ceph-mon[30178]: pgmap v797: 168 pgs: 168 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T22:25:41.927 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:41.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:41.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:42.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:42.067 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:42.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:42.169 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:42.198 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:42.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:42.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:42.300 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:42.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:42.345 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:42.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:42.446 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:42.486 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:42.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:42.588 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:42.635 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:42.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:42.687 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:25:42.695 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:25:42.712 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:25:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:25:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:25:42.737 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:42.771 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:42.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:42.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:42.906 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:42.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:43.008 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:43.060 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:43.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:43.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:43.196 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:43.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:43.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:43.298 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:43.334 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:43.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:43.436 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:43.475 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:43.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:43.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:43.617 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:43.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:43.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:43.719 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:43.742 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:43.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:43.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:43.844 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:43.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:43 smithi144 ceph-mon[33984]: pgmap v798: 168 pgs: 168 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-11T22:25:43.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:43 smithi144 ceph-mon[33984]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:25:43.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:43 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:25:43.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:43 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:25:43.879 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:43.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:43.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:43.981 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:44.011 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:44.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:44.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:44.034 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:43 smithi045 ceph-mon[30178]: pgmap v798: 168 pgs: 168 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-11T22:25:44.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:43 smithi045 ceph-mon[30178]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:25:44.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:43 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:25:44.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:43 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:25:44.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:43 smithi045 ceph-mon[39715]: pgmap v798: 168 pgs: 168 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-11T22:25:44.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:43 smithi045 ceph-mon[39715]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:25:44.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:43 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:25:44.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:43 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:25:44.114 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:44.143 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:44.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:44.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:44.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:44.290 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:44.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:44.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:44.432 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:44.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:44.533 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:44.588 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:44.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:44.690 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:44.721 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:44.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:44.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:44.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:44 smithi045 ceph-mon[30178]: osdmap e388: 8 total, 8 up, 8 in 2024-03-11T22:25:44.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:44 smithi045 ceph-mon[39715]: osdmap e388: 8 total, 8 up, 8 in 2024-03-11T22:25:44.823 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:44.856 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:44.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:44.957 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:44.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:44.995 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:44.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:45.055 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:45.096 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:45.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:44 smithi144 ceph-mon[33984]: osdmap e388: 8 total, 8 up, 8 in 2024-03-11T22:25:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:45.151 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:45.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:45.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:45.286 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:45.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:45.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:45.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:45.429 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:45.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:45.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:45.572 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:45.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:45.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:45.674 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:45.706 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:45.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:45.727 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":798,"stamp":"2024-03-11T22:25:42.420862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3467581812,"num_objects":53101,"num_object_clones":0,"num_object_copies":159303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53101,"num_whiteouts":0,"num_read":2118,"num_read_kb":1870,"num_write":211869,"num_write_kb":8449582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":2436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83737,"ondisk_log_size":83737,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14834632,"kb_used_data":10252168,"kb_used_omap":12,"kb_used_meta":4582195,"kb_avail":735126584,"statfs":{"total":767960285184,"available":752769622016,"internally_reserved":0,"allocated":10498220032,"data_stored":10495641932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":4692168243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1107755008,"num_objects":-16903,"num_object_clones":0,"num_object_copies":-50709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-16903,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":16903,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004090"},"pg_stats":[{"pgid":"10.2","version":"387'13198","reported_seq":13287,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:41.604541+0000","last_change":"2024-03-11T22:22:52.799244+0000","last_active":"2024-03-11T22:25:41.604541+0000","last_peered":"2024-03-11T22:25:41.604541+0000","last_clean":"2024-03-11T22:25:41.604541+0000","last_became_active":"2024-03-11T22:22:52.798875+0000","last_became_peered":"2024-03-11T22:22:52.798875+0000","last_unstale":"2024-03-11T22:25:41.604541+0000","last_undegraded":"2024-03-11T22:25:41.604541+0000","last_fullsized":"2024-03-11T22:25:41.604541+0000","mapping_epoch":348,"log_start":"373'8100","ondisk_log_start":"373'8100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":5098,"log_dups_size":0,"ondisk_log_size":5098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209190912,"num_objects":3192,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13198,"num_write_kb":524480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"294'25","reported_seq":508,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.558993+0000","last_change":"2024-03-11T22:24:39.621975+0000","last_active":"2024-03-11T22:25:27.558993+0000","last_peered":"2024-03-11T22:25:27.558993+0000","last_clean":"2024-03-11T22:25:27.558993+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:25:27.558993+0000","last_undegraded":"2024-03-11T22:25:27.558993+0000","last_fullsized":"2024-03-11T22:25:27.558993+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-03-11T22:24:39.621707+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.621707+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.425365+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":696,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.555291+0000","last_change":"2024-03-11T22:24:37.604756+0000","last_active":"2024-03-11T22:25:27.555291+0000","last_peered":"2024-03-11T22:25:27.555291+0000","last_clean":"2024-03-11T22:25:27.555291+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:25:27.555291+0000","last_undegraded":"2024-03-11T22:25:27.555291+0000","last_fullsized":"2024-03-11T22:25:27.555291+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.601240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.601240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:51.956237+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":935,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:33.554571+0000","last_change":"2024-03-11T22:24:01.449741+0000","last_active":"2024-03-11T22:25:33.554571+0000","last_peered":"2024-03-11T22:25:33.554571+0000","last_clean":"2024-03-11T22:25:33.554571+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:25:33.554571+0000","last_undegraded":"2024-03-11T22:25:33.554571+0000","last_fullsized":"2024-03-11T22:25:33.554571+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:24:01.447108+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:19:40.008325+0000","last_clean_scrub_stamp":"2024-03-11T22:24:01.447108+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:41.944709+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":462,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.547658+0000","last_change":"2024-03-11T22:23:07.686811+0000","last_active":"2024-03-11T22:25:27.547658+0000","last_peered":"2024-03-11T22:25:27.547658+0000","last_clean":"2024-03-11T22:25:27.547658+0000","last_became_active":"2024-03-11T22:14:52.243690+0000","last_became_peered":"2024-03-11T22:14:52.243690+0000","last_unstale":"2024-03-11T22:25:27.547658+0000","last_undegraded":"2024-03-11T22:25:27.547658+0000","last_fullsized":"2024-03-11T22:25:27.547658+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:07.686730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:07.686730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:29.530053+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":1019,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.554596+0000","last_change":"2024-03-11T22:23:43.670715+0000","last_active":"2024-03-11T22:25:27.554596+0000","last_peered":"2024-03-11T22:25:27.554596+0000","last_clean":"2024-03-11T22:25:27.554596+0000","last_became_active":"2024-03-11T22:20:32.293356+0000","last_became_peered":"2024-03-11T22:20:32.293356+0000","last_unstale":"2024-03-11T22:25:27.554596+0000","last_undegraded":"2024-03-11T22:25:27.554596+0000","last_fullsized":"2024-03-11T22:25:27.554596+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:43.670585+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:43.670585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754700+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"387'13283","reported_seq":13448,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:41.061204+0000","last_change":"2024-03-11T22:24:53.352685+0000","last_active":"2024-03-11T22:25:41.061204+0000","last_peered":"2024-03-11T22:25:41.061204+0000","last_clean":"2024-03-11T22:25:41.061204+0000","last_became_active":"2024-03-11T22:22:52.469327+0000","last_became_peered":"2024-03-11T22:22:52.469327+0000","last_unstale":"2024-03-11T22:25:41.061204+0000","last_undegraded":"2024-03-11T22:25:41.061204+0000","last_fullsized":"2024-03-11T22:25:41.061204+0000","mapping_epoch":348,"log_start":"371'7200","ondisk_log_start":"371'7200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9962","last_scrub_stamp":"2024-03-11T22:24:53.352359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.352359+0000","objects_scrubbed":6655,"log_size":6083,"log_dups_size":0,"ondisk_log_size":6083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:58.557123+0000","scrub_duration":2633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212140032,"num_objects":3237,"num_object_clones":0,"num_object_copies":9711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13283,"num_write_kb":528640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"353'39","reported_seq":562,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548813+0000","last_change":"2024-03-11T22:24:58.663588+0000","last_active":"2024-03-11T22:25:27.548813+0000","last_peered":"2024-03-11T22:25:27.548813+0000","last_clean":"2024-03-11T22:25:27.548813+0000","last_became_active":"2024-03-11T22:14:52.239649+0000","last_became_peered":"2024-03-11T22:14:52.239649+0000","last_unstale":"2024-03-11T22:25:27.548813+0000","last_undegraded":"2024-03-11T22:25:27.548813+0000","last_fullsized":"2024-03-11T22:25:27.548813+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'39","last_scrub_stamp":"2024-03-11T22:24:58.663290+0000","last_deep_scrub":"353'39","last_deep_scrub_stamp":"2024-03-11T22:23:05.461729+0000","last_clean_scrub_stamp":"2024-03-11T22:24:58.663290+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:01.646724+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":720,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.543089+0000","last_change":"2024-03-11T22:23:51.794264+0000","last_active":"2024-03-11T22:25:27.543089+0000","last_peered":"2024-03-11T22:25:27.543089+0000","last_clean":"2024-03-11T22:25:27.543089+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:25:27.543089+0000","last_undegraded":"2024-03-11T22:25:27.543089+0000","last_fullsized":"2024-03-11T22:25:27.543089+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:51.794140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.794140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":696,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548650+0000","last_change":"2024-03-11T22:23:48.515368+0000","last_active":"2024-03-11T22:25:27.548650+0000","last_peered":"2024-03-11T22:25:27.548650+0000","last_clean":"2024-03-11T22:25:27.548650+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:25:27.548650+0000","last_undegraded":"2024-03-11T22:25:27.548650+0000","last_fullsized":"2024-03-11T22:25:27.548650+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.515251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.515251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:29.530080+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":708,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.552008+0000","last_change":"2024-03-11T22:23:49.537509+0000","last_active":"2024-03-11T22:25:27.552008+0000","last_peered":"2024-03-11T22:25:27.552008+0000","last_clean":"2024-03-11T22:25:27.552008+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:25:27.552008+0000","last_undegraded":"2024-03-11T22:25:27.552008+0000","last_fullsized":"2024-03-11T22:25:27.552008+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:49.537151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:49.537151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:31.852934+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":995,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.551081+0000","last_change":"2024-03-11T22:22:57.117070+0000","last_active":"2024-03-11T22:25:27.551081+0000","last_peered":"2024-03-11T22:25:27.551081+0000","last_clean":"2024-03-11T22:25:27.551081+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:25:27.551081+0000","last_undegraded":"2024-03-11T22:25:27.551081+0000","last_fullsized":"2024-03-11T22:25:27.551081+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:22:57.116971+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:22:57.116971+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.992927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"387'13218","reported_seq":13307,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:40.604682+0000","last_change":"2024-03-11T22:22:52.799080+0000","last_active":"2024-03-11T22:25:40.604682+0000","last_peered":"2024-03-11T22:25:40.604682+0000","last_clean":"2024-03-11T22:25:40.604682+0000","last_became_active":"2024-03-11T22:22:52.798328+0000","last_became_peered":"2024-03-11T22:22:52.798328+0000","last_unstale":"2024-03-11T22:25:40.604682+0000","last_undegraded":"2024-03-11T22:25:40.604682+0000","last_fullsized":"2024-03-11T22:25:40.604682+0000","mapping_epoch":348,"log_start":"373'8200","ondisk_log_start":"373'8200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210894848,"num_objects":3218,"num_object_clones":0,"num_object_copies":9654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13218,"num_write_kb":525952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"273'19","reported_seq":528,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.543795+0000","last_change":"2024-03-11T22:24:57.716291+0000","last_active":"2024-03-11T22:25:27.543795+0000","last_peered":"2024-03-11T22:25:27.543795+0000","last_clean":"2024-03-11T22:25:27.543795+0000","last_became_active":"2024-03-11T22:14:52.240437+0000","last_became_peered":"2024-03-11T22:14:52.240437+0000","last_unstale":"2024-03-11T22:25:27.543795+0000","last_undegraded":"2024-03-11T22:25:27.543795+0000","last_fullsized":"2024-03-11T22:25:27.543795+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:24:57.716144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:57.716144+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:16.100178+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":487,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.547321+0000","last_change":"2024-03-11T22:24:57.690108+0000","last_active":"2024-03-11T22:25:27.547321+0000","last_peered":"2024-03-11T22:25:27.547321+0000","last_clean":"2024-03-11T22:25:27.547321+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:25:27.547321+0000","last_undegraded":"2024-03-11T22:25:27.547321+0000","last_fullsized":"2024-03-11T22:25:27.547321+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:57.689696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:57.689696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.466412+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":696,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.555249+0000","last_change":"2024-03-11T22:24:59.851270+0000","last_active":"2024-03-11T22:25:27.555249+0000","last_peered":"2024-03-11T22:25:27.555249+0000","last_clean":"2024-03-11T22:25:27.555249+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:25:27.555249+0000","last_undegraded":"2024-03-11T22:25:27.555249+0000","last_fullsized":"2024-03-11T22:25:27.555249+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:59.851040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:59.851040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:06.559357+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":676,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.558408+0000","last_change":"2024-03-11T22:23:48.716684+0000","last_active":"2024-03-11T22:25:27.558408+0000","last_peered":"2024-03-11T22:25:27.558408+0000","last_clean":"2024-03-11T22:25:27.558408+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:25:27.558408+0000","last_undegraded":"2024-03-11T22:25:27.558408+0000","last_fullsized":"2024-03-11T22:25:27.558408+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.716557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.716557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754789+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"93'130","reported_seq":1023,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.549448+0000","last_change":"2024-03-11T22:22:14.766914+0000","last_active":"2024-03-11T22:25:27.549448+0000","last_peered":"2024-03-11T22:25:27.549448+0000","last_clean":"2024-03-11T22:25:27.549448+0000","last_became_active":"2024-03-11T22:18:00.271764+0000","last_became_peered":"2024-03-11T22:18:00.271764+0000","last_unstale":"2024-03-11T22:25:27.549448+0000","last_undegraded":"2024-03-11T22:25:27.549448+0000","last_fullsized":"2024-03-11T22:25:27.549448+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:22:14.766775+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:22:14.766775+0000","last_clean_scrub_stamp":"2024-03-11T22:22:14.766775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:29.517323+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"387'17209","reported_seq":17473,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:37.547431+0000","last_change":"2024-03-11T22:25:26.549894+0000","last_active":"2024-03-11T22:25:37.547431+0000","last_peered":"2024-03-11T22:25:37.547431+0000","last_clean":"2024-03-11T22:25:37.547431+0000","last_became_active":"2024-03-11T22:22:52.797873+0000","last_became_peered":"2024-03-11T22:22:52.797873+0000","last_unstale":"2024-03-11T22:25:37.547431+0000","last_undegraded":"2024-03-11T22:25:37.547431+0000","last_fullsized":"2024-03-11T22:25:37.547431+0000","mapping_epoch":348,"log_start":"383'12402","ondisk_log_start":"383'12402","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":4242,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:37.951282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230490112,"num_objects":3517,"num_object_clones":0,"num_object_copies":10551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12967,"num_write_kb":527488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"273'34","reported_seq":758,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.550088+0000","last_change":"2024-03-11T22:21:49.811476+0000","last_active":"2024-03-11T22:25:27.550088+0000","last_peered":"2024-03-11T22:25:27.550088+0000","last_clean":"2024-03-11T22:25:27.550088+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:25:27.550088+0000","last_undegraded":"2024-03-11T22:25:27.550088+0000","last_fullsized":"2024-03-11T22:25:27.550088+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:21:49.811214+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:21:49.811214+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:29.517260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":708,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.554163+0000","last_change":"2024-03-11T22:25:06.617067+0000","last_active":"2024-03-11T22:25:27.554163+0000","last_peered":"2024-03-11T22:25:27.554163+0000","last_clean":"2024-03-11T22:25:27.554163+0000","last_became_active":"2024-03-11T22:20:29.426224+0000","last_became_peered":"2024-03-11T22:20:29.426224+0000","last_unstale":"2024-03-11T22:25:27.554163+0000","last_undegraded":"2024-03-11T22:25:27.554163+0000","last_fullsized":"2024-03-11T22:25:27.554163+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:06.616932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:25:06.616932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:21.802971+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":618,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:33.555581+0000","last_change":"2024-03-11T22:24:00.708144+0000","last_active":"2024-03-11T22:25:33.555581+0000","last_peered":"2024-03-11T22:25:33.555581+0000","last_clean":"2024-03-11T22:25:33.555581+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:25:33.555581+0000","last_undegraded":"2024-03-11T22:25:33.555581+0000","last_fullsized":"2024-03-11T22:25:33.555581+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:24:00.708018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.708018+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:43.100566+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":684,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.553374+0000","last_change":"2024-03-11T22:23:33.067353+0000","last_active":"2024-03-11T22:25:27.553374+0000","last_peered":"2024-03-11T22:25:27.553374+0000","last_clean":"2024-03-11T22:25:27.553374+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:25:27.553374+0000","last_undegraded":"2024-03-11T22:25:27.553374+0000","last_fullsized":"2024-03-11T22:25:27.553374+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:33.067236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:23:33.067236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.993077+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":1004,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.551358+0000","last_change":"2024-03-11T22:24:48.925640+0000","last_active":"2024-03-11T22:25:27.551358+0000","last_peered":"2024-03-11T22:25:27.551358+0000","last_clean":"2024-03-11T22:25:27.551358+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:25:27.551358+0000","last_undegraded":"2024-03-11T22:25:27.551358+0000","last_fullsized":"2024-03-11T22:25:27.551358+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:48.925477+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:48.925477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:02.846482+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"387'13368","reported_seq":13525,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:40.617259+0000","last_change":"2024-03-11T22:25:01.501408+0000","last_active":"2024-03-11T22:25:40.617259+0000","last_peered":"2024-03-11T22:25:40.617259+0000","last_clean":"2024-03-11T22:25:40.617259+0000","last_became_active":"2024-03-11T22:22:52.799804+0000","last_became_peered":"2024-03-11T22:22:52.799804+0000","last_unstale":"2024-03-11T22:25:40.617259+0000","last_undegraded":"2024-03-11T22:25:40.617259+0000","last_fullsized":"2024-03-11T22:25:40.617259+0000","mapping_epoch":348,"log_start":"373'8300","ondisk_log_start":"373'8300","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10662","last_scrub_stamp":"2024-03-11T22:25:01.500954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:01.500954+0000","objects_scrubbed":6034,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:26.252693+0000","scrub_duration":1796,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":3264,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13368,"num_write_kb":532224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":473,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.555311+0000","last_change":"2024-03-11T22:22:53.846450+0000","last_active":"2024-03-11T22:25:27.555311+0000","last_peered":"2024-03-11T22:25:27.555311+0000","last_clean":"2024-03-11T22:25:27.555311+0000","last_became_active":"2024-03-11T22:20:29.438896+0000","last_became_peered":"2024-03-11T22:20:29.438896+0000","last_unstale":"2024-03-11T22:25:27.555311+0000","last_undegraded":"2024-03-11T22:25:27.555311+0000","last_fullsized":"2024-03-11T22:25:27.555311+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:53.846355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:22:53.846355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"273'30","reported_seq":786,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.554252+0000","last_change":"2024-03-11T22:23:05.635969+0000","last_active":"2024-03-11T22:25:27.554252+0000","last_peered":"2024-03-11T22:25:27.554252+0000","last_clean":"2024-03-11T22:25:27.554252+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:25:27.554252+0000","last_undegraded":"2024-03-11T22:25:27.554252+0000","last_fullsized":"2024-03-11T22:25:27.554252+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'30","last_scrub_stamp":"2024-03-11T22:23:05.635556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:05.635556+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754520+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":708,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.559176+0000","last_change":"2024-03-11T22:23:57.680282+0000","last_active":"2024-03-11T22:25:27.559176+0000","last_peered":"2024-03-11T22:25:27.559176+0000","last_clean":"2024-03-11T22:25:27.559176+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:25:27.559176+0000","last_undegraded":"2024-03-11T22:25:27.559176+0000","last_fullsized":"2024-03-11T22:25:27.559176+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:57.680156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:57.680156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:30.767957+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":455,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.561800+0000","last_change":"2024-03-11T22:22:04.830996+0000","last_active":"2024-03-11T22:25:27.561800+0000","last_peered":"2024-03-11T22:25:27.561800+0000","last_clean":"2024-03-11T22:25:27.561800+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:25:27.561800+0000","last_undegraded":"2024-03-11T22:25:27.561800+0000","last_fullsized":"2024-03-11T22:25:27.561800+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:04.830875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:04.830875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754271+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":965,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.549045+0000","last_change":"2024-03-11T22:25:05.774090+0000","last_active":"2024-03-11T22:25:27.549045+0000","last_peered":"2024-03-11T22:25:27.549045+0000","last_clean":"2024-03-11T22:25:27.549045+0000","last_became_active":"2024-03-11T22:18:20.240728+0000","last_became_peered":"2024-03-11T22:18:20.240728+0000","last_unstale":"2024-03-11T22:25:27.549045+0000","last_undegraded":"2024-03-11T22:25:27.549045+0000","last_fullsized":"2024-03-11T22:25:27.549045+0000","mapping_epoch":175,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:25:05.773963+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:25:05.773963+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.773963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:17.950615+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.f","version":"387'13130","reported_seq":13287,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:40.598454+0000","last_change":"2024-03-11T22:25:00.782159+0000","last_active":"2024-03-11T22:25:40.598454+0000","last_peered":"2024-03-11T22:25:40.598454+0000","last_clean":"2024-03-11T22:25:40.598454+0000","last_became_active":"2024-03-11T22:22:52.469064+0000","last_became_peered":"2024-03-11T22:22:52.469064+0000","last_unstale":"2024-03-11T22:25:40.598454+0000","last_undegraded":"2024-03-11T22:25:40.598454+0000","last_fullsized":"2024-03-11T22:25:40.598454+0000","mapping_epoch":348,"log_start":"373'8100","ondisk_log_start":"373'8100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10438","last_scrub_stamp":"2024-03-11T22:25:00.781894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:00.781894+0000","objects_scrubbed":6054,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:20.149816+0000","scrub_duration":2051,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215482368,"num_objects":3288,"num_object_clones":0,"num_object_copies":9864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13130,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":895,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:33.554408+0000","last_change":"2024-03-11T22:23:07.693414+0000","last_active":"2024-03-11T22:25:33.554408+0000","last_peered":"2024-03-11T22:25:33.554408+0000","last_clean":"2024-03-11T22:25:33.554408+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:25:33.554408+0000","last_undegraded":"2024-03-11T22:25:33.554408+0000","last_fullsized":"2024-03-11T22:25:33.554408+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:23:07.693340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:07.693340+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:43.004075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"273'28","reported_seq":524,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.551414+0000","last_change":"2024-03-11T22:24:44.976788+0000","last_active":"2024-03-11T22:25:27.551414+0000","last_peered":"2024-03-11T22:25:27.551414+0000","last_clean":"2024-03-11T22:25:27.551414+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:25:27.551414+0000","last_undegraded":"2024-03-11T22:25:27.551414+0000","last_fullsized":"2024-03-11T22:25:27.551414+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:24:44.976524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:44.976524+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:13.465935+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":697,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.544816+0000","last_change":"2024-03-11T22:22:56.087742+0000","last_active":"2024-03-11T22:25:27.544816+0000","last_peered":"2024-03-11T22:25:27.544816+0000","last_clean":"2024-03-11T22:25:27.544816+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:25:27.544816+0000","last_undegraded":"2024-03-11T22:25:27.544816+0000","last_fullsized":"2024-03-11T22:25:27.544816+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:22:56.087614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:22:56.087614+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.868397+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":651,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.544990+0000","last_change":"2024-03-11T22:23:59.089317+0000","last_active":"2024-03-11T22:25:27.544990+0000","last_peered":"2024-03-11T22:25:27.544990+0000","last_clean":"2024-03-11T22:25:27.544990+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:25:27.544990+0000","last_undegraded":"2024-03-11T22:25:27.544990+0000","last_fullsized":"2024-03-11T22:25:27.544990+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:59.089227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:23:59.089227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":1012,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.555069+0000","last_change":"2024-03-11T22:23:39.755855+0000","last_active":"2024-03-11T22:25:27.555069+0000","last_peered":"2024-03-11T22:25:27.555069+0000","last_clean":"2024-03-11T22:25:27.555069+0000","last_became_active":"2024-03-11T22:20:32.293085+0000","last_became_peered":"2024-03-11T22:20:32.293085+0000","last_unstale":"2024-03-11T22:25:27.555069+0000","last_undegraded":"2024-03-11T22:25:27.555069+0000","last_fullsized":"2024-03-11T22:25:27.555069+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:39.755701+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:39.755701+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"387'12987","reported_seq":13076,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:37.583173+0000","last_change":"2024-03-11T22:22:52.799793+0000","last_active":"2024-03-11T22:25:37.583173+0000","last_peered":"2024-03-11T22:25:37.583173+0000","last_clean":"2024-03-11T22:25:37.583173+0000","last_became_active":"2024-03-11T22:22:52.799425+0000","last_became_peered":"2024-03-11T22:22:52.799425+0000","last_unstale":"2024-03-11T22:25:37.583173+0000","last_undegraded":"2024-03-11T22:25:37.583173+0000","last_fullsized":"2024-03-11T22:25:37.583173+0000","mapping_epoch":348,"log_start":"373'8200","ondisk_log_start":"373'8200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":4787,"log_dups_size":0,"ondisk_log_size":4787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:39.983766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227082240,"num_objects":3465,"num_object_clones":0,"num_object_copies":10395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12987,"num_write_kb":526464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"378'9","reported_seq":911,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:28.555883+0000","last_change":"2024-03-11T22:24:54.792194+0000","last_active":"2024-03-11T22:25:28.555883+0000","last_peered":"2024-03-11T22:25:28.555883+0000","last_clean":"2024-03-11T22:25:28.555883+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:25:28.555883+0000","last_undegraded":"2024-03-11T22:25:28.555883+0000","last_fullsized":"2024-03-11T22:25:28.555883+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9","last_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_deep_scrub":"378'9","last_deep_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_clean_scrub_stamp":"2024-03-11T22:24:54.792049+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:07.965072+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"273'20","reported_seq":739,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.551385+0000","last_change":"2024-03-11T22:24:47.479305+0000","last_active":"2024-03-11T22:25:27.551385+0000","last_peered":"2024-03-11T22:25:27.551385+0000","last_clean":"2024-03-11T22:25:27.551385+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:25:27.551385+0000","last_undegraded":"2024-03-11T22:25:27.551385+0000","last_fullsized":"2024-03-11T22:25:27.551385+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:24:47.479158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:47.479158+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:09.501559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":705,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.553482+0000","last_change":"2024-03-11T22:23:42.943711+0000","last_active":"2024-03-11T22:25:27.553482+0000","last_peered":"2024-03-11T22:25:27.553482+0000","last_clean":"2024-03-11T22:25:27.553482+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:25:27.553482+0000","last_undegraded":"2024-03-11T22:25:27.553482+0000","last_fullsized":"2024-03-11T22:25:27.553482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:42.943546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:42.943546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.993216+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":685,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548384+0000","last_change":"2024-03-11T22:22:55.110611+0000","last_active":"2024-03-11T22:25:27.548384+0000","last_peered":"2024-03-11T22:25:27.548384+0000","last_clean":"2024-03-11T22:25:27.548384+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:25:27.548384+0000","last_undegraded":"2024-03-11T22:25:27.548384+0000","last_fullsized":"2024-03-11T22:25:27.548384+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:55.110461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:55.110461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.872268+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":937,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.543656+0000","last_change":"2024-03-11T22:23:09.000258+0000","last_active":"2024-03-11T22:25:27.543656+0000","last_peered":"2024-03-11T22:25:27.543656+0000","last_clean":"2024-03-11T22:25:27.543656+0000","last_became_active":"2024-03-11T22:20:32.290512+0000","last_became_peered":"2024-03-11T22:20:32.290512+0000","last_unstale":"2024-03-11T22:25:27.543656+0000","last_undegraded":"2024-03-11T22:25:27.543656+0000","last_fullsized":"2024-03-11T22:25:27.543656+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:09.000091+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:09.000091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"387'13301","reported_seq":13427,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:40.593840+0000","last_change":"2024-03-11T22:25:07.810616+0000","last_active":"2024-03-11T22:25:40.593840+0000","last_peered":"2024-03-11T22:25:40.593840+0000","last_clean":"2024-03-11T22:25:40.593840+0000","last_became_active":"2024-03-11T22:22:52.797795+0000","last_became_peered":"2024-03-11T22:22:52.797795+0000","last_unstale":"2024-03-11T22:25:40.593840+0000","last_undegraded":"2024-03-11T22:25:40.593840+0000","last_fullsized":"2024-03-11T22:25:40.593840+0000","mapping_epoch":348,"log_start":"373'8300","ondisk_log_start":"373'8300","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":3299,"log_size":5001,"log_dups_size":0,"ondisk_log_size":5001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217841688,"num_objects":3325,"num_object_clones":0,"num_object_copies":9975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13301,"num_write_kb":531969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":713,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.543005+0000","last_change":"2024-03-11T22:24:00.941070+0000","last_active":"2024-03-11T22:25:27.543005+0000","last_peered":"2024-03-11T22:25:27.543005+0000","last_clean":"2024-03-11T22:25:27.543005+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:25:27.543005+0000","last_undegraded":"2024-03-11T22:25:27.543005+0000","last_fullsized":"2024-03-11T22:25:27.543005+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:00.940803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.940803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"273'29","reported_seq":744,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.547086+0000","last_change":"2024-03-11T22:23:37.154525+0000","last_active":"2024-03-11T22:25:27.547086+0000","last_peered":"2024-03-11T22:25:27.547086+0000","last_clean":"2024-03-11T22:25:27.547086+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:25:27.547086+0000","last_undegraded":"2024-03-11T22:25:27.547086+0000","last_fullsized":"2024-03-11T22:25:27.547086+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'29","last_scrub_stamp":"2024-03-11T22:23:37.154389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:37.154389+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.872540+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":705,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548463+0000","last_change":"2024-03-11T22:23:04.706330+0000","last_active":"2024-03-11T22:25:27.548463+0000","last_peered":"2024-03-11T22:25:27.548463+0000","last_clean":"2024-03-11T22:25:27.548463+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:25:27.548463+0000","last_undegraded":"2024-03-11T22:25:27.548463+0000","last_fullsized":"2024-03-11T22:25:27.548463+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:04.706082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:04.706082+0000","last_clean_scrub_stamp":"2024-03-11T22:23:04.706082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:29.517421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":693,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.546644+0000","last_change":"2024-03-11T22:22:58.736160+0000","last_active":"2024-03-11T22:25:27.546644+0000","last_peered":"2024-03-11T22:25:27.546644+0000","last_clean":"2024-03-11T22:25:27.546644+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:25:27.546644+0000","last_undegraded":"2024-03-11T22:25:27.546644+0000","last_fullsized":"2024-03-11T22:25:27.546644+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:58.736057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:58.736057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:29.530022+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":912,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.543917+0000","last_change":"2024-03-11T22:23:47.783004+0000","last_active":"2024-03-11T22:25:27.543917+0000","last_peered":"2024-03-11T22:25:27.543917+0000","last_clean":"2024-03-11T22:25:27.543917+0000","last_became_active":"2024-03-11T22:18:17.238247+0000","last_became_peered":"2024-03-11T22:18:17.238247+0000","last_unstale":"2024-03-11T22:25:27.543917+0000","last_undegraded":"2024-03-11T22:25:27.543917+0000","last_fullsized":"2024-03-11T22:25:27.543917+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:47.782891+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:47.782891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"387'13001","reported_seq":13156,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:37.572901+0000","last_change":"2024-03-11T22:25:05.585418+0000","last_active":"2024-03-11T22:25:37.572901+0000","last_peered":"2024-03-11T22:25:37.572901+0000","last_clean":"2024-03-11T22:25:37.572901+0000","last_became_active":"2024-03-11T22:22:52.496115+0000","last_became_peered":"2024-03-11T22:22:52.496115+0000","last_unstale":"2024-03-11T22:25:37.572901+0000","last_undegraded":"2024-03-11T22:25:37.572901+0000","last_fullsized":"2024-03-11T22:25:37.572901+0000","mapping_epoch":348,"log_start":"373'8200","ondisk_log_start":"373'8200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'10743","last_scrub_stamp":"2024-03-11T22:25:05.585079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.585079+0000","objects_scrubbed":5803,"log_size":4801,"log_dups_size":0,"ondisk_log_size":4801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:06.494058+0000","scrub_duration":1911,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228130816,"num_objects":3481,"num_object_clones":0,"num_object_copies":10443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13001,"num_write_kb":527424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":696,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.555336+0000","last_change":"2024-03-11T22:24:39.906388+0000","last_active":"2024-03-11T22:25:27.555336+0000","last_peered":"2024-03-11T22:25:27.555336+0000","last_clean":"2024-03-11T22:25:27.555336+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:25:27.555336+0000","last_undegraded":"2024-03-11T22:25:27.555336+0000","last_fullsized":"2024-03-11T22:25:27.555336+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:39.906243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.906243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:00.278120+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"273'24","reported_seq":736,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.544717+0000","last_change":"2024-03-11T22:23:41.208818+0000","last_active":"2024-03-11T22:25:27.544717+0000","last_peered":"2024-03-11T22:25:27.544717+0000","last_clean":"2024-03-11T22:25:27.544717+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:25:27.544717+0000","last_undegraded":"2024-03-11T22:25:27.544717+0000","last_fullsized":"2024-03-11T22:25:27.544717+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'24","last_scrub_stamp":"2024-03-11T22:23:41.208655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:41.208655+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.872420+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":1713,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:40.393282+0000","last_change":"2024-03-11T22:22:19.742008+0000","last_active":"2024-03-11T22:25:40.393282+0000","last_peered":"2024-03-11T22:25:40.393282+0000","last_clean":"2024-03-11T22:25:40.393282+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:25:40.393282+0000","last_undegraded":"2024-03-11T22:25:40.393282+0000","last_fullsized":"2024-03-11T22:25:40.393282+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:22:19.741884+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:22:19.741884+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":993,"num_read_kb":988,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":705,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.545168+0000","last_change":"2024-03-11T22:24:46.870653+0000","last_active":"2024-03-11T22:25:27.545168+0000","last_peered":"2024-03-11T22:25:27.545168+0000","last_clean":"2024-03-11T22:25:27.545168+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:25:27.545168+0000","last_undegraded":"2024-03-11T22:25:27.545168+0000","last_fullsized":"2024-03-11T22:25:27.545168+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:46.870527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.870527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:51.216116+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":701,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.546364+0000","last_change":"2024-03-11T22:22:22.750641+0000","last_active":"2024-03-11T22:25:27.546364+0000","last_peered":"2024-03-11T22:25:27.546364+0000","last_clean":"2024-03-11T22:25:27.546364+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:25:27.546364+0000","last_undegraded":"2024-03-11T22:25:27.546364+0000","last_fullsized":"2024-03-11T22:25:27.546364+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:22.750560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:22.750560+0000","last_clean_scrub_stamp":"2024-03-11T22:22:22.750560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:29.517352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":1002,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.549793+0000","last_change":"2024-03-11T22:24:45.428664+0000","last_active":"2024-03-11T22:25:27.549793+0000","last_peered":"2024-03-11T22:25:27.549793+0000","last_clean":"2024-03-11T22:25:27.549793+0000","last_became_active":"2024-03-11T22:14:38.164101+0000","last_became_peered":"2024-03-11T22:14:38.164101+0000","last_unstale":"2024-03-11T22:25:27.549793+0000","last_undegraded":"2024-03-11T22:25:27.549793+0000","last_fullsized":"2024-03-11T22:25:27.549793+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:45.428590+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:45.428590+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:04.484198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"387'13291","reported_seq":13380,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:40.613472+0000","last_change":"2024-03-11T22:22:52.472336+0000","last_active":"2024-03-11T22:25:40.613472+0000","last_peered":"2024-03-11T22:25:40.613472+0000","last_clean":"2024-03-11T22:25:40.613472+0000","last_became_active":"2024-03-11T22:22:52.472068+0000","last_became_peered":"2024-03-11T22:22:52.472068+0000","last_unstale":"2024-03-11T22:25:40.613472+0000","last_undegraded":"2024-03-11T22:25:40.613472+0000","last_fullsized":"2024-03-11T22:25:40.613472+0000","mapping_epoch":348,"log_start":"373'8200","ondisk_log_start":"373'8200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":5091,"log_dups_size":0,"ondisk_log_size":5091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209649664,"num_objects":3199,"num_object_clones":0,"num_object_copies":9597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13291,"num_write_kb":527680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":671,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548548+0000","last_change":"2024-03-11T22:24:45.777817+0000","last_active":"2024-03-11T22:25:27.548548+0000","last_peered":"2024-03-11T22:25:27.548548+0000","last_clean":"2024-03-11T22:25:27.548548+0000","last_became_active":"2024-03-11T22:20:29.420497+0000","last_became_peered":"2024-03-11T22:20:29.420497+0000","last_unstale":"2024-03-11T22:25:27.548548+0000","last_undegraded":"2024-03-11T22:25:27.548548+0000","last_fullsized":"2024-03-11T22:25:27.548548+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:45.777703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:24:45.777703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:54.742850+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"273'28","reported_seq":963,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:33.555813+0000","last_change":"2024-03-11T22:24:42.729950+0000","last_active":"2024-03-11T22:25:33.555813+0000","last_peered":"2024-03-11T22:25:33.555813+0000","last_clean":"2024-03-11T22:25:33.555813+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:25:33.555813+0000","last_undegraded":"2024-03-11T22:25:33.555813+0000","last_fullsized":"2024-03-11T22:25:33.555813+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:24:42.729799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:42.729799+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.455019+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":697,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.561487+0000","last_change":"2024-03-11T22:24:47.653085+0000","last_active":"2024-03-11T22:25:27.561487+0000","last_peered":"2024-03-11T22:25:27.561487+0000","last_clean":"2024-03-11T22:25:27.561487+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:25:27.561487+0000","last_undegraded":"2024-03-11T22:25:27.561487+0000","last_fullsized":"2024-03-11T22:25:27.561487+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:47.652970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:24:47.652970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.393586+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":1020,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548477+0000","last_change":"2024-03-11T22:24:42.773278+0000","last_active":"2024-03-11T22:25:27.548477+0000","last_peered":"2024-03-11T22:25:27.548477+0000","last_clean":"2024-03-11T22:25:27.548477+0000","last_became_active":"2024-03-11T22:20:32.293740+0000","last_became_peered":"2024-03-11T22:20:32.293740+0000","last_unstale":"2024-03-11T22:25:27.548477+0000","last_undegraded":"2024-03-11T22:25:27.548477+0000","last_fullsized":"2024-03-11T22:25:27.548477+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:42.773024+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:24:42.773024+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:47.036511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":692,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.554633+0000","last_change":"2024-03-11T22:24:54.991281+0000","last_active":"2024-03-11T22:25:27.554633+0000","last_peered":"2024-03-11T22:25:27.554633+0000","last_clean":"2024-03-11T22:25:27.554633+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:25:27.554633+0000","last_undegraded":"2024-03-11T22:25:27.554633+0000","last_fullsized":"2024-03-11T22:25:27.554633+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:54.991081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:54.991081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:59.396448+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"387'13403","reported_seq":13561,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:40.655763+0000","last_change":"2024-03-11T22:25:02.575828+0000","last_active":"2024-03-11T22:25:40.655763+0000","last_peered":"2024-03-11T22:25:40.655763+0000","last_clean":"2024-03-11T22:25:40.655763+0000","last_became_active":"2024-03-11T22:22:52.496328+0000","last_became_peered":"2024-03-11T22:22:52.496328+0000","last_unstale":"2024-03-11T22:25:40.655763+0000","last_undegraded":"2024-03-11T22:25:40.655763+0000","last_fullsized":"2024-03-11T22:25:40.655763+0000","mapping_epoch":348,"log_start":"373'8500","ondisk_log_start":"373'8500","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'10701","last_scrub_stamp":"2024-03-11T22:25:02.575600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:02.575600+0000","objects_scrubbed":6024,"log_size":4903,"log_dups_size":0,"ondisk_log_size":4903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:02.718025+0000","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215023616,"num_objects":3281,"num_object_clones":0,"num_object_copies":9843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13403,"num_write_kb":533888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":720,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.549867+0000","last_change":"2024-03-11T22:24:48.727176+0000","last_active":"2024-03-11T22:25:27.549867+0000","last_peered":"2024-03-11T22:25:27.549867+0000","last_clean":"2024-03-11T22:25:27.549867+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:25:27.549867+0000","last_undegraded":"2024-03-11T22:25:27.549867+0000","last_fullsized":"2024-03-11T22:25:27.549867+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:48.727061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:48.727061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:17.884566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"273'17","reported_seq":741,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.550334+0000","last_change":"2024-03-11T22:24:41.396924+0000","last_active":"2024-03-11T22:25:27.550334+0000","last_peered":"2024-03-11T22:25:27.550334+0000","last_clean":"2024-03-11T22:25:27.550334+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:25:27.550334+0000","last_undegraded":"2024-03-11T22:25:27.550334+0000","last_fullsized":"2024-03-11T22:25:27.550334+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'17","last_scrub_stamp":"2024-03-11T22:24:41.396754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.396754+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:02.581690+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":738,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.547485+0000","last_change":"2024-03-11T22:23:45.364864+0000","last_active":"2024-03-11T22:25:27.547485+0000","last_peered":"2024-03-11T22:25:27.547485+0000","last_clean":"2024-03-11T22:25:27.547485+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:25:27.547485+0000","last_undegraded":"2024-03-11T22:25:27.547485+0000","last_fullsized":"2024-03-11T22:25:27.547485+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:45.364724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:23:45.364724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:31.853062+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":713,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548506+0000","last_change":"2024-03-11T22:23:58.411257+0000","last_active":"2024-03-11T22:25:27.548506+0000","last_peered":"2024-03-11T22:25:27.548506+0000","last_clean":"2024-03-11T22:25:27.548506+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:25:27.548506+0000","last_undegraded":"2024-03-11T22:25:27.548506+0000","last_fullsized":"2024-03-11T22:25:27.548506+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:58.411052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.411052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:29.530170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":651,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.543507+0000","last_change":"2024-03-11T22:24:38.834963+0000","last_active":"2024-03-11T22:25:27.543507+0000","last_peered":"2024-03-11T22:25:27.543507+0000","last_clean":"2024-03-11T22:25:27.543507+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:25:27.543507+0000","last_undegraded":"2024-03-11T22:25:27.543507+0000","last_fullsized":"2024-03-11T22:25:27.543507+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:38.834828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:24:38.834828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.525769+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":1019,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.547370+0000","last_change":"2024-03-11T22:23:11.030109+0000","last_active":"2024-03-11T22:25:27.547370+0000","last_peered":"2024-03-11T22:25:27.547370+0000","last_clean":"2024-03-11T22:25:27.547370+0000","last_became_active":"2024-03-11T22:20:32.295175+0000","last_became_peered":"2024-03-11T22:20:32.295175+0000","last_unstale":"2024-03-11T22:25:27.547370+0000","last_undegraded":"2024-03-11T22:25:27.547370+0000","last_fullsized":"2024-03-11T22:25:27.547370+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:11.029981+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:11.029981+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.872299+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"387'13381","reported_seq":13536,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:40.578976+0000","last_change":"2024-03-11T22:25:05.305867+0000","last_active":"2024-03-11T22:25:40.578976+0000","last_peered":"2024-03-11T22:25:40.578976+0000","last_clean":"2024-03-11T22:25:40.578976+0000","last_became_active":"2024-03-11T22:22:52.467125+0000","last_became_peered":"2024-03-11T22:22:52.467125+0000","last_unstale":"2024-03-11T22:25:40.578976+0000","last_undegraded":"2024-03-11T22:25:40.578976+0000","last_fullsized":"2024-03-11T22:25:40.578976+0000","mapping_epoch":348,"log_start":"373'8300","ondisk_log_start":"373'8300","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'10771","last_scrub_stamp":"2024-03-11T22:25:05.305569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.305569+0000","objects_scrubbed":5888,"log_size":5081,"log_dups_size":0,"ondisk_log_size":5081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:07.737385+0000","scrub_duration":1552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212664320,"num_objects":3245,"num_object_clones":0,"num_object_copies":9735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13381,"num_write_kb":532032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":697,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.544490+0000","last_change":"2024-03-11T22:24:01.031042+0000","last_active":"2024-03-11T22:25:27.544490+0000","last_peered":"2024-03-11T22:25:27.544490+0000","last_clean":"2024-03-11T22:25:27.544490+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:25:27.544490+0000","last_undegraded":"2024-03-11T22:25:27.544490+0000","last_fullsized":"2024-03-11T22:25:27.544490+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:01.030925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:01.030925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"273'40","reported_seq":981,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:33.554848+0000","last_change":"2024-03-11T22:23:57.429937+0000","last_active":"2024-03-11T22:25:33.554848+0000","last_peered":"2024-03-11T22:25:33.554848+0000","last_clean":"2024-03-11T22:25:33.554848+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:25:33.554848+0000","last_undegraded":"2024-03-11T22:25:33.554848+0000","last_fullsized":"2024-03-11T22:25:33.554848+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'40","last_scrub_stamp":"2024-03-11T22:23:57.429825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:57.429825+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:40.661058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":464,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.546982+0000","last_change":"2024-03-11T22:23:56.935550+0000","last_active":"2024-03-11T22:25:27.546982+0000","last_peered":"2024-03-11T22:25:27.546982+0000","last_clean":"2024-03-11T22:25:27.546982+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:25:27.546982+0000","last_undegraded":"2024-03-11T22:25:27.546982+0000","last_fullsized":"2024-03-11T22:25:27.546982+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:56.934997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:56.934997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.872508+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":717,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.552115+0000","last_change":"2024-03-11T22:25:05.938232+0000","last_active":"2024-03-11T22:25:27.552115+0000","last_peered":"2024-03-11T22:25:27.552115+0000","last_clean":"2024-03-11T22:25:27.552115+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:25:27.552115+0000","last_undegraded":"2024-03-11T22:25:27.552115+0000","last_fullsized":"2024-03-11T22:25:27.552115+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:25:05.938003+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.938003+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:25.242523+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":688,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.558603+0000","last_change":"2024-03-11T22:23:03.679822+0000","last_active":"2024-03-11T22:25:27.558603+0000","last_peered":"2024-03-11T22:25:27.558603+0000","last_clean":"2024-03-11T22:25:27.558603+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:25:27.558603+0000","last_undegraded":"2024-03-11T22:25:27.558603+0000","last_fullsized":"2024-03-11T22:25:27.558603+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:03.679700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:23:03.679700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754376+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":1001,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.551092+0000","last_change":"2024-03-11T22:24:56.886620+0000","last_active":"2024-03-11T22:25:27.551092+0000","last_peered":"2024-03-11T22:25:27.551092+0000","last_clean":"2024-03-11T22:25:27.551092+0000","last_became_active":"2024-03-11T22:14:38.164859+0000","last_became_peered":"2024-03-11T22:14:38.164859+0000","last_unstale":"2024-03-11T22:25:27.551092+0000","last_undegraded":"2024-03-11T22:25:27.551092+0000","last_fullsized":"2024-03-11T22:25:27.551092+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:56.886500+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:56.886500+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:58.328021+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"387'13205","reported_seq":13365,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:41.061391+0000","last_change":"2024-03-11T22:24:55.556260+0000","last_active":"2024-03-11T22:25:41.061391+0000","last_peered":"2024-03-11T22:25:41.061391+0000","last_clean":"2024-03-11T22:25:41.061391+0000","last_became_active":"2024-03-11T22:22:52.469442+0000","last_became_peered":"2024-03-11T22:22:52.469442+0000","last_unstale":"2024-03-11T22:25:41.061391+0000","last_undegraded":"2024-03-11T22:25:41.061391+0000","last_fullsized":"2024-03-11T22:25:41.061391+0000","mapping_epoch":348,"log_start":"371'7200","ondisk_log_start":"371'7200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9999","last_scrub_stamp":"2024-03-11T22:24:55.555941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:24:55.555941+0000","objects_scrubbed":6582,"log_size":6005,"log_dups_size":0,"ondisk_log_size":6005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:00.936378+0000","scrub_duration":2782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214761472,"num_objects":3277,"num_object_clones":0,"num_object_copies":9831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13205,"num_write_kb":527424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"273'15","reported_seq":733,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.547334+0000","last_change":"2024-03-11T22:23:36.167248+0000","last_active":"2024-03-11T22:25:27.547334+0000","last_peered":"2024-03-11T22:25:27.547334+0000","last_clean":"2024-03-11T22:25:27.547334+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:25:27.547334+0000","last_undegraded":"2024-03-11T22:25:27.547334+0000","last_fullsized":"2024-03-11T22:25:27.547334+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:23:36.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:36.167121+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.872389+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":704,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.552405+0000","last_change":"2024-03-11T22:24:58.853739+0000","last_active":"2024-03-11T22:25:27.552405+0000","last_peered":"2024-03-11T22:25:27.552405+0000","last_clean":"2024-03-11T22:25:27.552405+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:25:27.552405+0000","last_undegraded":"2024-03-11T22:25:27.552405+0000","last_fullsized":"2024-03-11T22:25:27.552405+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:58.853630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:58.853630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:23.082251+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":720,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.547371+0000","last_change":"2024-03-11T22:24:00.422907+0000","last_active":"2024-03-11T22:25:27.547371+0000","last_peered":"2024-03-11T22:25:27.547371+0000","last_clean":"2024-03-11T22:25:27.547371+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:25:27.547371+0000","last_undegraded":"2024-03-11T22:25:27.547371+0000","last_fullsized":"2024-03-11T22:25:27.547371+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:00.422761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.422761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.780771+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":704,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.545365+0000","last_change":"2024-03-11T22:24:53.948958+0000","last_active":"2024-03-11T22:25:27.545365+0000","last_peered":"2024-03-11T22:25:27.545365+0000","last_clean":"2024-03-11T22:25:27.545365+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:25:27.545365+0000","last_undegraded":"2024-03-11T22:25:27.545365+0000","last_fullsized":"2024-03-11T22:25:27.545365+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:53.948853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.948853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.727979+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"93'130","reported_seq":846,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.558187+0000","last_change":"2024-03-11T22:23:32.711840+0000","last_active":"2024-03-11T22:25:27.558187+0000","last_peered":"2024-03-11T22:25:27.558187+0000","last_clean":"2024-03-11T22:25:27.558187+0000","last_became_active":"2024-03-11T22:20:32.292676+0000","last_became_peered":"2024-03-11T22:20:32.292676+0000","last_unstale":"2024-03-11T22:25:27.558187+0000","last_undegraded":"2024-03-11T22:25:27.558187+0000","last_fullsized":"2024-03-11T22:25:27.558187+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:32.711693+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:32.711693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754820+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"387'12875","reported_seq":12963,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:37.542188+0000","last_change":"2024-03-11T22:22:52.800885+0000","last_active":"2024-03-11T22:25:37.542188+0000","last_peered":"2024-03-11T22:25:37.542188+0000","last_clean":"2024-03-11T22:25:37.542188+0000","last_became_active":"2024-03-11T22:22:52.800526+0000","last_became_peered":"2024-03-11T22:22:52.800526+0000","last_unstale":"2024-03-11T22:25:37.542188+0000","last_undegraded":"2024-03-11T22:25:37.542188+0000","last_fullsized":"2024-03-11T22:25:37.542188+0000","mapping_epoch":348,"log_start":"373'8100","ondisk_log_start":"373'8100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:39.983702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228524032,"num_objects":3487,"num_object_clones":0,"num_object_copies":10461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12875,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"273'35","reported_seq":823,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.542893+0000","last_change":"2024-03-11T22:24:46.752170+0000","last_active":"2024-03-11T22:25:27.542893+0000","last_peered":"2024-03-11T22:25:27.542893+0000","last_clean":"2024-03-11T22:25:27.542893+0000","last_became_active":"2024-03-11T22:20:29.418041+0000","last_became_peered":"2024-03-11T22:20:29.418041+0000","last_unstale":"2024-03-11T22:25:27.542893+0000","last_undegraded":"2024-03-11T22:25:27.542893+0000","last_fullsized":"2024-03-11T22:25:27.542893+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'35","last_scrub_stamp":"2024-03-11T22:24:46.751969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.751969+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:58.638998+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":647,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.553768+0000","last_change":"2024-03-11T22:23:10.963581+0000","last_active":"2024-03-11T22:25:27.553768+0000","last_peered":"2024-03-11T22:25:27.553768+0000","last_clean":"2024-03-11T22:25:27.553768+0000","last_became_active":"2024-03-11T22:20:29.413664+0000","last_became_peered":"2024-03-11T22:20:29.413664+0000","last_unstale":"2024-03-11T22:25:27.553768+0000","last_undegraded":"2024-03-11T22:25:27.553768+0000","last_fullsized":"2024-03-11T22:25:27.553768+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:10.963448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:10.963448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.992955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":712,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548393+0000","last_change":"2024-03-11T22:23:40.405746+0000","last_active":"2024-03-11T22:25:27.548393+0000","last_peered":"2024-03-11T22:25:27.548393+0000","last_clean":"2024-03-11T22:25:27.548393+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:25:27.548393+0000","last_undegraded":"2024-03-11T22:25:27.548393+0000","last_fullsized":"2024-03-11T22:25:27.548393+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:40.405626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:23:40.405626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:31.853115+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":670,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.559161+0000","last_change":"2024-03-11T22:23:51.364137+0000","last_active":"2024-03-11T22:25:27.559161+0000","last_peered":"2024-03-11T22:25:27.559161+0000","last_clean":"2024-03-11T22:25:27.559161+0000","last_became_active":"2024-03-11T22:20:29.415565+0000","last_became_peered":"2024-03-11T22:20:29.415565+0000","last_unstale":"2024-03-11T22:25:27.559161+0000","last_undegraded":"2024-03-11T22:25:27.559161+0000","last_fullsized":"2024-03-11T22:25:27.559161+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:51.362666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:20.931122+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.362666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:31.853033+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"93'130","reported_seq":897,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.543509+0000","last_change":"2024-03-11T22:23:02.040555+0000","last_active":"2024-03-11T22:25:27.543509+0000","last_peered":"2024-03-11T22:25:27.543509+0000","last_clean":"2024-03-11T22:25:27.543509+0000","last_became_active":"2024-03-11T22:14:43.195433+0000","last_became_peered":"2024-03-11T22:14:43.195433+0000","last_unstale":"2024-03-11T22:25:27.543509+0000","last_undegraded":"2024-03-11T22:25:27.543509+0000","last_fullsized":"2024-03-11T22:25:27.543509+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:02.040354+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:02.040354+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"387'13139","reported_seq":13233,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:40.600370+0000","last_change":"2024-03-11T22:25:07.811931+0000","last_active":"2024-03-11T22:25:40.600370+0000","last_peered":"2024-03-11T22:25:40.600370+0000","last_clean":"2024-03-11T22:25:40.600370+0000","last_became_active":"2024-03-11T22:22:52.465673+0000","last_became_peered":"2024-03-11T22:22:52.465673+0000","last_unstale":"2024-03-11T22:25:40.600370+0000","last_undegraded":"2024-03-11T22:25:40.600370+0000","last_fullsized":"2024-03-11T22:25:40.600370+0000","mapping_epoch":348,"log_start":"373'8100","ondisk_log_start":"373'8100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":199,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214368256,"num_objects":3271,"num_object_clones":0,"num_object_copies":9813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13139,"num_write_kb":525120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":720,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.558901+0000","last_change":"2024-03-11T22:24:56.704048+0000","last_active":"2024-03-11T22:25:27.558901+0000","last_peered":"2024-03-11T22:25:27.558901+0000","last_clean":"2024-03-11T22:25:27.558901+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:25:27.558901+0000","last_undegraded":"2024-03-11T22:25:27.558901+0000","last_fullsized":"2024-03-11T22:25:27.558901+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:56.703933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:24:56.703933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:24.261508+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"273'18","reported_seq":520,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.551403+0000","last_change":"2024-03-11T22:23:46.897451+0000","last_active":"2024-03-11T22:25:27.551403+0000","last_peered":"2024-03-11T22:25:27.551403+0000","last_clean":"2024-03-11T22:25:27.551403+0000","last_became_active":"2024-03-11T22:14:52.247400+0000","last_became_peered":"2024-03-11T22:14:52.247400+0000","last_unstale":"2024-03-11T22:25:27.551403+0000","last_undegraded":"2024-03-11T22:25:27.551403+0000","last_fullsized":"2024-03-11T22:25:27.551403+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:23:46.897161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.897161+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.993137+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":486,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.547809+0000","last_change":"2024-03-11T22:24:53.435632+0000","last_active":"2024-03-11T22:25:27.547809+0000","last_peered":"2024-03-11T22:25:27.547809+0000","last_clean":"2024-03-11T22:25:27.547809+0000","last_became_active":"2024-03-11T22:20:29.416729+0000","last_became_peered":"2024-03-11T22:20:29.416729+0000","last_unstale":"2024-03-11T22:25:27.547809+0000","last_undegraded":"2024-03-11T22:25:27.547809+0000","last_fullsized":"2024-03-11T22:25:27.547809+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:53.435516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.435516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:08.499620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":698,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.555542+0000","last_change":"2024-03-11T22:23:01.776787+0000","last_active":"2024-03-11T22:25:27.555542+0000","last_peered":"2024-03-11T22:25:27.555542+0000","last_clean":"2024-03-11T22:25:27.555542+0000","last_became_active":"2024-03-11T22:20:29.438413+0000","last_became_peered":"2024-03-11T22:20:29.438413+0000","last_unstale":"2024-03-11T22:25:27.555542+0000","last_undegraded":"2024-03-11T22:25:27.555542+0000","last_fullsized":"2024-03-11T22:25:27.555542+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:23:01.776660+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:23:01.776660+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754641+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"93'130","reported_seq":1022,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.553862+0000","last_change":"2024-03-11T22:23:10.731794+0000","last_active":"2024-03-11T22:25:27.553862+0000","last_peered":"2024-03-11T22:25:27.553862+0000","last_clean":"2024-03-11T22:25:27.553862+0000","last_became_active":"2024-03-11T22:17:57.246241+0000","last_became_peered":"2024-03-11T22:17:57.246241+0000","last_unstale":"2024-03-11T22:25:27.553862+0000","last_undegraded":"2024-03-11T22:25:27.553862+0000","last_fullsized":"2024-03-11T22:25:27.553862+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:10.731483+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:10.731483+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754412+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"387'13400","reported_seq":13489,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:40.630226+0000","last_change":"2024-03-11T22:22:52.798264+0000","last_active":"2024-03-11T22:25:40.630226+0000","last_peered":"2024-03-11T22:25:40.630226+0000","last_clean":"2024-03-11T22:25:40.630226+0000","last_became_active":"2024-03-11T22:22:52.797922+0000","last_became_peered":"2024-03-11T22:22:52.797922+0000","last_unstale":"2024-03-11T22:25:40.630226+0000","last_undegraded":"2024-03-11T22:25:40.630226+0000","last_fullsized":"2024-03-11T22:25:40.630226+0000","mapping_epoch":348,"log_start":"374'8500","ondisk_log_start":"374'8500","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":4900,"log_dups_size":0,"ondisk_log_size":4900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:43.100140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206831616,"num_objects":3156,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13400,"num_write_kb":529792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"310'11","reported_seq":878,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:33.555695+0000","last_change":"2024-03-11T22:23:56.945205+0000","last_active":"2024-03-11T22:25:33.555695+0000","last_peered":"2024-03-11T22:25:33.555695+0000","last_clean":"2024-03-11T22:25:33.555695+0000","last_became_active":"2024-03-11T22:20:29.418355+0000","last_became_peered":"2024-03-11T22:20:29.418355+0000","last_unstale":"2024-03-11T22:25:33.555695+0000","last_undegraded":"2024-03-11T22:25:33.555695+0000","last_fullsized":"2024-03-11T22:25:33.555695+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:23:56.945075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:56.945075+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:39.952557+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"273'34","reported_seq":794,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.550831+0000","last_change":"2024-03-11T22:23:47.917460+0000","last_active":"2024-03-11T22:25:27.550831+0000","last_peered":"2024-03-11T22:25:27.550831+0000","last_clean":"2024-03-11T22:25:27.550831+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:25:27.550831+0000","last_undegraded":"2024-03-11T22:25:27.550831+0000","last_fullsized":"2024-03-11T22:25:27.550831+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:23:47.917308+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:23:47.917308+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.993107+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":496,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.544131+0000","last_change":"2024-03-11T22:23:38.153481+0000","last_active":"2024-03-11T22:25:27.544131+0000","last_peered":"2024-03-11T22:25:27.544131+0000","last_clean":"2024-03-11T22:25:27.544131+0000","last_became_active":"2024-03-11T22:20:29.415226+0000","last_became_peered":"2024-03-11T22:20:29.415226+0000","last_unstale":"2024-03-11T22:25:27.544131+0000","last_undegraded":"2024-03-11T22:25:27.544131+0000","last_fullsized":"2024-03-11T22:25:27.544131+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:23:38.153326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:38.153326+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.872571+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":684,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.552356+0000","last_change":"2024-03-11T22:23:44.941526+0000","last_active":"2024-03-11T22:25:27.552356+0000","last_peered":"2024-03-11T22:25:27.552356+0000","last_clean":"2024-03-11T22:25:27.552356+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:25:27.552356+0000","last_undegraded":"2024-03-11T22:25:27.552356+0000","last_fullsized":"2024-03-11T22:25:27.552356+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:44.941284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:44.941284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.993163+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"93'130","reported_seq":1039,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.547673+0000","last_change":"2024-03-11T22:23:44.564522+0000","last_active":"2024-03-11T22:25:27.547673+0000","last_peered":"2024-03-11T22:25:27.547673+0000","last_clean":"2024-03-11T22:25:27.547673+0000","last_became_active":"2024-03-11T22:20:32.295606+0000","last_became_peered":"2024-03-11T22:20:32.295606+0000","last_unstale":"2024-03-11T22:25:27.547673+0000","last_undegraded":"2024-03-11T22:25:27.547673+0000","last_fullsized":"2024-03-11T22:25:27.547673+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:44.564377+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:44.564377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:29.530143+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"273'12","reported_seq":730,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.561084+0000","last_change":"2024-03-11T22:22:16.742232+0000","last_active":"2024-03-11T22:25:27.561084+0000","last_peered":"2024-03-11T22:25:27.561084+0000","last_clean":"2024-03-11T22:25:27.561084+0000","last_became_active":"2024-03-11T22:20:29.441599+0000","last_became_peered":"2024-03-11T22:20:29.441599+0000","last_unstale":"2024-03-11T22:25:27.561084+0000","last_undegraded":"2024-03-11T22:25:27.561084+0000","last_fullsized":"2024-03-11T22:25:27.561084+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'12","last_scrub_stamp":"2024-03-11T22:22:16.742100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:22:16.742100+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":712,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.543139+0000","last_change":"2024-03-11T22:23:48.806202+0000","last_active":"2024-03-11T22:25:27.543139+0000","last_peered":"2024-03-11T22:25:27.543139+0000","last_clean":"2024-03-11T22:25:27.543139+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:25:27.543139+0000","last_undegraded":"2024-03-11T22:25:27.543139+0000","last_fullsized":"2024-03-11T22:25:27.543139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.805960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.805960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":696,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.554250+0000","last_change":"2024-03-11T22:23:59.713751+0000","last_active":"2024-03-11T22:25:27.554250+0000","last_peered":"2024-03-11T22:25:27.554250+0000","last_clean":"2024-03-11T22:25:27.554250+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:25:27.554250+0000","last_undegraded":"2024-03-11T22:25:27.554250+0000","last_fullsized":"2024-03-11T22:25:27.554250+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:59.713626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:23:59.713626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":642,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.543615+0000","last_change":"2024-03-11T22:23:46.876978+0000","last_active":"2024-03-11T22:25:27.543615+0000","last_peered":"2024-03-11T22:25:27.543615+0000","last_clean":"2024-03-11T22:25:27.543615+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:25:27.543615+0000","last_undegraded":"2024-03-11T22:25:27.543615+0000","last_fullsized":"2024-03-11T22:25:27.543615+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:46.876749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.876749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"93'130","reported_seq":1010,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.550922+0000","last_change":"2024-03-11T22:24:52.878462+0000","last_active":"2024-03-11T22:25:27.550922+0000","last_peered":"2024-03-11T22:25:27.550922+0000","last_clean":"2024-03-11T22:25:27.550922+0000","last_became_active":"2024-03-11T22:14:43.194413+0000","last_became_peered":"2024-03-11T22:14:43.194413+0000","last_unstale":"2024-03-11T22:25:27.550922+0000","last_undegraded":"2024-03-11T22:25:27.550922+0000","last_fullsized":"2024-03-11T22:25:27.550922+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:52.878201+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:52.878201+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:04.489712+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"93'130","reported_seq":1009,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.549339+0000","last_change":"2024-03-11T22:23:51.445370+0000","last_active":"2024-03-11T22:25:27.549339+0000","last_peered":"2024-03-11T22:25:27.549339+0000","last_clean":"2024-03-11T22:25:27.549339+0000","last_became_active":"2024-03-11T22:14:43.198216+0000","last_became_peered":"2024-03-11T22:14:43.198216+0000","last_unstale":"2024-03-11T22:25:27.549339+0000","last_undegraded":"2024-03-11T22:25:27.549339+0000","last_fullsized":"2024-03-11T22:25:27.549339+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:51.445038+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:23:51.445038+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.445038+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:29.530112+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"273'19","reported_seq":761,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.545079+0000","last_change":"2024-03-11T22:23:53.926458+0000","last_active":"2024-03-11T22:25:27.545079+0000","last_peered":"2024-03-11T22:25:27.545079+0000","last_clean":"2024-03-11T22:25:27.545079+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:25:27.545079+0000","last_undegraded":"2024-03-11T22:25:27.545079+0000","last_fullsized":"2024-03-11T22:25:27.545079+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_clean_scrub_stamp":"2024-03-11T22:23:53.926316+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:34.905303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":474,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.542634+0000","last_change":"2024-03-11T22:23:35.792737+0000","last_active":"2024-03-11T22:25:27.542634+0000","last_peered":"2024-03-11T22:25:27.542634+0000","last_clean":"2024-03-11T22:25:27.542634+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:25:27.542634+0000","last_undegraded":"2024-03-11T22:25:27.542634+0000","last_fullsized":"2024-03-11T22:25:27.542634+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:35.792609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:35.792609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":700,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.559927+0000","last_change":"2024-03-11T22:24:51.687892+0000","last_active":"2024-03-11T22:25:27.559927+0000","last_peered":"2024-03-11T22:25:27.559927+0000","last_clean":"2024-03-11T22:25:27.559927+0000","last_became_active":"2024-03-11T22:20:29.433005+0000","last_became_peered":"2024-03-11T22:20:29.433005+0000","last_unstale":"2024-03-11T22:25:27.559927+0000","last_undegraded":"2024-03-11T22:25:27.559927+0000","last_fullsized":"2024-03-11T22:25:27.559927+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:51.687767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:51.687767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:20.042381+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":716,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548443+0000","last_change":"2024-03-11T22:23:57.498120+0000","last_active":"2024-03-11T22:25:27.548443+0000","last_peered":"2024-03-11T22:25:27.548443+0000","last_clean":"2024-03-11T22:25:27.548443+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:25:27.548443+0000","last_undegraded":"2024-03-11T22:25:27.548443+0000","last_fullsized":"2024-03-11T22:25:27.548443+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:57.498047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:23:57.498047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:31.853212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"273'19","reported_seq":729,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.553634+0000","last_change":"2024-03-11T22:23:37.882805+0000","last_active":"2024-03-11T22:25:27.553634+0000","last_peered":"2024-03-11T22:25:27.553634+0000","last_clean":"2024-03-11T22:25:27.553634+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:25:27.553634+0000","last_undegraded":"2024-03-11T22:25:27.553634+0000","last_fullsized":"2024-03-11T22:25:27.553634+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:37.882661+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:23:37.882661+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":716,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.547257+0000","last_change":"2024-03-11T22:23:50.030718+0000","last_active":"2024-03-11T22:25:27.547257+0000","last_peered":"2024-03-11T22:25:27.547257+0000","last_clean":"2024-03-11T22:25:27.547257+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:25:27.547257+0000","last_undegraded":"2024-03-11T22:25:27.547257+0000","last_fullsized":"2024-03-11T22:25:27.547257+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:50.030272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:50.030272+0000","last_clean_scrub_stamp":"2024-03-11T22:23:50.030272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.872614+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'11","reported_seq":937,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:33.555012+0000","last_change":"2024-03-11T22:24:53.646061+0000","last_active":"2024-03-11T22:25:33.555012+0000","last_peered":"2024-03-11T22:25:33.555012+0000","last_clean":"2024-03-11T22:25:33.555012+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:25:33.555012+0000","last_undegraded":"2024-03-11T22:25:33.555012+0000","last_fullsized":"2024-03-11T22:25:33.555012+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:24:53.645906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.645906+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:03.406173+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":478,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.551853+0000","last_change":"2024-03-11T22:23:48.490897+0000","last_active":"2024-03-11T22:25:27.551853+0000","last_peered":"2024-03-11T22:25:27.551853+0000","last_clean":"2024-03-11T22:25:27.551853+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:25:27.551853+0000","last_undegraded":"2024-03-11T22:25:27.551853+0000","last_fullsized":"2024-03-11T22:25:27.551853+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.490757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:48.490757+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.490757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:31.853005+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"93'130","reported_seq":995,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.551188+0000","last_change":"2024-03-11T22:23:12.969623+0000","last_active":"2024-03-11T22:25:27.551188+0000","last_peered":"2024-03-11T22:25:27.551188+0000","last_clean":"2024-03-11T22:25:27.551188+0000","last_became_active":"2024-03-11T22:14:43.194350+0000","last_became_peered":"2024-03-11T22:14:43.194350+0000","last_unstale":"2024-03-11T22:25:27.551188+0000","last_undegraded":"2024-03-11T22:25:27.551188+0000","last_fullsized":"2024-03-11T22:25:27.551188+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:12.969362+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:23:12.969362+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.992986+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"273'20","reported_seq":732,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.558627+0000","last_change":"2024-03-11T22:23:00.635612+0000","last_active":"2024-03-11T22:25:27.558627+0000","last_peered":"2024-03-11T22:25:27.558627+0000","last_clean":"2024-03-11T22:25:27.558627+0000","last_became_active":"2024-03-11T22:16:15.815814+0000","last_became_peered":"2024-03-11T22:16:15.815814+0000","last_unstale":"2024-03-11T22:25:27.558627+0000","last_undegraded":"2024-03-11T22:25:27.558627+0000","last_fullsized":"2024-03-11T22:25:27.558627+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:23:00.635505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:00.635505+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":502,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548309+0000","last_change":"2024-03-11T22:24:49.777349+0000","last_active":"2024-03-11T22:25:27.548309+0000","last_peered":"2024-03-11T22:25:27.548309+0000","last_clean":"2024-03-11T22:25:27.548309+0000","last_became_active":"2024-03-11T22:20:29.415019+0000","last_became_peered":"2024-03-11T22:20:29.415019+0000","last_unstale":"2024-03-11T22:25:27.548309+0000","last_undegraded":"2024-03-11T22:25:27.548309+0000","last_fullsized":"2024-03-11T22:25:27.548309+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:49.777229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:49.777229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:00.825874+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":910,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:33.555520+0000","last_change":"2024-03-11T22:23:53.776480+0000","last_active":"2024-03-11T22:25:33.555520+0000","last_peered":"2024-03-11T22:25:33.555520+0000","last_clean":"2024-03-11T22:25:33.555520+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:25:33.555520+0000","last_undegraded":"2024-03-11T22:25:33.555520+0000","last_fullsized":"2024-03-11T22:25:33.555520+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:23:53.776342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:53.776342+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:43.100650+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":696,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.544661+0000","last_change":"2024-03-11T22:24:45.964526+0000","last_active":"2024-03-11T22:25:27.544661+0000","last_peered":"2024-03-11T22:25:27.544661+0000","last_clean":"2024-03-11T22:25:27.544661+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:25:27.544661+0000","last_undegraded":"2024-03-11T22:25:27.544661+0000","last_fullsized":"2024-03-11T22:25:27.544661+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:45.964401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:45.964401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:07.337646+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"93'130","reported_seq":951,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.544879+0000","last_change":"2024-03-11T22:23:10.014311+0000","last_active":"2024-03-11T22:25:27.544879+0000","last_peered":"2024-03-11T22:25:27.544879+0000","last_clean":"2024-03-11T22:25:27.544879+0000","last_became_active":"2024-03-11T22:14:43.191297+0000","last_became_peered":"2024-03-11T22:14:43.191297+0000","last_unstale":"2024-03-11T22:25:27.544879+0000","last_undegraded":"2024-03-11T22:25:27.544879+0000","last_fullsized":"2024-03-11T22:25:27.544879+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:10.014235+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:10.014235+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.872330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"273'19","reported_seq":520,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.544959+0000","last_change":"2024-03-11T22:23:45.819413+0000","last_active":"2024-03-11T22:25:27.544959+0000","last_peered":"2024-03-11T22:25:27.544959+0000","last_clean":"2024-03-11T22:25:27.544959+0000","last_became_active":"2024-03-11T22:14:52.239810+0000","last_became_peered":"2024-03-11T22:14:52.239810+0000","last_unstale":"2024-03-11T22:25:27.544959+0000","last_undegraded":"2024-03-11T22:25:27.544959+0000","last_fullsized":"2024-03-11T22:25:27.544959+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:45.818986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:45.818986+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":499,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.543051+0000","last_change":"2024-03-11T22:23:44.776567+0000","last_active":"2024-03-11T22:25:27.543051+0000","last_peered":"2024-03-11T22:25:27.543051+0000","last_clean":"2024-03-11T22:25:27.543051+0000","last_became_active":"2024-03-11T22:20:29.420867+0000","last_became_peered":"2024-03-11T22:20:29.420867+0000","last_unstale":"2024-03-11T22:25:27.543051+0000","last_undegraded":"2024-03-11T22:25:27.543051+0000","last_fullsized":"2024-03-11T22:25:27.543051+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:23:44.776433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:44.776433+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":696,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.551303+0000","last_change":"2024-03-11T22:21:51.817381+0000","last_active":"2024-03-11T22:25:27.551303+0000","last_peered":"2024-03-11T22:25:27.551303+0000","last_clean":"2024-03-11T22:25:27.551303+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:25:27.551303+0000","last_undegraded":"2024-03-11T22:25:27.551303+0000","last_fullsized":"2024-03-11T22:25:27.551303+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:51.817265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:21:51.817265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:29.517190+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":676,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.554752+0000","last_change":"2024-03-11T22:23:50.723665+0000","last_active":"2024-03-11T22:25:27.554752+0000","last_peered":"2024-03-11T22:25:27.554752+0000","last_clean":"2024-03-11T22:25:27.554752+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:25:27.554752+0000","last_undegraded":"2024-03-11T22:25:27.554752+0000","last_fullsized":"2024-03-11T22:25:27.554752+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:50.723528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:50.723528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754929+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":729,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.551474+0000","last_change":"2024-03-11T22:21:58.841578+0000","last_active":"2024-03-11T22:25:27.551474+0000","last_peered":"2024-03-11T22:25:27.551474+0000","last_clean":"2024-03-11T22:25:27.551474+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:25:27.551474+0000","last_undegraded":"2024-03-11T22:25:27.551474+0000","last_fullsized":"2024-03-11T22:25:27.551474+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:21:58.841451+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:21:58.841451+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:29.517157+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":639,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.558748+0000","last_change":"2024-03-11T22:24:41.612477+0000","last_active":"2024-03-11T22:25:27.558748+0000","last_peered":"2024-03-11T22:25:27.558748+0000","last_clean":"2024-03-11T22:25:27.558748+0000","last_became_active":"2024-03-11T22:20:29.422747+0000","last_became_peered":"2024-03-11T22:20:29.422747+0000","last_unstale":"2024-03-11T22:25:27.558748+0000","last_undegraded":"2024-03-11T22:25:27.558748+0000","last_fullsized":"2024-03-11T22:25:27.558748+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:41.612363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.612363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:56.508483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":688,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.555370+0000","last_change":"2024-03-11T22:24:49.679683+0000","last_active":"2024-03-11T22:25:27.555370+0000","last_peered":"2024-03-11T22:25:27.555370+0000","last_clean":"2024-03-11T22:25:27.555370+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:25:27.555370+0000","last_undegraded":"2024-03-11T22:25:27.555370+0000","last_fullsized":"2024-03-11T22:25:27.555370+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:49.679560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:24:49.679560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:13.908754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":684,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548285+0000","last_change":"2024-03-11T22:23:09.058314+0000","last_active":"2024-03-11T22:25:27.548285+0000","last_peered":"2024-03-11T22:25:27.548285+0000","last_clean":"2024-03-11T22:25:27.548285+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:25:27.548285+0000","last_undegraded":"2024-03-11T22:25:27.548285+0000","last_fullsized":"2024-03-11T22:25:27.548285+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:09.057970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:09.057970+0000","last_clean_scrub_stamp":"2024-03-11T22:23:09.057970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.872210+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"93'130","reported_seq":1012,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548569+0000","last_change":"2024-03-11T22:23:41.379256+0000","last_active":"2024-03-11T22:25:27.548569+0000","last_peered":"2024-03-11T22:25:27.548569+0000","last_clean":"2024-03-11T22:25:27.548569+0000","last_became_active":"2024-03-11T22:14:43.194955+0000","last_became_peered":"2024-03-11T22:14:43.194955+0000","last_unstale":"2024-03-11T22:25:27.548569+0000","last_undegraded":"2024-03-11T22:25:27.548569+0000","last_fullsized":"2024-03-11T22:25:27.548569+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:41.379110+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:18:43.722467+0000","last_clean_scrub_stamp":"2024-03-11T22:23:41.379110+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:31.852973+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"274'20","reported_seq":752,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.555050+0000","last_change":"2024-03-11T22:23:52.653463+0000","last_active":"2024-03-11T22:25:27.555050+0000","last_peered":"2024-03-11T22:25:27.555050+0000","last_clean":"2024-03-11T22:25:27.555050+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:25:27.555050+0000","last_undegraded":"2024-03-11T22:25:27.555050+0000","last_fullsized":"2024-03-11T22:25:27.555050+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'20","last_scrub_stamp":"2024-03-11T22:23:52.653309+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:23:52.653309+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:30.767926+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":716,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.554296+0000","last_change":"2024-03-11T22:24:41.895990+0000","last_active":"2024-03-11T22:25:27.554296+0000","last_peered":"2024-03-11T22:25:27.554296+0000","last_clean":"2024-03-11T22:25:27.554296+0000","last_became_active":"2024-03-11T22:20:29.416792+0000","last_became_peered":"2024-03-11T22:20:29.416792+0000","last_unstale":"2024-03-11T22:25:27.554296+0000","last_undegraded":"2024-03-11T22:25:27.554296+0000","last_fullsized":"2024-03-11T22:25:27.554296+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:41.895742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.895742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:45.455063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":473,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.542505+0000","last_change":"2024-03-11T22:23:33.856940+0000","last_active":"2024-03-11T22:25:27.542505+0000","last_peered":"2024-03-11T22:25:27.542505+0000","last_clean":"2024-03-11T22:25:27.542505+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:25:27.542505+0000","last_undegraded":"2024-03-11T22:25:27.542505+0000","last_fullsized":"2024-03-11T22:25:27.542505+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:33.856676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:23:33.856676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":684,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.555646+0000","last_change":"2024-03-11T22:24:59.829935+0000","last_active":"2024-03-11T22:25:27.555646+0000","last_peered":"2024-03-11T22:25:27.555646+0000","last_clean":"2024-03-11T22:25:27.555646+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:25:27.555646+0000","last_undegraded":"2024-03-11T22:25:27.555646+0000","last_fullsized":"2024-03-11T22:25:27.555646+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:59.829811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:59.829811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:17.495056+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"273'8","reported_seq":716,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.554069+0000","last_change":"2024-03-11T22:23:59.693653+0000","last_active":"2024-03-11T22:25:27.554069+0000","last_peered":"2024-03-11T22:25:27.554069+0000","last_clean":"2024-03-11T22:25:27.554069+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:25:27.554069+0000","last_undegraded":"2024-03-11T22:25:27.554069+0000","last_fullsized":"2024-03-11T22:25:27.554069+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'8","last_scrub_stamp":"2024-03-11T22:23:59.693510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:59.693510+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754904+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":696,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.544265+0000","last_change":"2024-03-11T22:23:12.019597+0000","last_active":"2024-03-11T22:25:27.544265+0000","last_peered":"2024-03-11T22:25:27.544265+0000","last_clean":"2024-03-11T22:25:27.544265+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:25:27.544265+0000","last_undegraded":"2024-03-11T22:25:27.544265+0000","last_fullsized":"2024-03-11T22:25:27.544265+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:12.019468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:12.019468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:34.905004+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":290,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.542537+0000","last_change":"2024-03-11T22:24:37.763829+0000","last_active":"2024-03-11T22:25:27.542537+0000","last_peered":"2024-03-11T22:25:27.542537+0000","last_clean":"2024-03-11T22:25:27.542537+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:25:27.542537+0000","last_undegraded":"2024-03-11T22:25:27.542537+0000","last_fullsized":"2024-03-11T22:25:27.542537+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.763709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.763709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.935899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":658,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.547582+0000","last_change":"2024-03-11T22:24:46.776804+0000","last_active":"2024-03-11T22:25:27.547582+0000","last_peered":"2024-03-11T22:25:27.547582+0000","last_clean":"2024-03-11T22:25:27.547582+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:25:27.547582+0000","last_undegraded":"2024-03-11T22:25:27.547582+0000","last_fullsized":"2024-03-11T22:25:27.547582+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:46.776600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.776600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:12.814680+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"273'19","reported_seq":740,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.543686+0000","last_change":"2024-03-11T22:23:42.850339+0000","last_active":"2024-03-11T22:25:27.543686+0000","last_peered":"2024-03-11T22:25:27.543686+0000","last_clean":"2024-03-11T22:25:27.543686+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:25:27.543686+0000","last_undegraded":"2024-03-11T22:25:27.543686+0000","last_fullsized":"2024-03-11T22:25:27.543686+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:42.850214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:42.850214+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":696,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.554017+0000","last_change":"2024-03-11T22:24:38.611323+0000","last_active":"2024-03-11T22:25:27.554017+0000","last_peered":"2024-03-11T22:25:27.554017+0000","last_clean":"2024-03-11T22:25:27.554017+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:25:27.554017+0000","last_undegraded":"2024-03-11T22:25:27.554017+0000","last_fullsized":"2024-03-11T22:25:27.554017+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:38.610924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:38.610924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:03.792615+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":697,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.545185+0000","last_change":"2024-03-11T22:23:46.048359+0000","last_active":"2024-03-11T22:25:27.545185+0000","last_peered":"2024-03-11T22:25:27.545185+0000","last_clean":"2024-03-11T22:25:27.545185+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:25:27.545185+0000","last_undegraded":"2024-03-11T22:25:27.545185+0000","last_fullsized":"2024-03-11T22:25:27.545185+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:46.048254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.048254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.872479+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":699,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548226+0000","last_change":"2024-03-11T22:25:05.513523+0000","last_active":"2024-03-11T22:25:27.548226+0000","last_peered":"2024-03-11T22:25:27.548226+0000","last_clean":"2024-03-11T22:25:27.548226+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:25:27.548226+0000","last_undegraded":"2024-03-11T22:25:27.548226+0000","last_fullsized":"2024-03-11T22:25:27.548226+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:05.513210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.513210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:06.854659+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"273'20","reported_seq":759,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.544915+0000","last_change":"2024-03-11T22:22:57.045033+0000","last_active":"2024-03-11T22:25:27.544915+0000","last_peered":"2024-03-11T22:25:27.544915+0000","last_clean":"2024-03-11T22:25:27.544915+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:25:27.544915+0000","last_undegraded":"2024-03-11T22:25:27.544915+0000","last_fullsized":"2024-03-11T22:25:27.544915+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:22:57.044931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:22:57.044931+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.868357+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":706,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.547135+0000","last_change":"2024-03-11T22:22:06.879148+0000","last_active":"2024-03-11T22:25:27.547135+0000","last_peered":"2024-03-11T22:25:27.547135+0000","last_clean":"2024-03-11T22:25:27.547135+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:25:27.547135+0000","last_undegraded":"2024-03-11T22:25:27.547135+0000","last_fullsized":"2024-03-11T22:25:27.547135+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:22:06.879026+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:22:06.879026+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:40.656802+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'11","reported_seq":952,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:33.556526+0000","last_change":"2024-03-11T22:23:50.837614+0000","last_active":"2024-03-11T22:25:33.556526+0000","last_peered":"2024-03-11T22:25:33.556526+0000","last_clean":"2024-03-11T22:25:33.556526+0000","last_became_active":"2024-03-11T22:20:29.415781+0000","last_became_peered":"2024-03-11T22:20:29.415781+0000","last_unstale":"2024-03-11T22:25:33.556526+0000","last_undegraded":"2024-03-11T22:25:33.556526+0000","last_fullsized":"2024-03-11T22:25:33.556526+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-11T22:23:50.837494+0000","last_deep_scrub":"315'11","last_deep_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_clean_scrub_stamp":"2024-03-11T22:23:50.837494+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:39.952240+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":645,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.552140+0000","last_change":"2024-03-11T22:23:51.821013+0000","last_active":"2024-03-11T22:25:27.552140+0000","last_peered":"2024-03-11T22:25:27.552140+0000","last_clean":"2024-03-11T22:25:27.552140+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:25:27.552140+0000","last_undegraded":"2024-03-11T22:25:27.552140+0000","last_fullsized":"2024-03-11T22:25:27.552140+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:23:51.820865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.820865+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:37.032536+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"273'18","reported_seq":723,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.559086+0000","last_change":"2024-03-11T22:25:01.629462+0000","last_active":"2024-03-11T22:25:27.559086+0000","last_peered":"2024-03-11T22:25:27.559086+0000","last_clean":"2024-03-11T22:25:27.559086+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:25:27.559086+0000","last_undegraded":"2024-03-11T22:25:27.559086+0000","last_fullsized":"2024-03-11T22:25:27.559086+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:25:01.629321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:01.629321+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:29.544545+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":471,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.551102+0000","last_change":"2024-03-11T22:25:04.394810+0000","last_active":"2024-03-11T22:25:27.551102+0000","last_peered":"2024-03-11T22:25:27.551102+0000","last_clean":"2024-03-11T22:25:27.551102+0000","last_became_active":"2024-03-11T22:14:52.245199+0000","last_became_peered":"2024-03-11T22:14:52.245199+0000","last_unstale":"2024-03-11T22:25:27.551102+0000","last_undegraded":"2024-03-11T22:25:27.551102+0000","last_fullsized":"2024-03-11T22:25:27.551102+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:04.394612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:25:04.394612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:30.314020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":454,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.558898+0000","last_change":"2024-03-11T22:25:05.656524+0000","last_active":"2024-03-11T22:25:27.558898+0000","last_peered":"2024-03-11T22:25:27.558898+0000","last_clean":"2024-03-11T22:25:27.558898+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:25:27.558898+0000","last_undegraded":"2024-03-11T22:25:27.558898+0000","last_fullsized":"2024-03-11T22:25:27.558898+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:05.656208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.656208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:14.267362+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":677,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.559018+0000","last_change":"2024-03-11T22:23:12.914118+0000","last_active":"2024-03-11T22:25:27.559018+0000","last_peered":"2024-03-11T22:25:27.559018+0000","last_clean":"2024-03-11T22:25:27.559018+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:25:27.559018+0000","last_undegraded":"2024-03-11T22:25:27.559018+0000","last_fullsized":"2024-03-11T22:25:27.559018+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:12.914007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:12.914007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754669+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"273'27","reported_seq":766,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548151+0000","last_change":"2024-03-11T22:23:46.333974+0000","last_active":"2024-03-11T22:25:27.548151+0000","last_peered":"2024-03-11T22:25:27.548151+0000","last_clean":"2024-03-11T22:25:27.548151+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:25:27.548151+0000","last_undegraded":"2024-03-11T22:25:27.548151+0000","last_fullsized":"2024-03-11T22:25:27.548151+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:23:46.333799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.333799+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:31.853151+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":715,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.547189+0000","last_change":"2024-03-11T22:24:55.760207+0000","last_active":"2024-03-11T22:25:27.547189+0000","last_peered":"2024-03-11T22:25:27.547189+0000","last_clean":"2024-03-11T22:25:27.547189+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:25:27.547189+0000","last_undegraded":"2024-03-11T22:25:27.547189+0000","last_fullsized":"2024-03-11T22:25:27.547189+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:55.760084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:24:55.760084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:21.111674+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":703,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548076+0000","last_change":"2024-03-11T22:24:37.481790+0000","last_active":"2024-03-11T22:25:27.548076+0000","last_peered":"2024-03-11T22:25:27.548076+0000","last_clean":"2024-03-11T22:25:27.548076+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:25:27.548076+0000","last_undegraded":"2024-03-11T22:25:27.548076+0000","last_fullsized":"2024-03-11T22:25:27.548076+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.481668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.481668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:03.122817+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":471,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.543884+0000","last_change":"2024-03-11T22:23:07.097233+0000","last_active":"2024-03-11T22:25:27.543884+0000","last_peered":"2024-03-11T22:25:27.543884+0000","last_clean":"2024-03-11T22:25:27.543884+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:25:27.543884+0000","last_undegraded":"2024-03-11T22:25:27.543884+0000","last_fullsized":"2024-03-11T22:25:27.543884+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:07.097049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:07.097049+0000","last_clean_scrub_stamp":"2024-03-11T22:23:07.097049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"273'9","reported_seq":531,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.555196+0000","last_change":"2024-03-11T22:23:58.648966+0000","last_active":"2024-03-11T22:25:27.555196+0000","last_peered":"2024-03-11T22:25:27.555196+0000","last_clean":"2024-03-11T22:25:27.555196+0000","last_became_active":"2024-03-11T22:14:52.238411+0000","last_became_peered":"2024-03-11T22:14:52.238411+0000","last_unstale":"2024-03-11T22:25:27.555196+0000","last_undegraded":"2024-03-11T22:25:27.555196+0000","last_fullsized":"2024-03-11T22:25:27.555196+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9","last_scrub_stamp":"2024-03-11T22:23:58.648385+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.648385+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":703,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.552307+0000","last_change":"2024-03-11T22:24:40.905435+0000","last_active":"2024-03-11T22:25:27.552307+0000","last_peered":"2024-03-11T22:25:27.552307+0000","last_clean":"2024-03-11T22:25:27.552307+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:25:27.552307+0000","last_undegraded":"2024-03-11T22:25:27.552307+0000","last_fullsized":"2024-03-11T22:25:27.552307+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:40.905314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:24:40.905314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:41.648326+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":465,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.550807+0000","last_change":"2024-03-11T22:22:03.796143+0000","last_active":"2024-03-11T22:25:27.550807+0000","last_peered":"2024-03-11T22:25:27.550807+0000","last_clean":"2024-03-11T22:25:27.550807+0000","last_became_active":"2024-03-11T22:14:52.239391+0000","last_became_peered":"2024-03-11T22:14:52.239391+0000","last_unstale":"2024-03-11T22:25:27.550807+0000","last_undegraded":"2024-03-11T22:25:27.550807+0000","last_fullsized":"2024-03-11T22:25:27.550807+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:03.796021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_clean_scrub_stamp":"2024-03-11T22:22:03.796021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:29.517289+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":684,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.551267+0000","last_change":"2024-03-11T22:22:59.182771+0000","last_active":"2024-03-11T22:25:27.551267+0000","last_peered":"2024-03-11T22:25:27.551267+0000","last_clean":"2024-03-11T22:25:27.551267+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:25:27.551267+0000","last_undegraded":"2024-03-11T22:25:27.551267+0000","last_fullsized":"2024-03-11T22:25:27.551267+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:59.182621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:59.182621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:43.100173+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"273'15","reported_seq":506,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.561639+0000","last_change":"2024-03-11T22:25:03.685762+0000","last_active":"2024-03-11T22:25:27.561639+0000","last_peered":"2024-03-11T22:25:27.561639+0000","last_clean":"2024-03-11T22:25:27.561639+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:25:27.561639+0000","last_undegraded":"2024-03-11T22:25:27.561639+0000","last_fullsized":"2024-03-11T22:25:27.561639+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:25:03.685629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:03.685629+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:21.379436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":703,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.553398+0000","last_change":"2024-03-11T22:23:58.700286+0000","last_active":"2024-03-11T22:25:27.553398+0000","last_peered":"2024-03-11T22:25:27.553398+0000","last_clean":"2024-03-11T22:25:27.553398+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:25:27.553398+0000","last_undegraded":"2024-03-11T22:25:27.553398+0000","last_fullsized":"2024-03-11T22:25:27.553398+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:58.700176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.700176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.993189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":707,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.553527+0000","last_change":"2024-03-11T22:24:49.941039+0000","last_active":"2024-03-11T22:25:27.553527+0000","last_peered":"2024-03-11T22:25:27.553527+0000","last_clean":"2024-03-11T22:25:27.553527+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:25:27.553527+0000","last_undegraded":"2024-03-11T22:25:27.553527+0000","last_fullsized":"2024-03-11T22:25:27.553527+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:49.940903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:24:49.940903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.554477+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":684,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.548671+0000","last_change":"2024-03-11T22:23:40.218666+0000","last_active":"2024-03-11T22:25:27.548671+0000","last_peered":"2024-03-11T22:25:27.548671+0000","last_clean":"2024-03-11T22:25:27.548671+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:25:27.548671+0000","last_undegraded":"2024-03-11T22:25:27.548671+0000","last_fullsized":"2024-03-11T22:25:27.548671+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:40.218559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:40.218559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:39.983977+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"273'27","reported_seq":727,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.555204+0000","last_change":"2024-03-11T22:23:47.683228+0000","last_active":"2024-03-11T22:25:27.555204+0000","last_peered":"2024-03-11T22:25:27.555204+0000","last_clean":"2024-03-11T22:25:27.555204+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:25:27.555204+0000","last_undegraded":"2024-03-11T22:25:27.555204+0000","last_fullsized":"2024-03-11T22:25:27.555204+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:23:47.683120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:47.683120+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":703,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.558355+0000","last_change":"2024-03-11T22:23:41.703918+0000","last_active":"2024-03-11T22:25:27.558355+0000","last_peered":"2024-03-11T22:25:27.558355+0000","last_clean":"2024-03-11T22:25:27.558355+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:25:27.558355+0000","last_undegraded":"2024-03-11T22:25:27.558355+0000","last_fullsized":"2024-03-11T22:25:27.558355+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:23:41.703753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:41.703753+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754763+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":473,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.542595+0000","last_change":"2024-03-11T22:22:58.001733+0000","last_active":"2024-03-11T22:25:27.542595+0000","last_peered":"2024-03-11T22:25:27.542595+0000","last_clean":"2024-03-11T22:25:27.542595+0000","last_became_active":"2024-03-11T22:14:52.247689+0000","last_became_peered":"2024-03-11T22:14:52.247689+0000","last_unstale":"2024-03-11T22:25:27.542595+0000","last_undegraded":"2024-03-11T22:25:27.542595+0000","last_fullsized":"2024-03-11T22:25:27.542595+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:58.001650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:22:58.001650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":687,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.555446+0000","last_change":"2024-03-11T22:23:08.833422+0000","last_active":"2024-03-11T22:25:27.555446+0000","last_peered":"2024-03-11T22:25:27.555446+0000","last_clean":"2024-03-11T22:25:27.555446+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:25:27.555446+0000","last_undegraded":"2024-03-11T22:25:27.555446+0000","last_fullsized":"2024-03-11T22:25:27.555446+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:08.833328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:08.833328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.754609+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"273'19","reported_seq":518,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.542816+0000","last_change":"2024-03-11T22:24:39.755038+0000","last_active":"2024-03-11T22:25:27.542816+0000","last_peered":"2024-03-11T22:25:27.542816+0000","last_clean":"2024-03-11T22:25:27.542816+0000","last_became_active":"2024-03-11T22:14:52.247411+0000","last_became_peered":"2024-03-11T22:14:52.247411+0000","last_unstale":"2024-03-11T22:25:27.542816+0000","last_undegraded":"2024-03-11T22:25:27.542816+0000","last_fullsized":"2024-03-11T22:25:27.542816+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:24:39.754894+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.754894+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:44.634751+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":705,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.543967+0000","last_change":"2024-03-11T22:24:00.015418+0000","last_active":"2024-03-11T22:25:27.543967+0000","last_peered":"2024-03-11T22:25:27.543967+0000","last_clean":"2024-03-11T22:25:27.543967+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:25:27.543967+0000","last_undegraded":"2024-03-11T22:25:27.543967+0000","last_fullsized":"2024-03-11T22:25:27.543967+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:00.015301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.015301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.872670+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":418,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.542635+0000","last_change":"2024-03-11T22:23:29.877294+0000","last_active":"2024-03-11T22:25:27.542635+0000","last_peered":"2024-03-11T22:25:27.542635+0000","last_clean":"2024-03-11T22:25:27.542635+0000","last_became_active":"2024-03-11T22:20:29.418227+0000","last_became_peered":"2024-03-11T22:20:29.418227+0000","last_unstale":"2024-03-11T22:25:27.542635+0000","last_undegraded":"2024-03-11T22:25:27.542635+0000","last_fullsized":"2024-03-11T22:25:27.542635+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:29.877172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:29.877172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":462,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-11T22:25:27.551902+0000","last_change":"2024-03-11T22:23:36.916038+0000","last_active":"2024-03-11T22:25:27.551902+0000","last_peered":"2024-03-11T22:25:27.551902+0000","last_clean":"2024-03-11T22:25:27.551902+0000","last_became_active":"2024-03-11T22:14:52.247273+0000","last_became_peered":"2024-03-11T22:14:52.247273+0000","last_unstale":"2024-03-11T22:25:27.551902+0000","last_undegraded":"2024-03-11T22:25:27.551902+0000","last_fullsized":"2024-03-11T22:25:27.551902+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:36.915904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:36.915904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:28.993013+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3466985496,"num_objects":52903,"num_object_clones":0,"num_object_copies":158709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52903,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211147,"num_write_kb":8449537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10362171392,"data_stored":10362159176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81487,"ondisk_log_size":81487,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":993,"num_read_kb":988,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":1430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005266,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1464344,"kb_used_data":847312,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92280808,"statfs":{"total":95995035648,"available":94495547392,"internally_reserved":0,"allocated":867647488,"data_stored":867326321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":631830987},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332311,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1987720,"kb_used_data":1487296,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":91757432,"statfs":{"total":95995035648,"available":93959610368,"internally_reserved":0,"allocated":1522991104,"data_stored":1522686155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":512424390},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575089,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1164916,"kb_used_data":606188,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92580236,"statfs":{"total":95995035648,"available":94802161664,"internally_reserved":0,"allocated":620736512,"data_stored":620386679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738614,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2914584,"kb_used_data":2261328,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":90830568,"statfs":{"total":95995035648,"available":93010501632,"internally_reserved":0,"allocated":2315599872,"data_stored":2315278061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411564,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1745220,"kb_used_data":1250364,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":91999932,"statfs":{"total":95995035648,"available":94207930368,"internally_reserved":0,"allocated":1280372736,"data_stored":1280072229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":506722757},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149926,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1651628,"kb_used_data":1042916,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":92093524,"statfs":{"total":95995035648,"available":94303768576,"internally_reserved":0,"allocated":1067945984,"data_stored":1067605197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":623311303},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920994,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2230944,"kb_used_data":1704088,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91514208,"statfs":{"total":95995035648,"available":93710548992,"internally_reserved":0,"allocated":1744986112,"data_stored":1744656318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":539425222},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659355,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1675276,"kb_used_data":1052676,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92069876,"statfs":{"total":95995035648,"available":94279553024,"internally_reserved":0,"allocated":1077940224,"data_stored":1077630972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":637532614},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2283933696,"data_stored":2283929624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":619184128,"data_stored":619184128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1274806272,"data_stored":1274806272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1047724032,"data_stored":1047724032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1716391936,"data_stored":1716387864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1062731776,"data_stored":1062731776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1494618112,"data_stored":1494614040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":862781440,"data_stored":862781440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:25:45.738 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T22:25:45.738 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:25:45.739 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-11T22:25:45.739 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-11T22:25:45.739 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:25:45.808 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:45.837 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:45.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:45.935 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:45 smithi045 ceph-mon[30178]: pgmap v800: 168 pgs: 168 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T22:25:45.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:45 smithi045 ceph-mon[39715]: pgmap v800: 168 pgs: 168 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T22:25:45.938 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:45.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:45.963 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:45.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:46.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:46.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:45 smithi144 ceph-mon[33984]: pgmap v800: 168 pgs: 168 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T22:25:46.109 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:46.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:46.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:46.211 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:46.242 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:46.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:46.344 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:46.379 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:46.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:46.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:46.482 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:46.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:46.518 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:46.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:46.619 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:46.647 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:46.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:46.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:46.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:46.784 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:46.784 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:25:46.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:25:46.798 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:25:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:46.818 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:46.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:46.837 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:25:46.888 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:25:46.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:46.932 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:25:46.969 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:46.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:46.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:46.979 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:25:47.028 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:25:47.071 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:47.085 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:25:47.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:25:47.098 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:25:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:25:47.117 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:47.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:47.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:47.138 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:25:47.144 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:25:47.144 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:25:47.144 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:25:47.144 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:25:47.144 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:25:47.144 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:25:47.144 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:25:47.144 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:25:47.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:47.265 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:47.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:47.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:47.367 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:47.400 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:47.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:47.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:47.538 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:47.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:47.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:47.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:47.666 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:25:47.685 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:47.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:47.787 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:47.821 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:47.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:47.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:47.956 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:47.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:48.034 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:47 smithi045 ceph-mon[39715]: pgmap v801: 168 pgs: 168 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-11T22:25:48.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:48.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:47 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3758616695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:25:48.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:47 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/651354125' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:25:48.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:47 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:25:48.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:47 smithi045 ceph-mon[30178]: pgmap v801: 168 pgs: 168 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-11T22:25:48.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:48.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3758616695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:25:48.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/651354125' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:25:48.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:47 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:25:48.058 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:48.094 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:48.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:48.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:48.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:47 smithi144 ceph-mon[33984]: pgmap v801: 168 pgs: 168 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-11T22:25:48.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:48.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:47 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3758616695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:25:48.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:47 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/651354125' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:25:48.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:47 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:25:48.196 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:48.231 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:48.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:48.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:48.370 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:48.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:48.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:48.509 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:48.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:48.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:48.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:48.648 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:48.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:48.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:48.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:48.790 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:48.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:48.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:48.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:48 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:25:48.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:48 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:25:48.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:48 smithi045 ceph-mon[39715]: osdmap e389: 8 total, 8 up, 8 in 2024-03-11T22:25:48.924 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:48 smithi045 ceph-mon[39715]: 10.d scrub starts 2024-03-11T22:25:48.925 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:48 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:25:48.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:48 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:25:48.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:48 smithi045 ceph-mon[30178]: osdmap e389: 8 total, 8 up, 8 in 2024-03-11T22:25:48.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:48 smithi045 ceph-mon[30178]: 10.d scrub starts 2024-03-11T22:25:48.926 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:48.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:48.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:49.028 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:49.062 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:49.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:49.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:49.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:48 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:25:49.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:48 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:25:49.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:48 smithi144 ceph-mon[33984]: osdmap e389: 8 total, 8 up, 8 in 2024-03-11T22:25:49.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:48 smithi144 ceph-mon[33984]: 10.d scrub starts 2024-03-11T22:25:49.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:49.201 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:49.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:49.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:49.303 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:49.339 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:49.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:49.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:49.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:49.474 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:49.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:49.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:49.610 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:49.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:49.712 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:49.744 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:49.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:49.846 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:49.885 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:49.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:49.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:49.987 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:50.019 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:50.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[39715]: pgmap v803: 168 pgs: 168 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-11T22:25:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:25:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:25:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[39715]: osdmap e390: 8 total, 8 up, 8 in 2024-03-11T22:25:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[39715]: 10.d scrub ok 2024-03-11T22:25:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[39715]: 10.1 scrub starts 2024-03-11T22:25:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[39715]: 10.7 scrub starts 2024-03-11T22:25:50.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[30178]: pgmap v803: 168 pgs: 168 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-11T22:25:50.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:25:50.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:25:50.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[30178]: osdmap e390: 8 total, 8 up, 8 in 2024-03-11T22:25:50.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[30178]: 10.d scrub ok 2024-03-11T22:25:50.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[30178]: 10.1 scrub starts 2024-03-11T22:25:50.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:49 smithi045 ceph-mon[30178]: 10.7 scrub starts 2024-03-11T22:25:50.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:49 smithi144 ceph-mon[33984]: pgmap v803: 168 pgs: 168 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-11T22:25:50.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:49 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:25:50.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:49 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:25:50.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:49 smithi144 ceph-mon[33984]: osdmap e390: 8 total, 8 up, 8 in 2024-03-11T22:25:50.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:49 smithi144 ceph-mon[33984]: 10.d scrub ok 2024-03-11T22:25:50.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:49 smithi144 ceph-mon[33984]: 10.1 scrub starts 2024-03-11T22:25:50.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:49 smithi144 ceph-mon[33984]: 10.7 scrub starts 2024-03-11T22:25:50.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:50.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:50.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:50.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:50.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:50.317 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:50.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:50.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:50.419 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:50.452 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:50.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:50.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:50.554 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:50.586 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:50.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:50.688 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:50.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:50.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:50.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:50 smithi045 ceph-mon[30178]: 10.1 scrub ok 2024-03-11T22:25:50.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:50 smithi045 ceph-mon[30178]: 10.7 scrub ok 2024-03-11T22:25:50.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:50 smithi045 ceph-mon[30178]: 10.0 scrub starts 2024-03-11T22:25:50.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:50 smithi045 ceph-mon[30178]: 4.1e scrub starts 2024-03-11T22:25:50.815 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:50 smithi045 ceph-mon[30178]: 4.1e scrub ok 2024-03-11T22:25:50.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:50 smithi045 ceph-mon[39715]: 10.1 scrub ok 2024-03-11T22:25:50.815 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:50 smithi045 ceph-mon[39715]: 10.7 scrub ok 2024-03-11T22:25:50.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:50 smithi045 ceph-mon[39715]: 10.0 scrub starts 2024-03-11T22:25:50.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:50 smithi045 ceph-mon[39715]: 4.1e scrub starts 2024-03-11T22:25:50.816 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:50 smithi045 ceph-mon[39715]: 4.1e scrub ok 2024-03-11T22:25:50.835 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:50.870 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:50.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:50.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:25:50.972 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:51.002 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:51.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:51.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:51.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:50 smithi144 ceph-mon[33984]: 10.1 scrub ok 2024-03-11T22:25:51.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:50 smithi144 ceph-mon[33984]: 10.7 scrub ok 2024-03-11T22:25:51.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:50 smithi144 ceph-mon[33984]: 10.0 scrub starts 2024-03-11T22:25:51.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:50 smithi144 ceph-mon[33984]: 4.1e scrub starts 2024-03-11T22:25:51.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:50 smithi144 ceph-mon[33984]: 4.1e scrub ok 2024-03-11T22:25:51.104 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:51.135 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:51.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:51.236 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:51.265 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:51.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:51.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:51.367 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:51.402 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:51.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:51.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:51.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:51.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:51.550 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:51.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:51.651 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:51.695 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:51.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:51.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:51.797 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:51.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:51.829 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:51.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:51.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:51.968 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:51.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:51.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:52.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:51 smithi045 ceph-mon[30178]: pgmap v805: 168 pgs: 168 active+clean; 2.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 898 op/s 2024-03-11T22:25:52.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:51 smithi045 ceph-mon[30178]: 10.0 scrub ok 2024-03-11T22:25:52.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:51 smithi045 ceph-mon[30178]: 6.f scrub starts 2024-03-11T22:25:52.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:51 smithi045 ceph-mon[30178]: 6.f scrub ok 2024-03-11T22:25:52.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:51 smithi045 ceph-mon[39715]: pgmap v805: 168 pgs: 168 active+clean; 2.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 898 op/s 2024-03-11T22:25:52.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:51 smithi045 ceph-mon[39715]: 10.0 scrub ok 2024-03-11T22:25:52.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:51 smithi045 ceph-mon[39715]: 6.f scrub starts 2024-03-11T22:25:52.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:51 smithi045 ceph-mon[39715]: 6.f scrub ok 2024-03-11T22:25:52.070 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:52.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:51 smithi144 ceph-mon[33984]: pgmap v805: 168 pgs: 168 active+clean; 2.8 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 898 op/s 2024-03-11T22:25:52.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:51 smithi144 ceph-mon[33984]: 10.0 scrub ok 2024-03-11T22:25:52.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:51 smithi144 ceph-mon[33984]: 6.f scrub starts 2024-03-11T22:25:52.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:51 smithi144 ceph-mon[33984]: 6.f scrub ok 2024-03-11T22:25:52.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:52.110 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:52.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:52.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:52.250 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:52.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:52.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:52.327 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:25:52.327 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:25:52.327 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-03-11T22:25:52.327 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-03-11T22:25:52.327 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:25:52.353 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:52.390 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:52.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:52.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:52.492 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:52.525 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:52.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:52.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:52.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:52.672 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:52.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:52.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:52.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:52.822 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:52.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:52.832 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:25:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:25:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:25:52.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:52 smithi045 ceph-mon[39715]: 6.2 scrub starts 2024-03-11T22:25:52.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:52 smithi045 ceph-mon[39715]: 6.2 scrub ok 2024-03-11T22:25:52.832 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:52 smithi045 ceph-mon[39715]: 3.6 scrub starts 2024-03-11T22:25:52.833 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:52 smithi045 ceph-mon[39715]: 3.6 scrub ok 2024-03-11T22:25:52.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:52 smithi045 ceph-mon[30178]: 6.2 scrub starts 2024-03-11T22:25:52.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:52 smithi045 ceph-mon[30178]: 6.2 scrub ok 2024-03-11T22:25:52.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:52 smithi045 ceph-mon[30178]: 3.6 scrub starts 2024-03-11T22:25:52.833 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:52 smithi045 ceph-mon[30178]: 3.6 scrub ok 2024-03-11T22:25:52.924 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:52.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:52.963 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:52.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:53.064 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:53.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:52 smithi144 ceph-mon[33984]: 6.2 scrub starts 2024-03-11T22:25:53.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:52 smithi144 ceph-mon[33984]: 6.2 scrub ok 2024-03-11T22:25:53.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:52 smithi144 ceph-mon[33984]: 3.6 scrub starts 2024-03-11T22:25:53.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:52 smithi144 ceph-mon[33984]: 3.6 scrub ok 2024-03-11T22:25:53.103 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:53.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:53.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:53.205 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:53.242 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:53.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:53.343 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:53.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T22:25:53.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:53.373 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:53.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:53.474 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:53.508 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:53.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:53.610 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:53.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:53.643 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:53.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:53.643 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:53.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:53.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:53.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:53.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:53.908 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:53.938 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:53.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:53.966 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:53 smithi045 ceph-mon[39715]: 4.17 scrub starts 2024-03-11T22:25:53.966 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:53 smithi045 ceph-mon[39715]: 4.17 scrub ok 2024-03-11T22:25:53.966 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:53 smithi045 ceph-mon[39715]: pgmap v806: 168 pgs: 168 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T22:25:53.966 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:53 smithi045 ceph-mon[39715]: 10.4 scrub starts 2024-03-11T22:25:53.966 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:53 smithi045 ceph-mon[30178]: 4.17 scrub starts 2024-03-11T22:25:53.966 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:53 smithi045 ceph-mon[30178]: 4.17 scrub ok 2024-03-11T22:25:53.966 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:53 smithi045 ceph-mon[30178]: pgmap v806: 168 pgs: 168 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T22:25:53.967 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:53 smithi045 ceph-mon[30178]: 10.4 scrub starts 2024-03-11T22:25:54.040 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:54.073 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:54.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:54.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:54.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:53 smithi144 ceph-mon[33984]: 4.17 scrub starts 2024-03-11T22:25:54.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:53 smithi144 ceph-mon[33984]: 4.17 scrub ok 2024-03-11T22:25:54.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:53 smithi144 ceph-mon[33984]: pgmap v806: 168 pgs: 168 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T22:25:54.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:53 smithi144 ceph-mon[33984]: 10.4 scrub starts 2024-03-11T22:25:54.148 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:25:54.151 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:25:54.163 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:25:54.175 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:54.205 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:54.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:54.306 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:54.355 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:54.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:54.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:54.484 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:54.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:54.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:54.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:54.637 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:54.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:54.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:25:54.778 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:54.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:54.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:25:54.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:54.904 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:54.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:54.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:55.007 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:55.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[39715]: 4.12 scrub starts 2024-03-11T22:25:55.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[39715]: 4.12 scrub ok 2024-03-11T22:25:55.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[39715]: 6.10 scrub starts 2024-03-11T22:25:55.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[39715]: 6.10 scrub ok 2024-03-11T22:25:55.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[39715]: 10.4 scrub ok 2024-03-11T22:25:55.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[39715]: 1.14 scrub starts 2024-03-11T22:25:55.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[39715]: 1.14 scrub ok 2024-03-11T22:25:55.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[30178]: 4.12 scrub starts 2024-03-11T22:25:55.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[30178]: 4.12 scrub ok 2024-03-11T22:25:55.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[30178]: 6.10 scrub starts 2024-03-11T22:25:55.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[30178]: 6.10 scrub ok 2024-03-11T22:25:55.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[30178]: 10.4 scrub ok 2024-03-11T22:25:55.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[30178]: 1.14 scrub starts 2024-03-11T22:25:55.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:54 smithi045 ceph-mon[30178]: 1.14 scrub ok 2024-03-11T22:25:55.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:55.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:55.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:55.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:54 smithi144 ceph-mon[33984]: 4.12 scrub starts 2024-03-11T22:25:55.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:54 smithi144 ceph-mon[33984]: 4.12 scrub ok 2024-03-11T22:25:55.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:54 smithi144 ceph-mon[33984]: 6.10 scrub starts 2024-03-11T22:25:55.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:54 smithi144 ceph-mon[33984]: 6.10 scrub ok 2024-03-11T22:25:55.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:54 smithi144 ceph-mon[33984]: 10.4 scrub ok 2024-03-11T22:25:55.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:54 smithi144 ceph-mon[33984]: 1.14 scrub starts 2024-03-11T22:25:55.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:54 smithi144 ceph-mon[33984]: 1.14 scrub ok 2024-03-11T22:25:55.145 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:55.181 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:55.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:55.253 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":806,"stamp":"2024-03-11T22:25:52.423786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2681411956,"num_objects":41105,"num_object_clones":0,"num_object_copies":123315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41105,"num_whiteouts":0,"num_read":2128,"num_read_kb":1880,"num_write":223865,"num_write_kb":8449582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":2436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83833,"ondisk_log_size":83833,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12494184,"kb_used_data":7940688,"kb_used_omap":12,"kb_used_meta":4553075,"kb_avail":737467032,"statfs":{"total":767960285184,"available":755166240768,"internally_reserved":0,"allocated":8131264512,"data_stored":8128596412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":4662349363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,8,2,2,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-685178880,"num_objects":-10455,"num_object_clones":0,"num_object_copies":-31365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10455,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10455,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002121"},"pg_stats":[{"pgid":"10.2","version":"390'13920","reported_seq":14014,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:51.633255+0000","last_change":"2024-03-11T22:22:52.799244+0000","last_active":"2024-03-11T22:25:51.633255+0000","last_peered":"2024-03-11T22:25:51.633255+0000","last_clean":"2024-03-11T22:25:51.633255+0000","last_became_active":"2024-03-11T22:22:52.798875+0000","last_became_peered":"2024-03-11T22:22:52.798875+0000","last_unstale":"2024-03-11T22:25:51.633255+0000","last_undegraded":"2024-03-11T22:25:51.633255+0000","last_fullsized":"2024-03-11T22:25:51.633255+0000","mapping_epoch":348,"log_start":"375'8900","ondisk_log_start":"375'8900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":5020,"log_dups_size":0,"ondisk_log_size":5020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161873920,"num_objects":2470,"num_object_clones":0,"num_object_copies":7410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13920,"num_write_kb":524480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"294'25","reported_seq":514,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.685725+0000","last_change":"2024-03-11T22:24:39.621975+0000","last_active":"2024-03-11T22:25:48.685725+0000","last_peered":"2024-03-11T22:25:48.685725+0000","last_clean":"2024-03-11T22:25:48.685725+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:25:48.685725+0000","last_undegraded":"2024-03-11T22:25:48.685725+0000","last_fullsized":"2024-03-11T22:25:48.685725+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-03-11T22:24:39.621707+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.621707+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.006435+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":702,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.682498+0000","last_change":"2024-03-11T22:24:37.604756+0000","last_active":"2024-03-11T22:25:48.682498+0000","last_peered":"2024-03-11T22:25:48.682498+0000","last_clean":"2024-03-11T22:25:48.682498+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:25:48.682498+0000","last_undegraded":"2024-03-11T22:25:48.682498+0000","last_fullsized":"2024-03-11T22:25:48.682498+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.601240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.601240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:51.956237+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":943,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.675319+0000","last_change":"2024-03-11T22:24:01.449741+0000","last_active":"2024-03-11T22:25:48.675319+0000","last_peered":"2024-03-11T22:25:48.675319+0000","last_clean":"2024-03-11T22:25:48.675319+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:25:48.675319+0000","last_undegraded":"2024-03-11T22:25:48.675319+0000","last_fullsized":"2024-03-11T22:25:48.675319+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:24:01.447108+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:19:40.008325+0000","last_clean_scrub_stamp":"2024-03-11T22:24:01.447108+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:49.910265+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":468,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.668530+0000","last_change":"2024-03-11T22:23:07.686811+0000","last_active":"2024-03-11T22:25:48.668530+0000","last_peered":"2024-03-11T22:25:48.668530+0000","last_clean":"2024-03-11T22:25:48.668530+0000","last_became_active":"2024-03-11T22:14:52.243690+0000","last_became_peered":"2024-03-11T22:14:52.243690+0000","last_unstale":"2024-03-11T22:25:48.668530+0000","last_undegraded":"2024-03-11T22:25:48.668530+0000","last_fullsized":"2024-03-11T22:25:48.668530+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:07.686730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:07.686730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602648+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":1025,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.683916+0000","last_change":"2024-03-11T22:23:43.670715+0000","last_active":"2024-03-11T22:25:48.683916+0000","last_peered":"2024-03-11T22:25:48.683916+0000","last_clean":"2024-03-11T22:25:48.683916+0000","last_became_active":"2024-03-11T22:20:32.293356+0000","last_became_peered":"2024-03-11T22:20:32.293356+0000","last_unstale":"2024-03-11T22:25:48.683916+0000","last_undegraded":"2024-03-11T22:25:48.683916+0000","last_fullsized":"2024-03-11T22:25:48.683916+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:43.670585+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:43.670585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966854+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"390'14043","reported_seq":14214,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:50.986519+0000","last_change":"2024-03-11T22:24:53.352685+0000","last_active":"2024-03-11T22:25:50.986519+0000","last_peered":"2024-03-11T22:25:50.986519+0000","last_clean":"2024-03-11T22:25:50.986519+0000","last_became_active":"2024-03-11T22:22:52.469327+0000","last_became_peered":"2024-03-11T22:22:52.469327+0000","last_unstale":"2024-03-11T22:25:50.986519+0000","last_undegraded":"2024-03-11T22:25:50.986519+0000","last_fullsized":"2024-03-11T22:25:50.986519+0000","mapping_epoch":348,"log_start":"373'8000","ondisk_log_start":"373'8000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9962","last_scrub_stamp":"2024-03-11T22:24:53.352359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.352359+0000","objects_scrubbed":6655,"log_size":6043,"log_dups_size":0,"ondisk_log_size":6043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:58.557123+0000","scrub_duration":2633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162332672,"num_objects":2477,"num_object_clones":0,"num_object_copies":7431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14043,"num_write_kb":528640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"353'39","reported_seq":568,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.675167+0000","last_change":"2024-03-11T22:24:58.663588+0000","last_active":"2024-03-11T22:25:48.675167+0000","last_peered":"2024-03-11T22:25:48.675167+0000","last_clean":"2024-03-11T22:25:48.675167+0000","last_became_active":"2024-03-11T22:14:52.239649+0000","last_became_peered":"2024-03-11T22:14:52.239649+0000","last_unstale":"2024-03-11T22:25:48.675167+0000","last_undegraded":"2024-03-11T22:25:48.675167+0000","last_fullsized":"2024-03-11T22:25:48.675167+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'39","last_scrub_stamp":"2024-03-11T22:24:58.663290+0000","last_deep_scrub":"353'39","last_deep_scrub_stamp":"2024-03-11T22:23:05.461729+0000","last_clean_scrub_stamp":"2024-03-11T22:24:58.663290+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:01.646724+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":726,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670450+0000","last_change":"2024-03-11T22:23:51.794264+0000","last_active":"2024-03-11T22:25:48.670450+0000","last_peered":"2024-03-11T22:25:48.670450+0000","last_clean":"2024-03-11T22:25:48.670450+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:25:48.670450+0000","last_undegraded":"2024-03-11T22:25:48.670450+0000","last_fullsized":"2024-03-11T22:25:48.670450+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:51.794140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.794140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":702,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670025+0000","last_change":"2024-03-11T22:23:48.515368+0000","last_active":"2024-03-11T22:25:48.670025+0000","last_peered":"2024-03-11T22:25:48.670025+0000","last_clean":"2024-03-11T22:25:48.670025+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:25:48.670025+0000","last_undegraded":"2024-03-11T22:25:48.670025+0000","last_fullsized":"2024-03-11T22:25:48.670025+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.515251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.515251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602673+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":722,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:51.099747+0000","last_change":"2024-03-11T22:25:51.099786+0000","last_active":"2024-03-11T22:25:51.099747+0000","last_peered":"2024-03-11T22:25:51.099747+0000","last_clean":"2024-03-11T22:25:51.099747+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:25:51.099747+0000","last_undegraded":"2024-03-11T22:25:51.099747+0000","last_fullsized":"2024-03-11T22:25:51.099747+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:51.099663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:51.099663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:13.674011+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":1001,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.673149+0000","last_change":"2024-03-11T22:22:57.117070+0000","last_active":"2024-03-11T22:25:48.673149+0000","last_peered":"2024-03-11T22:25:48.673149+0000","last_clean":"2024-03-11T22:25:48.673149+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:25:48.673149+0000","last_undegraded":"2024-03-11T22:25:48.673149+0000","last_fullsized":"2024-03-11T22:25:48.673149+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:22:57.116971+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:22:57.116971+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"390'13923","reported_seq":14051,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:50.618050+0000","last_change":"2024-03-11T22:25:50.467218+0000","last_active":"2024-03-11T22:25:50.618050+0000","last_peered":"2024-03-11T22:25:50.618050+0000","last_clean":"2024-03-11T22:25:50.618050+0000","last_became_active":"2024-03-11T22:22:52.798328+0000","last_became_peered":"2024-03-11T22:22:52.798328+0000","last_unstale":"2024-03-11T22:25:50.618050+0000","last_undegraded":"2024-03-11T22:25:50.618050+0000","last_fullsized":"2024-03-11T22:25:50.618050+0000","mapping_epoch":348,"log_start":"375'8900","ondisk_log_start":"375'8900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'13909","last_scrub_stamp":"2024-03-11T22:25:50.466823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:50.466823+0000","objects_scrubbed":2545,"log_size":5023,"log_dups_size":0,"ondisk_log_size":5023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:14.543146+0000","scrub_duration":622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164691968,"num_objects":2513,"num_object_clones":0,"num_object_copies":7539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13923,"num_write_kb":525952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"273'19","reported_seq":534,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670707+0000","last_change":"2024-03-11T22:24:57.716291+0000","last_active":"2024-03-11T22:25:48.670707+0000","last_peered":"2024-03-11T22:25:48.670707+0000","last_clean":"2024-03-11T22:25:48.670707+0000","last_became_active":"2024-03-11T22:14:52.240437+0000","last_became_peered":"2024-03-11T22:14:52.240437+0000","last_unstale":"2024-03-11T22:25:48.670707+0000","last_undegraded":"2024-03-11T22:25:48.670707+0000","last_fullsized":"2024-03-11T22:25:48.670707+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:24:57.716144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:57.716144+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:16.100178+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":494,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.671410+0000","last_change":"2024-03-11T22:24:57.690108+0000","last_active":"2024-03-11T22:25:48.671410+0000","last_peered":"2024-03-11T22:25:48.671410+0000","last_clean":"2024-03-11T22:25:48.671410+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:25:48.671410+0000","last_undegraded":"2024-03-11T22:25:48.671410+0000","last_fullsized":"2024-03-11T22:25:48.671410+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:57.689696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:57.689696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.466412+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":702,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.675165+0000","last_change":"2024-03-11T22:24:59.851270+0000","last_active":"2024-03-11T22:25:48.675165+0000","last_peered":"2024-03-11T22:25:48.675165+0000","last_clean":"2024-03-11T22:25:48.675165+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:25:48.675165+0000","last_undegraded":"2024-03-11T22:25:48.675165+0000","last_fullsized":"2024-03-11T22:25:48.675165+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:59.851040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:59.851040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:06.559357+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":682,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.685170+0000","last_change":"2024-03-11T22:23:48.716684+0000","last_active":"2024-03-11T22:25:48.685170+0000","last_peered":"2024-03-11T22:25:48.685170+0000","last_clean":"2024-03-11T22:25:48.685170+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:25:48.685170+0000","last_undegraded":"2024-03-11T22:25:48.685170+0000","last_fullsized":"2024-03-11T22:25:48.685170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.716557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.716557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966945+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"93'130","reported_seq":1029,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.667962+0000","last_change":"2024-03-11T22:22:14.766914+0000","last_active":"2024-03-11T22:25:48.667962+0000","last_peered":"2024-03-11T22:25:48.667962+0000","last_clean":"2024-03-11T22:25:48.667962+0000","last_became_active":"2024-03-11T22:18:00.271764+0000","last_became_peered":"2024-03-11T22:18:00.271764+0000","last_unstale":"2024-03-11T22:25:48.667962+0000","last_undegraded":"2024-03-11T22:25:48.667962+0000","last_fullsized":"2024-03-11T22:25:48.667962+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:22:14.766775+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:22:14.766775+0000","last_clean_scrub_stamp":"2024-03-11T22:22:14.766775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602485+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"388'17928","reported_seq":18194,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:47.577185+0000","last_change":"2024-03-11T22:25:26.549894+0000","last_active":"2024-03-11T22:25:47.577185+0000","last_peered":"2024-03-11T22:25:47.577185+0000","last_clean":"2024-03-11T22:25:47.577185+0000","last_became_active":"2024-03-11T22:22:52.797873+0000","last_became_peered":"2024-03-11T22:22:52.797873+0000","last_unstale":"2024-03-11T22:25:47.577185+0000","last_undegraded":"2024-03-11T22:25:47.577185+0000","last_fullsized":"2024-03-11T22:25:47.577185+0000","mapping_epoch":348,"log_start":"384'13102","ondisk_log_start":"384'13102","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":4242,"log_size":4826,"log_dups_size":0,"ondisk_log_size":4826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183369728,"num_objects":2798,"num_object_clones":0,"num_object_copies":8394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13686,"num_write_kb":527488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"273'34","reported_seq":764,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.669127+0000","last_change":"2024-03-11T22:21:49.811476+0000","last_active":"2024-03-11T22:25:48.669127+0000","last_peered":"2024-03-11T22:25:48.669127+0000","last_clean":"2024-03-11T22:25:48.669127+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:25:48.669127+0000","last_undegraded":"2024-03-11T22:25:48.669127+0000","last_fullsized":"2024-03-11T22:25:48.669127+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:21:49.811214+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:21:49.811214+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602421+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":714,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.682067+0000","last_change":"2024-03-11T22:25:06.617067+0000","last_active":"2024-03-11T22:25:48.682067+0000","last_peered":"2024-03-11T22:25:48.682067+0000","last_clean":"2024-03-11T22:25:48.682067+0000","last_became_active":"2024-03-11T22:20:29.426224+0000","last_became_peered":"2024-03-11T22:20:29.426224+0000","last_unstale":"2024-03-11T22:25:48.682067+0000","last_undegraded":"2024-03-11T22:25:48.682067+0000","last_fullsized":"2024-03-11T22:25:48.682067+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:06.616932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:25:06.616932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:21.802971+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":626,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.674960+0000","last_change":"2024-03-11T22:24:00.708144+0000","last_active":"2024-03-11T22:25:48.674960+0000","last_peered":"2024-03-11T22:25:48.674960+0000","last_clean":"2024-03-11T22:25:48.674960+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:25:48.674960+0000","last_undegraded":"2024-03-11T22:25:48.674960+0000","last_fullsized":"2024-03-11T22:25:48.674960+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:24:00.708018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.708018+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":690,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.674815+0000","last_change":"2024-03-11T22:23:33.067353+0000","last_active":"2024-03-11T22:25:48.674815+0000","last_peered":"2024-03-11T22:25:48.674815+0000","last_clean":"2024-03-11T22:25:48.674815+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:25:48.674815+0000","last_undegraded":"2024-03-11T22:25:48.674815+0000","last_fullsized":"2024-03-11T22:25:48.674815+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:33.067236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:23:33.067236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":1010,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.673665+0000","last_change":"2024-03-11T22:24:48.925640+0000","last_active":"2024-03-11T22:25:48.673665+0000","last_peered":"2024-03-11T22:25:48.673665+0000","last_clean":"2024-03-11T22:25:48.673665+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:25:48.673665+0000","last_undegraded":"2024-03-11T22:25:48.673665+0000","last_fullsized":"2024-03-11T22:25:48.673665+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:48.925477+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:48.925477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:02.846482+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.e","version":"390'14131","reported_seq":14294,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:50.612077+0000","last_change":"2024-03-11T22:25:01.501408+0000","last_active":"2024-03-11T22:25:50.612077+0000","last_peered":"2024-03-11T22:25:50.612077+0000","last_clean":"2024-03-11T22:25:50.612077+0000","last_became_active":"2024-03-11T22:22:52.799804+0000","last_became_peered":"2024-03-11T22:22:52.799804+0000","last_unstale":"2024-03-11T22:25:50.612077+0000","last_undegraded":"2024-03-11T22:25:50.612077+0000","last_fullsized":"2024-03-11T22:25:50.612077+0000","mapping_epoch":348,"log_start":"375'9100","ondisk_log_start":"375'9100","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10662","last_scrub_stamp":"2024-03-11T22:25:01.500954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:01.500954+0000","objects_scrubbed":6034,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:26.252693+0000","scrub_duration":1796,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163905536,"num_objects":2501,"num_object_clones":0,"num_object_copies":7503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14131,"num_write_kb":532224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":479,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.684794+0000","last_change":"2024-03-11T22:22:53.846450+0000","last_active":"2024-03-11T22:25:48.684794+0000","last_peered":"2024-03-11T22:25:48.684794+0000","last_clean":"2024-03-11T22:25:48.684794+0000","last_became_active":"2024-03-11T22:20:29.438896+0000","last_became_peered":"2024-03-11T22:20:29.438896+0000","last_unstale":"2024-03-11T22:25:48.684794+0000","last_undegraded":"2024-03-11T22:25:48.684794+0000","last_fullsized":"2024-03-11T22:25:48.684794+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:53.846355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:22:53.846355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"273'30","reported_seq":792,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.682119+0000","last_change":"2024-03-11T22:23:05.635969+0000","last_active":"2024-03-11T22:25:48.682119+0000","last_peered":"2024-03-11T22:25:48.682119+0000","last_clean":"2024-03-11T22:25:48.682119+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:25:48.682119+0000","last_undegraded":"2024-03-11T22:25:48.682119+0000","last_fullsized":"2024-03-11T22:25:48.682119+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'30","last_scrub_stamp":"2024-03-11T22:23:05.635556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:05.635556+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966677+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":714,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.694174+0000","last_change":"2024-03-11T22:23:57.680282+0000","last_active":"2024-03-11T22:25:48.694174+0000","last_peered":"2024-03-11T22:25:48.694174+0000","last_clean":"2024-03-11T22:25:48.694174+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:25:48.694174+0000","last_undegraded":"2024-03-11T22:25:48.694174+0000","last_fullsized":"2024-03-11T22:25:48.694174+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:57.680156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:57.680156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.967155+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":469,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:51.005303+0000","last_change":"2024-03-11T22:25:51.005350+0000","last_active":"2024-03-11T22:25:51.005303+0000","last_peered":"2024-03-11T22:25:51.005303+0000","last_clean":"2024-03-11T22:25:51.005303+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:25:51.005303+0000","last_undegraded":"2024-03-11T22:25:51.005303+0000","last_fullsized":"2024-03-11T22:25:51.005303+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:51.005219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:51.005219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:18.615750+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":971,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.675444+0000","last_change":"2024-03-11T22:25:05.774090+0000","last_active":"2024-03-11T22:25:48.675444+0000","last_peered":"2024-03-11T22:25:48.675444+0000","last_clean":"2024-03-11T22:25:48.675444+0000","last_became_active":"2024-03-11T22:18:20.240728+0000","last_became_peered":"2024-03-11T22:18:20.240728+0000","last_unstale":"2024-03-11T22:25:48.675444+0000","last_undegraded":"2024-03-11T22:25:48.675444+0000","last_fullsized":"2024-03-11T22:25:48.675444+0000","mapping_epoch":175,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:25:05.773963+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:25:05.773963+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.773963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:17.950615+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.f","version":"390'13855","reported_seq":14018,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:50.623314+0000","last_change":"2024-03-11T22:25:00.782159+0000","last_active":"2024-03-11T22:25:50.623314+0000","last_peered":"2024-03-11T22:25:50.623314+0000","last_clean":"2024-03-11T22:25:50.623314+0000","last_became_active":"2024-03-11T22:22:52.469064+0000","last_became_peered":"2024-03-11T22:22:52.469064+0000","last_unstale":"2024-03-11T22:25:50.623314+0000","last_undegraded":"2024-03-11T22:25:50.623314+0000","last_fullsized":"2024-03-11T22:25:50.623314+0000","mapping_epoch":348,"log_start":"375'8800","ondisk_log_start":"375'8800","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10438","last_scrub_stamp":"2024-03-11T22:25:00.781894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:00.781894+0000","objects_scrubbed":6054,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:20.149816+0000","scrub_duration":2051,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167968768,"num_objects":2563,"num_object_clones":0,"num_object_copies":7689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13855,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":903,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.682160+0000","last_change":"2024-03-11T22:23:07.693414+0000","last_active":"2024-03-11T22:25:48.682160+0000","last_peered":"2024-03-11T22:25:48.682160+0000","last_clean":"2024-03-11T22:25:48.682160+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:25:48.682160+0000","last_undegraded":"2024-03-11T22:25:48.682160+0000","last_fullsized":"2024-03-11T22:25:48.682160+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:23:07.693340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:07.693340+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"273'28","reported_seq":530,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.673535+0000","last_change":"2024-03-11T22:24:44.976788+0000","last_active":"2024-03-11T22:25:48.673535+0000","last_peered":"2024-03-11T22:25:48.673535+0000","last_clean":"2024-03-11T22:25:48.673535+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:25:48.673535+0000","last_undegraded":"2024-03-11T22:25:48.673535+0000","last_fullsized":"2024-03-11T22:25:48.673535+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:24:44.976524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:44.976524+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:13.465935+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":699,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.630998+0000","last_change":"2024-03-11T22:22:56.087742+0000","last_active":"2024-03-11T22:25:43.630998+0000","last_peered":"2024-03-11T22:25:43.630998+0000","last_clean":"2024-03-11T22:25:43.630998+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:25:43.630998+0000","last_undegraded":"2024-03-11T22:25:43.630998+0000","last_fullsized":"2024-03-11T22:25:43.630998+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:22:56.087614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:22:56.087614+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.927616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":657,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670169+0000","last_change":"2024-03-11T22:23:59.089317+0000","last_active":"2024-03-11T22:25:48.670169+0000","last_peered":"2024-03-11T22:25:48.670169+0000","last_clean":"2024-03-11T22:25:48.670169+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:25:48.670169+0000","last_undegraded":"2024-03-11T22:25:48.670169+0000","last_fullsized":"2024-03-11T22:25:48.670169+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:59.089227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:23:59.089227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":1018,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.684990+0000","last_change":"2024-03-11T22:23:39.755855+0000","last_active":"2024-03-11T22:25:48.684990+0000","last_peered":"2024-03-11T22:25:48.684990+0000","last_clean":"2024-03-11T22:25:48.684990+0000","last_became_active":"2024-03-11T22:20:32.293085+0000","last_became_peered":"2024-03-11T22:20:32.293085+0000","last_unstale":"2024-03-11T22:25:48.684990+0000","last_undegraded":"2024-03-11T22:25:48.684990+0000","last_fullsized":"2024-03-11T22:25:48.684990+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:39.755701+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:39.755701+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.967014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"388'13720","reported_seq":13810,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:47.628279+0000","last_change":"2024-03-11T22:22:52.799793+0000","last_active":"2024-03-11T22:25:47.628279+0000","last_peered":"2024-03-11T22:25:47.628279+0000","last_clean":"2024-03-11T22:25:47.628279+0000","last_became_active":"2024-03-11T22:22:52.799425+0000","last_became_peered":"2024-03-11T22:22:52.799425+0000","last_unstale":"2024-03-11T22:25:47.628279+0000","last_undegraded":"2024-03-11T22:25:47.628279+0000","last_fullsized":"2024-03-11T22:25:47.628279+0000","mapping_epoch":348,"log_start":"375'8900","ondisk_log_start":"375'8900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179044352,"num_objects":2732,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13720,"num_write_kb":526464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"378'9","reported_seq":915,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.630222+0000","last_change":"2024-03-11T22:24:54.792194+0000","last_active":"2024-03-11T22:25:43.630222+0000","last_peered":"2024-03-11T22:25:43.630222+0000","last_clean":"2024-03-11T22:25:43.630222+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:25:43.630222+0000","last_undegraded":"2024-03-11T22:25:43.630222+0000","last_fullsized":"2024-03-11T22:25:43.630222+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9","last_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_deep_scrub":"378'9","last_deep_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_clean_scrub_stamp":"2024-03-11T22:24:54.792049+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:07.965072+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"273'20","reported_seq":745,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670471+0000","last_change":"2024-03-11T22:24:47.479305+0000","last_active":"2024-03-11T22:25:48.670471+0000","last_peered":"2024-03-11T22:25:48.670471+0000","last_clean":"2024-03-11T22:25:48.670471+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:25:48.670471+0000","last_undegraded":"2024-03-11T22:25:48.670471+0000","last_fullsized":"2024-03-11T22:25:48.670471+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:24:47.479158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:47.479158+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:09.501559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":711,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.675105+0000","last_change":"2024-03-11T22:23:42.943711+0000","last_active":"2024-03-11T22:25:48.675105+0000","last_peered":"2024-03-11T22:25:48.675105+0000","last_clean":"2024-03-11T22:25:48.675105+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:25:48.675105+0000","last_undegraded":"2024-03-11T22:25:48.675105+0000","last_fullsized":"2024-03-11T22:25:48.675105+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:42.943546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:42.943546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":687,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.617742+0000","last_change":"2024-03-11T22:22:55.110611+0000","last_active":"2024-03-11T22:25:43.617742+0000","last_peered":"2024-03-11T22:25:43.617742+0000","last_clean":"2024-03-11T22:25:43.617742+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:25:43.617742+0000","last_undegraded":"2024-03-11T22:25:43.617742+0000","last_fullsized":"2024-03-11T22:25:43.617742+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:55.110461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:55.110461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934387+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":943,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670997+0000","last_change":"2024-03-11T22:23:09.000258+0000","last_active":"2024-03-11T22:25:48.670997+0000","last_peered":"2024-03-11T22:25:48.670997+0000","last_clean":"2024-03-11T22:25:48.670997+0000","last_became_active":"2024-03-11T22:20:32.290512+0000","last_became_peered":"2024-03-11T22:20:32.290512+0000","last_unstale":"2024-03-11T22:25:48.670997+0000","last_undegraded":"2024-03-11T22:25:48.670997+0000","last_fullsized":"2024-03-11T22:25:48.670997+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:09.000091+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:09.000091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"390'14087","reported_seq":14255,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:50.564098+0000","last_change":"2024-03-11T22:25:48.784665+0000","last_active":"2024-03-11T22:25:50.564098+0000","last_peered":"2024-03-11T22:25:50.564098+0000","last_clean":"2024-03-11T22:25:50.564098+0000","last_became_active":"2024-03-11T22:22:52.797795+0000","last_became_peered":"2024-03-11T22:22:52.797795+0000","last_unstale":"2024-03-11T22:25:50.564098+0000","last_undegraded":"2024-03-11T22:25:50.564098+0000","last_fullsized":"2024-03-11T22:25:50.564098+0000","mapping_epoch":348,"log_start":"375'9000","ondisk_log_start":"375'9000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'13952","last_scrub_stamp":"2024-03-11T22:25:48.784383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:48.784383+0000","objects_scrubbed":2709,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:51.707764+0000","scrub_duration":940,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166330392,"num_objects":2539,"num_object_clones":0,"num_object_copies":7617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2539,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14087,"num_write_kb":531969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":719,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670345+0000","last_change":"2024-03-11T22:24:00.941070+0000","last_active":"2024-03-11T22:25:48.670345+0000","last_peered":"2024-03-11T22:25:48.670345+0000","last_clean":"2024-03-11T22:25:48.670345+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:25:48.670345+0000","last_undegraded":"2024-03-11T22:25:48.670345+0000","last_fullsized":"2024-03-11T22:25:48.670345+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:00.940803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.940803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"273'29","reported_seq":746,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.617789+0000","last_change":"2024-03-11T22:23:37.154525+0000","last_active":"2024-03-11T22:25:43.617789+0000","last_peered":"2024-03-11T22:25:43.617789+0000","last_clean":"2024-03-11T22:25:43.617789+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:25:43.617789+0000","last_undegraded":"2024-03-11T22:25:43.617789+0000","last_fullsized":"2024-03-11T22:25:43.617789+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'29","last_scrub_stamp":"2024-03-11T22:23:37.154389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:37.154389+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934673+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":711,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.669818+0000","last_change":"2024-03-11T22:23:04.706330+0000","last_active":"2024-03-11T22:25:48.669818+0000","last_peered":"2024-03-11T22:25:48.669818+0000","last_clean":"2024-03-11T22:25:48.669818+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:25:48.669818+0000","last_undegraded":"2024-03-11T22:25:48.669818+0000","last_fullsized":"2024-03-11T22:25:48.669818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:04.706082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:04.706082+0000","last_clean_scrub_stamp":"2024-03-11T22:23:04.706082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":699,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.667263+0000","last_change":"2024-03-11T22:22:58.736160+0000","last_active":"2024-03-11T22:25:48.667263+0000","last_peered":"2024-03-11T22:25:48.667263+0000","last_clean":"2024-03-11T22:25:48.667263+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:25:48.667263+0000","last_undegraded":"2024-03-11T22:25:48.667263+0000","last_fullsized":"2024-03-11T22:25:48.667263+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:58.736057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:58.736057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602623+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":918,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670879+0000","last_change":"2024-03-11T22:23:47.783004+0000","last_active":"2024-03-11T22:25:48.670879+0000","last_peered":"2024-03-11T22:25:48.670879+0000","last_clean":"2024-03-11T22:25:48.670879+0000","last_became_active":"2024-03-11T22:18:17.238247+0000","last_became_peered":"2024-03-11T22:18:17.238247+0000","last_unstale":"2024-03-11T22:25:48.670879+0000","last_undegraded":"2024-03-11T22:25:48.670879+0000","last_fullsized":"2024-03-11T22:25:48.670879+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:47.782891+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:47.782891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"388'13803","reported_seq":13962,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:47.585797+0000","last_change":"2024-03-11T22:25:05.585418+0000","last_active":"2024-03-11T22:25:47.585797+0000","last_peered":"2024-03-11T22:25:47.585797+0000","last_clean":"2024-03-11T22:25:47.585797+0000","last_became_active":"2024-03-11T22:22:52.496115+0000","last_became_peered":"2024-03-11T22:22:52.496115+0000","last_unstale":"2024-03-11T22:25:47.585797+0000","last_undegraded":"2024-03-11T22:25:47.585797+0000","last_fullsized":"2024-03-11T22:25:47.585797+0000","mapping_epoch":348,"log_start":"375'9000","ondisk_log_start":"375'9000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'10743","last_scrub_stamp":"2024-03-11T22:25:05.585079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.585079+0000","objects_scrubbed":5803,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:06.494058+0000","scrub_duration":1911,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175570944,"num_objects":2679,"num_object_clones":0,"num_object_copies":8037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13803,"num_write_kb":527424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":702,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.675254+0000","last_change":"2024-03-11T22:24:39.906388+0000","last_active":"2024-03-11T22:25:48.675254+0000","last_peered":"2024-03-11T22:25:48.675254+0000","last_clean":"2024-03-11T22:25:48.675254+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:25:48.675254+0000","last_undegraded":"2024-03-11T22:25:48.675254+0000","last_fullsized":"2024-03-11T22:25:48.675254+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:39.906243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.906243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:00.278120+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"273'24","reported_seq":738,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.630841+0000","last_change":"2024-03-11T22:23:41.208818+0000","last_active":"2024-03-11T22:25:43.630841+0000","last_peered":"2024-03-11T22:25:43.630841+0000","last_clean":"2024-03-11T22:25:43.630841+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:25:43.630841+0000","last_undegraded":"2024-03-11T22:25:43.630841+0000","last_fullsized":"2024-03-11T22:25:43.630841+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'24","last_scrub_stamp":"2024-03-11T22:23:41.208655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:41.208655+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934550+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":1729,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:50.417189+0000","last_change":"2024-03-11T22:22:19.742008+0000","last_active":"2024-03-11T22:25:50.417189+0000","last_peered":"2024-03-11T22:25:50.417189+0000","last_clean":"2024-03-11T22:25:50.417189+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:25:50.417189+0000","last_undegraded":"2024-03-11T22:25:50.417189+0000","last_fullsized":"2024-03-11T22:25:50.417189+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:22:19.741884+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:22:19.741884+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602542+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1003,"num_read_kb":998,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":707,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.616698+0000","last_change":"2024-03-11T22:24:46.870653+0000","last_active":"2024-03-11T22:25:43.616698+0000","last_peered":"2024-03-11T22:25:43.616698+0000","last_clean":"2024-03-11T22:25:43.616698+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:25:43.616698+0000","last_undegraded":"2024-03-11T22:25:43.616698+0000","last_fullsized":"2024-03-11T22:25:43.616698+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:46.870527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.870527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:51.216116+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":707,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.667752+0000","last_change":"2024-03-11T22:22:22.750641+0000","last_active":"2024-03-11T22:25:48.667752+0000","last_peered":"2024-03-11T22:25:48.667752+0000","last_clean":"2024-03-11T22:25:48.667752+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:25:48.667752+0000","last_undegraded":"2024-03-11T22:25:48.667752+0000","last_fullsized":"2024-03-11T22:25:48.667752+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:22.750560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:22.750560+0000","last_clean_scrub_stamp":"2024-03-11T22:22:22.750560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":1008,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.668184+0000","last_change":"2024-03-11T22:24:45.428664+0000","last_active":"2024-03-11T22:25:48.668184+0000","last_peered":"2024-03-11T22:25:48.668184+0000","last_clean":"2024-03-11T22:25:48.668184+0000","last_became_active":"2024-03-11T22:14:38.164101+0000","last_became_peered":"2024-03-11T22:14:38.164101+0000","last_unstale":"2024-03-11T22:25:48.668184+0000","last_undegraded":"2024-03-11T22:25:48.668184+0000","last_fullsized":"2024-03-11T22:25:48.668184+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:45.428590+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:45.428590+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:04.484198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"390'14037","reported_seq":14131,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:50.599046+0000","last_change":"2024-03-11T22:22:52.472336+0000","last_active":"2024-03-11T22:25:50.599046+0000","last_peered":"2024-03-11T22:25:50.599046+0000","last_clean":"2024-03-11T22:25:50.599046+0000","last_became_active":"2024-03-11T22:22:52.472068+0000","last_became_peered":"2024-03-11T22:22:52.472068+0000","last_unstale":"2024-03-11T22:25:50.599046+0000","last_undegraded":"2024-03-11T22:25:50.599046+0000","last_fullsized":"2024-03-11T22:25:50.599046+0000","mapping_epoch":348,"log_start":"375'9000","ondisk_log_start":"375'9000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":5037,"log_dups_size":0,"ondisk_log_size":5037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160759808,"num_objects":2453,"num_object_clones":0,"num_object_copies":7359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14037,"num_write_kb":527680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":677,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.672526+0000","last_change":"2024-03-11T22:24:45.777817+0000","last_active":"2024-03-11T22:25:48.672526+0000","last_peered":"2024-03-11T22:25:48.672526+0000","last_clean":"2024-03-11T22:25:48.672526+0000","last_became_active":"2024-03-11T22:20:29.420497+0000","last_became_peered":"2024-03-11T22:20:29.420497+0000","last_unstale":"2024-03-11T22:25:48.672526+0000","last_undegraded":"2024-03-11T22:25:48.672526+0000","last_fullsized":"2024-03-11T22:25:48.672526+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:45.777703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:24:45.777703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:54.742850+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"273'28","reported_seq":971,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.671213+0000","last_change":"2024-03-11T22:24:42.729950+0000","last_active":"2024-03-11T22:25:48.671213+0000","last_peered":"2024-03-11T22:25:48.671213+0000","last_clean":"2024-03-11T22:25:48.671213+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:25:48.671213+0000","last_undegraded":"2024-03-11T22:25:48.671213+0000","last_fullsized":"2024-03-11T22:25:48.671213+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:24:42.729799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:42.729799+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.455019+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":703,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.682259+0000","last_change":"2024-03-11T22:24:47.653085+0000","last_active":"2024-03-11T22:25:48.682259+0000","last_peered":"2024-03-11T22:25:48.682259+0000","last_clean":"2024-03-11T22:25:48.682259+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:25:48.682259+0000","last_undegraded":"2024-03-11T22:25:48.682259+0000","last_fullsized":"2024-03-11T22:25:48.682259+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:47.652970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:24:47.652970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.393586+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":1022,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.617847+0000","last_change":"2024-03-11T22:24:42.773278+0000","last_active":"2024-03-11T22:25:43.617847+0000","last_peered":"2024-03-11T22:25:43.617847+0000","last_clean":"2024-03-11T22:25:43.617847+0000","last_became_active":"2024-03-11T22:20:32.293740+0000","last_became_peered":"2024-03-11T22:20:32.293740+0000","last_unstale":"2024-03-11T22:25:43.617847+0000","last_undegraded":"2024-03-11T22:25:43.617847+0000","last_fullsized":"2024-03-11T22:25:43.617847+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:42.773024+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:24:42.773024+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:47.036511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":698,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.674831+0000","last_change":"2024-03-11T22:24:54.991281+0000","last_active":"2024-03-11T22:25:48.674831+0000","last_peered":"2024-03-11T22:25:48.674831+0000","last_clean":"2024-03-11T22:25:48.674831+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:25:48.674831+0000","last_undegraded":"2024-03-11T22:25:48.674831+0000","last_fullsized":"2024-03-11T22:25:48.674831+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:54.991081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:54.991081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:59.396448+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.9","version":"390'14123","reported_seq":14287,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:50.649452+0000","last_change":"2024-03-11T22:25:02.575828+0000","last_active":"2024-03-11T22:25:50.649452+0000","last_peered":"2024-03-11T22:25:50.649452+0000","last_clean":"2024-03-11T22:25:50.649452+0000","last_became_active":"2024-03-11T22:22:52.496328+0000","last_became_peered":"2024-03-11T22:22:52.496328+0000","last_unstale":"2024-03-11T22:25:50.649452+0000","last_undegraded":"2024-03-11T22:25:50.649452+0000","last_fullsized":"2024-03-11T22:25:50.649452+0000","mapping_epoch":348,"log_start":"375'9200","ondisk_log_start":"375'9200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'10701","last_scrub_stamp":"2024-03-11T22:25:02.575600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:02.575600+0000","objects_scrubbed":6024,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:02.718025+0000","scrub_duration":1689,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167837696,"num_objects":2561,"num_object_clones":0,"num_object_copies":7683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14123,"num_write_kb":533888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":726,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.675447+0000","last_change":"2024-03-11T22:24:48.727176+0000","last_active":"2024-03-11T22:25:48.675447+0000","last_peered":"2024-03-11T22:25:48.675447+0000","last_clean":"2024-03-11T22:25:48.675447+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:25:48.675447+0000","last_undegraded":"2024-03-11T22:25:48.675447+0000","last_fullsized":"2024-03-11T22:25:48.675447+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:48.727061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:48.727061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:17.884566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"273'17","reported_seq":747,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.669279+0000","last_change":"2024-03-11T22:24:41.396924+0000","last_active":"2024-03-11T22:25:48.669279+0000","last_peered":"2024-03-11T22:25:48.669279+0000","last_clean":"2024-03-11T22:25:48.669279+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:25:48.669279+0000","last_undegraded":"2024-03-11T22:25:48.669279+0000","last_fullsized":"2024-03-11T22:25:48.669279+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'17","last_scrub_stamp":"2024-03-11T22:24:41.396754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.396754+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:02.581690+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":744,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.671807+0000","last_change":"2024-03-11T22:23:45.364864+0000","last_active":"2024-03-11T22:25:48.671807+0000","last_peered":"2024-03-11T22:25:48.671807+0000","last_clean":"2024-03-11T22:25:48.671807+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:25:48.671807+0000","last_undegraded":"2024-03-11T22:25:48.671807+0000","last_fullsized":"2024-03-11T22:25:48.671807+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:45.364724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:23:45.364724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:49.910169+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":719,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.671224+0000","last_change":"2024-03-11T22:23:58.411257+0000","last_active":"2024-03-11T22:25:48.671224+0000","last_peered":"2024-03-11T22:25:48.671224+0000","last_clean":"2024-03-11T22:25:48.671224+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:25:48.671224+0000","last_undegraded":"2024-03-11T22:25:48.671224+0000","last_fullsized":"2024-03-11T22:25:48.671224+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:58.411052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.411052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602759+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":657,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.671222+0000","last_change":"2024-03-11T22:24:38.834963+0000","last_active":"2024-03-11T22:25:48.671222+0000","last_peered":"2024-03-11T22:25:48.671222+0000","last_clean":"2024-03-11T22:25:48.671222+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:25:48.671222+0000","last_undegraded":"2024-03-11T22:25:48.671222+0000","last_fullsized":"2024-03-11T22:25:48.671222+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:38.834828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:24:38.834828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.525769+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":1021,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.630422+0000","last_change":"2024-03-11T22:23:11.030109+0000","last_active":"2024-03-11T22:25:43.630422+0000","last_peered":"2024-03-11T22:25:43.630422+0000","last_clean":"2024-03-11T22:25:43.630422+0000","last_became_active":"2024-03-11T22:20:32.295175+0000","last_became_peered":"2024-03-11T22:20:32.295175+0000","last_unstale":"2024-03-11T22:25:43.630422+0000","last_undegraded":"2024-03-11T22:25:43.630422+0000","last_fullsized":"2024-03-11T22:25:43.630422+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:11.029981+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:11.029981+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934420+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"390'14083","reported_seq":14244,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:50.605470+0000","last_change":"2024-03-11T22:25:05.305867+0000","last_active":"2024-03-11T22:25:50.605470+0000","last_peered":"2024-03-11T22:25:50.605470+0000","last_clean":"2024-03-11T22:25:50.605470+0000","last_became_active":"2024-03-11T22:22:52.467125+0000","last_became_peered":"2024-03-11T22:22:52.467125+0000","last_unstale":"2024-03-11T22:25:50.605470+0000","last_undegraded":"2024-03-11T22:25:50.605470+0000","last_fullsized":"2024-03-11T22:25:50.605470+0000","mapping_epoch":348,"log_start":"375'9000","ondisk_log_start":"375'9000","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'10771","last_scrub_stamp":"2024-03-11T22:25:05.305569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.305569+0000","objects_scrubbed":5888,"log_size":5083,"log_dups_size":0,"ondisk_log_size":5083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:07.737385+0000","scrub_duration":1552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166658048,"num_objects":2543,"num_object_clones":0,"num_object_copies":7629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14083,"num_write_kb":532032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":699,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.633418+0000","last_change":"2024-03-11T22:24:01.031042+0000","last_active":"2024-03-11T22:25:43.633418+0000","last_peered":"2024-03-11T22:25:43.633418+0000","last_clean":"2024-03-11T22:25:43.633418+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:25:43.633418+0000","last_undegraded":"2024-03-11T22:25:43.633418+0000","last_fullsized":"2024-03-11T22:25:43.633418+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:01.030925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:01.030925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934825+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"273'40","reported_seq":989,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.669921+0000","last_change":"2024-03-11T22:23:57.429937+0000","last_active":"2024-03-11T22:25:48.669921+0000","last_peered":"2024-03-11T22:25:48.669921+0000","last_clean":"2024-03-11T22:25:48.669921+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:25:48.669921+0000","last_undegraded":"2024-03-11T22:25:48.669921+0000","last_fullsized":"2024-03-11T22:25:48.669921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'40","last_scrub_stamp":"2024-03-11T22:23:57.429825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:57.429825+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":466,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.617696+0000","last_change":"2024-03-11T22:23:56.935550+0000","last_active":"2024-03-11T22:25:43.617696+0000","last_peered":"2024-03-11T22:25:43.617696+0000","last_clean":"2024-03-11T22:25:43.617696+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:25:43.617696+0000","last_undegraded":"2024-03-11T22:25:43.617696+0000","last_fullsized":"2024-03-11T22:25:43.617696+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:56.934997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:56.934997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934639+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":723,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.673926+0000","last_change":"2024-03-11T22:25:05.938232+0000","last_active":"2024-03-11T22:25:48.673926+0000","last_peered":"2024-03-11T22:25:48.673926+0000","last_clean":"2024-03-11T22:25:48.673926+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:25:48.673926+0000","last_undegraded":"2024-03-11T22:25:48.673926+0000","last_fullsized":"2024-03-11T22:25:48.673926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:25:05.938003+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.938003+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:25.242523+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":694,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.683821+0000","last_change":"2024-03-11T22:23:03.679822+0000","last_active":"2024-03-11T22:25:48.683821+0000","last_peered":"2024-03-11T22:25:48.683821+0000","last_clean":"2024-03-11T22:25:48.683821+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:25:48.683821+0000","last_undegraded":"2024-03-11T22:25:48.683821+0000","last_fullsized":"2024-03-11T22:25:48.683821+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:03.679700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:23:03.679700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966586+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":1007,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.673436+0000","last_change":"2024-03-11T22:24:56.886620+0000","last_active":"2024-03-11T22:25:48.673436+0000","last_peered":"2024-03-11T22:25:48.673436+0000","last_clean":"2024-03-11T22:25:48.673436+0000","last_became_active":"2024-03-11T22:14:38.164859+0000","last_became_peered":"2024-03-11T22:14:38.164859+0000","last_unstale":"2024-03-11T22:25:48.673436+0000","last_undegraded":"2024-03-11T22:25:48.673436+0000","last_fullsized":"2024-03-11T22:25:48.673436+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":144,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:56.886500+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:56.886500+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:58.328021+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"390'13958","reported_seq":14126,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:51.073160+0000","last_change":"2024-03-11T22:24:55.556260+0000","last_active":"2024-03-11T22:25:51.073160+0000","last_peered":"2024-03-11T22:25:51.073160+0000","last_clean":"2024-03-11T22:25:51.073160+0000","last_became_active":"2024-03-11T22:22:52.469442+0000","last_became_peered":"2024-03-11T22:22:52.469442+0000","last_unstale":"2024-03-11T22:25:51.073160+0000","last_undegraded":"2024-03-11T22:25:51.073160+0000","last_fullsized":"2024-03-11T22:25:51.073160+0000","mapping_epoch":348,"log_start":"373'7900","ondisk_log_start":"373'7900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9999","last_scrub_stamp":"2024-03-11T22:24:55.555941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:24:55.555941+0000","objects_scrubbed":6582,"log_size":6058,"log_dups_size":0,"ondisk_log_size":6058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:00.936378+0000","scrub_duration":2782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165412864,"num_objects":2524,"num_object_clones":0,"num_object_copies":7572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13958,"num_write_kb":527424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"273'15","reported_seq":735,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.630119+0000","last_change":"2024-03-11T22:23:36.167248+0000","last_active":"2024-03-11T22:25:43.630119+0000","last_peered":"2024-03-11T22:25:43.630119+0000","last_clean":"2024-03-11T22:25:43.630119+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:25:43.630119+0000","last_undegraded":"2024-03-11T22:25:43.630119+0000","last_fullsized":"2024-03-11T22:25:43.630119+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:23:36.167121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:36.167121+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934517+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":710,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.674404+0000","last_change":"2024-03-11T22:24:58.853739+0000","last_active":"2024-03-11T22:25:48.674404+0000","last_peered":"2024-03-11T22:25:48.674404+0000","last_clean":"2024-03-11T22:25:48.674404+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:25:48.674404+0000","last_undegraded":"2024-03-11T22:25:48.674404+0000","last_fullsized":"2024-03-11T22:25:48.674404+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:58.853630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:58.853630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:23.082251+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":726,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.671366+0000","last_change":"2024-03-11T22:24:00.422907+0000","last_active":"2024-03-11T22:25:48.671366+0000","last_peered":"2024-03-11T22:25:48.671366+0000","last_clean":"2024-03-11T22:25:48.671366+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:25:48.671366+0000","last_undegraded":"2024-03-11T22:25:48.671366+0000","last_fullsized":"2024-03-11T22:25:48.671366+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:00.422761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.422761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:49.910317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":706,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.616914+0000","last_change":"2024-03-11T22:24:53.948958+0000","last_active":"2024-03-11T22:25:43.616914+0000","last_peered":"2024-03-11T22:25:43.616914+0000","last_clean":"2024-03-11T22:25:43.616914+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:25:43.616914+0000","last_undegraded":"2024-03-11T22:25:43.616914+0000","last_fullsized":"2024-03-11T22:25:43.616914+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:53.948853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.948853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:05.727979+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"93'130","reported_seq":852,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.683527+0000","last_change":"2024-03-11T22:23:32.711840+0000","last_active":"2024-03-11T22:25:48.683527+0000","last_peered":"2024-03-11T22:25:48.683527+0000","last_clean":"2024-03-11T22:25:48.683527+0000","last_became_active":"2024-03-11T22:20:32.292676+0000","last_became_peered":"2024-03-11T22:20:32.292676+0000","last_unstale":"2024-03-11T22:25:48.683527+0000","last_undegraded":"2024-03-11T22:25:48.683527+0000","last_fullsized":"2024-03-11T22:25:48.683527+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:32.711693+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:32.711693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966980+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"388'13688","reported_seq":13779,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:47.603784+0000","last_change":"2024-03-11T22:22:52.800885+0000","last_active":"2024-03-11T22:25:47.603784+0000","last_peered":"2024-03-11T22:25:47.603784+0000","last_clean":"2024-03-11T22:25:47.603784+0000","last_became_active":"2024-03-11T22:22:52.800526+0000","last_became_peered":"2024-03-11T22:22:52.800526+0000","last_unstale":"2024-03-11T22:25:47.603784+0000","last_undegraded":"2024-03-11T22:25:47.603784+0000","last_fullsized":"2024-03-11T22:25:47.603784+0000","mapping_epoch":348,"log_start":"375'8900","ondisk_log_start":"375'8900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":4788,"log_dups_size":0,"ondisk_log_size":4788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175243264,"num_objects":2674,"num_object_clones":0,"num_object_copies":8022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13688,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"273'35","reported_seq":829,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670321+0000","last_change":"2024-03-11T22:24:46.752170+0000","last_active":"2024-03-11T22:25:48.670321+0000","last_peered":"2024-03-11T22:25:48.670321+0000","last_clean":"2024-03-11T22:25:48.670321+0000","last_became_active":"2024-03-11T22:20:29.418041+0000","last_became_peered":"2024-03-11T22:20:29.418041+0000","last_unstale":"2024-03-11T22:25:48.670321+0000","last_undegraded":"2024-03-11T22:25:48.670321+0000","last_fullsized":"2024-03-11T22:25:48.670321+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'35","last_scrub_stamp":"2024-03-11T22:24:46.751969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.751969+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:58.638998+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":653,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.675191+0000","last_change":"2024-03-11T22:23:10.963581+0000","last_active":"2024-03-11T22:25:48.675191+0000","last_peered":"2024-03-11T22:25:48.675191+0000","last_clean":"2024-03-11T22:25:48.675191+0000","last_became_active":"2024-03-11T22:20:29.413664+0000","last_became_peered":"2024-03-11T22:20:29.413664+0000","last_unstale":"2024-03-11T22:25:48.675191+0000","last_undegraded":"2024-03-11T22:25:48.675191+0000","last_fullsized":"2024-03-11T22:25:48.675191+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:10.963448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:10.963448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":718,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.672545+0000","last_change":"2024-03-11T22:23:40.405746+0000","last_active":"2024-03-11T22:25:48.672545+0000","last_peered":"2024-03-11T22:25:48.672545+0000","last_clean":"2024-03-11T22:25:48.672545+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:25:48.672545+0000","last_undegraded":"2024-03-11T22:25:48.672545+0000","last_fullsized":"2024-03-11T22:25:48.672545+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:40.405626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:23:40.405626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:49.910197+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":677,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.673574+0000","last_change":"2024-03-11T22:23:51.364137+0000","last_active":"2024-03-11T22:25:48.673574+0000","last_peered":"2024-03-11T22:25:48.673574+0000","last_clean":"2024-03-11T22:25:48.673574+0000","last_became_active":"2024-03-11T22:20:29.415565+0000","last_became_peered":"2024-03-11T22:20:29.415565+0000","last_unstale":"2024-03-11T22:25:48.673574+0000","last_undegraded":"2024-03-11T22:25:48.673574+0000","last_fullsized":"2024-03-11T22:25:48.673574+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:51.362666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:20.931122+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.362666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:49.910141+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"93'130","reported_seq":903,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670896+0000","last_change":"2024-03-11T22:23:02.040555+0000","last_active":"2024-03-11T22:25:48.670896+0000","last_peered":"2024-03-11T22:25:48.670896+0000","last_clean":"2024-03-11T22:25:48.670896+0000","last_became_active":"2024-03-11T22:14:43.195433+0000","last_became_peered":"2024-03-11T22:14:43.195433+0000","last_unstale":"2024-03-11T22:25:48.670896+0000","last_undegraded":"2024-03-11T22:25:48.670896+0000","last_fullsized":"2024-03-11T22:25:48.670896+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:02.040354+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:02.040354+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"390'13949","reported_seq":14083,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:50.587277+0000","last_change":"2024-03-11T22:25:49.839979+0000","last_active":"2024-03-11T22:25:50.587277+0000","last_peered":"2024-03-11T22:25:50.587277+0000","last_clean":"2024-03-11T22:25:50.587277+0000","last_became_active":"2024-03-11T22:22:52.465673+0000","last_became_peered":"2024-03-11T22:22:52.465673+0000","last_unstale":"2024-03-11T22:25:50.587277+0000","last_undegraded":"2024-03-11T22:25:50.587277+0000","last_fullsized":"2024-03-11T22:25:50.587277+0000","mapping_epoch":348,"log_start":"375'8900","ondisk_log_start":"375'8900","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'13902","last_scrub_stamp":"2024-03-11T22:25:49.839666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:25:49.839666+0000","objects_scrubbed":2559,"log_size":5049,"log_dups_size":0,"ondisk_log_size":5049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:57.749717+0000","scrub_duration":991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161284096,"num_objects":2461,"num_object_clones":0,"num_object_copies":7383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13949,"num_write_kb":525120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":726,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.673316+0000","last_change":"2024-03-11T22:24:56.704048+0000","last_active":"2024-03-11T22:25:48.673316+0000","last_peered":"2024-03-11T22:25:48.673316+0000","last_clean":"2024-03-11T22:25:48.673316+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:25:48.673316+0000","last_undegraded":"2024-03-11T22:25:48.673316+0000","last_fullsized":"2024-03-11T22:25:48.673316+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:56.703933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:24:56.703933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:24.261508+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"273'18","reported_seq":527,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.673774+0000","last_change":"2024-03-11T22:23:46.897451+0000","last_active":"2024-03-11T22:25:48.673774+0000","last_peered":"2024-03-11T22:25:48.673774+0000","last_clean":"2024-03-11T22:25:48.673774+0000","last_became_active":"2024-03-11T22:14:52.247400+0000","last_became_peered":"2024-03-11T22:14:52.247400+0000","last_unstale":"2024-03-11T22:25:48.673774+0000","last_undegraded":"2024-03-11T22:25:48.673774+0000","last_fullsized":"2024-03-11T22:25:48.673774+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:23:46.897161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.897161+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":492,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670111+0000","last_change":"2024-03-11T22:24:53.435632+0000","last_active":"2024-03-11T22:25:48.670111+0000","last_peered":"2024-03-11T22:25:48.670111+0000","last_clean":"2024-03-11T22:25:48.670111+0000","last_became_active":"2024-03-11T22:20:29.416729+0000","last_became_peered":"2024-03-11T22:20:29.416729+0000","last_unstale":"2024-03-11T22:25:48.670111+0000","last_undegraded":"2024-03-11T22:25:48.670111+0000","last_fullsized":"2024-03-11T22:25:48.670111+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:53.435516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.435516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:08.499620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":704,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.682739+0000","last_change":"2024-03-11T22:23:01.776787+0000","last_active":"2024-03-11T22:25:48.682739+0000","last_peered":"2024-03-11T22:25:48.682739+0000","last_clean":"2024-03-11T22:25:48.682739+0000","last_became_active":"2024-03-11T22:20:29.438413+0000","last_became_peered":"2024-03-11T22:20:29.438413+0000","last_unstale":"2024-03-11T22:25:48.682739+0000","last_undegraded":"2024-03-11T22:25:48.682739+0000","last_fullsized":"2024-03-11T22:25:48.682739+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:23:01.776660+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:23:01.776660+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966790+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"93'130","reported_seq":1028,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.684886+0000","last_change":"2024-03-11T22:23:10.731794+0000","last_active":"2024-03-11T22:25:48.684886+0000","last_peered":"2024-03-11T22:25:48.684886+0000","last_clean":"2024-03-11T22:25:48.684886+0000","last_became_active":"2024-03-11T22:17:57.246241+0000","last_became_peered":"2024-03-11T22:17:57.246241+0000","last_unstale":"2024-03-11T22:25:48.684886+0000","last_undegraded":"2024-03-11T22:25:48.684886+0000","last_fullsized":"2024-03-11T22:25:48.684886+0000","mapping_epoch":173,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:10.731483+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:10.731483+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966635+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"390'14137","reported_seq":14232,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:50.649293+0000","last_change":"2024-03-11T22:22:52.798264+0000","last_active":"2024-03-11T22:25:50.649293+0000","last_peered":"2024-03-11T22:25:50.649293+0000","last_clean":"2024-03-11T22:25:50.649293+0000","last_became_active":"2024-03-11T22:22:52.797922+0000","last_became_peered":"2024-03-11T22:22:52.797922+0000","last_unstale":"2024-03-11T22:25:50.649293+0000","last_undegraded":"2024-03-11T22:25:50.649293+0000","last_fullsized":"2024-03-11T22:25:50.649293+0000","mapping_epoch":348,"log_start":"375'9200","ondisk_log_start":"375'9200","created":348,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:51.447385+0000","last_clean_scrub_stamp":"2024-03-11T22:22:51.447385+0000","objects_scrubbed":0,"log_size":4937,"log_dups_size":0,"ondisk_log_size":4937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158531584,"num_objects":2419,"num_object_clones":0,"num_object_copies":7257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14137,"num_write_kb":529792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"310'11","reported_seq":887,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670046+0000","last_change":"2024-03-11T22:23:56.945205+0000","last_active":"2024-03-11T22:25:48.670046+0000","last_peered":"2024-03-11T22:25:48.670046+0000","last_clean":"2024-03-11T22:25:48.670046+0000","last_became_active":"2024-03-11T22:20:29.418355+0000","last_became_peered":"2024-03-11T22:20:29.418355+0000","last_unstale":"2024-03-11T22:25:48.670046+0000","last_undegraded":"2024-03-11T22:25:48.670046+0000","last_fullsized":"2024-03-11T22:25:48.670046+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:23:56.945075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:56.945075+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"273'34","reported_seq":800,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.672968+0000","last_change":"2024-03-11T22:23:47.917460+0000","last_active":"2024-03-11T22:25:48.672968+0000","last_peered":"2024-03-11T22:25:48.672968+0000","last_clean":"2024-03-11T22:25:48.672968+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:25:48.672968+0000","last_undegraded":"2024-03-11T22:25:48.672968+0000","last_fullsized":"2024-03-11T22:25:48.672968+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:23:47.917308+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:23:47.917308+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":498,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.629670+0000","last_change":"2024-03-11T22:23:38.153481+0000","last_active":"2024-03-11T22:25:43.629670+0000","last_peered":"2024-03-11T22:25:43.629670+0000","last_clean":"2024-03-11T22:25:43.629670+0000","last_became_active":"2024-03-11T22:20:29.415226+0000","last_became_peered":"2024-03-11T22:20:29.415226+0000","last_unstale":"2024-03-11T22:25:43.629670+0000","last_undegraded":"2024-03-11T22:25:43.629670+0000","last_fullsized":"2024-03-11T22:25:43.629670+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-11T22:23:38.153326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:38.153326+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934706+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":690,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.674069+0000","last_change":"2024-03-11T22:23:44.941526+0000","last_active":"2024-03-11T22:25:48.674069+0000","last_peered":"2024-03-11T22:25:48.674069+0000","last_clean":"2024-03-11T22:25:48.674069+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:25:48.674069+0000","last_undegraded":"2024-03-11T22:25:48.674069+0000","last_fullsized":"2024-03-11T22:25:48.674069+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:44.941284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:44.941284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"93'130","reported_seq":1045,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.667892+0000","last_change":"2024-03-11T22:23:44.564522+0000","last_active":"2024-03-11T22:25:48.667892+0000","last_peered":"2024-03-11T22:25:48.667892+0000","last_clean":"2024-03-11T22:25:48.667892+0000","last_became_active":"2024-03-11T22:20:32.295606+0000","last_became_peered":"2024-03-11T22:20:32.295606+0000","last_unstale":"2024-03-11T22:25:48.667892+0000","last_undegraded":"2024-03-11T22:25:48.667892+0000","last_fullsized":"2024-03-11T22:25:48.667892+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:44.564377+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:44.564377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602733+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"273'12","reported_seq":736,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.694445+0000","last_change":"2024-03-11T22:22:16.742232+0000","last_active":"2024-03-11T22:25:48.694445+0000","last_peered":"2024-03-11T22:25:48.694445+0000","last_clean":"2024-03-11T22:25:48.694445+0000","last_became_active":"2024-03-11T22:20:29.441599+0000","last_became_peered":"2024-03-11T22:20:29.441599+0000","last_unstale":"2024-03-11T22:25:48.694445+0000","last_undegraded":"2024-03-11T22:25:48.694445+0000","last_fullsized":"2024-03-11T22:25:48.694445+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'12","last_scrub_stamp":"2024-03-11T22:22:16.742100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:22:16.742100+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966507+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":718,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.669657+0000","last_change":"2024-03-11T22:23:48.806202+0000","last_active":"2024-03-11T22:25:48.669657+0000","last_peered":"2024-03-11T22:25:48.669657+0000","last_clean":"2024-03-11T22:25:48.669657+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:25:48.669657+0000","last_undegraded":"2024-03-11T22:25:48.669657+0000","last_fullsized":"2024-03-11T22:25:48.669657+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.805960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.805960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":702,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.674707+0000","last_change":"2024-03-11T22:23:59.713751+0000","last_active":"2024-03-11T22:25:48.674707+0000","last_peered":"2024-03-11T22:25:48.674707+0000","last_clean":"2024-03-11T22:25:48.674707+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:25:48.674707+0000","last_undegraded":"2024-03-11T22:25:48.674707+0000","last_fullsized":"2024-03-11T22:25:48.674707+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:59.713626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:23:59.713626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":648,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.671294+0000","last_change":"2024-03-11T22:23:46.876978+0000","last_active":"2024-03-11T22:25:48.671294+0000","last_peered":"2024-03-11T22:25:48.671294+0000","last_clean":"2024-03-11T22:25:48.671294+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:25:48.671294+0000","last_undegraded":"2024-03-11T22:25:48.671294+0000","last_fullsized":"2024-03-11T22:25:48.671294+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:46.876749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.876749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"93'130","reported_seq":1016,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.673569+0000","last_change":"2024-03-11T22:24:52.878462+0000","last_active":"2024-03-11T22:25:48.673569+0000","last_peered":"2024-03-11T22:25:48.673569+0000","last_clean":"2024-03-11T22:25:48.673569+0000","last_became_active":"2024-03-11T22:14:43.194413+0000","last_became_peered":"2024-03-11T22:14:43.194413+0000","last_unstale":"2024-03-11T22:25:48.673569+0000","last_undegraded":"2024-03-11T22:25:48.673569+0000","last_fullsized":"2024-03-11T22:25:48.673569+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:52.878201+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:52.878201+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:04.489712+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"93'130","reported_seq":1015,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.667852+0000","last_change":"2024-03-11T22:23:51.445370+0000","last_active":"2024-03-11T22:25:48.667852+0000","last_peered":"2024-03-11T22:25:48.667852+0000","last_clean":"2024-03-11T22:25:48.667852+0000","last_became_active":"2024-03-11T22:14:43.198216+0000","last_became_peered":"2024-03-11T22:14:43.198216+0000","last_unstale":"2024-03-11T22:25:48.667852+0000","last_undegraded":"2024-03-11T22:25:48.667852+0000","last_fullsized":"2024-03-11T22:25:48.667852+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:51.445038+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:23:51.445038+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.445038+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602703+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"273'19","reported_seq":763,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.631223+0000","last_change":"2024-03-11T22:23:53.926458+0000","last_active":"2024-03-11T22:25:43.631223+0000","last_peered":"2024-03-11T22:25:43.631223+0000","last_clean":"2024-03-11T22:25:43.631223+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:25:43.631223+0000","last_undegraded":"2024-03-11T22:25:43.631223+0000","last_fullsized":"2024-03-11T22:25:43.631223+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_clean_scrub_stamp":"2024-03-11T22:23:53.926316+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":480,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.669644+0000","last_change":"2024-03-11T22:23:35.792737+0000","last_active":"2024-03-11T22:25:48.669644+0000","last_peered":"2024-03-11T22:25:48.669644+0000","last_clean":"2024-03-11T22:25:48.669644+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:25:48.669644+0000","last_undegraded":"2024-03-11T22:25:48.669644+0000","last_fullsized":"2024-03-11T22:25:48.669644+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:35.792609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:35.792609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":706,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.694364+0000","last_change":"2024-03-11T22:24:51.687892+0000","last_active":"2024-03-11T22:25:48.694364+0000","last_peered":"2024-03-11T22:25:48.694364+0000","last_clean":"2024-03-11T22:25:48.694364+0000","last_became_active":"2024-03-11T22:20:29.433005+0000","last_became_peered":"2024-03-11T22:20:29.433005+0000","last_unstale":"2024-03-11T22:25:48.694364+0000","last_undegraded":"2024-03-11T22:25:48.694364+0000","last_fullsized":"2024-03-11T22:25:48.694364+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:51.687767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:51.687767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:20.042381+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":722,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.672434+0000","last_change":"2024-03-11T22:23:57.498120+0000","last_active":"2024-03-11T22:25:48.672434+0000","last_peered":"2024-03-11T22:25:48.672434+0000","last_clean":"2024-03-11T22:25:48.672434+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:25:48.672434+0000","last_undegraded":"2024-03-11T22:25:48.672434+0000","last_fullsized":"2024-03-11T22:25:48.672434+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:57.498047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:23:57.498047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:49.910291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"273'19","reported_seq":735,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.675224+0000","last_change":"2024-03-11T22:23:37.882805+0000","last_active":"2024-03-11T22:25:48.675224+0000","last_peered":"2024-03-11T22:25:48.675224+0000","last_clean":"2024-03-11T22:25:48.675224+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:25:48.675224+0000","last_undegraded":"2024-03-11T22:25:48.675224+0000","last_fullsized":"2024-03-11T22:25:48.675224+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:37.882661+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:23:37.882661+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":718,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.630327+0000","last_change":"2024-03-11T22:23:50.030718+0000","last_active":"2024-03-11T22:25:43.630327+0000","last_peered":"2024-03-11T22:25:43.630327+0000","last_clean":"2024-03-11T22:25:43.630327+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:25:43.630327+0000","last_undegraded":"2024-03-11T22:25:43.630327+0000","last_fullsized":"2024-03-11T22:25:43.630327+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:50.030272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:50.030272+0000","last_clean_scrub_stamp":"2024-03-11T22:23:50.030272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934736+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'11","reported_seq":945,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.685482+0000","last_change":"2024-03-11T22:24:53.646061+0000","last_active":"2024-03-11T22:25:48.685482+0000","last_peered":"2024-03-11T22:25:48.685482+0000","last_clean":"2024-03-11T22:25:48.685482+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:25:48.685482+0000","last_undegraded":"2024-03-11T22:25:48.685482+0000","last_fullsized":"2024-03-11T22:25:48.685482+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:24:53.645906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.645906+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:03.406173+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":484,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.677366+0000","last_change":"2024-03-11T22:23:48.490897+0000","last_active":"2024-03-11T22:25:48.677366+0000","last_peered":"2024-03-11T22:25:48.677366+0000","last_clean":"2024-03-11T22:25:48.677366+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:25:48.677366+0000","last_undegraded":"2024-03-11T22:25:48.677366+0000","last_fullsized":"2024-03-11T22:25:48.677366+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.490757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:48.490757+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.490757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:49.910113+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"93'130","reported_seq":1001,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.673157+0000","last_change":"2024-03-11T22:23:12.969623+0000","last_active":"2024-03-11T22:25:48.673157+0000","last_peered":"2024-03-11T22:25:48.673157+0000","last_clean":"2024-03-11T22:25:48.673157+0000","last_became_active":"2024-03-11T22:14:43.194350+0000","last_became_peered":"2024-03-11T22:14:43.194350+0000","last_unstale":"2024-03-11T22:25:48.673157+0000","last_undegraded":"2024-03-11T22:25:48.673157+0000","last_fullsized":"2024-03-11T22:25:48.673157+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:12.969362+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:23:12.969362+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"273'20","reported_seq":738,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.685384+0000","last_change":"2024-03-11T22:23:00.635612+0000","last_active":"2024-03-11T22:25:48.685384+0000","last_peered":"2024-03-11T22:25:48.685384+0000","last_clean":"2024-03-11T22:25:48.685384+0000","last_became_active":"2024-03-11T22:16:15.815814+0000","last_became_peered":"2024-03-11T22:16:15.815814+0000","last_unstale":"2024-03-11T22:25:48.685384+0000","last_undegraded":"2024-03-11T22:25:48.685384+0000","last_fullsized":"2024-03-11T22:25:48.685384+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:23:00.635505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:00.635505+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":508,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.672448+0000","last_change":"2024-03-11T22:24:49.777349+0000","last_active":"2024-03-11T22:25:48.672448+0000","last_peered":"2024-03-11T22:25:48.672448+0000","last_clean":"2024-03-11T22:25:48.672448+0000","last_became_active":"2024-03-11T22:20:29.415019+0000","last_became_peered":"2024-03-11T22:20:29.415019+0000","last_unstale":"2024-03-11T22:25:48.672448+0000","last_undegraded":"2024-03-11T22:25:48.672448+0000","last_fullsized":"2024-03-11T22:25:48.672448+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:49.777229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:49.777229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:00.825874+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":918,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.674997+0000","last_change":"2024-03-11T22:23:53.776480+0000","last_active":"2024-03-11T22:25:48.674997+0000","last_peered":"2024-03-11T22:25:48.674997+0000","last_clean":"2024-03-11T22:25:48.674997+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:25:48.674997+0000","last_undegraded":"2024-03-11T22:25:48.674997+0000","last_fullsized":"2024-03-11T22:25:48.674997+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:23:53.776342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:53.776342+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":698,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.630702+0000","last_change":"2024-03-11T22:24:45.964526+0000","last_active":"2024-03-11T22:25:43.630702+0000","last_peered":"2024-03-11T22:25:43.630702+0000","last_clean":"2024-03-11T22:25:43.630702+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:25:43.630702+0000","last_undegraded":"2024-03-11T22:25:43.630702+0000","last_fullsized":"2024-03-11T22:25:43.630702+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:45.964401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:45.964401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:07.337646+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"93'130","reported_seq":953,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.616478+0000","last_change":"2024-03-11T22:23:10.014311+0000","last_active":"2024-03-11T22:25:43.616478+0000","last_peered":"2024-03-11T22:25:43.616478+0000","last_clean":"2024-03-11T22:25:43.616478+0000","last_became_active":"2024-03-11T22:14:43.191297+0000","last_became_peered":"2024-03-11T22:14:43.191297+0000","last_unstale":"2024-03-11T22:25:43.616478+0000","last_undegraded":"2024-03-11T22:25:43.616478+0000","last_fullsized":"2024-03-11T22:25:43.616478+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:10.014235+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:23:10.014235+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"273'19","reported_seq":526,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670088+0000","last_change":"2024-03-11T22:23:45.819413+0000","last_active":"2024-03-11T22:25:48.670088+0000","last_peered":"2024-03-11T22:25:48.670088+0000","last_clean":"2024-03-11T22:25:48.670088+0000","last_became_active":"2024-03-11T22:14:52.239810+0000","last_became_peered":"2024-03-11T22:14:52.239810+0000","last_unstale":"2024-03-11T22:25:48.670088+0000","last_undegraded":"2024-03-11T22:25:48.670088+0000","last_fullsized":"2024-03-11T22:25:48.670088+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:45.818986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:45.818986+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":505,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670461+0000","last_change":"2024-03-11T22:23:44.776567+0000","last_active":"2024-03-11T22:25:48.670461+0000","last_peered":"2024-03-11T22:25:48.670461+0000","last_clean":"2024-03-11T22:25:48.670461+0000","last_became_active":"2024-03-11T22:20:29.420867+0000","last_became_peered":"2024-03-11T22:20:29.420867+0000","last_unstale":"2024-03-11T22:25:48.670461+0000","last_undegraded":"2024-03-11T22:25:48.670461+0000","last_fullsized":"2024-03-11T22:25:48.670461+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:23:44.776433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:44.776433+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":702,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670385+0000","last_change":"2024-03-11T22:21:51.817381+0000","last_active":"2024-03-11T22:25:48.670385+0000","last_peered":"2024-03-11T22:25:48.670385+0000","last_clean":"2024-03-11T22:25:48.670385+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:25:48.670385+0000","last_undegraded":"2024-03-11T22:25:48.670385+0000","last_fullsized":"2024-03-11T22:25:48.670385+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:21:51.817265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:21:51.817265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602356+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":682,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.684101+0000","last_change":"2024-03-11T22:23:50.723665+0000","last_active":"2024-03-11T22:25:48.684101+0000","last_peered":"2024-03-11T22:25:48.684101+0000","last_clean":"2024-03-11T22:25:48.684101+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:25:48.684101+0000","last_undegraded":"2024-03-11T22:25:48.684101+0000","last_fullsized":"2024-03-11T22:25:48.684101+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:50.723528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:50.723528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.967080+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":735,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670567+0000","last_change":"2024-03-11T22:21:58.841578+0000","last_active":"2024-03-11T22:25:48.670567+0000","last_peered":"2024-03-11T22:25:48.670567+0000","last_clean":"2024-03-11T22:25:48.670567+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:25:48.670567+0000","last_undegraded":"2024-03-11T22:25:48.670567+0000","last_fullsized":"2024-03-11T22:25:48.670567+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:21:58.841451+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:21:58.841451+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602327+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":645,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.684177+0000","last_change":"2024-03-11T22:24:41.612477+0000","last_active":"2024-03-11T22:25:48.684177+0000","last_peered":"2024-03-11T22:25:48.684177+0000","last_clean":"2024-03-11T22:25:48.684177+0000","last_became_active":"2024-03-11T22:20:29.422747+0000","last_became_peered":"2024-03-11T22:20:29.422747+0000","last_unstale":"2024-03-11T22:25:48.684177+0000","last_undegraded":"2024-03-11T22:25:48.684177+0000","last_fullsized":"2024-03-11T22:25:48.684177+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:41.612363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.612363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:56.508483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":694,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.682576+0000","last_change":"2024-03-11T22:24:49.679683+0000","last_active":"2024-03-11T22:25:48.682576+0000","last_peered":"2024-03-11T22:25:48.682576+0000","last_clean":"2024-03-11T22:25:48.682576+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:25:48.682576+0000","last_undegraded":"2024-03-11T22:25:48.682576+0000","last_fullsized":"2024-03-11T22:25:48.682576+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:49.679560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:24:49.679560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:13.908754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":686,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.617638+0000","last_change":"2024-03-11T22:23:09.058314+0000","last_active":"2024-03-11T22:25:43.617638+0000","last_peered":"2024-03-11T22:25:43.617638+0000","last_clean":"2024-03-11T22:25:43.617638+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:25:43.617638+0000","last_undegraded":"2024-03-11T22:25:43.617638+0000","last_fullsized":"2024-03-11T22:25:43.617638+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:09.057970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:09.057970+0000","last_clean_scrub_stamp":"2024-03-11T22:23:09.057970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.927674+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"93'130","reported_seq":1018,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.671298+0000","last_change":"2024-03-11T22:23:41.379256+0000","last_active":"2024-03-11T22:25:48.671298+0000","last_peered":"2024-03-11T22:25:48.671298+0000","last_clean":"2024-03-11T22:25:48.671298+0000","last_became_active":"2024-03-11T22:14:43.194955+0000","last_became_peered":"2024-03-11T22:14:43.194955+0000","last_unstale":"2024-03-11T22:25:48.671298+0000","last_undegraded":"2024-03-11T22:25:48.671298+0000","last_fullsized":"2024-03-11T22:25:48.671298+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":168,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:23:41.379110+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:18:43.722467+0000","last_clean_scrub_stamp":"2024-03-11T22:23:41.379110+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:49.910085+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"274'20","reported_seq":758,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.684896+0000","last_change":"2024-03-11T22:23:52.653463+0000","last_active":"2024-03-11T22:25:48.684896+0000","last_peered":"2024-03-11T22:25:48.684896+0000","last_clean":"2024-03-11T22:25:48.684896+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:25:48.684896+0000","last_undegraded":"2024-03-11T22:25:48.684896+0000","last_fullsized":"2024-03-11T22:25:48.684896+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'20","last_scrub_stamp":"2024-03-11T22:23:52.653309+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:23:52.653309+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.967114+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":722,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.674814+0000","last_change":"2024-03-11T22:24:41.895990+0000","last_active":"2024-03-11T22:25:48.674814+0000","last_peered":"2024-03-11T22:25:48.674814+0000","last_clean":"2024-03-11T22:25:48.674814+0000","last_became_active":"2024-03-11T22:20:29.416792+0000","last_became_peered":"2024-03-11T22:20:29.416792+0000","last_unstale":"2024-03-11T22:25:48.674814+0000","last_undegraded":"2024-03-11T22:25:48.674814+0000","last_fullsized":"2024-03-11T22:25:48.674814+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:41.895742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.895742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:51.086325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":479,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.669578+0000","last_change":"2024-03-11T22:23:33.856940+0000","last_active":"2024-03-11T22:25:48.669578+0000","last_peered":"2024-03-11T22:25:48.669578+0000","last_clean":"2024-03-11T22:25:48.669578+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:25:48.669578+0000","last_undegraded":"2024-03-11T22:25:48.669578+0000","last_fullsized":"2024-03-11T22:25:48.669578+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:33.856676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:23:33.856676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":690,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.682829+0000","last_change":"2024-03-11T22:24:59.829935+0000","last_active":"2024-03-11T22:25:48.682829+0000","last_peered":"2024-03-11T22:25:48.682829+0000","last_clean":"2024-03-11T22:25:48.682829+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:25:48.682829+0000","last_undegraded":"2024-03-11T22:25:48.682829+0000","last_fullsized":"2024-03-11T22:25:48.682829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:59.829811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:59.829811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:17.495056+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"273'8","reported_seq":722,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.682015+0000","last_change":"2024-03-11T22:23:59.693653+0000","last_active":"2024-03-11T22:25:48.682015+0000","last_peered":"2024-03-11T22:25:48.682015+0000","last_clean":"2024-03-11T22:25:48.682015+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:25:48.682015+0000","last_undegraded":"2024-03-11T22:25:48.682015+0000","last_fullsized":"2024-03-11T22:25:48.682015+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'8","last_scrub_stamp":"2024-03-11T22:23:59.693510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:59.693510+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.967049+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":698,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.629763+0000","last_change":"2024-03-11T22:23:12.019597+0000","last_active":"2024-03-11T22:25:43.629763+0000","last_peered":"2024-03-11T22:25:43.629763+0000","last_clean":"2024-03-11T22:25:43.629763+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:25:43.629763+0000","last_undegraded":"2024-03-11T22:25:43.629763+0000","last_fullsized":"2024-03-11T22:25:43.629763+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:12.019468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:12.019468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934484+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":296,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.669554+0000","last_change":"2024-03-11T22:24:37.763829+0000","last_active":"2024-03-11T22:25:48.669554+0000","last_peered":"2024-03-11T22:25:48.669554+0000","last_clean":"2024-03-11T22:25:48.669554+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:25:48.669554+0000","last_undegraded":"2024-03-11T22:25:48.669554+0000","last_fullsized":"2024-03-11T22:25:48.669554+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.763709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.763709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.935899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":664,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.672091+0000","last_change":"2024-03-11T22:24:46.776804+0000","last_active":"2024-03-11T22:25:48.672091+0000","last_peered":"2024-03-11T22:25:48.672091+0000","last_clean":"2024-03-11T22:25:48.672091+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:25:48.672091+0000","last_undegraded":"2024-03-11T22:25:48.672091+0000","last_fullsized":"2024-03-11T22:25:48.672091+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:46.776600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.776600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:12.814680+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"273'19","reported_seq":746,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.671093+0000","last_change":"2024-03-11T22:23:42.850339+0000","last_active":"2024-03-11T22:25:48.671093+0000","last_peered":"2024-03-11T22:25:48.671093+0000","last_clean":"2024-03-11T22:25:48.671093+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:25:48.671093+0000","last_undegraded":"2024-03-11T22:25:48.671093+0000","last_fullsized":"2024-03-11T22:25:48.671093+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:42.850214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:42.850214+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":702,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.681968+0000","last_change":"2024-03-11T22:24:38.611323+0000","last_active":"2024-03-11T22:25:48.681968+0000","last_peered":"2024-03-11T22:25:48.681968+0000","last_clean":"2024-03-11T22:25:48.681968+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:25:48.681968+0000","last_undegraded":"2024-03-11T22:25:48.681968+0000","last_fullsized":"2024-03-11T22:25:48.681968+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:38.610924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:38.610924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:03.792615+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":699,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.631367+0000","last_change":"2024-03-11T22:23:46.048359+0000","last_active":"2024-03-11T22:25:43.631367+0000","last_peered":"2024-03-11T22:25:43.631367+0000","last_clean":"2024-03-11T22:25:43.631367+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:25:43.631367+0000","last_undegraded":"2024-03-11T22:25:43.631367+0000","last_fullsized":"2024-03-11T22:25:43.631367+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:46.048254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.048254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934609+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":705,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.668424+0000","last_change":"2024-03-11T22:25:05.513523+0000","last_active":"2024-03-11T22:25:48.668424+0000","last_peered":"2024-03-11T22:25:48.668424+0000","last_clean":"2024-03-11T22:25:48.668424+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:25:48.668424+0000","last_undegraded":"2024-03-11T22:25:48.668424+0000","last_fullsized":"2024-03-11T22:25:48.668424+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:05.513210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.513210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:06.854659+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"273'20","reported_seq":761,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.631125+0000","last_change":"2024-03-11T22:22:57.045033+0000","last_active":"2024-03-11T22:25:43.631125+0000","last_peered":"2024-03-11T22:25:43.631125+0000","last_clean":"2024-03-11T22:25:43.631125+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:25:43.631125+0000","last_undegraded":"2024-03-11T22:25:43.631125+0000","last_fullsized":"2024-03-11T22:25:43.631125+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:22:57.044931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:22:57.044931+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.927583+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":712,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.669588+0000","last_change":"2024-03-11T22:22:06.879148+0000","last_active":"2024-03-11T22:25:48.669588+0000","last_peered":"2024-03-11T22:25:48.669588+0000","last_clean":"2024-03-11T22:25:48.669588+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:25:48.669588+0000","last_undegraded":"2024-03-11T22:25:48.669588+0000","last_fullsized":"2024-03-11T22:25:48.669588+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:22:06.879026+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:22:06.879026+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602389+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'11","reported_seq":960,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670982+0000","last_change":"2024-03-11T22:23:50.837614+0000","last_active":"2024-03-11T22:25:48.670982+0000","last_peered":"2024-03-11T22:25:48.670982+0000","last_clean":"2024-03-11T22:25:48.670982+0000","last_became_active":"2024-03-11T22:20:29.415781+0000","last_became_peered":"2024-03-11T22:20:29.415781+0000","last_unstale":"2024-03-11T22:25:48.670982+0000","last_undegraded":"2024-03-11T22:25:48.670982+0000","last_fullsized":"2024-03-11T22:25:48.670982+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-11T22:23:50.837494+0000","last_deep_scrub":"315'11","last_deep_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_clean_scrub_stamp":"2024-03-11T22:23:50.837494+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":651,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.674241+0000","last_change":"2024-03-11T22:23:51.821013+0000","last_active":"2024-03-11T22:25:48.674241+0000","last_peered":"2024-03-11T22:25:48.674241+0000","last_clean":"2024-03-11T22:25:48.674241+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:25:48.674241+0000","last_undegraded":"2024-03-11T22:25:48.674241+0000","last_fullsized":"2024-03-11T22:25:48.674241+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:23:51.820865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.820865+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"273'18","reported_seq":729,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.684452+0000","last_change":"2024-03-11T22:25:01.629462+0000","last_active":"2024-03-11T22:25:48.684452+0000","last_peered":"2024-03-11T22:25:48.684452+0000","last_clean":"2024-03-11T22:25:48.684452+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:25:48.684452+0000","last_undegraded":"2024-03-11T22:25:48.684452+0000","last_fullsized":"2024-03-11T22:25:48.684452+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:25:01.629321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:01.629321+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:29.544545+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":477,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670246+0000","last_change":"2024-03-11T22:25:04.394810+0000","last_active":"2024-03-11T22:25:48.670246+0000","last_peered":"2024-03-11T22:25:48.670246+0000","last_clean":"2024-03-11T22:25:48.670246+0000","last_became_active":"2024-03-11T22:14:52.245199+0000","last_became_peered":"2024-03-11T22:14:52.245199+0000","last_unstale":"2024-03-11T22:25:48.670246+0000","last_undegraded":"2024-03-11T22:25:48.670246+0000","last_fullsized":"2024-03-11T22:25:48.670246+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:04.394612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:25:04.394612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:30.314020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":460,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.685637+0000","last_change":"2024-03-11T22:25:05.656524+0000","last_active":"2024-03-11T22:25:48.685637+0000","last_peered":"2024-03-11T22:25:48.685637+0000","last_clean":"2024-03-11T22:25:48.685637+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:25:48.685637+0000","last_undegraded":"2024-03-11T22:25:48.685637+0000","last_fullsized":"2024-03-11T22:25:48.685637+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:05.656208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.656208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:14.267362+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":683,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.694006+0000","last_change":"2024-03-11T22:23:12.914118+0000","last_active":"2024-03-11T22:25:48.694006+0000","last_peered":"2024-03-11T22:25:48.694006+0000","last_clean":"2024-03-11T22:25:48.694006+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:25:48.694006+0000","last_undegraded":"2024-03-11T22:25:48.694006+0000","last_fullsized":"2024-03-11T22:25:48.694006+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:12.914007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:12.914007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966817+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"273'27","reported_seq":772,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.671483+0000","last_change":"2024-03-11T22:23:46.333974+0000","last_active":"2024-03-11T22:25:48.671483+0000","last_peered":"2024-03-11T22:25:48.671483+0000","last_clean":"2024-03-11T22:25:48.671483+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:25:48.671483+0000","last_undegraded":"2024-03-11T22:25:48.671483+0000","last_fullsized":"2024-03-11T22:25:48.671483+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:23:46.333799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.333799+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:49.910233+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":717,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.617905+0000","last_change":"2024-03-11T22:24:55.760207+0000","last_active":"2024-03-11T22:25:43.617905+0000","last_peered":"2024-03-11T22:25:43.617905+0000","last_clean":"2024-03-11T22:25:43.617905+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:25:43.617905+0000","last_undegraded":"2024-03-11T22:25:43.617905+0000","last_fullsized":"2024-03-11T22:25:43.617905+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:55.760084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:24:55.760084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:21.111674+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":709,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.668276+0000","last_change":"2024-03-11T22:24:37.481790+0000","last_active":"2024-03-11T22:25:48.668276+0000","last_peered":"2024-03-11T22:25:48.668276+0000","last_clean":"2024-03-11T22:25:48.668276+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:25:48.668276+0000","last_undegraded":"2024-03-11T22:25:48.668276+0000","last_fullsized":"2024-03-11T22:25:48.668276+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.481668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.481668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:03.122817+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":477,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670849+0000","last_change":"2024-03-11T22:23:07.097233+0000","last_active":"2024-03-11T22:25:48.670849+0000","last_peered":"2024-03-11T22:25:48.670849+0000","last_clean":"2024-03-11T22:25:48.670849+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:25:48.670849+0000","last_undegraded":"2024-03-11T22:25:48.670849+0000","last_fullsized":"2024-03-11T22:25:48.670849+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:07.097049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:07.097049+0000","last_clean_scrub_stamp":"2024-03-11T22:23:07.097049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"273'9","reported_seq":537,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.682409+0000","last_change":"2024-03-11T22:23:58.648966+0000","last_active":"2024-03-11T22:25:48.682409+0000","last_peered":"2024-03-11T22:25:48.682409+0000","last_clean":"2024-03-11T22:25:48.682409+0000","last_became_active":"2024-03-11T22:14:52.238411+0000","last_became_peered":"2024-03-11T22:14:52.238411+0000","last_unstale":"2024-03-11T22:25:48.682409+0000","last_undegraded":"2024-03-11T22:25:48.682409+0000","last_fullsized":"2024-03-11T22:25:48.682409+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9","last_scrub_stamp":"2024-03-11T22:23:58.648385+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.648385+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.967190+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":709,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.674371+0000","last_change":"2024-03-11T22:24:40.905435+0000","last_active":"2024-03-11T22:25:48.674371+0000","last_peered":"2024-03-11T22:25:48.674371+0000","last_clean":"2024-03-11T22:25:48.674371+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:25:48.674371+0000","last_undegraded":"2024-03-11T22:25:48.674371+0000","last_fullsized":"2024-03-11T22:25:48.674371+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:40.905314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:24:40.905314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":471,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.670047+0000","last_change":"2024-03-11T22:22:03.796143+0000","last_active":"2024-03-11T22:25:48.670047+0000","last_peered":"2024-03-11T22:25:48.670047+0000","last_clean":"2024-03-11T22:25:48.670047+0000","last_became_active":"2024-03-11T22:14:52.239391+0000","last_became_peered":"2024-03-11T22:14:52.239391+0000","last_unstale":"2024-03-11T22:25:48.670047+0000","last_undegraded":"2024-03-11T22:25:48.670047+0000","last_fullsized":"2024-03-11T22:25:48.670047+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:03.796021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:28.434334+0000","last_clean_scrub_stamp":"2024-03-11T22:22:03.796021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.602452+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":690,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.673629+0000","last_change":"2024-03-11T22:22:59.182771+0000","last_active":"2024-03-11T22:25:48.673629+0000","last_peered":"2024-03-11T22:25:48.673629+0000","last_clean":"2024-03-11T22:25:48.673629+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:25:48.673629+0000","last_undegraded":"2024-03-11T22:25:48.673629+0000","last_fullsized":"2024-03-11T22:25:48.673629+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:59.182621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:22:59.182621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"273'15","reported_seq":512,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.682400+0000","last_change":"2024-03-11T22:25:03.685762+0000","last_active":"2024-03-11T22:25:48.682400+0000","last_peered":"2024-03-11T22:25:48.682400+0000","last_clean":"2024-03-11T22:25:48.682400+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:25:48.682400+0000","last_undegraded":"2024-03-11T22:25:48.682400+0000","last_fullsized":"2024-03-11T22:25:48.682400+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:25:03.685629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:03.685629+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:21.379436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":709,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.674904+0000","last_change":"2024-03-11T22:23:58.700286+0000","last_active":"2024-03-11T22:25:48.674904+0000","last_peered":"2024-03-11T22:25:48.674904+0000","last_clean":"2024-03-11T22:25:48.674904+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:25:48.674904+0000","last_undegraded":"2024-03-11T22:25:48.674904+0000","last_fullsized":"2024-03-11T22:25:48.674904+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:58.700176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.700176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":713,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.673033+0000","last_change":"2024-03-11T22:24:49.941039+0000","last_active":"2024-03-11T22:25:48.673033+0000","last_peered":"2024-03-11T22:25:48.673033+0000","last_clean":"2024-03-11T22:25:48.673033+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:25:48.673033+0000","last_undegraded":"2024-03-11T22:25:48.673033+0000","last_fullsized":"2024-03-11T22:25:48.673033+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:49.940903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:24:49.940903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:52.554477+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":686,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.630339+0000","last_change":"2024-03-11T22:23:40.218666+0000","last_active":"2024-03-11T22:25:43.630339+0000","last_peered":"2024-03-11T22:25:43.630339+0000","last_clean":"2024-03-11T22:25:43.630339+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:25:43.630339+0000","last_undegraded":"2024-03-11T22:25:43.630339+0000","last_fullsized":"2024-03-11T22:25:43.630339+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:40.218559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:40.218559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934579+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"273'27","reported_seq":733,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.684991+0000","last_change":"2024-03-11T22:23:47.683228+0000","last_active":"2024-03-11T22:25:48.684991+0000","last_peered":"2024-03-11T22:25:48.684991+0000","last_clean":"2024-03-11T22:25:48.684991+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:25:48.684991+0000","last_undegraded":"2024-03-11T22:25:48.684991+0000","last_fullsized":"2024-03-11T22:25:48.684991+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:23:47.683120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:47.683120+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966885+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":709,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.683597+0000","last_change":"2024-03-11T22:23:41.703918+0000","last_active":"2024-03-11T22:25:48.683597+0000","last_peered":"2024-03-11T22:25:48.683597+0000","last_clean":"2024-03-11T22:25:48.683597+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:25:48.683597+0000","last_undegraded":"2024-03-11T22:25:48.683597+0000","last_fullsized":"2024-03-11T22:25:48.683597+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:23:41.703753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:41.703753+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966916+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":479,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.669743+0000","last_change":"2024-03-11T22:22:58.001733+0000","last_active":"2024-03-11T22:25:48.669743+0000","last_peered":"2024-03-11T22:25:48.669743+0000","last_clean":"2024-03-11T22:25:48.669743+0000","last_became_active":"2024-03-11T22:14:52.247689+0000","last_became_peered":"2024-03-11T22:14:52.247689+0000","last_unstale":"2024-03-11T22:25:48.669743+0000","last_undegraded":"2024-03-11T22:25:48.669743+0000","last_fullsized":"2024-03-11T22:25:48.669743+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:22:58.001650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:22:58.001650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":693,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.682655+0000","last_change":"2024-03-11T22:23:08.833422+0000","last_active":"2024-03-11T22:25:48.682655+0000","last_peered":"2024-03-11T22:25:48.682655+0000","last_clean":"2024-03-11T22:25:48.682655+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:25:48.682655+0000","last_undegraded":"2024-03-11T22:25:48.682655+0000","last_fullsized":"2024-03-11T22:25:48.682655+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:08.833328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:08.833328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:50.966760+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"273'19","reported_seq":524,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.669885+0000","last_change":"2024-03-11T22:24:39.755038+0000","last_active":"2024-03-11T22:25:48.669885+0000","last_peered":"2024-03-11T22:25:48.669885+0000","last_clean":"2024-03-11T22:25:48.669885+0000","last_became_active":"2024-03-11T22:14:52.247411+0000","last_became_peered":"2024-03-11T22:14:52.247411+0000","last_unstale":"2024-03-11T22:25:48.669885+0000","last_undegraded":"2024-03-11T22:25:48.669885+0000","last_fullsized":"2024-03-11T22:25:48.669885+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:24:39.754894+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.754894+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":707,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-11T22:25:43.617865+0000","last_change":"2024-03-11T22:24:00.015418+0000","last_active":"2024-03-11T22:25:43.617865+0000","last_peered":"2024-03-11T22:25:43.617865+0000","last_clean":"2024-03-11T22:25:43.617865+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:25:43.617865+0000","last_undegraded":"2024-03-11T22:25:43.617865+0000","last_fullsized":"2024-03-11T22:25:43.617865+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:00.015301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.015301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:25:46.934797+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":424,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.669580+0000","last_change":"2024-03-11T22:23:29.877294+0000","last_active":"2024-03-11T22:25:48.669580+0000","last_peered":"2024-03-11T22:25:48.669580+0000","last_clean":"2024-03-11T22:25:48.669580+0000","last_became_active":"2024-03-11T22:20:29.418227+0000","last_became_peered":"2024-03-11T22:20:29.418227+0000","last_unstale":"2024-03-11T22:25:48.669580+0000","last_undegraded":"2024-03-11T22:25:48.669580+0000","last_fullsized":"2024-03-11T22:25:48.669580+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:29.877172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:29.877172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":468,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-11T22:25:48.673751+0000","last_change":"2024-03-11T22:23:36.916038+0000","last_active":"2024-03-11T22:25:48.673751+0000","last_peered":"2024-03-11T22:25:48.673751+0000","last_clean":"2024-03-11T22:25:48.673751+0000","last_became_active":"2024-03-11T22:14:52.247273+0000","last_became_peered":"2024-03-11T22:14:52.247273+0000","last_unstale":"2024-03-11T22:25:48.673751+0000","last_undegraded":"2024-03-11T22:25:48.673751+0000","last_fullsized":"2024-03-11T22:25:48.673751+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:36.915904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:36.915904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2680815640,"num_objects":40907,"num_object_clones":0,"num_object_copies":122721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40907,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":223143,"num_write_kb":8449537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8008642560,"data_stored":8008630344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81583,"ondisk_log_size":81583,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1003,"num_read_kb":998,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":1430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005268,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1269564,"kb_used_data":656692,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92475588,"statfs":{"total":95995035648,"available":94695002112,"internally_reserved":0,"allocated":672452608,"data_stored":672119166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":627571147},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332313,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1654512,"kb_used_data":1149860,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92090640,"statfs":{"total":95995035648,"available":94300815360,"internally_reserved":0,"allocated":1177456640,"data_stored":1177139416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":516684230},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575091,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019288,"kb_used_data":468880,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":92725864,"statfs":{"total":95995035648,"available":94951284736,"internally_reserved":0,"allocated":480133120,"data_stored":479771012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738616,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2431588,"kb_used_data":1782556,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91313564,"statfs":{"total":95995035648,"available":93505089536,"internally_reserved":0,"allocated":1825337344,"data_stored":1825011458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411566,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1442472,"kb_used_data":964192,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92302680,"statfs":{"total":95995035648,"available":94517944320,"internally_reserved":0,"allocated":987332608,"data_stored":987019826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149928,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1402068,"kb_used_data":797384,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92343084,"statfs":{"total":95995035648,"available":94559318016,"internally_reserved":0,"allocated":816521216,"data_stored":816168154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":619051463},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920996,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1827596,"kb_used_data":1309116,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":91917556,"statfs":{"total":95995035648,"available":94123577344,"internally_reserved":0,"allocated":1340534784,"data_stored":1340192715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":530905542},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659357,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1447096,"kb_used_data":812008,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92298056,"statfs":{"total":95995035648,"available":94513209344,"internally_reserved":0,"allocated":831496192,"data_stored":831174665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":650312134},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1784025088,"data_stored":1784021016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":478019584,"data_stored":478019584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":982253568,"data_stored":982253568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":801243136,"data_stored":801243136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1324552192,"data_stored":1324548120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":817758208,"data_stored":817758208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1151406080,"data_stored":1151402008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":669384704,"data_stored":669384704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:25:55.260 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-03-11T22:25:55.260 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-11T22:25:55.283 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:55.318 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:55.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:55.420 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:55.452 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:55.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:55.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:55.554 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:55.598 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:55.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:55.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:55.700 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:55.735 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:55.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:55.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:55.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:55.878 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:55.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:55.980 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:56.016 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:56.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:56.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:55 smithi045 ceph-mon[30178]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:25:56.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:55 smithi045 ceph-mon[30178]: 10.6 scrub starts 2024-03-11T22:25:56.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:55 smithi045 ceph-mon[30178]: pgmap v807: 168 pgs: 168 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 969 op/s 2024-03-11T22:25:56.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:55 smithi045 ceph-mon[30178]: 10.6 scrub ok 2024-03-11T22:25:56.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:55 smithi045 ceph-mon[30178]: 5.18 scrub starts 2024-03-11T22:25:56.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:55 smithi045 ceph-mon[30178]: 5.18 scrub ok 2024-03-11T22:25:56.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:55 smithi045 ceph-mon[39715]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:25:56.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:55 smithi045 ceph-mon[39715]: 10.6 scrub starts 2024-03-11T22:25:56.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:55 smithi045 ceph-mon[39715]: pgmap v807: 168 pgs: 168 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 969 op/s 2024-03-11T22:25:56.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:55 smithi045 ceph-mon[39715]: 10.6 scrub ok 2024-03-11T22:25:56.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:55 smithi045 ceph-mon[39715]: 5.18 scrub starts 2024-03-11T22:25:56.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:55 smithi045 ceph-mon[39715]: 5.18 scrub ok 2024-03-11T22:25:56.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:55 smithi144 ceph-mon[33984]: from='client.25529 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:25:56.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:55 smithi144 ceph-mon[33984]: 10.6 scrub starts 2024-03-11T22:25:56.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:55 smithi144 ceph-mon[33984]: pgmap v807: 168 pgs: 168 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 969 op/s 2024-03-11T22:25:56.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:55 smithi144 ceph-mon[33984]: 10.6 scrub ok 2024-03-11T22:25:56.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:55 smithi144 ceph-mon[33984]: 5.18 scrub starts 2024-03-11T22:25:56.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:55 smithi144 ceph-mon[33984]: 5.18 scrub ok 2024-03-11T22:25:56.117 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:56.158 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:56.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:56.260 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:56.295 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:56.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:56.396 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:56.430 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:56.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:56.531 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:56.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:56.576 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:56.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:56.677 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:56.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:56.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:56.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:56.855 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:56.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:56.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:56.939 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:25:56.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:57.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:57.033 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:57.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:57.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:56 smithi045 ceph-mon[30178]: 5.12 scrub starts 2024-03-11T22:25:57.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:56 smithi045 ceph-mon[30178]: 5.12 scrub ok 2024-03-11T22:25:57.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:56 smithi045 ceph-mon[30178]: 10.b scrub starts 2024-03-11T22:25:57.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:56 smithi045 ceph-mon[39715]: 5.12 scrub starts 2024-03-11T22:25:57.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:56 smithi045 ceph-mon[39715]: 5.12 scrub ok 2024-03-11T22:25:57.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:56 smithi045 ceph-mon[39715]: 10.b scrub starts 2024-03-11T22:25:57.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:56 smithi144 ceph-mon[33984]: 5.12 scrub starts 2024-03-11T22:25:57.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:56 smithi144 ceph-mon[33984]: 5.12 scrub ok 2024-03-11T22:25:57.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:56 smithi144 ceph-mon[33984]: 10.b scrub starts 2024-03-11T22:25:57.099 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:25:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:25:57.134 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:57.181 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:57.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:57.282 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:57.336 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:57.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:57.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:57.438 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:57.469 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:57.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:57.571 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:57.604 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:57.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:57.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:57.706 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:57.739 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:57.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:57.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:57.841 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:57.872 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:57.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:57.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:57.974 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:58.005 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:58.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:58.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:57 smithi045 ceph-mon[39715]: 4.14 scrub starts 2024-03-11T22:25:58.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:57 smithi045 ceph-mon[39715]: 4.14 scrub ok 2024-03-11T22:25:58.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:57 smithi045 ceph-mon[39715]: pgmap v808: 168 pgs: 168 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 582 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-11T22:25:58.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:57 smithi045 ceph-mon[39715]: 10.b scrub ok 2024-03-11T22:25:58.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:58.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:57 smithi045 ceph-mon[30178]: 4.14 scrub starts 2024-03-11T22:25:58.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:57 smithi045 ceph-mon[30178]: 4.14 scrub ok 2024-03-11T22:25:58.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:57 smithi045 ceph-mon[30178]: pgmap v808: 168 pgs: 168 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 582 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-11T22:25:58.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:57 smithi045 ceph-mon[30178]: 10.b scrub ok 2024-03-11T22:25:58.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:58.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:57 smithi144 ceph-mon[33984]: 4.14 scrub starts 2024-03-11T22:25:58.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:57 smithi144 ceph-mon[33984]: 4.14 scrub ok 2024-03-11T22:25:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:57 smithi144 ceph-mon[33984]: pgmap v808: 168 pgs: 168 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 582 B/s rd, 0 B/s wr, 1.29k op/s 2024-03-11T22:25:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:57 smithi144 ceph-mon[33984]: 10.b scrub ok 2024-03-11T22:25:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:25:58.107 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:58.139 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:58.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:58.241 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:25:58.277 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:58.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:25:58.379 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:25:58.412 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:58.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:58.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:25:58.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:58.554 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:58.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:58.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:58.656 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:58.688 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:58.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:58.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:58.769 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:25:58.790 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:58.822 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:58.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:58.924 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:58.959 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:58.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:59.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:58 smithi045 ceph-mon[39715]: 1.7 scrub starts 2024-03-11T22:25:59.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:58 smithi045 ceph-mon[39715]: 1.7 scrub ok 2024-03-11T22:25:59.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:58 smithi045 ceph-mon[39715]: 6.17 scrub starts 2024-03-11T22:25:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:58 smithi045 ceph-mon[39715]: 6.17 scrub ok 2024-03-11T22:25:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:58 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1009365049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:25:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:58 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:25:59.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:58 smithi045 ceph-mon[30178]: 1.7 scrub starts 2024-03-11T22:25:59.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:58 smithi045 ceph-mon[30178]: 1.7 scrub ok 2024-03-11T22:25:59.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:58 smithi045 ceph-mon[30178]: 6.17 scrub starts 2024-03-11T22:25:59.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:58 smithi045 ceph-mon[30178]: 6.17 scrub ok 2024-03-11T22:25:59.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:58 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1009365049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:25:59.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:58 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:25:59.062 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:25:59.095 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:59.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:25:59.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:58 smithi144 ceph-mon[33984]: 1.7 scrub starts 2024-03-11T22:25:59.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:58 smithi144 ceph-mon[33984]: 1.7 scrub ok 2024-03-11T22:25:59.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:58 smithi144 ceph-mon[33984]: 6.17 scrub starts 2024-03-11T22:25:59.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:58 smithi144 ceph-mon[33984]: 6.17 scrub ok 2024-03-11T22:25:59.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:58 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1009365049' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:25:59.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:58 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:25:59.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:25:59.243 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:59.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:59.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:25:59.344 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:59.378 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:59.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:59.479 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:25:59.517 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:59.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:59.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:25:59.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:59.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:59.658 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:59.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:25:59.759 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:25:59.788 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:25:59.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:59.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:25:59.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:25:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:25:59.926 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:25:59.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:00.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:00.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[39715]: 6.19 scrub starts 2024-03-11T22:26:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[39715]: 6.6 scrub starts 2024-03-11T22:26:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[39715]: 6.6 scrub ok 2024-03-11T22:26:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[39715]: 6.19 scrub ok 2024-03-11T22:26:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[39715]: pgmap v809: 168 pgs: 168 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-11T22:26:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[39715]: 10.c scrub starts 2024-03-11T22:26:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:26:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:26:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:26:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[39715]: osdmap e391: 8 total, 8 up, 8 in 2024-03-11T22:26:00.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[30178]: 6.19 scrub starts 2024-03-11T22:26:00.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[30178]: 6.6 scrub starts 2024-03-11T22:26:00.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[30178]: 6.6 scrub ok 2024-03-11T22:26:00.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[30178]: 6.19 scrub ok 2024-03-11T22:26:00.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[30178]: pgmap v809: 168 pgs: 168 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-11T22:26:00.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[30178]: 10.c scrub starts 2024-03-11T22:26:00.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:26:00.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:26:00.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:26:00.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:25:59 smithi045 ceph-mon[30178]: osdmap e391: 8 total, 8 up, 8 in 2024-03-11T22:26:00.071 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:00.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:59 smithi144 ceph-mon[33984]: 6.19 scrub starts 2024-03-11T22:26:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:59 smithi144 ceph-mon[33984]: 6.6 scrub starts 2024-03-11T22:26:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:59 smithi144 ceph-mon[33984]: 6.6 scrub ok 2024-03-11T22:26:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:59 smithi144 ceph-mon[33984]: 6.19 scrub ok 2024-03-11T22:26:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:59 smithi144 ceph-mon[33984]: pgmap v809: 168 pgs: 168 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-11T22:26:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:59 smithi144 ceph-mon[33984]: 10.c scrub starts 2024-03-11T22:26:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:59 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:26:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:59 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:26:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:59 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:26:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:25:59 smithi144 ceph-mon[33984]: osdmap e391: 8 total, 8 up, 8 in 2024-03-11T22:26:00.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:00.210 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:00.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:00.312 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:00.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:00.347 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:00.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:00.416 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:00.417 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:00.444 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:26:00.444 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-03-11T22:26:00.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-03-11T22:26:00.444 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:26:00.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.444 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25142.0:27626 9.7 e282/273 9:e5cd1be8:::benchmark_data_smithi045_141553_object27625:head v 282'1698, mlcod=282'1696)", 2024-03-11T22:26:00.444 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:19:05.235532+0000", 2024-03-11T22:26:00.445 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 415.19982445699998, 2024-03-11T22:26:00.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.205663929, 2024-03-11T22:26:00.445 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.445 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.445 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235532+0000", 2024-03-11T22:26:00.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235532+0000", 2024-03-11T22:26:00.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.446 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.446 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235533+0000", 2024-03-11T22:26:00.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5000000000000002e-07 2024-03-11T22:26:00.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.446 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.446 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235574+0000", 2024-03-11T22:26:00.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1770999999999997e-05 2024-03-11T22:26:00.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235585+0000", 2024-03-11T22:26:00.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0305999999999999e-05 2024-03-11T22:26:00.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235592+0000", 2024-03-11T22:26:00.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3839999999999999e-06 2024-03-11T22:26:00.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235790+0000", 2024-03-11T22:26:00.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019756999999999999 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235810+0000", 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9998000000000001e-05 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.441154+0000", 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20534397900000001 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.441196+0000", 2024-03-11T22:26:00.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2271000000000002e-05 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:27631 9.c 9:3254e176:::benchmark_data_smithi045_141553_object27630:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:19:05.235886+0000", 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 415.19947077099999, 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.2058566, 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.449 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27631 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235886+0000", 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235886+0000", 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235887+0000", 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3030000000000001e-06 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235909+0000", 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2572e-05 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235921+0000", 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.148e-05 2024-03-11T22:26:00.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235929+0000", 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1610000000000003e-06 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235985+0000", 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5853000000000002e-05 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.236136+0000", 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000150848 2024-03-11T22:26:00.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,7", 2024-03-11T22:26:00.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.236284+0000", 2024-03-11T22:26:00.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014817300000000001 2024-03-11T22:26:00.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.441320+0000", 2024-03-11T22:26:00.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20503602200000001 2024-03-11T22:26:00.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.441338+0000", 2024-03-11T22:26:00.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7876999999999998e-05 2024-03-11T22:26:00.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.454 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.441604+0000", 2024-03-11T22:26:00.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00026633400000000002 2024-03-11T22:26:00.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.455 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.441690+0000", 2024-03-11T22:26:00.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5502999999999998e-05 2024-03-11T22:26:00.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.455 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.455 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.441742+0000", 2024-03-11T22:26:00.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2473999999999999e-05 2024-03-11T22:26:00.455 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.455 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.455 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25142.0:27630 9.f e282/273 9:f7a6d69f:::benchmark_data_smithi045_141553_object27629:head v 282'1725, mlcod=282'1723)", 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:19:05.237455+0000", 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 415.19790183999999, 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20679778900000001, 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237455+0000", 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237455+0000", 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237455+0000", 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4899999999999995e-07 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237490+0000", 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4353000000000001e-05 2024-03-11T22:26:00.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.441363+0000", 2024-03-11T22:26:00.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20387355099999999 2024-03-11T22:26:00.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.441370+0000", 2024-03-11T22:26:00.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9519999999999997e-06 2024-03-11T22:26:00.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.442038+0000", 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00066784799999999997 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.442058+0000", 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0001999999999999e-05 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.444222+0000", 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021643270000000002 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.444252+0000", 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0107000000000002e-05 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop_reply(client.25142.0:27631 9.c e282/273 ondisk, result = 0)", 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:19:05.238321+0000", 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 415.19703562699999, 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20301830700000001, 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238321+0000", 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238321+0000", 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238322+0000", 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8700000000000005e-07 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.461 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.462 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.462 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238326+0000", 2024-03-11T22:26:00.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9230000000000001e-06 2024-03-11T22:26:00.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.462 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.462 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.462 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.440515+0000", 2024-03-11T22:26:00.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20218873200000001 2024-03-11T22:26:00.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.462 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.462 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.462 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.440533+0000", 2024-03-11T22:26:00.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7496e-05 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.441305+0000", 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00077241099999999997 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.441313+0000", 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2989999999999997e-06 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.441339+0000", 2024-03-11T22:26:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5658999999999999e-05 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:146716 9.1 9:88bee207:::benchmark_data_smithi045_141553_object22596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:46.127935+0000", 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.30742169600001, 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.28884823700000001, 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 146716 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.127935+0000", 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.127935+0000", 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.127935+0000", 2024-03-11T22:26:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0299999999999998e-07 2024-03-11T22:26:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.127938+0000", 2024-03-11T22:26:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6469999999999998e-06 2024-03-11T22:26:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.127949+0000", 2024-03-11T22:26:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0709e-05 2024-03-11T22:26:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.127955+0000", 2024-03-11T22:26:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4180000000000002e-06 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.128371+0000", 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00041588900000000002 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.128488+0000", 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117015 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,6,7", 2024-03-11T22:26:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.128613+0000", 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012474799999999999 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.135745+0000", 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0071317840000000004 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.135785+0000", 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0821000000000001e-05 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.135806+0000", 2024-03-11T22:26:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.065e-05 2024-03-11T22:26:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.416599+0000", 2024-03-11T22:26:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.28079263799999998 2024-03-11T22:26:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.416653+0000", 2024-03-11T22:26:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4713000000000003e-05 2024-03-11T22:26:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.416783+0000", 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012950200000000001 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:146721 9.1 9:8c58b4a2:::benchmark_data_smithi045_141553_object22601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:46.132467+0000", 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.30288965599999, 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29064284400000001, 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 146721 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.132467+0000", 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.132467+0000", 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.473 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.132468+0000", 2024-03-11T22:26:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.176e-06 2024-03-11T22:26:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.132472+0000", 2024-03-11T22:26:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3649999999999997e-06 2024-03-11T22:26:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.132483+0000", 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0740000000000001e-05 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.132491+0000", 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0989999999999999e-06 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.132627+0000", 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000135759 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.133059+0000", 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00043170399999999999 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,6,7", 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.133234+0000", 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017525799999999999 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.135814+0000", 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025798319999999998 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.142736+0000", 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0069223219999999999 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.142784+0000", 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7679999999999998e-05 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.422859+0000", 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.28007526700000002 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.422971+0000", 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000111863 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.423110+0000", 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013877900000000001 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:146727 9.1 9:89668253:::benchmark_data_smithi045_141553_object22607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:46.135897+0000", 2024-03-11T22:26:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.29945979899998, 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.28866863799999998, 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 146727 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.135897+0000", 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.135897+0000", 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.135898+0000", 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2100000000000001e-06 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.135903+0000", 2024-03-11T22:26:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.468e-06 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.135914+0000", 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0706999999999999e-05 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.135922+0000", 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6899999999999992e-06 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.136040+0000", 2024-03-11T22:26:00.481 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118306 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.136172+0000", 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013204300000000001 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,6,7", 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.136309+0000", 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000136604 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.482 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.139039+0000", 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027305849999999998 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.142830+0000", 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003790631 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.142881+0000", 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1317999999999999e-05 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.424368+0000", 2024-03-11T22:26:00.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.28148646100000002 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.424429+0000", 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1503999999999999e-05 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.424565+0000", 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013611100000000001 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.484 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:146729 9.1 9:8371710f:::benchmark_data_smithi045_141553_object22609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:46.139516+0000", 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.29584068000003, 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29306660600000001, 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 146729 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.139516+0000", 2024-03-11T22:26:00.485 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.139516+0000", 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.139517+0000", 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3990000000000001e-06 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.486 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.139523+0000", 2024-03-11T22:26:00.487 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4620000000000004e-06 2024-03-11T22:26:00.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.487 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.487 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.139539+0000", 2024-03-11T22:26:00.487 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.609e-05 2024-03-11T22:26:00.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.487 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.487 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.139547+0000", 2024-03-11T22:26:00.487 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0269999999999995e-06 2024-03-11T22:26:00.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.139753+0000", 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020647600000000001 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.140271+0000", 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000517634 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,6,7", 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.140411+0000", 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000140302 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.488 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.142932+0000", 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025207720000000001 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.143287+0000", 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00035547500000000001 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.143304+0000", 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6895e-05 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.432314+0000", 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.28900930699999999 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.432441+0000", 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012714200000000001 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.432582+0000", 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014162499999999999 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:146748 9.1 9:897a94b4:::benchmark_data_smithi045_141553_object22628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:46.155774+0000", 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.27958274999997, 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27704340700000002, 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 146748 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.491 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.155774+0000", 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.155774+0000", 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.155774+0000", 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.13e-07 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.492 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.155777+0000", 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.948e-06 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.155784+0000", 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3679999999999996e-06 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.155791+0000", 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.601e-06 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.156208+0000", 2024-03-11T22:26:00.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00041646899999999999 2024-03-11T22:26:00.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.494 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.494 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.156343+0000", 2024-03-11T22:26:00.494 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013519199999999999 2024-03-11T22:26:00.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.494 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,6,7", 2024-03-11T22:26:00.494 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.156490+0000", 2024-03-11T22:26:00.494 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014776399999999999 2024-03-11T22:26:00.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.494 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.160127+0000", 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0036367080000000002 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.162005+0000", 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001877755 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.163493+0000", 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001488119 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.432638+0000", 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26914530399999997 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.432695+0000", 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6279000000000001e-05 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.432817+0000", 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012238700000000001 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.496 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.497 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.497 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:146761 9.1 9:83eec256:::benchmark_data_smithi045_141553_object22641:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:26:00.497 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:46.169342+0000", 2024-03-11T22:26:00.497 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.26601438900002, 2024-03-11T22:26:00.497 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26373824800000001, 2024-03-11T22:26:00.497 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.497 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.497 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.497 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.497 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.497 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 146761 2024-03-11T22:26:00.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.498 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.498 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.498 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.169342+0000", 2024-03-11T22:26:00.498 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.498 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.498 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.169342+0000", 2024-03-11T22:26:00.498 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.498 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.169348+0000", 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5879999999999997e-06 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.169357+0000", 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9590000000000001e-06 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.169369+0000", 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2233e-05 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.499 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.169377+0000", 2024-03-11T22:26:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2579999999999995e-06 2024-03-11T22:26:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.169433+0000", 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5687999999999999e-05 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.169555+0000", 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012204199999999999 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,6,7", 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.169683+0000", 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000128353 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.173798+0000", 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0041147880000000003 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.173812+0000", 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3833e-05 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.173856+0000", 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4233999999999997e-05 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.432880+0000", 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25902452100000001 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.432939+0000", 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8554000000000002e-05 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.433080+0000", 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000141197 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:146766 9.1 9:851fb8a5:::benchmark_data_smithi045_141553_object22646:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:46.176155+0000", 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.259201362, 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25717282200000002, 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 146766 2024-03-11T22:26:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.176155+0000", 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.176155+0000", 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.176156+0000", 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2399999999999996e-07 2024-03-11T22:26:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.176160+0000", 2024-03-11T22:26:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2799999999999997e-06 2024-03-11T22:26:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.176243+0000", 2024-03-11T22:26:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2967999999999996e-05 2024-03-11T22:26:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.176253+0000", 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5359999999999998e-06 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.176650+0000", 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000397506 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.178066+0000", 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014161600000000001 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,6,7", 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.178209+0000", 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014256199999999999 2024-03-11T22:26:00.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.181165+0000", 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029562659999999999 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.181209+0000", 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3888999999999999e-05 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.182750+0000", 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001540884 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.433126+0000", 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25037552400000002 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.433185+0000", 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9079999999999997e-05 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.433328+0000", 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014324300000000001 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.509 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:146777 9.1 9:82dacef9:::benchmark_data_smithi045_141553_object22657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:46.189972+0000", 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.24538450300003, 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24368198899999999, 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 146777 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.510 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.189972+0000", 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.189972+0000", 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.189973+0000", 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2440000000000001e-06 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.189978+0000", 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7140000000000001e-06 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.189990+0000", 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2543e-05 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.189998+0000", 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7840000000000004e-06 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.190060+0000", 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1804000000000007e-05 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.190270+0000", 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000209741 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,6,7", 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.190432+0000", 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016195399999999999 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.194017+0000", 2024-03-11T22:26:00.513 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003585473 2024-03-11T22:26:00.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.514 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.514 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.194033+0000", 2024-03-11T22:26:00.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5529999999999999e-05 2024-03-11T22:26:00.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.514 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.514 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.194075+0000", 2024-03-11T22:26:00.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2117999999999999e-05 2024-03-11T22:26:00.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.514 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.433409+0000", 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23933373699999999 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.433527+0000", 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118675 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.433654+0000", 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012667200000000001 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:146780 9.1 9:877c71d6:::benchmark_data_smithi045_141553_object22660:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:46.194511+0000", 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 314.24084539900002, 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23945071100000001, 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 146780 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.516 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.194511+0000", 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.194511+0000", 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.194512+0000", 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1599999999999999e-06 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.194517+0000", 2024-03-11T22:26:00.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0950000000000003e-06 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.194530+0000", 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2979999999999999e-05 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.194540+0000", 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6160000000000008e-06 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.194594+0000", 2024-03-11T22:26:00.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4552999999999998e-05 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.194687+0000", 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2275000000000007e-05 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,6,7", 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.194768+0000", 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1788999999999999e-05 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.196794+0000", 2024-03-11T22:26:00.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020256860000000001 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.196802+0000", 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8210000000000002e-06 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.197252+0000", 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00045014099999999999 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.433747+0000", 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23649473900000001 2024-03-11T22:26:00.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.433850+0000", 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000102989 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.433962+0000", 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000111867 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:220289 9.2 9:4943b343:::benchmark_data_smithi045_141553_object96169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-11T22:26:00.521 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:59.688792+0000", 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 240.74656468699999, 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23182599700000001, 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 220289 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.688792+0000", 2024-03-11T22:26:00.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.688792+0000", 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.688792+0000", 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4499999999999996e-07 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.688796+0000", 2024-03-11T22:26:00.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.298e-06 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.688805+0000", 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.901e-06 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.688811+0000", 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4300000000000003e-06 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.688887+0000", 2024-03-11T22:26:00.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6181999999999997e-05 2024-03-11T22:26:00.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.688999+0000", 2024-03-11T22:26:00.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011129300000000001 2024-03-11T22:26:00.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:26:00.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.689105+0000", 2024-03-11T22:26:00.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000106833 2024-03-11T22:26:00.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.694418+0000", 2024-03-11T22:26:00.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0053126500000000004 2024-03-11T22:26:00.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.694778+0000", 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00036006299999999998 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.920489+0000", 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22571054199999999 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.920546+0000", 2024-03-11T22:26:00.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7581000000000003e-05 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.920618+0000", 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1478999999999998e-05 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:220315 9.2 9:433aaee0:::benchmark_data_smithi045_141553_object96195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:59.706039+0000", 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 240.72931748400001, 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21843927199999999, 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.528 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 220315 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.706039+0000", 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.706039+0000", 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.706040+0000", 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8299999999999995e-07 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.706044+0000", 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9310000000000001e-06 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.706055+0000", 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.131e-05 2024-03-11T22:26:00.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.706063+0000", 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7130000000000005e-06 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.706141+0000", 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8238999999999994e-05 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.706823+0000", 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00068191499999999995 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.706943+0000", 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011979 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.710095+0000", 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031520929999999999 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.715489+0000", 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0053943079999999996 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.924370+0000", 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.208881183 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.924429+0000", 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8896e-05 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.924478+0000", 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9011000000000002e-05 2024-03-11T22:26:00.533 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.534 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.534 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.534 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:220342 9.2 9:44832c11:::benchmark_data_smithi045_141553_object96222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-11T22:26:00.534 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:59.722940+0000", 2024-03-11T22:26:00.534 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 240.712416039, 2024-03-11T22:26:00.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20205993999999999, 2024-03-11T22:26:00.534 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.534 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.534 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.534 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.534 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 220342 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.722940+0000", 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.722940+0000", 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.722941+0000", 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3600000000000003e-07 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.722945+0000", 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8410000000000002e-06 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.722966+0000", 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0628999999999998e-05 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.722972+0000", 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5509999999999996e-06 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.725665+0000", 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026930740000000002 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.725770+0000", 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000104557 2024-03-11T22:26:00.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.725884+0000", 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011401200000000001 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.729012+0000", 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0031287590000000001 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.729787+0000", 2024-03-11T22:26:00.538 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00077477899999999996 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.924862+0000", 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19507448399999999 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.924960+0000", 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8479000000000004e-05 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.925000+0000", 2024-03-11T22:26:00.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0039000000000001e-05 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:220952 9.3 9:c2d4e581:::benchmark_data_smithi045_141553_object96832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:22:00.364245+0000", 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 240.071110944, 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20776457700000001, 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 220952 2024-03-11T22:26:00.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364245+0000", 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364245+0000", 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364247+0000", 2024-03-11T22:26:00.541 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.091e-06 2024-03-11T22:26:00.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.542 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.542 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364250+0000", 2024-03-11T22:26:00.542 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8979999999999998e-06 2024-03-11T22:26:00.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.542 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.542 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364304+0000", 2024-03-11T22:26:00.542 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3695e-05 2024-03-11T22:26:00.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.542 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364313+0000", 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2499999999999995e-06 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364437+0000", 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000123299 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364770+0000", 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00033360699999999998 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364887+0000", 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117218 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.369071+0000", 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0041831869999999997 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.561786+0000", 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19271545400000001 2024-03-11T22:26:00.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.568747+0000", 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0069609859999999997 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.571670+0000", 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002923372 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.572010+0000", 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00033952 2024-03-11T22:26:00.545 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:220953 9.2 9:4d5c1e78:::benchmark_data_smithi045_141553_object96833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:22:00.364331+0000", 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 240.07102522700001, 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20279325300000001, 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.546 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 220953 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364331+0000", 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364331+0000", 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.547 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364332+0000", 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0999999999999999e-07 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364335+0000", 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0730000000000001e-06 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364350+0000", 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4779e-05 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364355+0000", 2024-03-11T22:26:00.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.801e-06 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364567+0000", 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000212007 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364743+0000", 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000175205 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:26:00.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.364861+0000", 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118235 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.368255+0000", 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033944750000000001 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.562187+0000", 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.193931775 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.550 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.567030+0000", 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0048428810000000003 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.567079+0000", 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9054999999999997e-05 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.567124+0000", 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5457000000000002e-05 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.551 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.552 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:220959 9.3 9:cc95939e:::benchmark_data_smithi045_141553_object96839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-11T22:26:00.552 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:22:00.368180+0000", 2024-03-11T22:26:00.552 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 240.06717597100001, 2024-03-11T22:26:00.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20667877400000001, 2024-03-11T22:26:00.552 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.552 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.552 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.552 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.552 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.552 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 220959 2024-03-11T22:26:00.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.553 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.553 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.368180+0000", 2024-03-11T22:26:00.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.553 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.368180+0000", 2024-03-11T22:26:00.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.554 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.368181+0000", 2024-03-11T22:26:00.554 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3e-07 2024-03-11T22:26:00.554 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.554 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.554 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.554 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.368184+0000", 2024-03-11T22:26:00.554 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8899999999999999e-06 2024-03-11T22:26:00.554 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.554 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.554 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.554 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.368192+0000", 2024-03-11T22:26:00.554 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4610000000000008e-06 2024-03-11T22:26:00.554 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.368199+0000", 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5899999999999996e-06 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.562715+0000", 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19451560600000001 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.562827+0000", 2024-03-11T22:26:00.555 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000112834 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,7", 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.562942+0000", 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114177 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.568701+0000", 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0057598479999999997 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.572083+0000", 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00338135 2024-03-11T22:26:00.556 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.574766+0000", 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026827909999999999 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.574813+0000", 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7207999999999999e-05 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.574859+0000", 2024-03-11T22:26:00.557 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6388999999999999e-05 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:220962 9.1 9:82072f7a:::benchmark_data_smithi045_141553_object96842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:22:00.372615+0000", 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 240.06274178000001, 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.206369107, 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:26:00.558 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 220962 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.372615+0000", 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.372615+0000", 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.559 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.372616+0000", 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0359999999999999e-06 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.372620+0000", 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4279999999999998e-06 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.562204+0000", 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.18958391999999999 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.560 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.562213+0000", 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4879999999999996e-06 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.564433+0000", 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0022193489999999998 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.564540+0000", 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000107169 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.561 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 6,7", 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.564650+0000", 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110018 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.578651+0000", 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.014001055 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.578702+0000", 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0875000000000003e-05 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.562 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.578888+0000", 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018561200000000001 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.578935+0000", 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7479999999999999e-05 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:22:00.578984+0000", 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8677000000000002e-05 2024-03-11T22:26:00.563 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.564 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.564 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.564 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:26:00.564 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:26:00.564 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:26:00.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:00.565 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:00.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:00.666 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:00.700 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:00.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:00.752 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:00 smithi045 ceph-mon[30178]: 10.c scrub ok 2024-03-11T22:26:00.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:00.845 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:00.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:00.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:00.946 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:00.979 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:00.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:00.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:01.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:00 smithi045 ceph-mon[30178]: 6.0 scrub starts 2024-03-11T22:26:01.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:00 smithi045 ceph-mon[30178]: 6.0 scrub ok 2024-03-11T22:26:01.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:00 smithi045 ceph-mon[30178]: 6.1e scrub starts 2024-03-11T22:26:01.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:00 smithi045 ceph-mon[30178]: 6.1e scrub ok 2024-03-11T22:26:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:00 smithi045 ceph-mon[39715]: 10.c scrub ok 2024-03-11T22:26:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:00 smithi045 ceph-mon[39715]: 6.0 scrub starts 2024-03-11T22:26:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:00 smithi045 ceph-mon[39715]: 6.0 scrub ok 2024-03-11T22:26:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:00 smithi045 ceph-mon[39715]: 6.1e scrub starts 2024-03-11T22:26:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:00 smithi045 ceph-mon[39715]: 6.1e scrub ok 2024-03-11T22:26:01.081 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:01.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:00 smithi144 ceph-mon[33984]: 10.c scrub ok 2024-03-11T22:26:01.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:00 smithi144 ceph-mon[33984]: 6.0 scrub starts 2024-03-11T22:26:01.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:00 smithi144 ceph-mon[33984]: 6.0 scrub ok 2024-03-11T22:26:01.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:00 smithi144 ceph-mon[33984]: 6.1e scrub starts 2024-03-11T22:26:01.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:00 smithi144 ceph-mon[33984]: 6.1e scrub ok 2024-03-11T22:26:01.119 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:01.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:01.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:01.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:01.262 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:01.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:01.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:01.403 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:01.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:01.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:01.541 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:01.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:01.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:01.643 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:01.684 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:01.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:01.755 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[30178]: 1.c scrub starts 2024-03-11T22:26:01.785 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:01.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:26:01.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:01.835 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:01.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:01.936 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:01.967 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:01.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:01.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:02.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[30178]: 1.c scrub ok 2024-03-11T22:26:02.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[30178]: pgmap v811: 168 pgs: 168 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-11T22:26:02.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[30178]: 1.2 scrub starts 2024-03-11T22:26:02.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[30178]: 1.2 scrub ok 2024-03-11T22:26:02.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:02.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[39715]: 1.c scrub starts 2024-03-11T22:26:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[39715]: 1.c scrub ok 2024-03-11T22:26:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[39715]: pgmap v811: 168 pgs: 168 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-11T22:26:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[39715]: 1.2 scrub starts 2024-03-11T22:26:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[39715]: 1.2 scrub ok 2024-03-11T22:26:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:02.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:02.069 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:02.098 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:01 smithi144 ceph-mon[33984]: 1.c scrub starts 2024-03-11T22:26:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:01 smithi144 ceph-mon[33984]: 1.c scrub ok 2024-03-11T22:26:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:01 smithi144 ceph-mon[33984]: pgmap v811: 168 pgs: 168 active+clean; 2.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-11T22:26:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:01 smithi144 ceph-mon[33984]: 1.2 scrub starts 2024-03-11T22:26:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:01 smithi144 ceph-mon[33984]: 1.2 scrub ok 2024-03-11T22:26:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:02.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:02.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:02.201 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:02.235 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:02.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:02.336 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:02.369 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:02.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:02.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:02.513 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:02.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:02.614 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:02.649 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:02.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:02.751 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:02.784 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:02.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:02.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:02.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:02.916 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:02.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:03.017 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:03.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:02 smithi045 ceph-mon[30178]: 1.11 scrub starts 2024-03-11T22:26:03.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:02 smithi045 ceph-mon[30178]: 1.11 scrub ok 2024-03-11T22:26:03.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:02 smithi045 ceph-mon[30178]: 1.f scrub starts 2024-03-11T22:26:03.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:02 smithi045 ceph-mon[30178]: 1.f scrub ok 2024-03-11T22:26:03.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:02 smithi045 ceph-mon[30178]: 6.8 scrub starts 2024-03-11T22:26:03.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:02 smithi045 ceph-mon[30178]: 6.8 scrub ok 2024-03-11T22:26:03.036 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:26:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:26:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:26:03.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:02 smithi045 ceph-mon[39715]: 1.11 scrub starts 2024-03-11T22:26:03.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:02 smithi045 ceph-mon[39715]: 1.11 scrub ok 2024-03-11T22:26:03.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:02 smithi045 ceph-mon[39715]: 1.f scrub starts 2024-03-11T22:26:03.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:02 smithi045 ceph-mon[39715]: 1.f scrub ok 2024-03-11T22:26:03.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:02 smithi045 ceph-mon[39715]: 6.8 scrub starts 2024-03-11T22:26:03.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:02 smithi045 ceph-mon[39715]: 6.8 scrub ok 2024-03-11T22:26:03.050 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:03.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:03.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:02 smithi144 ceph-mon[33984]: 1.11 scrub starts 2024-03-11T22:26:03.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:02 smithi144 ceph-mon[33984]: 1.11 scrub ok 2024-03-11T22:26:03.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:02 smithi144 ceph-mon[33984]: 1.f scrub starts 2024-03-11T22:26:03.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:02 smithi144 ceph-mon[33984]: 1.f scrub ok 2024-03-11T22:26:03.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:02 smithi144 ceph-mon[33984]: 6.8 scrub starts 2024-03-11T22:26:03.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:02 smithi144 ceph-mon[33984]: 6.8 scrub ok 2024-03-11T22:26:03.151 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:03.194 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:03.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:03.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:03.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:03.325 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:03.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:03.427 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:03.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:03.463 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:03.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:03.564 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:03.594 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:03.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:03.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:03.696 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:03.736 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:03.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:03.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:03.838 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:03.889 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:03.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:03.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:03.991 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:04.027 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:04.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:04.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:03 smithi045 ceph-mon[30178]: pgmap v812: 168 pgs: 168 active+clean; 1.8 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:26:04.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:03 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:26:04.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:03 smithi045 ceph-mon[30178]: 1.8 scrub starts 2024-03-11T22:26:04.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:03 smithi045 ceph-mon[30178]: 1.8 scrub ok 2024-03-11T22:26:04.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:03 smithi045 ceph-mon[39715]: pgmap v812: 168 pgs: 168 active+clean; 1.8 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:26:04.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:03 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:26:04.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:03 smithi045 ceph-mon[39715]: 1.8 scrub starts 2024-03-11T22:26:04.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:03 smithi045 ceph-mon[39715]: 1.8 scrub ok 2024-03-11T22:26:04.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:03 smithi144 ceph-mon[33984]: pgmap v812: 168 pgs: 168 active+clean; 1.8 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:26:04.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:03 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:26:04.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:03 smithi144 ceph-mon[33984]: 1.8 scrub starts 2024-03-11T22:26:04.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:03 smithi144 ceph-mon[33984]: 1.8 scrub ok 2024-03-11T22:26:04.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:04.171 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:04.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:04.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:04.310 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:04.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:04.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:04.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:04.449 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:04.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:04.551 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:04.577 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:04.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:04.680 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:04.722 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:04.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:04.823 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:04.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:04.866 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:04.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:04.967 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:05.003 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:05.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:05.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:04 smithi045 ceph-mon[39715]: 3.19 scrub starts 2024-03-11T22:26:05.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:04 smithi045 ceph-mon[39715]: 3.19 scrub ok 2024-03-11T22:26:05.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:04 smithi045 ceph-mon[39715]: 3.d scrub starts 2024-03-11T22:26:05.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:04 smithi045 ceph-mon[39715]: 3.d scrub ok 2024-03-11T22:26:05.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:04 smithi045 ceph-mon[30178]: 3.19 scrub starts 2024-03-11T22:26:05.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:04 smithi045 ceph-mon[30178]: 3.19 scrub ok 2024-03-11T22:26:05.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:04 smithi045 ceph-mon[30178]: 3.d scrub starts 2024-03-11T22:26:05.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:04 smithi045 ceph-mon[30178]: 3.d scrub ok 2024-03-11T22:26:05.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:04 smithi144 ceph-mon[33984]: 3.19 scrub starts 2024-03-11T22:26:05.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:04 smithi144 ceph-mon[33984]: 3.19 scrub ok 2024-03-11T22:26:05.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:04 smithi144 ceph-mon[33984]: 3.d scrub starts 2024-03-11T22:26:05.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:04 smithi144 ceph-mon[33984]: 3.d scrub ok 2024-03-11T22:26:05.105 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:05.147 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:05.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:05.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:05.249 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:05.295 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:05.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:05.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:05.438 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:05.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:05.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:05.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:05.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:05.580 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:05.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:05.681 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:05.720 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:05.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:05.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:05.822 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:05.856 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:05.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:05.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:05.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:05.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:05.996 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:05.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:06.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:05 smithi045 ceph-mon[30178]: pgmap v813: 168 pgs: 168 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:26:06.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:05 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:06.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:05 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:06.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:05 smithi045 ceph-mon[39715]: pgmap v813: 168 pgs: 168 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:26:06.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:05 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:06.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:05 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:06.097 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:06.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:05 smithi144 ceph-mon[33984]: pgmap v813: 168 pgs: 168 active+clean; 1.7 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-11T22:26:06.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:05 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:06.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:05 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:06.129 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:06.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:06.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:06.231 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:06.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:06.261 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:06.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:06.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:06.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:06.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:06.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:06.547 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:06.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:06.649 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:06.687 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:06.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:06.788 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:06.822 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:06.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:06.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:06.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:06.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:06.966 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:06.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:07.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:07.098 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:26:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:26:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:06 smithi144 ceph-mon[33984]: 4.0 scrub starts 2024-03-11T22:26:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:06 smithi144 ceph-mon[33984]: 4.0 scrub ok 2024-03-11T22:26:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:06 smithi144 ceph-mon[33984]: 4.f scrub starts 2024-03-11T22:26:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:06 smithi144 ceph-mon[33984]: 4.f scrub ok 2024-03-11T22:26:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:06 smithi144 ceph-mon[33984]: 3.17 scrub starts 2024-03-11T22:26:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:06 smithi144 ceph-mon[33984]: 3.17 scrub ok 2024-03-11T22:26:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:06 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:26:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:06 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:26:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:06 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:06 smithi144 ceph-mon[33984]: 2.1 scrub starts 2024-03-11T22:26:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:06 smithi144 ceph-mon[33984]: 2.1 scrub ok 2024-03-11T22:26:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:07.103 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:07.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:07.205 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:07.245 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:07.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:07.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:07.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[39715]: 4.0 scrub starts 2024-03-11T22:26:07.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[39715]: 4.0 scrub ok 2024-03-11T22:26:07.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[39715]: 4.f scrub starts 2024-03-11T22:26:07.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[39715]: 4.f scrub ok 2024-03-11T22:26:07.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[39715]: 3.17 scrub starts 2024-03-11T22:26:07.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[39715]: 3.17 scrub ok 2024-03-11T22:26:07.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:26:07.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:26:07.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:07.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[39715]: 2.1 scrub starts 2024-03-11T22:26:07.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[39715]: 2.1 scrub ok 2024-03-11T22:26:07.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[30178]: 4.0 scrub starts 2024-03-11T22:26:07.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[30178]: 4.0 scrub ok 2024-03-11T22:26:07.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[30178]: 4.f scrub starts 2024-03-11T22:26:07.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[30178]: 4.f scrub ok 2024-03-11T22:26:07.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[30178]: 3.17 scrub starts 2024-03-11T22:26:07.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[30178]: 3.17 scrub ok 2024-03-11T22:26:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:26:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:26:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[30178]: 2.1 scrub starts 2024-03-11T22:26:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:06 smithi045 ceph-mon[30178]: 2.1 scrub ok 2024-03-11T22:26:07.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:07.387 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:07.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:07.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:07.489 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:07.528 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:07.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:07.630 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:07.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:07.665 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:07.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:07.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:07.800 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:07.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:07.902 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:07.926 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T22:26:07.934 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:07.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:07.939 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:08.036 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:08.068 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:08.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:08.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:08.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:08.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[30178]: pgmap v814: 168 pgs: 168 active+clean; 1.4 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-11T22:26:08.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:08.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[30178]: 5.19 scrub starts 2024-03-11T22:26:08.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[30178]: 5.19 scrub ok 2024-03-11T22:26:08.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[30178]: 1.10 scrub starts 2024-03-11T22:26:08.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[30178]: 1.10 scrub ok 2024-03-11T22:26:08.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:26:08.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:26:08.190 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[39715]: pgmap v814: 168 pgs: 168 active+clean; 1.4 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-11T22:26:08.190 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:08.190 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[39715]: 5.19 scrub starts 2024-03-11T22:26:08.190 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[39715]: 5.19 scrub ok 2024-03-11T22:26:08.190 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[39715]: 1.10 scrub starts 2024-03-11T22:26:08.190 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[39715]: 1.10 scrub ok 2024-03-11T22:26:08.190 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:26:08.190 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:07 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:26:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:08.223 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:08.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:08.324 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:08.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:07 smithi144 ceph-mon[33984]: pgmap v814: 168 pgs: 168 active+clean; 1.4 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-11T22:26:08.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:08.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:07 smithi144 ceph-mon[33984]: 5.19 scrub starts 2024-03-11T22:26:08.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:07 smithi144 ceph-mon[33984]: 5.19 scrub ok 2024-03-11T22:26:08.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:07 smithi144 ceph-mon[33984]: 1.10 scrub starts 2024-03-11T22:26:08.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:07 smithi144 ceph-mon[33984]: 1.10 scrub ok 2024-03-11T22:26:08.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:07 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:26:08.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:07 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:26:08.358 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:08.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:08.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:08.460 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:08.501 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:08.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:08.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:08.603 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:08.642 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:08.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:08.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:08.744 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:08.788 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:08.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:08.889 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:08.933 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:08.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:09.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:09.068 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:09.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:09.077 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:08 smithi045 ceph-mon[30178]: osdmap e392: 8 total, 8 up, 8 in 2024-03-11T22:26:09.077 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:08 smithi045 ceph-mon[39715]: osdmap e392: 8 total, 8 up, 8 in 2024-03-11T22:26:09.169 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:09.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:09.215 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:09.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:09.303 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:26:09.316 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:09.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:08 smithi144 ceph-mon[33984]: osdmap e392: 8 total, 8 up, 8 in 2024-03-11T22:26:09.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:09.351 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:09.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:09.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:09.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:09.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:09.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:09.521 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:26:09.521 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:26:09.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:26:09.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:26:09.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:26:09.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:26:09.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:26:09.522 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:26:09.522 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:26:09.522 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:26:09.522 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:26:09.522 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:26:09.523 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:26:09.523 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:26:09.523 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:26:09.523 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:26:09.597 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:09.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:09.633 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:09.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:09.734 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:09.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:09.771 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:09.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:09.873 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:09.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:09.906 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:09.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:10.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:10.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:10.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:10.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:10.145 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:10.169 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[30178]: pgmap v816: 168 pgs: 168 active+clean; 1.3 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-11T22:26:10.169 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/515358790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:10.169 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[30178]: 5.1 scrub starts 2024-03-11T22:26:10.169 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[30178]: 5.b scrub starts 2024-03-11T22:26:10.169 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[30178]: 5.1 scrub ok 2024-03-11T22:26:10.170 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[30178]: 5.b scrub ok 2024-03-11T22:26:10.170 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/559096795' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T22:26:10.170 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T22:26:10.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[39715]: pgmap v816: 168 pgs: 168 active+clean; 1.3 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-11T22:26:10.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/515358790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:10.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[39715]: 5.1 scrub starts 2024-03-11T22:26:10.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[39715]: 5.b scrub starts 2024-03-11T22:26:10.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[39715]: 5.1 scrub ok 2024-03-11T22:26:10.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[39715]: 5.b scrub ok 2024-03-11T22:26:10.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/559096795' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T22:26:10.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:09 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T22:26:10.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:10.182 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:10.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:10.283 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:10.320 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:10.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:10.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:10.336 INFO:teuthology.orchestra.run.smithi045.stderr:set pool 1 pg_num to 11 2024-03-11T22:26:10.348 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:09 smithi144 ceph-mon[33984]: pgmap v816: 168 pgs: 168 active+clean; 1.3 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-11T22:26:10.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:09 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/515358790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:10.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:09 smithi144 ceph-mon[33984]: 5.1 scrub starts 2024-03-11T22:26:10.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:09 smithi144 ceph-mon[33984]: 5.b scrub starts 2024-03-11T22:26:10.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:09 smithi144 ceph-mon[33984]: 5.1 scrub ok 2024-03-11T22:26:10.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:09 smithi144 ceph-mon[33984]: 5.b scrub ok 2024-03-11T22:26:10.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:09 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/559096795' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T22:26:10.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:09 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-11T22:26:10.421 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:10.458 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:10.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:10.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:10.560 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:10.595 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:10.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:10.697 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:10.744 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:10.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:10.845 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:10.888 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:10.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:10.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:11.017 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[30178]: 5.1a deep-scrub starts 2024-03-11T22:26:11.017 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[30178]: 5.1a deep-scrub ok 2024-03-11T22:26:11.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-11T22:26:11.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[30178]: osdmap e393: 8 total, 8 up, 8 in 2024-03-11T22:26:11.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[30178]: 4.1c scrub starts 2024-03-11T22:26:11.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[30178]: 4.1c scrub ok 2024-03-11T22:26:11.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[30178]: 6.1a scrub starts 2024-03-11T22:26:11.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[30178]: 6.1a scrub ok 2024-03-11T22:26:11.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T22:26:11.018 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[39715]: 5.1a deep-scrub starts 2024-03-11T22:26:11.018 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[39715]: 5.1a deep-scrub ok 2024-03-11T22:26:11.018 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-11T22:26:11.018 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[39715]: osdmap e393: 8 total, 8 up, 8 in 2024-03-11T22:26:11.018 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[39715]: 4.1c scrub starts 2024-03-11T22:26:11.019 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[39715]: 4.1c scrub ok 2024-03-11T22:26:11.019 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[39715]: 6.1a scrub starts 2024-03-11T22:26:11.019 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[39715]: 6.1a scrub ok 2024-03-11T22:26:11.019 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:10 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T22:26:11.025 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:11.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:11.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:10 smithi144 ceph-mon[33984]: 5.1a deep-scrub starts 2024-03-11T22:26:11.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:10 smithi144 ceph-mon[33984]: 5.1a deep-scrub ok 2024-03-11T22:26:11.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:10 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-11T22:26:11.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:10 smithi144 ceph-mon[33984]: osdmap e393: 8 total, 8 up, 8 in 2024-03-11T22:26:11.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:10 smithi144 ceph-mon[33984]: 4.1c scrub starts 2024-03-11T22:26:11.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:10 smithi144 ceph-mon[33984]: 4.1c scrub ok 2024-03-11T22:26:11.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:10 smithi144 ceph-mon[33984]: 6.1a scrub starts 2024-03-11T22:26:11.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:10 smithi144 ceph-mon[33984]: 6.1a scrub ok 2024-03-11T22:26:11.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:10 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-11T22:26:11.127 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:11.167 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:11.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:11.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:11.269 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:11.305 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:11.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:11.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:11.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:11.445 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:11.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:11.547 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:11.582 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:11.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:11.684 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:11.721 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:11.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:11.823 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:11.855 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:11.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:11.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:11.956 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:11.991 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:11.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:12.092 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:12.098 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:11 smithi144 ceph-mon[33984]: pgmap v818: 168 pgs: 168 active+clean; 1.3 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 929 op/s 2024-03-11T22:26:12.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:11 smithi144 ceph-mon[33984]: 1.b scrub starts 2024-03-11T22:26:12.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:11 smithi144 ceph-mon[33984]: 1.b scrub ok 2024-03-11T22:26:12.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:11 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:26:12.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:11 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:26:12.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:11 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T22:26:12.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:11 smithi144 ceph-mon[33984]: osdmap e394: 8 total, 8 up, 8 in 2024-03-11T22:26:12.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:11 smithi144 ceph-mon[33984]: osdmap e395: 8 total, 8 up, 8 in 2024-03-11T22:26:12.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:12.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:12.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:26:12.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[39715]: pgmap v818: 168 pgs: 168 active+clean; 1.3 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 929 op/s 2024-03-11T22:26:12.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[39715]: 1.b scrub starts 2024-03-11T22:26:12.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[39715]: 1.b scrub ok 2024-03-11T22:26:12.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:26:12.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:26:12.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T22:26:12.227 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[39715]: osdmap e394: 8 total, 8 up, 8 in 2024-03-11T22:26:12.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[39715]: osdmap e395: 8 total, 8 up, 8 in 2024-03-11T22:26:12.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[30178]: pgmap v818: 168 pgs: 168 active+clean; 1.3 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 929 op/s 2024-03-11T22:26:12.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[30178]: 1.b scrub starts 2024-03-11T22:26:12.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[30178]: 1.b scrub ok 2024-03-11T22:26:12.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:26:12.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:26:12.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-11T22:26:12.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[30178]: osdmap e394: 8 total, 8 up, 8 in 2024-03-11T22:26:12.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:11 smithi045 ceph-mon[30178]: osdmap e395: 8 total, 8 up, 8 in 2024-03-11T22:26:12.234 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:12.264 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:12.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:12.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:12.412 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:12.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:12.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:12.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:12.584 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:12.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:12.671 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:12.685 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:12.741 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:12.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:12.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:12.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:12.854 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:26:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:26:12.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:12.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:12.983 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:13.017 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:13.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:12 smithi144 ceph-mon[33984]: 4.13 scrub starts 2024-03-11T22:26:13.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:12 smithi144 ceph-mon[33984]: 4.13 scrub ok 2024-03-11T22:26:13.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:12 smithi144 ceph-mon[33984]: 5.d deep-scrub starts 2024-03-11T22:26:13.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:12 smithi144 ceph-mon[33984]: 5.d deep-scrub ok 2024-03-11T22:26:13.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:12 smithi144 ceph-mon[33984]: osdmap e396: 8 total, 8 up, 8 in 2024-03-11T22:26:13.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:12 smithi144 ceph-mon[33984]: 1.c starting backfill to osd.5 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:13.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:12 smithi144 ceph-mon[33984]: 1.c starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:13.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:12 smithi144 ceph-mon[33984]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:13.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:12 smithi144 ceph-mon[33984]: 1.e starting backfill to osd.6 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:13.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:12 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2561812160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:26:13.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:12 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:26:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:13.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:13.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:13.158 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:13.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:13.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[30178]: 4.13 scrub starts 2024-03-11T22:26:13.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[30178]: 4.13 scrub ok 2024-03-11T22:26:13.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[30178]: 5.d deep-scrub starts 2024-03-11T22:26:13.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[30178]: 5.d deep-scrub ok 2024-03-11T22:26:13.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[30178]: osdmap e396: 8 total, 8 up, 8 in 2024-03-11T22:26:13.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[30178]: 1.c starting backfill to osd.5 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:13.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[30178]: 1.c starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:13.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[30178]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:13.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[30178]: 1.e starting backfill to osd.6 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:13.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2561812160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:26:13.198 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:26:13.199 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[39715]: 4.13 scrub starts 2024-03-11T22:26:13.199 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[39715]: 4.13 scrub ok 2024-03-11T22:26:13.199 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[39715]: 5.d deep-scrub starts 2024-03-11T22:26:13.199 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[39715]: 5.d deep-scrub ok 2024-03-11T22:26:13.199 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[39715]: osdmap e396: 8 total, 8 up, 8 in 2024-03-11T22:26:13.199 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[39715]: 1.c starting backfill to osd.5 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:13.199 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[39715]: 1.c starting backfill to osd.7 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:13.199 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[39715]: 1.e starting backfill to osd.0 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:13.199 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[39715]: 1.e starting backfill to osd.6 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:13.199 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2561812160' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:26:13.199 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:12 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:26:13.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:13.294 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:13.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:13.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:13.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:13.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:13.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:13.505 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:26:13.506 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:26:13.538 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:13.555 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:26:13.582 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:13.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:13.684 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:13.717 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:13.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:13.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:13.855 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:13.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:13.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:13.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:13.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:13.993 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:13.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:14.094 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:14.127 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:14.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:14.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:14.168 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: 4.15 scrub starts 2024-03-11T22:26:14.168 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: 4.15 scrub ok 2024-03-11T22:26:14.168 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: pgmap v821: 168 pgs: 6 remapped+peering, 162 active+clean; 1.0 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T22:26:14.168 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: 1.d starting backfill to osd.2 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:14.168 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:14.168 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:14.168 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: 1.b starting backfill to osd.6 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:14.169 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: 4.b scrub starts 2024-03-11T22:26:14.169 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: 4.b scrub ok 2024-03-11T22:26:14.169 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:26:14.169 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: 6.14 scrub starts 2024-03-11T22:26:14.169 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: 6.14 scrub ok 2024-03-11T22:26:14.169 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: 5.0 scrub starts 2024-03-11T22:26:14.169 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: 5.0 scrub ok 2024-03-11T22:26:14.169 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:26:14.169 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:26:14.169 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:13 smithi144 ceph-mon[33984]: osdmap e397: 8 total, 8 up, 8 in 2024-03-11T22:26:14.219 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T22:26:14.229 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:14.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: 4.15 scrub starts 2024-03-11T22:26:14.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: 4.15 scrub ok 2024-03-11T22:26:14.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: pgmap v821: 168 pgs: 6 remapped+peering, 162 active+clean; 1.0 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T22:26:14.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: 1.d starting backfill to osd.2 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:14.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:14.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:14.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: 1.b starting backfill to osd.6 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: 4.b scrub starts 2024-03-11T22:26:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: 4.b scrub ok 2024-03-11T22:26:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:26:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: 6.14 scrub starts 2024-03-11T22:26:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: 6.14 scrub ok 2024-03-11T22:26:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: 5.0 scrub starts 2024-03-11T22:26:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: 5.0 scrub ok 2024-03-11T22:26:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:26:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:26:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[39715]: osdmap e397: 8 total, 8 up, 8 in 2024-03-11T22:26:14.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: 4.15 scrub starts 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: 4.15 scrub ok 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: pgmap v821: 168 pgs: 6 remapped+peering, 162 active+clean; 1.0 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: 1.d starting backfill to osd.2 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: 1.d starting backfill to osd.6 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: 1.b starting backfill to osd.4 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: 1.b starting backfill to osd.6 from (0'0,0'0] MAX to 93'130 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: 4.b scrub starts 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: 4.b scrub ok 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: 6.14 scrub starts 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: 6.14 scrub ok 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: 5.0 scrub starts 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: 5.0 scrub ok 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:26:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:13 smithi045 ceph-mon[30178]: osdmap e397: 8 total, 8 up, 8 in 2024-03-11T22:26:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:14.304 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:14.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:14.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:14.453 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:14.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:14.554 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:14.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:14.604 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:14.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:14.685 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:26:14.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:14.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:14.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:14.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:14.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:14.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:14.883 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:14.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:14.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:15.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:15.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:15.134 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:15.167 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:15.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:15.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:15.269 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:15.302 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:15.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:15.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:15.404 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:15.440 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:15.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:15.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:15.581 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:15.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:15.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:15 smithi144 ceph-mon[33984]: pgmap v824: 168 pgs: 6 remapped+peering, 162 active+clean; 970 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-11T22:26:15.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:15 smithi144 ceph-mon[33984]: osdmap e398: 8 total, 8 up, 8 in 2024-03-11T22:26:15.683 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:15.719 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:15.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:15.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:15 smithi045 ceph-mon[39715]: pgmap v824: 168 pgs: 6 remapped+peering, 162 active+clean; 970 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-11T22:26:15.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:15 smithi045 ceph-mon[39715]: osdmap e398: 8 total, 8 up, 8 in 2024-03-11T22:26:15.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:15 smithi045 ceph-mon[30178]: pgmap v824: 168 pgs: 6 remapped+peering, 162 active+clean; 970 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-11T22:26:15.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:15 smithi045 ceph-mon[30178]: osdmap e398: 8 total, 8 up, 8 in 2024-03-11T22:26:15.821 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:15.845 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:15.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:15.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:15.947 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:15.981 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:15.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:16.067 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:26:16.067 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:26:16.067 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-03-11T22:26:16.067 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T22:26:16.067 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T22:26:16.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-03-11T22:26:16.083 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:16.117 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:16.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:16.219 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:16.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:16.264 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:16.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:16.365 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:16.397 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:16.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:16.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:16.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:26:16.499 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:16.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:16.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:16.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:16.650 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:16.678 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:16.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:16.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:16.780 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:16.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:16.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:16.849 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:26:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:26:16.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:16.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:16.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:17.047 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:26:17.047 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:26:17.047 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:26:17.048 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:26:17.048 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:26:17.048 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:26:17.048 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:26:17.048 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:26:17.048 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:26:17.048 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:26:17.048 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:26:17.048 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:26:17.049 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:26:17.049 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:26:17.049 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:26:17.049 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:26:17.056 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:26:17.056 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-03-11T22:26:17.056 INFO:teuthology.orchestra.run.smithi144.stdout: "complaint_time": 30, 2024-03-11T22:26:17.056 INFO:teuthology.orchestra.run.smithi144.stdout: "num_blocked_ops": 0 2024-03-11T22:26:17.056 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:26:17.071 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:17.102 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:17.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:17.203 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:17.241 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:17.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:17.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:17.375 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:17.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:17.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:17.411 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:17 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:26:17.411 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:17 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:26:17.411 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:17 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:17.411 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:17 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:17.411 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:17 smithi144 ceph-mon[33984]: pgmap v826: 168 pgs: 5 remapped+peering, 163 active+clean; 700 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-11T22:26:17.412 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:17 smithi144 ceph-mon[33984]: osdmap e399: 8 total, 8 up, 8 in 2024-03-11T22:26:17.412 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:17 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3153204839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:17.412 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:17.477 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:17.514 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:17.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:17.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:26:17.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:26:17.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:17.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:17.535 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[39715]: pgmap v826: 168 pgs: 5 remapped+peering, 163 active+clean; 700 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-11T22:26:17.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[39715]: osdmap e399: 8 total, 8 up, 8 in 2024-03-11T22:26:17.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3153204839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:17.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:17.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:26:17.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:26:17.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:17.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:17.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[30178]: pgmap v826: 168 pgs: 5 remapped+peering, 163 active+clean; 700 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-11T22:26:17.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[30178]: osdmap e399: 8 total, 8 up, 8 in 2024-03-11T22:26:17.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3153204839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:17.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:17.615 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:17.648 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:17.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:17.750 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:17.788 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:17.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:17.870 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T22:26:17.890 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:17.950 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:17.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:18.051 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:18.096 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:18.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:18.197 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:18.248 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:18.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:18.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:18.313 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:26:18.350 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:18.385 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:18.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:18.486 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:18.519 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:18.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:18.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:18.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:18.631 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:18 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-11T22:26:18.631 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:18 smithi045 ceph-mon[39715]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:26:18.631 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:18 smithi045 ceph-mon[39715]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-11T22:26:18.631 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:18 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-11T22:26:18.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:18 smithi045 ceph-mon[30178]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:26:18.632 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:18 smithi045 ceph-mon[30178]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-11T22:26:18.672 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:18.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:18.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:18.752 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:18 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-03-11T22:26:18.752 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:18 smithi144 ceph-mon[33984]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:26:18.752 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:18 smithi144 ceph-mon[33984]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-11T22:26:18.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:18.782 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:18.813 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:18.851 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:18.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:18.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:18.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:19.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:19.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:19.144 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:19.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:19.245 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:19.289 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:19.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:19.391 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:19.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:19.430 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:19.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:19.531 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:19.569 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:19.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:19.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:19.670 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:19.713 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:19.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:19.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:19.761 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:19 smithi144 ceph-mon[33984]: pgmap v828: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 577 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 868 B/s rd, 0 B/s wr, 1.32k op/s; 54 KiB/s, 0 objects/s recovering 2024-03-11T22:26:19.761 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-11T22:26:19.761 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:19 smithi144 ceph-mon[33984]: osdmap e400: 8 total, 8 up, 8 in 2024-03-11T22:26:19.770 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:19 smithi045 ceph-mon[30178]: pgmap v828: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 577 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 868 B/s rd, 0 B/s wr, 1.32k op/s; 54 KiB/s, 0 objects/s recovering 2024-03-11T22:26:19.770 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-11T22:26:19.770 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:19 smithi045 ceph-mon[30178]: osdmap e400: 8 total, 8 up, 8 in 2024-03-11T22:26:19.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:19 smithi045 ceph-mon[39715]: pgmap v828: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 577 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 868 B/s rd, 0 B/s wr, 1.32k op/s; 54 KiB/s, 0 objects/s recovering 2024-03-11T22:26:19.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-03-11T22:26:19.771 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:19 smithi045 ceph-mon[39715]: osdmap e400: 8 total, 8 up, 8 in 2024-03-11T22:26:19.815 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:19.852 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:19.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:19.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:19.954 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:19.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:19.990 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:19.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:20.091 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:20.123 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:20.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:20.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:20.242 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:26:20.242 INFO:teuthology.orchestra.run.smithi144.stdout: "size": 20, 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 600, 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [ 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:6046 10.0 e353/348 10:00090e9b:::benchmark_data_smithi045_169175_object6045:head v 353'369, mlcod=353'367)", 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:03.768845+0000", 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 196.467480373, 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.082820458, 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.768845+0000", 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.243 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.768845+0000", 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.768846+0000", 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8299999999999996e-07 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769060+0000", 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00021431799999999999 2024-03-11T22:26:20.244 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769068+0000", 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4449999999999999e-06 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769073+0000", 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0320000000000003e-06 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769129+0000", 2024-03-11T22:26:20.245 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.5763000000000001e-05 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769164+0000", 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5583999999999997e-05 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769792+0000", 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00062805499999999998 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.851666+0000", 2024-03-11T22:26:20.246 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.081873578000000002 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:6049 10.2 e353/348 10:4440ab9c:::benchmark_data_smithi045_169175_object6048:head v 353'419, mlcod=353'416)", 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:03.769505+0000", 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 196.46682029300001, 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.082168621999999997, 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.247 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769505+0000", 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769505+0000", 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769506+0000", 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.3300000000000002e-07 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.248 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769560+0000", 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3934999999999998e-05 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769570+0000", 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0550000000000001e-05 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769577+0000", 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.922e-06 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.249 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769621+0000", 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.3331000000000002e-05 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.769639+0000", 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8666000000000001e-05 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.770303+0000", 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000663636 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.250 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.851674+0000", 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.081370948999999998 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:6050 10.d e353/348 10:b0fc312e:::benchmark_data_smithi045_169175_object6049:head v 353'366, mlcod=353'364)", 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:03.770361+0000", 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 196.46596487400001, 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.082769639000000006, 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.251 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.770361+0000", 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.770361+0000", 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.770362+0000", 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.9800000000000003e-07 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.252 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.770407+0000", 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.5080000000000002e-05 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.851627+0000", 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.081219956999999995 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.851644+0000", 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7357999999999999e-05 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.253 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.851710+0000", 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6288e-05 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.851730+0000", 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0038999999999999e-05 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.853102+0000", 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001371452 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.254 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.255 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.255 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.853130+0000", 2024-03-11T22:26:20.255 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.8666999999999998e-05 2024-03-11T22:26:20.255 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.255 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.255 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.255 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.255 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.255 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:6056 10.2 e353/348 10:47de54f1:::benchmark_data_smithi045_169175_object6055:head v 353'420, mlcod=353'416)", 2024-03-11T22:26:20.255 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:23:03.771306+0000", 2024-03-11T22:26:20.255 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 196.46501924699999, 2024-03-11T22:26:20.255 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.081142977000000005, 2024-03-11T22:26:20.255 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.256 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.256 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.256 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.256 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.771306+0000", 2024-03-11T22:26:20.256 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.256 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.256 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.256 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.771306+0000", 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.771307+0000", 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0800000000000004e-07 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.771362+0000", 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.4824000000000002e-05 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.257 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.851626+0000", 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.080263714999999999 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.851644+0000", 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8297000000000001e-05 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.851698+0000", 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.3807999999999997e-05 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.258 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.851717+0000", 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8804000000000002e-05 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.852409+0000", 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000692839 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:23:03.852449+0000", 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.9981999999999998e-05 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.259 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:145028 10.1 e375/348 10:8c1655b8:::benchmark_data_smithi045_169175_object13001:head v 375'9069, mlcod=375'9067)", 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:24:41.953299+0000", 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 98.283026782999997, 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.066437450999999995, 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:24:41.953299+0000", 2024-03-11T22:26:20.260 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:24:41.953299+0000", 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:24:41.953299+0000", 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.34e-07 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:24:41.953302+0000", 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.0939999999999999e-06 2024-03-11T22:26:20.261 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:24:41.953307+0000", 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.0719999999999999e-06 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:24:41.953311+0000", 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.5200000000000002e-06 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:24:41.953345+0000", 2024-03-11T22:26:20.262 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.4134000000000002e-05 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:24:41.953358+0000", 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.261e-05 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:24:41.955107+0000", 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.001749684 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:24:42.019736+0000", 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.064628903000000001 2024-03-11T22:26:20.263 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(osd.0.0:221002 10.1 e385/348 10:8ab186e0:::benchmark_data_smithi045_169175_object107242:head v 385'15299, mlcod=385'15270)", 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:25:22.194754+0000", 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 58.041571273000002, 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.043716923999999997, 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.194754+0000", 2024-03-11T22:26:20.264 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.194754+0000", 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.194755+0000", 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8199999999999999e-07 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.194761+0000", 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.609e-06 2024-03-11T22:26:20.265 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.194773+0000", 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2451e-05 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.194780+0000", 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.0380000000000002e-06 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.216501+0000", 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.021721120999999999 2024-03-11T22:26:20.266 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.216520+0000", 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8335999999999999e-05 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.238409+0000", 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.021889077 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.238471+0000", 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2609999999999999e-05 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.267 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(osd.0.0:221003 10.1 e385/348 10:8ab31a13:::benchmark_data_smithi045_169175_object91785:head v 385'15300, mlcod=385'15270)", 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:25:22.195866+0000", 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 58.040460162999999, 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.045469081000000001, 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.195866+0000", 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.268 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.195866+0000", 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.195866+0000", 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8400000000000005e-07 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.195871+0000", 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7720000000000002e-06 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.269 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.195880+0000", 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9840000000000007e-06 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.195886+0000", 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2840000000000003e-06 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.219407+0000", 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.023521180999999999 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.270 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.219425+0000", 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7571000000000002e-05 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.241293+0000", 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.021868519999999999 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.241335+0000", 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.1285000000000002e-05 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.271 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:189878 10.1 e385/348 10:82ed4f42:::benchmark_data_smithi045_169175_object57851:head v 385'15301, mlcod=385'15289)", 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:25:22.201089+0000", 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 58.035236626, 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.040274103999999998, 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.201089+0000", 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.272 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.201089+0000", 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.201089+0000", 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6199999999999998e-07 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.201094+0000", 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0670000000000002e-06 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.273 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.201101+0000", 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3810000000000001e-06 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.201107+0000", 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.6210000000000001e-06 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.222418+0000", 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.021311093999999999 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.274 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.222435+0000", 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7728999999999999e-05 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.241338+0000", 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.018902935999999999 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:22.241363+0000", 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4814e-05 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.275 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(osd.0.0:221647 10.1 e385/348 10:8b8615da:::benchmark_data_smithi045_169175_object108143:head v 385'15636, mlcod=385'15623)", 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:25:23.692692+0000", 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 56.543633988000003, 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.041474265000000003, 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.692692+0000", 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.276 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.692692+0000", 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.692693+0000", 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.09e-07 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.692699+0000", 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.2210000000000002e-06 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.277 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.692713+0000", 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4647000000000001e-05 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.692721+0000", 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2790000000000001e-06 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.696380+0000", 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0036588440000000001 2024-03-11T22:26:20.278 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.696428+0000", 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.8624000000000001e-05 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.734062+0000", 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.037634293999999999 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.734166+0000", 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000103447 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.279 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:191584 10.1 e385/348 10:8a52c7f3:::benchmark_data_smithi045_169175_object59557:head v 385'15656, mlcod=385'15650)", 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:25:23.749763+0000", 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 56.486562270999997, 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.040463008000000002, 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.749763+0000", 2024-03-11T22:26:20.280 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.749763+0000", 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.749764+0000", 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.61e-07 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.281 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.749771+0000", 2024-03-11T22:26:20.282 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:20.282 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6510000000000004e-06 2024-03-11T22:26:20.282 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.282 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.282 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.282 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.749784+0000", 2024-03-11T22:26:20.282 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.3407999999999999e-05 2024-03-11T22:26:20.282 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.749791+0000", 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1890000000000001e-06 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.750773+0000", 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00098197600000000007 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.750791+0000", 2024-03-11T22:26:20.283 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7396000000000001e-05 2024-03-11T22:26:20.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.284 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.284 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.284 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.790148+0000", 2024-03-11T22:26:20.284 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.039357401 2024-03-11T22:26:20.284 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.284 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.285 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.285 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.790226+0000", 2024-03-11T22:26:20.285 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.8226000000000006e-05 2024-03-11T22:26:20.285 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.285 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.285 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:191637 10.1 e385/348 10:8726f8a7:::benchmark_data_smithi045_169175_object59610:head v 385'15662, mlcod=385'15654)", 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:25:23.786372+0000", 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 56.449953483000002, 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.046788940000000001, 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.786372+0000", 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.286 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.786372+0000", 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.786373+0000", 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.063e-06 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.786380+0000", 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9120000000000001e-06 2024-03-11T22:26:20.287 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.786395+0000", 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5099999999999999e-05 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.786403+0000", 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.0590000000000003e-06 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.798641+0000", 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.01223749 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.288 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.798659+0000", 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8260000000000001e-05 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.833043+0000", 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.034384297000000001 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.833161+0000", 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.000117759 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.289 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(osd.0.0:221692 10.1 e385/348 10:8b9365f9:::benchmark_data_smithi045_169175_object90144:head v 385'15663, mlcod=385'15654)", 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:25:23.791560+0000", 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 56.444765904999997, 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.041633326999999998, 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.791560+0000", 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.290 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.791560+0000", 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.791561+0000", 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.2500000000000004e-07 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.791567+0000", 2024-03-11T22:26:20.291 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.6749999999999996e-06 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.791583+0000", 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5732000000000001e-05 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.791590+0000", 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3479999999999998e-06 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.801558+0000", 2024-03-11T22:26:20.292 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0099670680000000008 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.801582+0000", 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.4756000000000002e-05 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.833164+0000", 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.031581811000000001 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:23.833193+0000", 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9011999999999999e-05 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.293 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:192063 10.1 e385/348 10:83161436:::benchmark_data_smithi045_169175_object60036:head v 385'15770, mlcod=385'15768)", 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:25:24.227014+0000", 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 56.009311461999999, 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.039579463000000002, 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.227014+0000", 2024-03-11T22:26:20.294 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.227014+0000", 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.227015+0000", 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.6199999999999998e-07 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.227024+0000", 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.9509999999999995e-06 2024-03-11T22:26:20.295 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.296 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.296 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.296 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.227031+0000", 2024-03-11T22:26:20.296 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4560000000000003e-06 2024-03-11T22:26:20.296 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.296 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.296 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.296 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.227036+0000", 2024-03-11T22:26:20.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:20.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:20.297 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.7049999999999998e-06 2024-03-11T22:26:20.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.298 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.298 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.298 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.247355+0000", 2024-03-11T22:26:20.298 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.020319272999999999 2024-03-11T22:26:20.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.298 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.298 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.298 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.247375+0000", 2024-03-11T22:26:20.298 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.0389000000000001e-05 2024-03-11T22:26:20.298 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.298 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.266450+0000", 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.019074244000000001 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.266594+0000", 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00014398300000000001 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(osd.0.0:221890 10.1 e385/348 10:8bdedcab:::benchmark_data_smithi045_169175_object74430:head v 385'15778, mlcod=385'15768)", 2024-03-11T22:26:20.299 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:25:24.260224+0000", 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 55.976101774999997, 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.039362670000000002, 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.260224+0000", 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.260224+0000", 2024-03-11T22:26:20.300 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.260225+0000", 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5499999999999997e-07 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.260231+0000", 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.9959999999999999e-06 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.301 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.260246+0000", 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.5699e-05 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.260254+0000", 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.4379999999999998e-06 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.266957+0000", 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0067031030000000002 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.302 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.266974+0000", 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.7306e-05 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.299518+0000", 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.032543571 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.299587+0000", 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8801999999999996e-05 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.303 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:192173 10.1 e385/348 10:8c1cce2e:::benchmark_data_smithi045_169175_object60146:head v 385'15803, mlcod=385'15789)", 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:25:24.329486+0000", 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 55.906839515000001, 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.039932576999999997, 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.329486+0000", 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.304 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.329486+0000", 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.329487+0000", 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9800000000000003e-07 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.329493+0000", 2024-03-11T22:26:20.305 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7740000000000001e-06 2024-03-11T22:26:20.306 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.306 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.306 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.306 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.329501+0000", 2024-03-11T22:26:20.306 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.8410000000000006e-06 2024-03-11T22:26:20.306 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.306 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.306 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.306 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.329508+0000", 2024-03-11T22:26:20.306 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.7680000000000003e-06 2024-03-11T22:26:20.306 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.306 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.306 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.336972+0000", 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0074634269999999999 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.337010+0000", 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8532000000000003e-05 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.369379+0000", 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.032368314000000002 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.307 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.369419+0000", 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 4.0222999999999998e-05 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(osd.0.0:222152 10.1 e385/348 10:8c37ab4c:::benchmark_data_smithi045_169175_object126062:head v 385'15920, mlcod=385'15905)", 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:25:24.837912+0000", 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 55.398413857000001, 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.043757167, 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.308 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.837912+0000", 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.837912+0000", 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.837913+0000", 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0100000000000001e-06 2024-03-11T22:26:20.309 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.837920+0000", 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5950000000000001e-06 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.837931+0000", 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.0854e-05 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.837939+0000", 2024-03-11T22:26:20.310 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.3520000000000001e-06 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.841328+0000", 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0033896019999999998 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.841349+0000", 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.1188000000000001e-05 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.311 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.881356+0000", 2024-03-11T22:26:20.312 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.040006892000000002 2024-03-11T22:26:20.312 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.312 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.312 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.312 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.881669+0000", 2024-03-11T22:26:20.312 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00031267399999999998 2024-03-11T22:26:20.312 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.312 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.312 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.312 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(osd.0.0:222153 10.1 e385/348 10:8c37d1c9:::benchmark_data_smithi045_169175_object81543:head v 385'15921, mlcod=385'15905)", 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:25:24.841597+0000", 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 55.394728688000001, 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.040106649000000001, 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.841597+0000", 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.313 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.841597+0000", 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.841598+0000", 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.78e-07 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.841605+0000", 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6410000000000002e-06 2024-03-11T22:26:20.314 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.841616+0000", 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.093e-05 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.841624+0000", 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.6340000000000001e-06 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.844090+0000", 2024-03-11T22:26:20.315 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0024662489999999998 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.844163+0000", 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.2521000000000006e-05 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.881674+0000", 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.037511559 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:24.881704+0000", 2024-03-11T22:26:20.316 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 2.9237000000000001e-05 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:193876 10.1 e385/348 10:8bdf7f62:::benchmark_data_smithi045_169175_object61849:head v 385'16193, mlcod=385'16189)", 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:25:25.920866+0000", 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 54.315459644000001, 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.041023451000000002, 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.317 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:25.920866+0000", 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:25.920866+0000", 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:25.920867+0000", 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.7700000000000004e-07 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.318 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:25.920872+0000", 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.2830000000000001e-06 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:25.920885+0000", 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.2887999999999999e-05 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:25.920891+0000", 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.674e-06 2024-03-11T22:26:20.319 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:25.923938+0000", 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0030468819999999999 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:25.923958+0000", 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.9534e-05 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:25.961796+0000", 2024-03-11T22:26:20.320 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.037838003000000002 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:25:25.961889+0000", 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.3610000000000007e-05 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:257020 10.c e399/348 10:3f47d994:::benchmark_data_smithi045_169175_object124993:head v 399'16016, mlcod=399'16014)", 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:26:18.211243+0000", 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 2.0250829650000002, 2024-03-11T22:26:20.321 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.026586217999999998, 2024-03-11T22:26:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:18.211243+0000", 2024-03-11T22:26:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:18.211243+0000", 2024-03-11T22:26:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:18.211243+0000", 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.9299999999999997e-07 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:18.211249+0000", 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 5.356e-06 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:18.211263+0000", 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.4741e-05 2024-03-11T22:26:20.323 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:18.211271+0000", 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.5399999999999998e-06 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:18.219877+0000", 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0086062139999999992 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:18.219896+0000", 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8856999999999999e-05 2024-03-11T22:26:20.324 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:18.237561+0000", 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.017665146 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:18.237829+0000", 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.00026767099999999998 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.325 INFO:teuthology.orchestra.run.smithi144.stdout: "description": "osd_repop(client.15540.0:259179 10.6 e401/348 10:6f6721ed:::benchmark_data_smithi045_169175_object127152:head v 401'16242, mlcod=401'16239)", 2024-03-11T22:26:20.326 INFO:teuthology.orchestra.run.smithi144.stdout: "initiated_at": "2024-03-11T22:26:19.974716+0000", 2024-03-11T22:26:20.326 INFO:teuthology.orchestra.run.smithi144.stdout: "age": 0.26161003799999999, 2024-03-11T22:26:20.326 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.025232791000000001, 2024-03-11T22:26:20.326 INFO:teuthology.orchestra.run.smithi144.stdout: "type_data": { 2024-03-11T22:26:20.326 INFO:teuthology.orchestra.run.smithi144.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:26:20.326 INFO:teuthology.orchestra.run.smithi144.stdout: "events": [ 2024-03-11T22:26:20.326 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.326 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "initiated", 2024-03-11T22:26:20.326 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:19.974716+0000", 2024-03-11T22:26:20.326 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.326 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.326 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.326 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "header_read", 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:19.974716+0000", 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "throttled", 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:19.974716+0000", 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 8.1900000000000001e-07 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "all_read", 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:19.974724+0000", 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 7.1389999999999998e-06 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "dispatched", 2024-03-11T22:26:20.327 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:19.974734+0000", 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 9.9429999999999995e-06 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "queued_for_pg", 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:19.974740+0000", 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 6.8399999999999997e-06 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "reached_pg", 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:19.977382+0000", 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.0026421080000000002 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "started", 2024-03-11T22:26:20.328 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:19.977401+0000", 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 1.8722000000000001e-05 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "commit_sent", 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:19.999910+0000", 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 0.022508878 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: }, 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: { 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: "event": "done", 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: "time": "2024-03-11T22:26:19.999948+0000", 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: "duration": 3.8342e-05 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: } 2024-03-11T22:26:20.329 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-03-11T22:26:20.330 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:26:20.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:20.455 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:20.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:20.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:20.557 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:20.589 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:20.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:20.598 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:20 smithi144 ceph-mon[33984]: osdmap e401: 8 total, 8 up, 8 in 2024-03-11T22:26:20.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:26:20.692 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:20.720 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:20 smithi045 ceph-mon[30178]: osdmap e401: 8 total, 8 up, 8 in 2024-03-11T22:26:20.720 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:20 smithi045 ceph-mon[39715]: osdmap e401: 8 total, 8 up, 8 in 2024-03-11T22:26:20.720 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:20.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:20.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:20.822 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:20.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:20.850 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:20.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:20.951 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:20.989 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:20.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:21.090 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:21.128 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:21.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:21.229 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:21.264 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:21.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:21.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:21.366 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:21.399 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:21.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:21.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:21.501 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:21.536 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:21.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:21.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:21.637 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:21.661 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:21 smithi045 ceph-mon[30178]: pgmap v831: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 577 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 868 B/s rd, 0 B/s wr, 1.07k op/s; 54 KiB/s, 0 objects/s recovering 2024-03-11T22:26:21.662 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:21 smithi045 ceph-mon[30178]: osdmap e402: 8 total, 8 up, 8 in 2024-03-11T22:26:21.662 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:21 smithi045 ceph-mon[39715]: pgmap v831: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 577 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 868 B/s rd, 0 B/s wr, 1.07k op/s; 54 KiB/s, 0 objects/s recovering 2024-03-11T22:26:21.662 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:21 smithi045 ceph-mon[39715]: osdmap e402: 8 total, 8 up, 8 in 2024-03-11T22:26:21.685 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:21.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:21.787 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:21.817 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:21.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:21.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:21.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:21 smithi144 ceph-mon[33984]: pgmap v831: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 577 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 868 B/s rd, 0 B/s wr, 1.07k op/s; 54 KiB/s, 0 objects/s recovering 2024-03-11T22:26:21.848 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:21 smithi144 ceph-mon[33984]: osdmap e402: 8 total, 8 up, 8 in 2024-03-11T22:26:21.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:21.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:21.956 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:21.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:22.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:22.091 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:22.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:22.193 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:22.228 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:22.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:22.330 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:22.363 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:22.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:22.465 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:22.501 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:22.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:22.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:22.638 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:22.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:22.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:22.777 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:22.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:22.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:22.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:22 smithi144 ceph-mon[33984]: osdmap e403: 8 total, 8 up, 8 in 2024-03-11T22:26:22.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:22 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-11T22:26:22.879 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:22.912 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:22.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:23.014 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:23.032 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T22:26:23.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:22 smithi045 ceph-mon[30178]: osdmap e403: 8 total, 8 up, 8 in 2024-03-11T22:26:23.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:22 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-11T22:26:23.035 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:26:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:26:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:26:23.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:22 smithi045 ceph-mon[39715]: osdmap e403: 8 total, 8 up, 8 in 2024-03-11T22:26:23.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:22 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-03-11T22:26:23.054 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:23.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:23.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:23.156 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:23.193 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:23.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:23.294 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:23.327 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:23.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:23.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:23.429 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:23.468 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:23.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:23.505 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:26:23.570 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:23.605 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:23.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:23.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:23.749 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:23.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:23.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:23 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:26:23.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:23 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:26:23.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:23 smithi045 ceph-mon[39715]: pgmap v834: 167 pgs: 2 active+remapped+backfill_toofull, 165 active+clean; 297 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-11T22:26:23.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:23 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-11T22:26:23.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:23 smithi045 ceph-mon[39715]: osdmap e404: 8 total, 8 up, 8 in 2024-03-11T22:26:23.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:23 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:26:23.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:23 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:26:23.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:23 smithi045 ceph-mon[30178]: pgmap v834: 167 pgs: 2 active+remapped+backfill_toofull, 165 active+clean; 297 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-11T22:26:23.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:23 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-11T22:26:23.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:23 smithi045 ceph-mon[30178]: osdmap e404: 8 total, 8 up, 8 in 2024-03-11T22:26:23.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:23 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:26:23.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:23 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:26:23.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:23 smithi144 ceph-mon[33984]: pgmap v834: 167 pgs: 2 active+remapped+backfill_toofull, 165 active+clean; 297 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-11T22:26:23.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:23 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-03-11T22:26:23.839 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:23 smithi144 ceph-mon[33984]: osdmap e404: 8 total, 8 up, 8 in 2024-03-11T22:26:23.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:23.890 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:23.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:23.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:23.916 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Removed 132024 objects 2024-03-11T22:26:23.917 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Clean up completed and total clean up time :112.078 2024-03-11T22:26:23.992 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:24.018 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:24.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:24.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:24.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:24.142 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:24.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:24.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:24.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:24.250 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:26:24.251 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T22:26:24.251 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:26:24.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:24.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:24.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:24.371 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:24.395 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:24.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:24.497 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:24.520 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:24.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:24.577 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:26:24.622 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:24.646 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:24.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:24.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:24.748 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:24.751 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:24 smithi144 ceph-mon[33984]: osdmap e405: 8 total, 8 up, 8 in 2024-03-11T22:26:24.751 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:24 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2474017838' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:26:24.773 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:24.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:24.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:24.868 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:24 smithi045 ceph-mon[39715]: osdmap e405: 8 total, 8 up, 8 in 2024-03-11T22:26:24.868 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:24 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2474017838' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:26:24.868 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:24 smithi045 ceph-mon[30178]: osdmap e405: 8 total, 8 up, 8 in 2024-03-11T22:26:24.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:24 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2474017838' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:26:24.875 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:24.899 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:24.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:25.000 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:25.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:25.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:25.117 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:25.127 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:25.150 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:25.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:25.252 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:25.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:25.276 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:25.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:25.350 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:26:25.350 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-03-11T22:26:25.350 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-03-11T22:26:25.350 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:26:25.376 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:25.399 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:25.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:25.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:25.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:25.527 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:25.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:25.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:25.629 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:25.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:25.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:25.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:25.668 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:25 smithi144 ceph-mon[33984]: pgmap v837: 167 pgs: 2 active+remapped+backfill_toofull, 165 active+clean; 202 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-03-11T22:26:25.668 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:25 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:26:25.668 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:25 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2474017838' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:26:25.668 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:25 smithi144 ceph-mon[33984]: osdmap e406: 8 total, 8 up, 8 in 2024-03-11T22:26:25.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:25.778 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:25.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:25.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:25.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:25 smithi045 ceph-mon[39715]: pgmap v837: 167 pgs: 2 active+remapped+backfill_toofull, 165 active+clean; 202 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-03-11T22:26:25.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:25 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:26:25.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:25 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2474017838' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:26:25.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:25 smithi045 ceph-mon[39715]: osdmap e406: 8 total, 8 up, 8 in 2024-03-11T22:26:25.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:25 smithi045 ceph-mon[30178]: pgmap v837: 167 pgs: 2 active+remapped+backfill_toofull, 165 active+clean; 202 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-03-11T22:26:25.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:25 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:26:25.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:25 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2474017838' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:26:25.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:25 smithi045 ceph-mon[30178]: osdmap e406: 8 total, 8 up, 8 in 2024-03-11T22:26:25.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:25.904 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:25.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:25.960 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T22:26:26.005 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:26.028 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:26.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:26.130 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:26.154 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:26.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:26.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:26.221 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:26:26.256 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:26.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:26.281 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:26.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:26.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:26.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:26.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:26.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:26.508 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:26.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:26.532 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:26.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:26.599 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:26 smithi144 ceph-mon[33984]: osdmap e407: 8 total, 8 up, 8 in 2024-03-11T22:26:26.599 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:26 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-11T22:26:26.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:26.658 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:26.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:26.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:26.782 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:26.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:26.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:26 smithi045 ceph-mon[30178]: osdmap e407: 8 total, 8 up, 8 in 2024-03-11T22:26:26.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:26 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-11T22:26:26.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:26 smithi045 ceph-mon[39715]: osdmap e407: 8 total, 8 up, 8 in 2024-03-11T22:26:26.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:26 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-03-11T22:26:26.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:26.909 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:26.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:27.010 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:27.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:27.035 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:27.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:27.099 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:26:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:26:27.136 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:27.208 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:27.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:27.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:27.332 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:27.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:27.434 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:27.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:27.459 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:27.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:27.560 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:27.586 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:27.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:27.688 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:27.713 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:27.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:27.815 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:27.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:27 smithi045 ceph-mon[39715]: pgmap v840: 166 pgs: 2 active+remapped+backfill_toofull, 164 active+clean; 33 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-03-11T22:26:27.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:27 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-11T22:26:27.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:27 smithi045 ceph-mon[39715]: osdmap e408: 8 total, 8 up, 8 in 2024-03-11T22:26:27.818 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:27.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:27 smithi045 ceph-mon[30178]: pgmap v840: 166 pgs: 2 active+remapped+backfill_toofull, 164 active+clean; 33 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-03-11T22:26:27.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:27 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-11T22:26:27.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:27 smithi045 ceph-mon[30178]: osdmap e408: 8 total, 8 up, 8 in 2024-03-11T22:26:27.819 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:27.834 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:27 smithi144 ceph-mon[33984]: pgmap v840: 166 pgs: 2 active+remapped+backfill_toofull, 164 active+clean; 33 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-03-11T22:26:27.835 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:27 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-03-11T22:26:27.835 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:27 smithi144 ceph-mon[33984]: osdmap e408: 8 total, 8 up, 8 in 2024-03-11T22:26:27.835 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:27.841 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:27.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:27.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:27.886 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:26:27.893 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:26:27.908 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:26:27.928 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:26:27.943 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:27.948 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:26:27.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:27.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:27.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:27.969 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:26:27.989 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:26:28.009 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:26:28.031 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:26:28.031 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:26:28.031 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:26:28.031 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:26:28.031 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:26:28.032 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:26:28.032 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:26:28.032 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:26:28.032 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:26:28.068 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:28.092 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:28.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:28.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:28.217 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:28.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:28.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:28.319 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:28.343 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:28.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:28.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:28.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:28.469 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:28.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:28.540 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:26:28.540 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:26:28.571 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:28.598 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:28.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:28.699 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:28.724 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:28.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:28.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:28 smithi045 ceph-mon[39715]: osdmap e409: 8 total, 8 up, 8 in 2024-03-11T22:26:28.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:28 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3780049793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:28.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:28 smithi045 ceph-mon[30178]: osdmap e409: 8 total, 8 up, 8 in 2024-03-11T22:26:28.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:28 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3780049793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:28.826 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:28.834 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:28 smithi144 ceph-mon[33984]: osdmap e409: 8 total, 8 up, 8 in 2024-03-11T22:26:28.834 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:28 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3780049793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:28.850 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:28.850 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:28.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:28.896 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-11T22:26:28.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-11T22:26:28.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:28.976 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:28.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:28.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:29.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:29.103 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:29.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:29.144 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T22:26:29.204 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:29.252 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:29.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:29.353 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:29.377 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:29.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:29.448 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:26:29.478 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:29.501 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:29.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:29.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:29.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:29.629 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:29.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:29.730 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:29.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:29.755 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:29.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:29.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:29 smithi045 ceph-mon[39715]: pgmap v843: 166 pgs: 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 162 active+clean; 582 KiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 805 op/s 2024-03-11T22:26:29.781 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:29 smithi045 ceph-mon[39715]: osdmap e410: 8 total, 8 up, 8 in 2024-03-11T22:26:29.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:29 smithi045 ceph-mon[30178]: pgmap v843: 166 pgs: 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 162 active+clean; 582 KiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 805 op/s 2024-03-11T22:26:29.781 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:29 smithi045 ceph-mon[30178]: osdmap e410: 8 total, 8 up, 8 in 2024-03-11T22:26:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:29 smithi144 ceph-mon[33984]: pgmap v843: 166 pgs: 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 162 active+clean; 582 KiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 805 op/s 2024-03-11T22:26:29.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:29 smithi144 ceph-mon[33984]: osdmap e410: 8 total, 8 up, 8 in 2024-03-11T22:26:29.856 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:29.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:29.881 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:29.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:29.895 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:26:29.896 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:26:29.896 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-11T22:26:29.896 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T22:26:29.896 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T22:26:29.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-11T22:26:29.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:30.006 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:30.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:30.110 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:30.128 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:30.134 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:30.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:30.159 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:30.236 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:30.261 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:30.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:30.362 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:30.388 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:30.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:30.489 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:30.514 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:30.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:30.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:30.616 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:30.640 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:30.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:30.658 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:30 smithi045 ceph-mon[30178]: osdmap e411: 8 total, 8 up, 8 in 2024-03-11T22:26:30.658 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:30 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-11T22:26:30.658 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:30 smithi045 ceph-mon[39715]: osdmap e411: 8 total, 8 up, 8 in 2024-03-11T22:26:30.659 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:30 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-11T22:26:30.741 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:30.766 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:30.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:30.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:26:30.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:30 smithi144 ceph-mon[33984]: osdmap e411: 8 total, 8 up, 8 in 2024-03-11T22:26:30.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:30 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-03-11T22:26:30.869 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:30.908 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:30.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:31.010 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:31.035 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:31.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:31.137 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:31.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:31.161 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:31.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:31.262 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:31.268 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:26:31.268 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:26:31.287 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:31.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:31.389 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:31.412 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:31.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:31.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:31.514 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:31.537 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:31.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:31.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:31.610 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:26:31.639 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:31.663 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:31.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:31.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:31.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:31 smithi045 ceph-mon[39715]: pgmap v846: 165 pgs: 1 clean+premerge+peered, 2 active+remapped+backfill_toofull, 162 active+clean; 582 KiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 131 op/s 2024-03-11T22:26:31.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-11T22:26:31.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:31 smithi045 ceph-mon[39715]: osdmap e412: 8 total, 8 up, 8 in 2024-03-11T22:26:31.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:31.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:31.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:31 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1168593288' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:26:31.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:31 smithi045 ceph-mon[30178]: pgmap v846: 165 pgs: 1 clean+premerge+peered, 2 active+remapped+backfill_toofull, 162 active+clean; 582 KiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 131 op/s 2024-03-11T22:26:31.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-11T22:26:31.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:31 smithi045 ceph-mon[30178]: osdmap e412: 8 total, 8 up, 8 in 2024-03-11T22:26:31.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:31.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:31.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:31 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1168593288' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:26:31.789 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:31.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:31.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:31 smithi144 ceph-mon[33984]: pgmap v846: 165 pgs: 1 clean+premerge+peered, 2 active+remapped+backfill_toofull, 162 active+clean; 582 KiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 131 op/s 2024-03-11T22:26:31.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-03-11T22:26:31.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:31 smithi144 ceph-mon[33984]: osdmap e412: 8 total, 8 up, 8 in 2024-03-11T22:26:31.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:31.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:31.842 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:31 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1168593288' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:26:31.868 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T22:26:31.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:31.915 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:31.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:32.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:32.040 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:32.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:32.111 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:26:32.141 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:32.164 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:32.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:32.266 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:32.290 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:32.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:32.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:32.392 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:32.417 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:32.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:32.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:32.543 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:32.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:32.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:32.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:32.770 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:32.794 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:32.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:32.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:32.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:32 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:26:32.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:32 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1168593288' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:26:32.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:32 smithi144 ceph-mon[33984]: osdmap e413: 8 total, 8 up, 8 in 2024-03-11T22:26:32.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:32 smithi144 ceph-mon[33984]: 1.e scrub starts 2024-03-11T22:26:32.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:32 smithi144 ceph-mon[33984]: 1.e scrub ok 2024-03-11T22:26:32.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:32 smithi144 ceph-mon[33984]: 2.2 scrub starts 2024-03-11T22:26:32.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:32 smithi144 ceph-mon[33984]: 2.2 scrub ok 2024-03-11T22:26:32.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:32 smithi144 ceph-mon[33984]: osdmap e414: 8 total, 8 up, 8 in 2024-03-11T22:26:32.896 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:32.919 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:32.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:33.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:33.035 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:26:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:26:33.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:26:33.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1168593288' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:26:33.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[30178]: osdmap e413: 8 total, 8 up, 8 in 2024-03-11T22:26:33.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[30178]: 1.e scrub starts 2024-03-11T22:26:33.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[30178]: 1.e scrub ok 2024-03-11T22:26:33.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[30178]: 2.2 scrub starts 2024-03-11T22:26:33.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[30178]: 2.2 scrub ok 2024-03-11T22:26:33.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[30178]: osdmap e414: 8 total, 8 up, 8 in 2024-03-11T22:26:33.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:26:33.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1168593288' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:26:33.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[39715]: osdmap e413: 8 total, 8 up, 8 in 2024-03-11T22:26:33.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[39715]: 1.e scrub starts 2024-03-11T22:26:33.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[39715]: 1.e scrub ok 2024-03-11T22:26:33.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[39715]: 2.2 scrub starts 2024-03-11T22:26:33.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[39715]: 2.2 scrub ok 2024-03-11T22:26:33.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:32 smithi045 ceph-mon[39715]: osdmap e414: 8 total, 8 up, 8 in 2024-03-11T22:26:33.047 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:33.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:33.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:33.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:33.173 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:33.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:33.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:33.274 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:33.298 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:33.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:33.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:33.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:33.425 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:33.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:33.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:33.527 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:33.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:33.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:33.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:33.653 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:33.677 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:33.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:33.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:33.779 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:33 smithi045 ceph-mon[39715]: pgmap v849: 165 pgs: 2 peering, 1 clean+premerge+peered, 2 active+remapped+backfill_toofull, 160 active+clean; 582 KiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:33 smithi045 ceph-mon[39715]: 1.2 scrub starts 2024-03-11T22:26:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:33 smithi045 ceph-mon[39715]: 1.2 scrub ok 2024-03-11T22:26:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:33 smithi045 ceph-mon[39715]: osdmap e415: 8 total, 8 up, 8 in 2024-03-11T22:26:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:33 smithi045 ceph-mon[30178]: pgmap v849: 165 pgs: 2 peering, 1 clean+premerge+peered, 2 active+remapped+backfill_toofull, 160 active+clean; 582 KiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:33 smithi045 ceph-mon[30178]: 1.2 scrub starts 2024-03-11T22:26:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:33 smithi045 ceph-mon[30178]: 1.2 scrub ok 2024-03-11T22:26:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:33 smithi045 ceph-mon[30178]: osdmap e415: 8 total, 8 up, 8 in 2024-03-11T22:26:33.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:33.806 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:33.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:33.817 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:33 smithi144 ceph-mon[33984]: pgmap v849: 165 pgs: 2 peering, 1 clean+premerge+peered, 2 active+remapped+backfill_toofull, 160 active+clean; 582 KiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:33.817 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:33 smithi144 ceph-mon[33984]: 1.2 scrub starts 2024-03-11T22:26:33.817 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:33 smithi144 ceph-mon[33984]: 1.2 scrub ok 2024-03-11T22:26:33.817 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:33 smithi144 ceph-mon[33984]: osdmap e415: 8 total, 8 up, 8 in 2024-03-11T22:26:33.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:33.933 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:33.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:34.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:34.059 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:34.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:34.160 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:34.184 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:34.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:34.193 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:26:34.193 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:26:34.286 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:34.310 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:34.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:34.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:34.438 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:34.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:34.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:34.564 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:34.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:34.634 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'unique_pool_3' removed 2024-03-11T22:26:34.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:34.719 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:34.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:34.799 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:34 smithi144 ceph-mon[33984]: 6.4 scrub starts 2024-03-11T22:26:34.800 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:34 smithi144 ceph-mon[33984]: 6.4 scrub ok 2024-03-11T22:26:34.800 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:34 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:26:34.800 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:34 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:26:34.800 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:34 smithi144 ceph-mon[33984]: 1.1 scrub starts 2024-03-11T22:26:34.800 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:34 smithi144 ceph-mon[33984]: 1.1 scrub ok 2024-03-11T22:26:34.800 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:34 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:26:34.800 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:34 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:26:34.800 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:34 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-11T22:26:34.800 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:34 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/528946914' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:26:34.821 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:34.826 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T22:26:34.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:34.846 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:34.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:34.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[39715]: 6.4 scrub starts 2024-03-11T22:26:34.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[39715]: 6.4 scrub ok 2024-03-11T22:26:34.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:26:34.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:26:34.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[39715]: 1.1 scrub starts 2024-03-11T22:26:34.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[39715]: 1.1 scrub ok 2024-03-11T22:26:34.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:26:34.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:26:34.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-11T22:26:34.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/528946914' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:26:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[30178]: 6.4 scrub starts 2024-03-11T22:26:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[30178]: 6.4 scrub ok 2024-03-11T22:26:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:26:34.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:26:34.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[30178]: 1.1 scrub starts 2024-03-11T22:26:34.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[30178]: 1.1 scrub ok 2024-03-11T22:26:34.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:26:34.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:26:34.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-03-11T22:26:34.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:34 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/528946914' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:26:34.947 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:34.970 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:34.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:35.072 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:35.087 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:26:35.096 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:35.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:35.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:35.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:26:35.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:35.223 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:35.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:35.324 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:35.350 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:35.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:35.450 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:35.475 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:35.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:35.576 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:35.605 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:35.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:35.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:35.667 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T22:26:35.667 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T22:26:35.667 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-11T22:26:35.667 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-11T22:26:35.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:35.737 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:35.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:35.745 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:35 smithi144 ceph-mon[33984]: 6.1b scrub starts 2024-03-11T22:26:35.745 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:35 smithi144 ceph-mon[33984]: 6.1b scrub ok 2024-03-11T22:26:35.745 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:35 smithi144 ceph-mon[33984]: pgmap v852: 164 pgs: 1 peering, 2 active+remapped+backfill_toofull, 161 active+clean; 582 KiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:35.746 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:35 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-11T22:26:35.746 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:35 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/528946914' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:26:35.746 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:35 smithi144 ceph-mon[33984]: osdmap e416: 8 total, 8 up, 8 in 2024-03-11T22:26:35.838 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:35.849 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:26:35.849 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "" 2024-03-11T22:26:35.849 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:26:35.862 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:35.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:35.963 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:35.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:35.987 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:35.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:35 smithi045 ceph-mon[30178]: 6.1b scrub starts 2024-03-11T22:26:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:35 smithi045 ceph-mon[30178]: 6.1b scrub ok 2024-03-11T22:26:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:35 smithi045 ceph-mon[30178]: pgmap v852: 164 pgs: 1 peering, 2 active+remapped+backfill_toofull, 161 active+clean; 582 KiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:35 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-11T22:26:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:35 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/528946914' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:26:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:35 smithi045 ceph-mon[30178]: osdmap e416: 8 total, 8 up, 8 in 2024-03-11T22:26:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:35 smithi045 ceph-mon[39715]: 6.1b scrub starts 2024-03-11T22:26:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:35 smithi045 ceph-mon[39715]: 6.1b scrub ok 2024-03-11T22:26:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:35 smithi045 ceph-mon[39715]: pgmap v852: 164 pgs: 1 peering, 2 active+remapped+backfill_toofull, 161 active+clean; 582 KiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:35 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-03-11T22:26:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:35 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/528946914' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:26:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:35 smithi045 ceph-mon[39715]: osdmap e416: 8 total, 8 up, 8 in 2024-03-11T22:26:36.049 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:36.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:36.111 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:36.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:36.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:36.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:36.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:36.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:36.339 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:36.363 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:36.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:36.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:36.465 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:36.488 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:36.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:36.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:36.590 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:36.614 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:36.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:36.715 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:36.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[39715]: 6.b scrub starts 2024-03-11T22:26:36.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[39715]: 6.b scrub ok 2024-03-11T22:26:36.728 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[39715]: osdmap e417: 8 total, 8 up, 8 in 2024-03-11T22:26:36.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[39715]: 4.6 scrub starts 2024-03-11T22:26:36.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[39715]: 4.6 scrub ok 2024-03-11T22:26:36.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[39715]: 2.0 scrub starts 2024-03-11T22:26:36.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[39715]: 2.0 scrub ok 2024-03-11T22:26:36.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[39715]: 6.d scrub starts 2024-03-11T22:26:36.729 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[39715]: 6.d scrub ok 2024-03-11T22:26:36.729 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[30178]: 6.b scrub starts 2024-03-11T22:26:36.729 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[30178]: 6.b scrub ok 2024-03-11T22:26:36.729 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[30178]: osdmap e417: 8 total, 8 up, 8 in 2024-03-11T22:26:36.729 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[30178]: 4.6 scrub starts 2024-03-11T22:26:36.729 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[30178]: 4.6 scrub ok 2024-03-11T22:26:36.729 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[30178]: 2.0 scrub starts 2024-03-11T22:26:36.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[30178]: 2.0 scrub ok 2024-03-11T22:26:36.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[30178]: 6.d scrub starts 2024-03-11T22:26:36.730 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:36 smithi045 ceph-mon[30178]: 6.d scrub ok 2024-03-11T22:26:36.738 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:36.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:36.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:36.840 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:36.849 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:26:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:26:36.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:36 smithi144 ceph-mon[33984]: 6.b scrub starts 2024-03-11T22:26:36.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:36 smithi144 ceph-mon[33984]: 6.b scrub ok 2024-03-11T22:26:36.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:36 smithi144 ceph-mon[33984]: osdmap e417: 8 total, 8 up, 8 in 2024-03-11T22:26:36.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:36 smithi144 ceph-mon[33984]: 4.6 scrub starts 2024-03-11T22:26:36.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:36 smithi144 ceph-mon[33984]: 4.6 scrub ok 2024-03-11T22:26:36.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:36 smithi144 ceph-mon[33984]: 2.0 scrub starts 2024-03-11T22:26:36.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:36 smithi144 ceph-mon[33984]: 2.0 scrub ok 2024-03-11T22:26:36.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:36 smithi144 ceph-mon[33984]: 6.d scrub starts 2024-03-11T22:26:36.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:36 smithi144 ceph-mon[33984]: 6.d scrub ok 2024-03-11T22:26:36.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:36.864 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:36.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:36.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:36.990 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:36.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:36.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:37.092 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:37.116 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:37.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:37.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:37.220 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:37.221 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:37.244 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:37.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:37.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:37.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:37.370 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:37.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:37.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:37.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:37.495 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:37.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:37.595 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:26:37.595 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:26:37.596 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:37.620 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:37.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:37.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:37.722 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:37.745 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:37.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:37.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:37.847 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:37.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:37 smithi144 ceph-mon[33984]: pgmap v855: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-03-11T22:26:37.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:37 smithi144 ceph-mon[33984]: osdmap e418: 8 total, 8 up, 8 in 2024-03-11T22:26:37.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:37.874 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:37.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:37.976 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:37.999 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:38.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:38.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:38.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:37 smithi045 ceph-mon[39715]: pgmap v855: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-03-11T22:26:38.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:37 smithi045 ceph-mon[39715]: osdmap e418: 8 total, 8 up, 8 in 2024-03-11T22:26:38.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:38.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:37 smithi045 ceph-mon[30178]: pgmap v855: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-03-11T22:26:38.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:37 smithi045 ceph-mon[30178]: osdmap e418: 8 total, 8 up, 8 in 2024-03-11T22:26:38.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:38.101 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:38.125 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:38.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:38.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:38.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:38.252 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:38.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:38.349 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T22:26:38.353 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:38.417 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:38.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:38.518 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:38.543 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:38.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:38.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:38.610 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:26:38.646 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:38.671 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:38.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:38.681 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[39715]: 6.1f scrub starts 2024-03-11T22:26:38.681 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[39715]: 6.1f scrub ok 2024-03-11T22:26:38.681 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[39715]: osdmap e419: 8 total, 8 up, 8 in 2024-03-11T22:26:38.682 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[39715]: 6.18 scrub starts 2024-03-11T22:26:38.682 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[39715]: 6.18 scrub ok 2024-03-11T22:26:38.682 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[39715]: 1.6 scrub starts 2024-03-11T22:26:38.682 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[39715]: 1.6 scrub ok 2024-03-11T22:26:38.682 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-11T22:26:38.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[30178]: 6.1f scrub starts 2024-03-11T22:26:38.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[30178]: 6.1f scrub ok 2024-03-11T22:26:38.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[30178]: osdmap e419: 8 total, 8 up, 8 in 2024-03-11T22:26:38.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[30178]: 6.18 scrub starts 2024-03-11T22:26:38.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[30178]: 6.18 scrub ok 2024-03-11T22:26:38.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[30178]: 1.6 scrub starts 2024-03-11T22:26:38.683 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[30178]: 1.6 scrub ok 2024-03-11T22:26:38.683 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:38 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-11T22:26:38.772 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:38.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:38.797 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:38.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:38.894 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:38 smithi144 ceph-mon[33984]: 6.1f scrub starts 2024-03-11T22:26:38.894 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:38 smithi144 ceph-mon[33984]: 6.1f scrub ok 2024-03-11T22:26:38.894 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:38 smithi144 ceph-mon[33984]: osdmap e419: 8 total, 8 up, 8 in 2024-03-11T22:26:38.894 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:38 smithi144 ceph-mon[33984]: 6.18 scrub starts 2024-03-11T22:26:38.894 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:38 smithi144 ceph-mon[33984]: 6.18 scrub ok 2024-03-11T22:26:38.894 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:38 smithi144 ceph-mon[33984]: 1.6 scrub starts 2024-03-11T22:26:38.894 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:38 smithi144 ceph-mon[33984]: 1.6 scrub ok 2024-03-11T22:26:38.894 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:38 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-03-11T22:26:38.898 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:38.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:38.923 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:38.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:39.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:39.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:39.049 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:39.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:39.151 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:39.174 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:39.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:39.276 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:39.301 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:39.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:39.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:39.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:39.428 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:39.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:39.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:39.554 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:39.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:39.655 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:39.680 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:39.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:39.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:39 smithi045 ceph-mon[39715]: pgmap v858: 147 pgs: 2 active+remapped+backfill_toofull, 145 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:26:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:39 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-11T22:26:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:39 smithi045 ceph-mon[39715]: osdmap e420: 8 total, 8 up, 8 in 2024-03-11T22:26:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:39 smithi045 ceph-mon[39715]: 3.14 scrub starts 2024-03-11T22:26:39.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:39 smithi045 ceph-mon[39715]: 3.14 scrub ok 2024-03-11T22:26:39.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:39 smithi045 ceph-mon[30178]: pgmap v858: 147 pgs: 2 active+remapped+backfill_toofull, 145 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:26:39.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:39 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-11T22:26:39.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:39 smithi045 ceph-mon[30178]: osdmap e420: 8 total, 8 up, 8 in 2024-03-11T22:26:39.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:39 smithi045 ceph-mon[30178]: 3.14 scrub starts 2024-03-11T22:26:39.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:39 smithi045 ceph-mon[30178]: 3.14 scrub ok 2024-03-11T22:26:39.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:39.806 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:39.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:39.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:39.930 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:39.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:39.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:40.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:40.055 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:40.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:40.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:40.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:39 smithi144 ceph-mon[33984]: pgmap v858: 147 pgs: 2 active+remapped+backfill_toofull, 145 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:26:40.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:39 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-03-11T22:26:40.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:39 smithi144 ceph-mon[33984]: osdmap e420: 8 total, 8 up, 8 in 2024-03-11T22:26:40.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:39 smithi144 ceph-mon[33984]: 3.14 scrub starts 2024-03-11T22:26:40.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:39 smithi144 ceph-mon[33984]: 3.14 scrub ok 2024-03-11T22:26:40.158 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:40.186 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:40.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:40.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:40.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:40.311 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:40.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:40.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:40.436 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:40.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:40.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:40.561 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:40.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:40.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:40.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[39715]: 4.19 scrub starts 2024-03-11T22:26:40.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[39715]: 4.19 scrub ok 2024-03-11T22:26:40.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[39715]: 3.4 scrub starts 2024-03-11T22:26:40.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[39715]: 3.4 scrub ok 2024-03-11T22:26:40.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[39715]: osdmap e421: 8 total, 8 up, 8 in 2024-03-11T22:26:40.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[39715]: 5.1f scrub starts 2024-03-11T22:26:40.655 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[39715]: 5.1f scrub ok 2024-03-11T22:26:40.655 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[30178]: 4.19 scrub starts 2024-03-11T22:26:40.655 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[30178]: 4.19 scrub ok 2024-03-11T22:26:40.655 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[30178]: 3.4 scrub starts 2024-03-11T22:26:40.656 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[30178]: 3.4 scrub ok 2024-03-11T22:26:40.656 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[30178]: osdmap e421: 8 total, 8 up, 8 in 2024-03-11T22:26:40.656 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[30178]: 5.1f scrub starts 2024-03-11T22:26:40.656 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:40 smithi045 ceph-mon[30178]: 5.1f scrub ok 2024-03-11T22:26:40.663 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:40.688 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:40.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:40.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:40.790 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:40.795 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:40 smithi144 ceph-mon[33984]: 4.19 scrub starts 2024-03-11T22:26:40.795 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:40 smithi144 ceph-mon[33984]: 4.19 scrub ok 2024-03-11T22:26:40.795 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:40 smithi144 ceph-mon[33984]: 3.4 scrub starts 2024-03-11T22:26:40.795 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:40 smithi144 ceph-mon[33984]: 3.4 scrub ok 2024-03-11T22:26:40.795 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:40 smithi144 ceph-mon[33984]: osdmap e421: 8 total, 8 up, 8 in 2024-03-11T22:26:40.795 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:40 smithi144 ceph-mon[33984]: 5.1f scrub starts 2024-03-11T22:26:40.795 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:40 smithi144 ceph-mon[33984]: 5.1f scrub ok 2024-03-11T22:26:40.816 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:40.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:40.917 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:40.924 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:26:40.924 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:26:40.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:40.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:41.043 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:41.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:41.068 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:41.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:41.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:41.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:41.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:41.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:41.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:41.310 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:26:41.321 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:41.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:41.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:41.331 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:26:41.351 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:26:41.370 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:26:41.391 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:26:41.410 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:26:41.423 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:41.431 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:26:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:41.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:41.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:41.451 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:26:41.453 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:26:41.453 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:26:41.453 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:26:41.453 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:26:41.453 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:26:41.453 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:26:41.453 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:26:41.453 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:26:41.488 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:26:41.488 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:26:41.488 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-03-11T22:26:41.488 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T22:26:41.489 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T22:26:41.489 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-03-11T22:26:41.548 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:41.571 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:41.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:41.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:41.612 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T22:26:41.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:41.698 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:41.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:41.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:41.756 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:26:41.799 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:41.824 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:41.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:41.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:41 smithi144 ceph-mon[33984]: 3.1e scrub starts 2024-03-11T22:26:41.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:41 smithi144 ceph-mon[33984]: 3.1e scrub ok 2024-03-11T22:26:41.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:41 smithi144 ceph-mon[33984]: pgmap v861: 147 pgs: 2 active+remapped+backfill_toofull, 145 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:41.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:41 smithi144 ceph-mon[33984]: osdmap e422: 8 total, 8 up, 8 in 2024-03-11T22:26:41.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:41 smithi144 ceph-mon[33984]: 4.1f scrub starts 2024-03-11T22:26:41.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:41 smithi144 ceph-mon[33984]: 4.1f scrub ok 2024-03-11T22:26:41.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:41 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2678344273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:41.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[30178]: 3.1e scrub starts 2024-03-11T22:26:41.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[30178]: 3.1e scrub ok 2024-03-11T22:26:41.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[30178]: pgmap v861: 147 pgs: 2 active+remapped+backfill_toofull, 145 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:41.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[30178]: osdmap e422: 8 total, 8 up, 8 in 2024-03-11T22:26:41.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[30178]: 4.1f scrub starts 2024-03-11T22:26:41.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[30178]: 4.1f scrub ok 2024-03-11T22:26:41.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2678344273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:41.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[39715]: 3.1e scrub starts 2024-03-11T22:26:41.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[39715]: 3.1e scrub ok 2024-03-11T22:26:41.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[39715]: pgmap v861: 147 pgs: 2 active+remapped+backfill_toofull, 145 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:41.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[39715]: osdmap e422: 8 total, 8 up, 8 in 2024-03-11T22:26:41.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[39715]: 4.1f scrub starts 2024-03-11T22:26:41.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[39715]: 4.1f scrub ok 2024-03-11T22:26:41.881 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:41 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2678344273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:41.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:41.949 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:41.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:42.050 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:42.076 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:42.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:42.177 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:42.203 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:42.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:42.304 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:42.328 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:42.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:42.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:42.430 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:42.455 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:42.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:42.498 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:26:42.558 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:42.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:42.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:42.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:42.684 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:42.708 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:42.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:42.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:42.810 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:42.833 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:42.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:42.835 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:26:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:26:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:26:42.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:42 smithi045 ceph-mon[39715]: osdmap e423: 8 total, 8 up, 8 in 2024-03-11T22:26:42.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:42 smithi045 ceph-mon[39715]: 5.a scrub starts 2024-03-11T22:26:42.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:42 smithi045 ceph-mon[39715]: 5.a scrub ok 2024-03-11T22:26:42.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:42 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-11T22:26:42.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:42 smithi045 ceph-mon[30178]: osdmap e423: 8 total, 8 up, 8 in 2024-03-11T22:26:42.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:42 smithi045 ceph-mon[30178]: 5.a scrub starts 2024-03-11T22:26:42.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:42 smithi045 ceph-mon[30178]: 5.a scrub ok 2024-03-11T22:26:42.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:42 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-11T22:26:42.935 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:42.958 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:42.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:42.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:42.992 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:42 smithi144 ceph-mon[33984]: osdmap e423: 8 total, 8 up, 8 in 2024-03-11T22:26:42.993 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:42 smithi144 ceph-mon[33984]: 5.a scrub starts 2024-03-11T22:26:42.993 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:42 smithi144 ceph-mon[33984]: 5.a scrub ok 2024-03-11T22:26:42.993 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:42 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-03-11T22:26:43.061 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:43.085 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:43.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:43.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:43.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:43.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:43.312 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:43.337 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:43.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:43.439 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:43.464 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:43.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:43.565 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:43.589 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:43.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:43.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:43.675 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:26:43.691 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:43.714 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:43.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:43.818 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:43.843 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:43.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:43.943 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:43 smithi144 ceph-mon[33984]: pgmap v864: 146 pgs: 1 peering, 2 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:43.943 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:43 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-11T22:26:43.943 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:43 smithi144 ceph-mon[33984]: osdmap e424: 8 total, 8 up, 8 in 2024-03-11T22:26:43.943 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:43 smithi144 ceph-mon[33984]: 5.4 scrub starts 2024-03-11T22:26:43.943 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:43 smithi144 ceph-mon[33984]: 5.4 scrub ok 2024-03-11T22:26:43.943 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:43 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/163333059' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:26:43.944 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:43.968 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:43.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:43.998 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:43 smithi045 ceph-mon[39715]: pgmap v864: 146 pgs: 1 peering, 2 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:43.998 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:43 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-11T22:26:43.998 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:43 smithi045 ceph-mon[39715]: osdmap e424: 8 total, 8 up, 8 in 2024-03-11T22:26:43.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:43 smithi045 ceph-mon[39715]: 5.4 scrub starts 2024-03-11T22:26:43.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:43 smithi045 ceph-mon[39715]: 5.4 scrub ok 2024-03-11T22:26:43.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:43 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/163333059' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:26:43.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:43 smithi045 ceph-mon[30178]: pgmap v864: 146 pgs: 1 peering, 2 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:43.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:43 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-03-11T22:26:43.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:43 smithi045 ceph-mon[30178]: osdmap e424: 8 total, 8 up, 8 in 2024-03-11T22:26:43.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:43 smithi045 ceph-mon[30178]: 5.4 scrub starts 2024-03-11T22:26:43.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:43 smithi045 ceph-mon[30178]: 5.4 scrub ok 2024-03-11T22:26:43.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:43 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/163333059' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:26:44.069 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:44.093 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:44.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:44.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:44.195 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:44.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:44.219 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:44.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:44.311 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:26:44.312 INFO:teuthology.orchestra.run.smithi144.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T22:26:44.312 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:26:44.320 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:44.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:26:44.346 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:44.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:44.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:44.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:44.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:44.472 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:44.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:44.573 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:44.584 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:44.597 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:44.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:44.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:44.687 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'unique_pool_4' created 2024-03-11T22:26:44.699 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:44.723 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:44.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:44.825 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:44.851 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:44.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:44.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:44.976 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:44.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:44.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:45.004 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:44 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:26:45.004 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:44 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/163333059' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:26:45.004 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:44 smithi144 ceph-mon[33984]: osdmap e425: 8 total, 8 up, 8 in 2024-03-11T22:26:45.004 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:44 smithi144 ceph-mon[33984]: 5.1e scrub starts 2024-03-11T22:26:45.004 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:44 smithi144 ceph-mon[33984]: 5.1e scrub ok 2024-03-11T22:26:45.004 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:44 smithi144 ceph-mon[33984]: 3.11 scrub starts 2024-03-11T22:26:45.004 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:44 smithi144 ceph-mon[33984]: 3.11 scrub ok 2024-03-11T22:26:45.005 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:44 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3667790810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T22:26:45.005 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:44 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T22:26:45.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:26:45.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/163333059' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:26:45.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[39715]: osdmap e425: 8 total, 8 up, 8 in 2024-03-11T22:26:45.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[39715]: 5.1e scrub starts 2024-03-11T22:26:45.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[39715]: 5.1e scrub ok 2024-03-11T22:26:45.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[39715]: 3.11 scrub starts 2024-03-11T22:26:45.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[39715]: 3.11 scrub ok 2024-03-11T22:26:45.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3667790810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T22:26:45.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T22:26:45.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:26:45.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/163333059' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:26:45.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[30178]: osdmap e425: 8 total, 8 up, 8 in 2024-03-11T22:26:45.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[30178]: 5.1e scrub starts 2024-03-11T22:26:45.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[30178]: 5.1e scrub ok 2024-03-11T22:26:45.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[30178]: 3.11 scrub starts 2024-03-11T22:26:45.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[30178]: 3.11 scrub ok 2024-03-11T22:26:45.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3667790810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T22:26:45.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:44 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T22:26:45.078 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:45.104 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:45.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:45.205 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:45.229 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:45.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:45.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:45.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:45.354 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:45.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:45.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:45.456 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:45.483 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:45.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:45.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:45.585 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:45.608 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:45.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:45.710 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:45.736 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:45.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:45.837 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:45.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:45 smithi144 ceph-mon[33984]: 1.9 scrub starts 2024-03-11T22:26:45.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:45 smithi144 ceph-mon[33984]: 1.9 scrub ok 2024-03-11T22:26:45.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:45 smithi144 ceph-mon[33984]: pgmap v867: 146 pgs: 1 peering, 2 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:45.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:45 smithi144 ceph-mon[33984]: 6.1 scrub starts 2024-03-11T22:26:45.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:45 smithi144 ceph-mon[33984]: 6.1 scrub ok 2024-03-11T22:26:45.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:45 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-11T22:26:45.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:45 smithi144 ceph-mon[33984]: osdmap e426: 8 total, 8 up, 8 in 2024-03-11T22:26:45.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:45 smithi144 ceph-mon[33984]: 5.10 scrub starts 2024-03-11T22:26:45.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:45 smithi144 ceph-mon[33984]: 5.10 scrub ok 2024-03-11T22:26:45.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:45 smithi144 ceph-mon[33984]: 3.3 scrub starts 2024-03-11T22:26:45.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:45 smithi144 ceph-mon[33984]: 3.3 scrub ok 2024-03-11T22:26:45.862 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:45.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:45.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[39715]: 1.9 scrub starts 2024-03-11T22:26:45.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[39715]: 1.9 scrub ok 2024-03-11T22:26:45.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[39715]: pgmap v867: 146 pgs: 1 peering, 2 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:45.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[39715]: 6.1 scrub starts 2024-03-11T22:26:45.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[39715]: 6.1 scrub ok 2024-03-11T22:26:45.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-11T22:26:45.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[39715]: osdmap e426: 8 total, 8 up, 8 in 2024-03-11T22:26:45.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[39715]: 5.10 scrub starts 2024-03-11T22:26:45.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[39715]: 5.10 scrub ok 2024-03-11T22:26:45.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[39715]: 3.3 scrub starts 2024-03-11T22:26:45.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[39715]: 3.3 scrub ok 2024-03-11T22:26:45.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[30178]: 1.9 scrub starts 2024-03-11T22:26:45.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[30178]: 1.9 scrub ok 2024-03-11T22:26:45.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[30178]: pgmap v867: 146 pgs: 1 peering, 2 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:45.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[30178]: 6.1 scrub starts 2024-03-11T22:26:45.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[30178]: 6.1 scrub ok 2024-03-11T22:26:45.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-11T22:26:45.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[30178]: osdmap e426: 8 total, 8 up, 8 in 2024-03-11T22:26:45.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[30178]: 5.10 scrub starts 2024-03-11T22:26:45.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[30178]: 5.10 scrub ok 2024-03-11T22:26:45.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[30178]: 3.3 scrub starts 2024-03-11T22:26:45.880 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:45 smithi045 ceph-mon[30178]: 3.3 scrub ok 2024-03-11T22:26:45.902 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-11T22:26:45.963 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:45.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:45.987 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:45.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:46.088 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:46.112 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:46.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:46.138 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-03-11T22:26:46.138 INFO:teuthology.orchestra.run.smithi144.stdout: "ops": [], 2024-03-11T22:26:46.138 INFO:teuthology.orchestra.run.smithi144.stdout: "num_ops": 0 2024-03-11T22:26:46.139 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-03-11T22:26:46.214 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:46.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:46.239 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:46.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:46.339 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:46.365 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:46.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:46.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:26:46.467 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:46.491 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:46.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:46.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:46.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:46.620 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:46.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:46.713 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:46 smithi144 ceph-mon[33984]: 4.8 scrub starts 2024-03-11T22:26:46.713 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:46 smithi144 ceph-mon[33984]: 4.8 scrub ok 2024-03-11T22:26:46.713 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:46 smithi144 ceph-mon[33984]: osdmap e427: 8 total, 8 up, 8 in 2024-03-11T22:26:46.713 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:46.714 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:46.714 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-11T22:26:46.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:46.746 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:46.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:46.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:46.759 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T22:26:46.848 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:46.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:46.873 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:46.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:46.925 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:46 smithi045 ceph-mon[30178]: 4.8 scrub starts 2024-03-11T22:26:46.925 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:46 smithi045 ceph-mon[30178]: 4.8 scrub ok 2024-03-11T22:26:46.925 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:46 smithi045 ceph-mon[30178]: osdmap e427: 8 total, 8 up, 8 in 2024-03-11T22:26:46.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:46.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:46.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-11T22:26:46.926 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:46 smithi045 ceph-mon[39715]: 4.8 scrub starts 2024-03-11T22:26:46.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:46 smithi045 ceph-mon[39715]: 4.8 scrub ok 2024-03-11T22:26:46.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:46 smithi045 ceph-mon[39715]: osdmap e427: 8 total, 8 up, 8 in 2024-03-11T22:26:46.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:26:46.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:26:46.927 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-03-11T22:26:46.940 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:26:46.965 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:26:46.972 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:46.975 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:46.995 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:26:47.000 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:47.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:47.003 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:26:47.004 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:47.024 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:26:47.044 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:26:47.065 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:26:47.086 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:26:47.099 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:26:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:26:47.101 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:47.106 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:26:47.107 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:26:47.107 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:26:47.107 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:26:47.107 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:26:47.108 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:26:47.108 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:26:47.108 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:26:47.108 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:26:47.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:47.125 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:47.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:47.226 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:47.250 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:47.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:47.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:47.352 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:47.376 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:47.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:47.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:47.478 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:47.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:47.503 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:47.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:47.603 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:47.627 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:47.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:47.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:47.729 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:47.754 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:47.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:47.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:47.857 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:47.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[30178]: 1.4 scrub starts 2024-03-11T22:26:47.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[30178]: 1.4 scrub ok 2024-03-11T22:26:47.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[30178]: 4.9 scrub starts 2024-03-11T22:26:47.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[30178]: 4.9 scrub ok 2024-03-11T22:26:47.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[30178]: pgmap v870: 161 pgs: 3 creating+activating, 9 unknown, 1 active+remapped+backfill_toofull, 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-03-11T22:26:47.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-11T22:26:47.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[30178]: osdmap e428: 8 total, 8 up, 8 in 2024-03-11T22:26:47.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:47.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4053661566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:47.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[30178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:26:47.867 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[30178]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:26:47.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[39715]: 1.4 scrub starts 2024-03-11T22:26:47.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[39715]: 1.4 scrub ok 2024-03-11T22:26:47.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[39715]: 4.9 scrub starts 2024-03-11T22:26:47.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[39715]: 4.9 scrub ok 2024-03-11T22:26:47.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[39715]: pgmap v870: 161 pgs: 3 creating+activating, 9 unknown, 1 active+remapped+backfill_toofull, 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-03-11T22:26:47.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-11T22:26:47.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[39715]: osdmap e428: 8 total, 8 up, 8 in 2024-03-11T22:26:47.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:47.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4053661566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:47.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[39715]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:26:47.867 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:47 smithi045 ceph-mon[39715]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:26:47.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:47.881 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:47.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:47.982 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:48.006 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:48.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:48.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:48.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:47 smithi144 ceph-mon[33984]: 1.4 scrub starts 2024-03-11T22:26:48.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:47 smithi144 ceph-mon[33984]: 1.4 scrub ok 2024-03-11T22:26:48.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:47 smithi144 ceph-mon[33984]: 4.9 scrub starts 2024-03-11T22:26:48.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:47 smithi144 ceph-mon[33984]: 4.9 scrub ok 2024-03-11T22:26:48.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:47 smithi144 ceph-mon[33984]: pgmap v870: 161 pgs: 3 creating+activating, 9 unknown, 1 active+remapped+backfill_toofull, 148 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-03-11T22:26:48.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:47 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-03-11T22:26:48.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:47 smithi144 ceph-mon[33984]: osdmap e428: 8 total, 8 up, 8 in 2024-03-11T22:26:48.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:48.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:47 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4053661566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:48.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:47 smithi144 ceph-mon[33984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:26:48.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:47 smithi144 ceph-mon[33984]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:26:48.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:48.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:48.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:48.233 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:48.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:48.258 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:48.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:48.359 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:48.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:48.384 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:48.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:48.484 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:48.507 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:48.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:48.609 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:48.634 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:48.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:48.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:48.736 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:48.761 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:48.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:48.790 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:26:48.790 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:26:48.790 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:48 smithi144 ceph-mon[33984]: osdmap e429: 8 total, 8 up, 8 in 2024-03-11T22:26:48.790 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:48 smithi144 ceph-mon[33984]: 4.3 deep-scrub starts 2024-03-11T22:26:48.790 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:48 smithi144 ceph-mon[33984]: 4.3 deep-scrub ok 2024-03-11T22:26:48.863 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:48.887 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:48.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:48.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:49.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:49.015 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:49.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:49.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:48 smithi045 ceph-mon[30178]: osdmap e429: 8 total, 8 up, 8 in 2024-03-11T22:26:49.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:48 smithi045 ceph-mon[30178]: 4.3 deep-scrub starts 2024-03-11T22:26:49.028 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:48 smithi045 ceph-mon[30178]: 4.3 deep-scrub ok 2024-03-11T22:26:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:48 smithi045 ceph-mon[39715]: osdmap e429: 8 total, 8 up, 8 in 2024-03-11T22:26:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:48 smithi045 ceph-mon[39715]: 4.3 deep-scrub starts 2024-03-11T22:26:49.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:48 smithi045 ceph-mon[39715]: 4.3 deep-scrub ok 2024-03-11T22:26:49.116 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:49.140 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:49.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:49.241 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:49.266 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:49.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:49.369 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:49.393 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:49.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:49.416 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T22:26:49.494 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:49.524 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:49.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:49.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:49.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:49.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:49.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:49.701 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:26:49.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:49.780 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:49.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:49.881 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:49.906 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:49.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:49.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:50.007 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:50.034 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:50.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:49 smithi045 ceph-mon[39715]: pgmap v873: 161 pgs: 3 creating+activating, 158 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:49 smithi045 ceph-mon[39715]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T22:26:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:49 smithi045 ceph-mon[39715]: osdmap e430: 8 total, 8 up, 8 in 2024-03-11T22:26:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:49 smithi045 ceph-mon[39715]: osdmap e431: 8 total, 8 up, 8 in 2024-03-11T22:26:50.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:49 smithi045 ceph-mon[30178]: pgmap v873: 161 pgs: 3 creating+activating, 158 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:50.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:49 smithi045 ceph-mon[30178]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T22:26:50.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:49 smithi045 ceph-mon[30178]: osdmap e430: 8 total, 8 up, 8 in 2024-03-11T22:26:50.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:49 smithi045 ceph-mon[30178]: osdmap e431: 8 total, 8 up, 8 in 2024-03-11T22:26:50.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:49 smithi144 ceph-mon[33984]: pgmap v873: 161 pgs: 3 creating+activating, 158 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:50.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:49 smithi144 ceph-mon[33984]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T22:26:50.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:49 smithi144 ceph-mon[33984]: osdmap e430: 8 total, 8 up, 8 in 2024-03-11T22:26:50.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:49 smithi144 ceph-mon[33984]: osdmap e431: 8 total, 8 up, 8 in 2024-03-11T22:26:50.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:50.159 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:50.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:50.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:50.285 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:50.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:50.386 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:50.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:50.411 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:50.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:50.512 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:50.538 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:50.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:50.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:50.580 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:26:50.580 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:26:50.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:26:50.641 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:50.667 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:50.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:50.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[39715]: 6.c scrub starts 2024-03-11T22:26:50.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[39715]: 6.c scrub ok 2024-03-11T22:26:50.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[39715]: 6.a deep-scrub starts 2024-03-11T22:26:50.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[39715]: 6.a deep-scrub ok 2024-03-11T22:26:50.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[39715]: 6.e deep-scrub starts 2024-03-11T22:26:50.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[39715]: 6.e deep-scrub ok 2024-03-11T22:26:50.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-11T22:26:50.759 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[30178]: 6.c scrub starts 2024-03-11T22:26:50.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[30178]: 6.c scrub ok 2024-03-11T22:26:50.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[30178]: 6.a deep-scrub starts 2024-03-11T22:26:50.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[30178]: 6.a deep-scrub ok 2024-03-11T22:26:50.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[30178]: 6.e deep-scrub starts 2024-03-11T22:26:50.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[30178]: 6.e deep-scrub ok 2024-03-11T22:26:50.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:50 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-11T22:26:50.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:50.771 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:26:50.793 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:50.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:50.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:50.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:50.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:50.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:50.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:51.021 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:51.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:51.046 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:51.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:51.073 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:50 smithi144 ceph-mon[33984]: 6.c scrub starts 2024-03-11T22:26:51.073 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:50 smithi144 ceph-mon[33984]: 6.c scrub ok 2024-03-11T22:26:51.073 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:50 smithi144 ceph-mon[33984]: 6.a deep-scrub starts 2024-03-11T22:26:51.073 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:50 smithi144 ceph-mon[33984]: 6.a deep-scrub ok 2024-03-11T22:26:51.073 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:50 smithi144 ceph-mon[33984]: 6.e deep-scrub starts 2024-03-11T22:26:51.073 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:50 smithi144 ceph-mon[33984]: 6.e deep-scrub ok 2024-03-11T22:26:51.073 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:50 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-03-11T22:26:51.147 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:51.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:51.172 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:51.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:51.273 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:51.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:51.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:51.399 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:51.424 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:51.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:51.502 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:51.503 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:51.525 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:51.553 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:51.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:51.557 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:26:51.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:51.675 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:26:51.675 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:26:51.679 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:51.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:51.780 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:51.804 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:51.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:51.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:51.905 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:51.930 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:51.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:51.998 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: pgmap v876: 160 pgs: 3 creating+activating, 157 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:51.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: 5.c scrub starts 2024-03-11T22:26:51.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: 5.c scrub ok 2024-03-11T22:26:51.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-11T22:26:51.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: osdmap e432: 8 total, 8 up, 8 in 2024-03-11T22:26:51.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: 3.f deep-scrub starts 2024-03-11T22:26:51.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: 3.f deep-scrub ok 2024-03-11T22:26:51.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: 3.13 deep-scrub starts 2024-03-11T22:26:51.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: 3.13 deep-scrub ok 2024-03-11T22:26:51.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2603219998' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:26:51.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:26:51.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2603219998' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:26:51.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: osdmap e433: 8 total, 8 up, 8 in 2024-03-11T22:26:51.999 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2492264574' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:26:52.000 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:51 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:26:52.020 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: pgmap v876: 160 pgs: 3 creating+activating, 157 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:52.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: 5.c scrub starts 2024-03-11T22:26:52.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: 5.c scrub ok 2024-03-11T22:26:52.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-11T22:26:52.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: osdmap e432: 8 total, 8 up, 8 in 2024-03-11T22:26:52.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: 3.f deep-scrub starts 2024-03-11T22:26:52.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: 3.f deep-scrub ok 2024-03-11T22:26:52.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: 3.13 deep-scrub starts 2024-03-11T22:26:52.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: 3.13 deep-scrub ok 2024-03-11T22:26:52.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2603219998' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:26:52.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:26:52.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2603219998' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:26:52.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: osdmap e433: 8 total, 8 up, 8 in 2024-03-11T22:26:52.021 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2492264574' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:26:52.022 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:26:52.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: pgmap v876: 160 pgs: 3 creating+activating, 157 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:52.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: 5.c scrub starts 2024-03-11T22:26:52.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: 5.c scrub ok 2024-03-11T22:26:52.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-03-11T22:26:52.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: osdmap e432: 8 total, 8 up, 8 in 2024-03-11T22:26:52.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: 3.f deep-scrub starts 2024-03-11T22:26:52.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: 3.f deep-scrub ok 2024-03-11T22:26:52.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: 3.13 deep-scrub starts 2024-03-11T22:26:52.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: 3.13 deep-scrub ok 2024-03-11T22:26:52.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2603219998' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:26:52.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:26:52.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2603219998' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:26:52.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: osdmap e433: 8 total, 8 up, 8 in 2024-03-11T22:26:52.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2492264574' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:26:52.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:51 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:26:52.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:52.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:52.056 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:52.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:52.157 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:52.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:52.181 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:52.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:52.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:52.306 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:52.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:52.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T22:26:52.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:52.431 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:52.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:52.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:52.533 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:52.558 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:52.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:52.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:52.560 INFO:teuthology.orchestra.run.smithi045.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-11T22:26:52.660 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:52.684 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:52.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:52.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:52.715 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:26:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:26:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:26:52.716 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:52 smithi045 ceph-mon[30178]: 4.10 scrub starts 2024-03-11T22:26:52.786 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:52.811 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:52.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:52.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:52.913 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:52.937 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:52.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:52.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:52.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:52 smithi045 ceph-mon[39715]: 4.10 scrub starts 2024-03-11T22:26:52.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:52 smithi045 ceph-mon[39715]: 4.10 scrub ok 2024-03-11T22:26:52.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:52 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:26:52.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:52 smithi045 ceph-mon[39715]: osdmap e434: 8 total, 8 up, 8 in 2024-03-11T22:26:53.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:52 smithi045 ceph-mon[30178]: 4.10 scrub ok 2024-03-11T22:26:53.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:52 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:26:53.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:52 smithi045 ceph-mon[30178]: osdmap e434: 8 total, 8 up, 8 in 2024-03-11T22:26:53.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:53.062 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:53.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:53.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:52 smithi144 ceph-mon[33984]: 4.10 scrub starts 2024-03-11T22:26:53.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:52 smithi144 ceph-mon[33984]: 4.10 scrub ok 2024-03-11T22:26:53.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:52 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:26:53.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:52 smithi144 ceph-mon[33984]: osdmap e434: 8 total, 8 up, 8 in 2024-03-11T22:26:53.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:53.188 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:53.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:53.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:53.289 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:53.313 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:53.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:53.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:53.415 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:53.441 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:53.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:53.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:53.543 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:53.570 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:53.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:53.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:53.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:53.700 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:53.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:53.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:53.825 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:53.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:53.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:53.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:53.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:53.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:53.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:54.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:53 smithi045 ceph-mon[39715]: pgmap v879: 160 pgs: 2 clean+premerge+peered, 158 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:54.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:53 smithi045 ceph-mon[39715]: osdmap e435: 8 total, 8 up, 8 in 2024-03-11T22:26:54.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:53 smithi045 ceph-mon[30178]: pgmap v879: 160 pgs: 2 clean+premerge+peered, 158 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:54.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:53 smithi045 ceph-mon[30178]: osdmap e435: 8 total, 8 up, 8 in 2024-03-11T22:26:54.054 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:54.082 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:54.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:54.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:53 smithi144 ceph-mon[33984]: pgmap v879: 160 pgs: 2 clean+premerge+peered, 158 active+clean; 582 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:54.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:53 smithi144 ceph-mon[33984]: osdmap e435: 8 total, 8 up, 8 in 2024-03-11T22:26:54.184 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:54.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:54.212 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:54.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:54.313 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:54.339 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:54.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:54.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:54.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:54.466 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:54.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:54.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:54.568 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:54.597 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:54.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:54.698 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:54.728 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:54.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:54.831 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:54.858 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:54.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:54.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:54.922 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:54 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:26:54.922 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:54 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:26:54.922 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:54 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T22:26:54.922 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:54 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:26:54.922 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:54 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:26:54.923 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:54 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T22:26:54.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:54.984 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:54.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:55.086 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:55.093 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:55.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:54 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:26:55.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:54 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:26:55.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:54 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-11T22:26:55.114 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:55.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:55.117 DEBUG:teuthology.orchestra.run.smithi045:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-11T22:26:55.120 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T22:26:55.120 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T22:26:55.169 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:hints = 1 2024-03-11T22:26:55.169 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T22:26:55.169 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Object prefix: benchmark_data_smithi045_196393 2024-03-11T22:26:55.170 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:26:55.170 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 0 0 0 0 0 0 - 0 2024-03-11T22:26:55.216 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:55.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:55.261 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:55.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:55.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:55.404 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:55.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:55.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:55.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:55.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:55.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:55.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:55.695 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:55.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:55.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:55.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:55 smithi045 ceph-mon[39715]: pgmap v882: 159 pgs: 1 clean+premerge+peered, 158 active+clean; 262 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:55.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:55 smithi045 ceph-mon[39715]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:26:55.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:55 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T22:26:55.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:55 smithi045 ceph-mon[39715]: osdmap e436: 8 total, 8 up, 8 in 2024-03-11T22:26:55.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:55 smithi045 ceph-mon[30178]: pgmap v882: 159 pgs: 1 clean+premerge+peered, 158 active+clean; 262 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:55.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:55 smithi045 ceph-mon[30178]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:26:55.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:55 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T22:26:55.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:55 smithi045 ceph-mon[30178]: osdmap e436: 8 total, 8 up, 8 in 2024-03-11T22:26:55.798 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:55.829 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:55.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:55.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:55.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:55.971 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:55.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:55.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:56.074 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:56.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:55 smithi144 ceph-mon[33984]: pgmap v882: 159 pgs: 1 clean+premerge+peered, 158 active+clean; 262 KiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:26:56.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:55 smithi144 ceph-mon[33984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:26:56.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:55 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-11T22:26:56.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:55 smithi144 ceph-mon[33984]: osdmap e436: 8 total, 8 up, 8 in 2024-03-11T22:26:56.105 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:56.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:56.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:56.169 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 1 15 3089 3074 192.114 192.125 0.0099017 0.00518483 2024-03-11T22:26:56.208 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:56.236 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:56.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:56.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:56.338 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:26:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:56.377 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:56.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:56.380 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:26:56.478 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:56.510 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:56.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:56.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:56.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:56.650 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:56.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:56.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:56.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:56.794 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:56.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:56.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:56.896 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:56.927 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:56.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:56.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:56.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:26:56.998 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:56 smithi045 ceph-mon[39715]: osdmap e437: 8 total, 8 up, 8 in 2024-03-11T22:26:56.998 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:56 smithi045 ceph-mon[39715]: osdmap e438: 8 total, 8 up, 8 in 2024-03-11T22:26:56.998 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:56 smithi045 ceph-mon[30178]: osdmap e437: 8 total, 8 up, 8 in 2024-03-11T22:26:56.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:56 smithi045 ceph-mon[30178]: osdmap e438: 8 total, 8 up, 8 in 2024-03-11T22:26:57.029 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:57.063 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:57.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:57.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:56 smithi144 ceph-mon[33984]: osdmap e437: 8 total, 8 up, 8 in 2024-03-11T22:26:57.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:56 smithi144 ceph-mon[33984]: osdmap e438: 8 total, 8 up, 8 in 2024-03-11T22:26:57.099 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:26:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:26:57.165 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:57.169 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 2 16 6024 6008 187.704 183.375 0.00708893 0.00531386 2024-03-11T22:26:57.197 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:57.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:57.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:57.298 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:57.329 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:57.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:57.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:57.431 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:57.464 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:57.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:57.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:57.567 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:57.600 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:57.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:57.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:57.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:57.747 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:57.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:57.749 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:26:57.797 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:26:57.842 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:26:57.850 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:57.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:57.882 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:57.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:57.890 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:26:57.890 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:57 smithi045 ceph-mon[39715]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:26:57.890 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:57 smithi045 ceph-mon[39715]: pgmap v885: 159 pgs: 1 clean+premerge+peered, 158 active+clean; 60 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 15 MiB/s wr, 240 op/s 2024-03-11T22:26:57.890 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:57.890 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:57 smithi045 ceph-mon[39715]: osdmap e439: 8 total, 8 up, 8 in 2024-03-11T22:26:57.890 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:57 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/868196040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:57.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:57 smithi045 ceph-mon[30178]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:26:57.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:57 smithi045 ceph-mon[30178]: pgmap v885: 159 pgs: 1 clean+premerge+peered, 158 active+clean; 60 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 15 MiB/s wr, 240 op/s 2024-03-11T22:26:57.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:57.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:57 smithi045 ceph-mon[30178]: osdmap e439: 8 total, 8 up, 8 in 2024-03-11T22:26:57.891 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:57 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/868196040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:57.932 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:26:57.973 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:26:57.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:58.015 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:26:58.024 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:58.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:58.059 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:26:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:26:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:26:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:26:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:26:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:26:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:26:58.064 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:26:58.065 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:26:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:57 smithi144 ceph-mon[33984]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:26:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:57 smithi144 ceph-mon[33984]: pgmap v885: 159 pgs: 1 clean+premerge+peered, 158 active+clean; 60 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 15 MiB/s wr, 240 op/s 2024-03-11T22:26:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:26:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:57 smithi144 ceph-mon[33984]: osdmap e439: 8 total, 8 up, 8 in 2024-03-11T22:26:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:57 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/868196040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:26:58.126 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:26:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:58.157 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:58.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:26:58.170 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 3 15 8710 8695 181.111 167.938 0.00823749 0.00551244 2024-03-11T22:26:58.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:58.301 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:58.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:58.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:58.403 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:58.436 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:58.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:58.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:58.538 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:58.570 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:58.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:58.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:26:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:58.713 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:58.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:26:58.814 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:26:58.844 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:58.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:58.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:26:58.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:26:58.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:58.988 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:58.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:26:59.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:26:59.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:59.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:26:59.170 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 4 16 11500 11484 179.408 174.312 0.00651132 0.00556587 2024-03-11T22:26:59.234 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:59.259 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:59.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:59.285 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":884,"stamp":"2024-03-11T22:26:55.722408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":38082908,"num_objects":774,"num_object_clones":0,"num_object_copies":2322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":774,"num_whiteouts":0,"num_read":2187,"num_read_kb":1939,"num_write":1299,"num_write_kb":36973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":2436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1407,"ondisk_log_size":1407,"up":477,"acting":477,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":480,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4845428,"kb_used_data":18740,"kb_used_omap":12,"kb_used_meta":4826547,"kb_avail":745115788,"statfs":{"total":767960285184,"available":762998566912,"internally_reserved":0,"allocated":19189760,"data_stored":16229924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":4942384691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37486592,"num_objects":576,"num_object_clones":0,"num_object_copies":1728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":577,"num_write_kb":36928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.175550"},"pg_stats":[{"pgid":"11.3","version":"436'127","reported_seq":156,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.683739+0000","last_change":"2024-03-11T22:26:45.836547+0000","last_active":"2024-03-11T22:26:55.683739+0000","last_peered":"2024-03-11T22:26:55.683739+0000","last_clean":"2024-03-11T22:26:55.683739+0000","last_became_active":"2024-03-11T22:26:45.836245+0000","last_became_peered":"2024-03-11T22:26:45.836245+0000","last_unstale":"2024-03-11T22:26:55.683739+0000","last_undegraded":"2024-03-11T22:26:55.683739+0000","last_fullsized":"2024-03-11T22:26:55.683739+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:47.122244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8323072,"num_objects":127,"num_object_clones":0,"num_object_copies":381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127,"num_write_kb":8128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"294'25","reported_seq":600,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.553395+0000","last_change":"2024-03-11T22:24:39.621975+0000","last_active":"2024-03-11T22:26:51.553395+0000","last_peered":"2024-03-11T22:26:51.553395+0000","last_clean":"2024-03-11T22:26:51.553395+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:26:51.553395+0000","last_undegraded":"2024-03-11T22:26:51.553395+0000","last_fullsized":"2024-03-11T22:26:51.553395+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-03-11T22:24:39.621707+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.621707+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":788,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552263+0000","last_change":"2024-03-11T22:24:37.604756+0000","last_active":"2024-03-11T22:26:51.552263+0000","last_peered":"2024-03-11T22:26:51.552263+0000","last_clean":"2024-03-11T22:26:51.552263+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:26:51.552263+0000","last_undegraded":"2024-03-11T22:26:51.552263+0000","last_fullsized":"2024-03-11T22:26:51.552263+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.601240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.601240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":1048,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829847+0000","last_change":"2024-03-11T22:26:11.940151+0000","last_active":"2024-03-11T22:26:50.829847+0000","last_peered":"2024-03-11T22:26:50.829847+0000","last_clean":"2024-03-11T22:26:50.829847+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:26:50.829847+0000","last_undegraded":"2024-03-11T22:26:50.829847+0000","last_fullsized":"2024-03-11T22:26:50.829847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:26:11.939962+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:26:11.939962+0000","last_clean_scrub_stamp":"2024-03-11T22:26:11.939962+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:24.442117+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":563,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.549183+0000","last_change":"2024-03-11T22:26:49.602037+0000","last_active":"2024-03-11T22:26:51.549183+0000","last_peered":"2024-03-11T22:26:51.549183+0000","last_clean":"2024-03-11T22:26:51.549183+0000","last_became_active":"2024-03-11T22:14:52.243690+0000","last_became_peered":"2024-03-11T22:14:52.243690+0000","last_unstale":"2024-03-11T22:26:51.549183+0000","last_undegraded":"2024-03-11T22:26:51.549183+0000","last_fullsized":"2024-03-11T22:26:51.549183+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:49.601908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:49.601908+0000","last_clean_scrub_stamp":"2024-03-11T22:26:49.601908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:05.732044+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":1138,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551626+0000","last_change":"2024-03-11T22:26:43.904135+0000","last_active":"2024-03-11T22:26:51.551626+0000","last_peered":"2024-03-11T22:26:51.551626+0000","last_clean":"2024-03-11T22:26:51.551626+0000","last_became_active":"2024-03-11T22:20:32.293356+0000","last_became_peered":"2024-03-11T22:20:32.293356+0000","last_unstale":"2024-03-11T22:26:51.551626+0000","last_undegraded":"2024-03-11T22:26:51.551626+0000","last_fullsized":"2024-03-11T22:26:51.551626+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:43.904050+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:43.904050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:53.974234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":21,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829194+0000","last_change":"2024-03-11T22:26:45.687974+0000","last_active":"2024-03-11T22:26:50.829194+0000","last_peered":"2024-03-11T22:26:50.829194+0000","last_clean":"2024-03-11T22:26:50.829194+0000","last_became_active":"2024-03-11T22:26:45.687642+0000","last_became_peered":"2024-03-11T22:26:45.687642+0000","last_unstale":"2024-03-11T22:26:50.829194+0000","last_undegraded":"2024-03-11T22:26:50.829194+0000","last_fullsized":"2024-03-11T22:26:50.829194+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:05.974776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"353'39","reported_seq":653,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829691+0000","last_change":"2024-03-11T22:24:58.663588+0000","last_active":"2024-03-11T22:26:50.829691+0000","last_peered":"2024-03-11T22:26:50.829691+0000","last_clean":"2024-03-11T22:26:50.829691+0000","last_became_active":"2024-03-11T22:14:52.239649+0000","last_became_peered":"2024-03-11T22:14:52.239649+0000","last_unstale":"2024-03-11T22:26:50.829691+0000","last_undegraded":"2024-03-11T22:26:50.829691+0000","last_fullsized":"2024-03-11T22:26:50.829691+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'39","last_scrub_stamp":"2024-03-11T22:24:58.663290+0000","last_deep_scrub":"353'39","last_deep_scrub_stamp":"2024-03-11T22:23:05.461729+0000","last_clean_scrub_stamp":"2024-03-11T22:24:58.663290+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":818,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721323+0000","last_change":"2024-03-11T22:23:51.794264+0000","last_active":"2024-03-11T22:26:54.721323+0000","last_peered":"2024-03-11T22:26:54.721323+0000","last_clean":"2024-03-11T22:26:54.721323+0000","last_became_active":"2024-03-11T22:08:33.037966+0000","last_became_peered":"2024-03-11T22:08:33.037966+0000","last_unstale":"2024-03-11T22:26:54.721323+0000","last_undegraded":"2024-03-11T22:26:54.721323+0000","last_fullsized":"2024-03-11T22:26:54.721323+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:51.794140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.794140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":797,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.549278+0000","last_change":"2024-03-11T22:26:50.571917+0000","last_active":"2024-03-11T22:26:51.549278+0000","last_peered":"2024-03-11T22:26:51.549278+0000","last_clean":"2024-03-11T22:26:51.549278+0000","last_became_active":"2024-03-11T22:08:37.078553+0000","last_became_peered":"2024-03-11T22:08:37.078553+0000","last_unstale":"2024-03-11T22:26:51.549278+0000","last_undegraded":"2024-03-11T22:26:51.549278+0000","last_fullsized":"2024-03-11T22:26:51.549278+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:50.571821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.571821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:08.186639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":807,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.830068+0000","last_change":"2024-03-11T22:25:51.099786+0000","last_active":"2024-03-11T22:26:50.830068+0000","last_peered":"2024-03-11T22:26:50.830068+0000","last_clean":"2024-03-11T22:26:50.830068+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:26:50.830068+0000","last_undegraded":"2024-03-11T22:26:50.830068+0000","last_fullsized":"2024-03-11T22:26:50.830068+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:51.099663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:51.099663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:13.674011+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":1122,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.172930+0000","last_change":"2024-03-11T22:26:03.097798+0000","last_active":"2024-03-11T22:26:55.172930+0000","last_peered":"2024-03-11T22:26:55.172930+0000","last_clean":"2024-03-11T22:26:55.172930+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:26:55.172930+0000","last_undegraded":"2024-03-11T22:26:55.172930+0000","last_fullsized":"2024-03-11T22:26:55.172930+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:03.097636+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:03.097636+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:31.883276+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"436'103","reported_seq":132,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.680227+0000","last_change":"2024-03-11T22:26:45.687410+0000","last_active":"2024-03-11T22:26:55.680227+0000","last_peered":"2024-03-11T22:26:55.680227+0000","last_clean":"2024-03-11T22:26:55.680227+0000","last_became_active":"2024-03-11T22:26:45.687157+0000","last_became_peered":"2024-03-11T22:26:45.687157+0000","last_unstale":"2024-03-11T22:26:55.680227+0000","last_undegraded":"2024-03-11T22:26:55.680227+0000","last_fullsized":"2024-03-11T22:26:55.680227+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:54.836187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6750208,"num_objects":103,"num_object_clones":0,"num_object_copies":309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":6592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"273'19","reported_seq":626,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721496+0000","last_change":"2024-03-11T22:24:57.716291+0000","last_active":"2024-03-11T22:26:54.721496+0000","last_peered":"2024-03-11T22:26:54.721496+0000","last_clean":"2024-03-11T22:26:54.721496+0000","last_became_active":"2024-03-11T22:14:52.240437+0000","last_became_peered":"2024-03-11T22:14:52.240437+0000","last_unstale":"2024-03-11T22:26:54.721496+0000","last_undegraded":"2024-03-11T22:26:54.721496+0000","last_fullsized":"2024-03-11T22:26:54.721496+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:24:57.716144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:57.716144+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":578,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829190+0000","last_change":"2024-03-11T22:24:57.690108+0000","last_active":"2024-03-11T22:26:50.829190+0000","last_peered":"2024-03-11T22:26:50.829190+0000","last_clean":"2024-03-11T22:26:50.829190+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:26:50.829190+0000","last_undegraded":"2024-03-11T22:26:50.829190+0000","last_fullsized":"2024-03-11T22:26:50.829190+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:57.689696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:57.689696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":794,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.174571+0000","last_change":"2024-03-11T22:24:59.851270+0000","last_active":"2024-03-11T22:26:55.174571+0000","last_peered":"2024-03-11T22:26:55.174571+0000","last_clean":"2024-03-11T22:26:55.174571+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:26:55.174571+0000","last_undegraded":"2024-03-11T22:26:55.174571+0000","last_fullsized":"2024-03-11T22:26:55.174571+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:59.851040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:59.851040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":776,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552793+0000","last_change":"2024-03-11T22:26:48.845265+0000","last_active":"2024-03-11T22:26:51.552793+0000","last_peered":"2024-03-11T22:26:51.552793+0000","last_clean":"2024-03-11T22:26:51.552793+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:26:51.552793+0000","last_undegraded":"2024-03-11T22:26:51.552793+0000","last_fullsized":"2024-03-11T22:26:51.552793+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:48.845177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:48.845177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:06.252440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"93'130","reported_seq":1161,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.550514+0000","last_change":"2024-03-11T22:26:43.667499+0000","last_active":"2024-03-11T22:26:51.550514+0000","last_peered":"2024-03-11T22:26:51.550514+0000","last_clean":"2024-03-11T22:26:51.550514+0000","last_became_active":"2024-03-11T22:26:43.666975+0000","last_became_peered":"2024-03-11T22:26:43.666975+0000","last_unstale":"2024-03-11T22:26:51.550514+0000","last_undegraded":"2024-03-11T22:26:51.550514+0000","last_fullsized":"2024-03-11T22:26:51.550514+0000","mapping_epoch":424,"log_start":"93'130","ondisk_log_start":"93'130","created":168,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:10.541158+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:22:14.766775+0000","last_clean_scrub_stamp":"2024-03-11T22:26:10.541158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:38.403771+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":23,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.548729+0000","last_change":"2024-03-11T22:26:45.835278+0000","last_active":"2024-03-11T22:26:51.548729+0000","last_peered":"2024-03-11T22:26:51.548729+0000","last_clean":"2024-03-11T22:26:51.548729+0000","last_became_active":"2024-03-11T22:26:45.834959+0000","last_became_peered":"2024-03-11T22:26:45.834959+0000","last_unstale":"2024-03-11T22:26:51.548729+0000","last_undegraded":"2024-03-11T22:26:51.548729+0000","last_fullsized":"2024-03-11T22:26:51.548729+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:45.336174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"273'34","reported_seq":859,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.548843+0000","last_change":"2024-03-11T22:26:05.664352+0000","last_active":"2024-03-11T22:26:51.548843+0000","last_peered":"2024-03-11T22:26:51.548843+0000","last_clean":"2024-03-11T22:26:51.548843+0000","last_became_active":"2024-03-11T22:08:35.060381+0000","last_became_peered":"2024-03-11T22:08:35.060381+0000","last_unstale":"2024-03-11T22:26:51.548843+0000","last_undegraded":"2024-03-11T22:26:51.548843+0000","last_fullsized":"2024-03-11T22:26:51.548843+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:26:05.664197+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:26:05.664197+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:17.372688+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":800,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551729+0000","last_change":"2024-03-11T22:25:06.617067+0000","last_active":"2024-03-11T22:26:51.551729+0000","last_peered":"2024-03-11T22:26:51.551729+0000","last_clean":"2024-03-11T22:26:51.551729+0000","last_became_active":"2024-03-11T22:20:29.426224+0000","last_became_peered":"2024-03-11T22:20:29.426224+0000","last_unstale":"2024-03-11T22:26:51.551729+0000","last_undegraded":"2024-03-11T22:26:51.551729+0000","last_fullsized":"2024-03-11T22:26:51.551729+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:06.616932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:25:06.616932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":731,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173941+0000","last_change":"2024-03-11T22:24:00.708144+0000","last_active":"2024-03-11T22:26:55.173941+0000","last_peered":"2024-03-11T22:26:55.173941+0000","last_clean":"2024-03-11T22:26:55.173941+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:26:55.173941+0000","last_undegraded":"2024-03-11T22:26:55.173941+0000","last_fullsized":"2024-03-11T22:26:55.173941+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:24:00.708018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.708018+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":790,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.172457+0000","last_change":"2024-03-11T22:26:36.096107+0000","last_active":"2024-03-11T22:26:55.172457+0000","last_peered":"2024-03-11T22:26:55.172457+0000","last_clean":"2024-03-11T22:26:55.172457+0000","last_became_active":"2024-03-11T22:08:39.079996+0000","last_became_peered":"2024-03-11T22:08:39.079996+0000","last_unstale":"2024-03-11T22:26:55.172457+0000","last_undegraded":"2024-03-11T22:26:55.172457+0000","last_fullsized":"2024-03-11T22:26:55.172457+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:36.095984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:26:36.095984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:51.186884+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":1123,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173780+0000","last_change":"2024-03-11T22:24:48.925640+0000","last_active":"2024-03-11T22:26:55.173780+0000","last_peered":"2024-03-11T22:26:55.173780+0000","last_clean":"2024-03-11T22:26:55.173780+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:26:55.173780+0000","last_undegraded":"2024-03-11T22:26:55.173780+0000","last_fullsized":"2024-03-11T22:26:55.173780+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:48.925477+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:48.925477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":23,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.548743+0000","last_change":"2024-03-11T22:26:45.835313+0000","last_active":"2024-03-11T22:26:51.548743+0000","last_peered":"2024-03-11T22:26:51.548743+0000","last_clean":"2024-03-11T22:26:51.548743+0000","last_became_active":"2024-03-11T22:26:45.834996+0000","last_became_peered":"2024-03-11T22:26:45.834996+0000","last_unstale":"2024-03-11T22:26:51.548743+0000","last_undegraded":"2024-03-11T22:26:51.548743+0000","last_fullsized":"2024-03-11T22:26:51.548743+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:04.983509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":572,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552588+0000","last_change":"2024-03-11T22:26:09.040467+0000","last_active":"2024-03-11T22:26:51.552588+0000","last_peered":"2024-03-11T22:26:51.552588+0000","last_clean":"2024-03-11T22:26:51.552588+0000","last_became_active":"2024-03-11T22:20:29.438896+0000","last_became_peered":"2024-03-11T22:20:29.438896+0000","last_unstale":"2024-03-11T22:26:51.552588+0000","last_undegraded":"2024-03-11T22:26:51.552588+0000","last_fullsized":"2024-03-11T22:26:51.552588+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:09.040331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:09.040331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:17.396549+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"273'30","reported_seq":886,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551847+0000","last_change":"2024-03-11T22:26:05.020921+0000","last_active":"2024-03-11T22:26:51.551847+0000","last_peered":"2024-03-11T22:26:51.551847+0000","last_clean":"2024-03-11T22:26:51.551847+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:26:51.551847+0000","last_undegraded":"2024-03-11T22:26:51.551847+0000","last_fullsized":"2024-03-11T22:26:51.551847+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'30","last_scrub_stamp":"2024-03-11T22:26:05.020726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:05.020726+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:25.683138+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":800,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.553001+0000","last_change":"2024-03-11T22:23:57.680282+0000","last_active":"2024-03-11T22:26:51.553001+0000","last_peered":"2024-03-11T22:26:51.553001+0000","last_clean":"2024-03-11T22:26:51.553001+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:26:51.553001+0000","last_undegraded":"2024-03-11T22:26:51.553001+0000","last_fullsized":"2024-03-11T22:26:51.553001+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:57.680156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:57.680156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":554,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552040+0000","last_change":"2024-03-11T22:25:51.005350+0000","last_active":"2024-03-11T22:26:51.552040+0000","last_peered":"2024-03-11T22:26:51.552040+0000","last_clean":"2024-03-11T22:26:51.552040+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:26:51.552040+0000","last_undegraded":"2024-03-11T22:26:51.552040+0000","last_fullsized":"2024-03-11T22:26:51.552040+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:51.005219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:51.005219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:18.615750+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":1084,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829945+0000","last_change":"2024-03-11T22:26:49.699105+0000","last_active":"2024-03-11T22:26:50.829945+0000","last_peered":"2024-03-11T22:26:50.829945+0000","last_clean":"2024-03-11T22:26:50.829945+0000","last_became_active":"2024-03-11T22:26:49.698651+0000","last_became_peered":"2024-03-11T22:26:49.698651+0000","last_unstale":"2024-03-11T22:26:50.829945+0000","last_undegraded":"2024-03-11T22:26:50.829945+0000","last_fullsized":"2024-03-11T22:26:50.829945+0000","mapping_epoch":175,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:25:05.773963+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:25:05.773963+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.773963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":23,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.549274+0000","last_change":"2024-03-11T22:26:45.835270+0000","last_active":"2024-03-11T22:26:51.549274+0000","last_peered":"2024-03-11T22:26:51.549274+0000","last_clean":"2024-03-11T22:26:51.549274+0000","last_became_active":"2024-03-11T22:26:45.834922+0000","last_became_peered":"2024-03-11T22:26:45.834922+0000","last_unstale":"2024-03-11T22:26:51.549274+0000","last_undegraded":"2024-03-11T22:26:51.549274+0000","last_fullsized":"2024-03-11T22:26:51.549274+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:49.606249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":1009,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551961+0000","last_change":"2024-03-11T22:26:13.061283+0000","last_active":"2024-03-11T22:26:51.551961+0000","last_peered":"2024-03-11T22:26:51.551961+0000","last_clean":"2024-03-11T22:26:51.551961+0000","last_became_active":"2024-03-11T22:08:37.078566+0000","last_became_peered":"2024-03-11T22:08:37.078566+0000","last_unstale":"2024-03-11T22:26:51.551961+0000","last_undegraded":"2024-03-11T22:26:51.551961+0000","last_fullsized":"2024-03-11T22:26:51.551961+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:26:13.061151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:13.061151+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:22.425153+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"273'28","reported_seq":621,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.172798+0000","last_change":"2024-03-11T22:24:44.976788+0000","last_active":"2024-03-11T22:26:55.172798+0000","last_peered":"2024-03-11T22:26:55.172798+0000","last_clean":"2024-03-11T22:26:55.172798+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:26:55.172798+0000","last_undegraded":"2024-03-11T22:26:55.172798+0000","last_fullsized":"2024-03-11T22:26:55.172798+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:24:44.976524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:44.976524+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":798,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583628+0000","last_change":"2024-03-11T22:25:51.793304+0000","last_active":"2024-03-11T22:26:51.583628+0000","last_peered":"2024-03-11T22:26:51.583628+0000","last_clean":"2024-03-11T22:26:51.583628+0000","last_became_active":"2024-03-11T22:08:33.038773+0000","last_became_peered":"2024-03-11T22:08:33.038773+0000","last_unstale":"2024-03-11T22:26:51.583628+0000","last_undegraded":"2024-03-11T22:26:51.583628+0000","last_fullsized":"2024-03-11T22:26:51.583628+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:25:51.793162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:25:51.793162+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:52.184919+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":748,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721219+0000","last_change":"2024-03-11T22:23:59.089317+0000","last_active":"2024-03-11T22:26:54.721219+0000","last_peered":"2024-03-11T22:26:54.721219+0000","last_clean":"2024-03-11T22:26:54.721219+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:26:54.721219+0000","last_undegraded":"2024-03-11T22:26:54.721219+0000","last_fullsized":"2024-03-11T22:26:54.721219+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:59.089227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:23:59.089227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":1146,"reported_epoch":433,"state":"clean+premerge+peered","last_fresh":"2024-03-11T22:26:51.555359+0000","last_change":"2024-03-11T22:26:51.555359+0000","last_active":"2024-03-11T22:26:50.704984+0000","last_peered":"2024-03-11T22:26:51.555359+0000","last_clean":"2024-03-11T22:26:51.555359+0000","last_became_active":"2024-03-11T22:26:21.556125+0000","last_became_peered":"2024-03-11T22:26:51.554944+0000","last_unstale":"2024-03-11T22:26:51.555359+0000","last_undegraded":"2024-03-11T22:26:51.555359+0000","last_fullsized":"2024-03-11T22:26:51.555359+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:45.837466+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:45.837466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:12.059714+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":21,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829737+0000","last_change":"2024-03-11T22:26:45.682819+0000","last_active":"2024-03-11T22:26:50.829737+0000","last_peered":"2024-03-11T22:26:50.829737+0000","last_clean":"2024-03-11T22:26:50.829737+0000","last_became_active":"2024-03-11T22:26:45.682468+0000","last_became_peered":"2024-03-11T22:26:45.682468+0000","last_unstale":"2024-03-11T22:26:50.829737+0000","last_undegraded":"2024-03-11T22:26:50.829737+0000","last_fullsized":"2024-03-11T22:26:50.829737+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:08.905356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"378'9","reported_seq":1018,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583452+0000","last_change":"2024-03-11T22:24:54.792194+0000","last_active":"2024-03-11T22:26:51.583452+0000","last_peered":"2024-03-11T22:26:51.583452+0000","last_clean":"2024-03-11T22:26:51.583452+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:26:51.583452+0000","last_undegraded":"2024-03-11T22:26:51.583452+0000","last_fullsized":"2024-03-11T22:26:51.583452+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9","last_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_deep_scrub":"378'9","last_deep_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_clean_scrub_stamp":"2024-03-11T22:24:54.792049+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"273'20","reported_seq":831,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551071+0000","last_change":"2024-03-11T22:24:47.479305+0000","last_active":"2024-03-11T22:26:51.551071+0000","last_peered":"2024-03-11T22:26:51.551071+0000","last_clean":"2024-03-11T22:26:51.551071+0000","last_became_active":"2024-03-11T22:08:35.057386+0000","last_became_peered":"2024-03-11T22:08:35.057386+0000","last_unstale":"2024-03-11T22:26:51.551071+0000","last_undegraded":"2024-03-11T22:26:51.551071+0000","last_fullsized":"2024-03-11T22:26:51.551071+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:24:47.479158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:47.479158+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":802,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173847+0000","last_change":"2024-03-11T22:23:42.943711+0000","last_active":"2024-03-11T22:26:55.173847+0000","last_peered":"2024-03-11T22:26:55.173847+0000","last_clean":"2024-03-11T22:26:55.173847+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:26:55.173847+0000","last_undegraded":"2024-03-11T22:26:55.173847+0000","last_fullsized":"2024-03-11T22:26:55.173847+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:42.943546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:42.943546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":786,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.582881+0000","last_change":"2024-03-11T22:25:59.788866+0000","last_active":"2024-03-11T22:26:51.582881+0000","last_peered":"2024-03-11T22:26:51.582881+0000","last_clean":"2024-03-11T22:26:51.582881+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:26:51.582881+0000","last_undegraded":"2024-03-11T22:26:51.582881+0000","last_fullsized":"2024-03-11T22:26:51.582881+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:59.788747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:59.788747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:04.132355+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":1074,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721797+0000","last_change":"2024-03-11T22:26:41.661797+0000","last_active":"2024-03-11T22:26:54.721797+0000","last_peered":"2024-03-11T22:26:54.721797+0000","last_clean":"2024-03-11T22:26:54.721797+0000","last_became_active":"2024-03-11T22:26:41.661375+0000","last_became_peered":"2024-03-11T22:26:41.661375+0000","last_unstale":"2024-03-11T22:26:54.721797+0000","last_undegraded":"2024-03-11T22:26:54.721797+0000","last_fullsized":"2024-03-11T22:26:54.721797+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":423,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:25:57.758735+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:25:57.758735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:23.778277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":23,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.548784+0000","last_change":"2024-03-11T22:26:45.685624+0000","last_active":"2024-03-11T22:26:51.548784+0000","last_peered":"2024-03-11T22:26:51.548784+0000","last_clean":"2024-03-11T22:26:51.548784+0000","last_became_active":"2024-03-11T22:26:45.685318+0000","last_became_peered":"2024-03-11T22:26:45.685318+0000","last_unstale":"2024-03-11T22:26:51.548784+0000","last_undegraded":"2024-03-11T22:26:51.548784+0000","last_fullsized":"2024-03-11T22:26:51.548784+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:02.396864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":810,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721010+0000","last_change":"2024-03-11T22:24:00.941070+0000","last_active":"2024-03-11T22:26:54.721010+0000","last_peered":"2024-03-11T22:26:54.721010+0000","last_clean":"2024-03-11T22:26:54.721010+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:26:54.721010+0000","last_undegraded":"2024-03-11T22:26:54.721010+0000","last_fullsized":"2024-03-11T22:26:54.721010+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:00.940803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.940803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"429'37","reported_seq":852,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583277+0000","last_change":"2024-03-11T22:26:47.889297+0000","last_active":"2024-03-11T22:26:51.583277+0000","last_peered":"2024-03-11T22:26:51.583277+0000","last_clean":"2024-03-11T22:26:51.583277+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:26:51.583277+0000","last_undegraded":"2024-03-11T22:26:51.583277+0000","last_fullsized":"2024-03-11T22:26:51.583277+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'37","last_scrub_stamp":"2024-03-11T22:26:47.889156+0000","last_deep_scrub":"429'37","last_deep_scrub_stamp":"2024-03-11T22:26:47.889156+0000","last_clean_scrub_stamp":"2024-03-11T22:26:47.889156+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:07.369409+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":805,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.549064+0000","last_change":"2024-03-11T22:26:39.629292+0000","last_active":"2024-03-11T22:26:51.549064+0000","last_peered":"2024-03-11T22:26:51.549064+0000","last_clean":"2024-03-11T22:26:51.549064+0000","last_became_active":"2024-03-11T22:08:33.298240+0000","last_became_peered":"2024-03-11T22:08:33.298240+0000","last_unstale":"2024-03-11T22:26:51.549064+0000","last_undegraded":"2024-03-11T22:26:51.549064+0000","last_fullsized":"2024-03-11T22:26:51.549064+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:39.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:04.706082+0000","last_clean_scrub_stamp":"2024-03-11T22:26:39.629202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:51.361679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":793,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.548231+0000","last_change":"2024-03-11T22:26:44.604306+0000","last_active":"2024-03-11T22:26:51.548231+0000","last_peered":"2024-03-11T22:26:51.548231+0000","last_clean":"2024-03-11T22:26:51.548231+0000","last_became_active":"2024-03-11T22:08:39.075977+0000","last_became_peered":"2024-03-11T22:08:39.075977+0000","last_unstale":"2024-03-11T22:26:51.548231+0000","last_undegraded":"2024-03-11T22:26:51.548231+0000","last_fullsized":"2024-03-11T22:26:51.548231+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.604198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.604198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:13.635987+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":1049,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721761+0000","last_change":"2024-03-11T22:26:45.688875+0000","last_active":"2024-03-11T22:26:54.721761+0000","last_peered":"2024-03-11T22:26:54.721761+0000","last_clean":"2024-03-11T22:26:54.721761+0000","last_became_active":"2024-03-11T22:26:45.688609+0000","last_became_peered":"2024-03-11T22:26:45.688609+0000","last_unstale":"2024-03-11T22:26:54.721761+0000","last_undegraded":"2024-03-11T22:26:54.721761+0000","last_fullsized":"2024-03-11T22:26:54.721761+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:37.856736+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:37.856736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:05.906770+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":22,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551859+0000","last_change":"2024-03-11T22:26:45.685167+0000","last_active":"2024-03-11T22:26:51.551859+0000","last_peered":"2024-03-11T22:26:51.551859+0000","last_clean":"2024-03-11T22:26:51.551859+0000","last_became_active":"2024-03-11T22:26:45.684876+0000","last_became_peered":"2024-03-11T22:26:45.684876+0000","last_unstale":"2024-03-11T22:26:51.551859+0000","last_undegraded":"2024-03-11T22:26:51.551859+0000","last_fullsized":"2024-03-11T22:26:51.551859+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:06.555793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":794,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.174476+0000","last_change":"2024-03-11T22:24:39.906388+0000","last_active":"2024-03-11T22:26:55.174476+0000","last_peered":"2024-03-11T22:26:55.174476+0000","last_clean":"2024-03-11T22:26:55.174476+0000","last_became_active":"2024-03-11T22:08:37.081248+0000","last_became_peered":"2024-03-11T22:08:37.081248+0000","last_unstale":"2024-03-11T22:26:55.174476+0000","last_undegraded":"2024-03-11T22:26:55.174476+0000","last_fullsized":"2024-03-11T22:26:55.174476+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:39.906243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.906243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"273'24","reported_seq":837,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583569+0000","last_change":"2024-03-11T22:26:35.660852+0000","last_active":"2024-03-11T22:26:51.583569+0000","last_peered":"2024-03-11T22:26:51.583569+0000","last_clean":"2024-03-11T22:26:51.583569+0000","last_became_active":"2024-03-11T22:08:35.058190+0000","last_became_peered":"2024-03-11T22:08:35.058190+0000","last_unstale":"2024-03-11T22:26:51.583569+0000","last_undegraded":"2024-03-11T22:26:51.583569+0000","last_fullsized":"2024-03-11T22:26:51.583569+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'24","last_scrub_stamp":"2024-03-11T22:26:35.660739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:35.660739+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:51.081996+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":1884,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.548510+0000","last_change":"2024-03-11T22:26:35.670062+0000","last_active":"2024-03-11T22:26:51.548510+0000","last_peered":"2024-03-11T22:26:51.548510+0000","last_clean":"2024-03-11T22:26:51.548510+0000","last_became_active":"2024-03-11T22:10:35.419376+0000","last_became_peered":"2024-03-11T22:10:35.419376+0000","last_unstale":"2024-03-11T22:26:51.548510+0000","last_undegraded":"2024-03-11T22:26:51.548510+0000","last_fullsized":"2024-03-11T22:26:51.548510+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:26:35.669961+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:26:35.669961+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:39.940036+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1063,"num_read_kb":1058,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":797,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.582844+0000","last_change":"2024-03-11T22:24:46.870653+0000","last_active":"2024-03-11T22:26:51.582844+0000","last_peered":"2024-03-11T22:26:51.582844+0000","last_clean":"2024-03-11T22:26:51.582844+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:26:51.582844+0000","last_undegraded":"2024-03-11T22:26:51.582844+0000","last_fullsized":"2024-03-11T22:26:51.582844+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:46.870527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.870527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":801,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.548620+0000","last_change":"2024-03-11T22:26:33.609508+0000","last_active":"2024-03-11T22:26:51.548620+0000","last_peered":"2024-03-11T22:26:51.548620+0000","last_clean":"2024-03-11T22:26:51.548620+0000","last_became_active":"2024-03-11T22:08:39.076662+0000","last_became_peered":"2024-03-11T22:08:39.076662+0000","last_unstale":"2024-03-11T22:26:51.548620+0000","last_undegraded":"2024-03-11T22:26:51.548620+0000","last_fullsized":"2024-03-11T22:26:51.548620+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:33.609412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:22.750560+0000","last_clean_scrub_stamp":"2024-03-11T22:26:33.609412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:43.463193+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":1124,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.550620+0000","last_change":"2024-03-11T22:26:25.579952+0000","last_active":"2024-03-11T22:26:51.550620+0000","last_peered":"2024-03-11T22:26:51.550620+0000","last_clean":"2024-03-11T22:26:51.550620+0000","last_became_active":"2024-03-11T22:26:25.579321+0000","last_became_peered":"2024-03-11T22:26:25.579321+0000","last_unstale":"2024-03-11T22:26:51.550620+0000","last_undegraded":"2024-03-11T22:26:51.550620+0000","last_fullsized":"2024-03-11T22:26:51.550620+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:45.428590+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:45.428590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":22,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551599+0000","last_change":"2024-03-11T22:26:45.687046+0000","last_active":"2024-03-11T22:26:51.551599+0000","last_peered":"2024-03-11T22:26:51.551599+0000","last_clean":"2024-03-11T22:26:51.551599+0000","last_became_active":"2024-03-11T22:26:45.686711+0000","last_became_peered":"2024-03-11T22:26:45.686711+0000","last_unstale":"2024-03-11T22:26:51.551599+0000","last_undegraded":"2024-03-11T22:26:51.551599+0000","last_fullsized":"2024-03-11T22:26:51.551599+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:54.022765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":769,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.830069+0000","last_change":"2024-03-11T22:26:42.960839+0000","last_active":"2024-03-11T22:26:50.830069+0000","last_peered":"2024-03-11T22:26:50.830069+0000","last_clean":"2024-03-11T22:26:50.830069+0000","last_became_active":"2024-03-11T22:20:29.420497+0000","last_became_peered":"2024-03-11T22:20:29.420497+0000","last_unstale":"2024-03-11T22:26:50.830069+0000","last_undegraded":"2024-03-11T22:26:50.830069+0000","last_fullsized":"2024-03-11T22:26:50.830069+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:42.960739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:26:42.960739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:02.582899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"273'28","reported_seq":1075,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721473+0000","last_change":"2024-03-11T22:24:42.729950+0000","last_active":"2024-03-11T22:26:54.721473+0000","last_peered":"2024-03-11T22:26:54.721473+0000","last_clean":"2024-03-11T22:26:54.721473+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:26:54.721473+0000","last_undegraded":"2024-03-11T22:26:54.721473+0000","last_fullsized":"2024-03-11T22:26:54.721473+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:24:42.729799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:42.729799+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":788,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551693+0000","last_change":"2024-03-11T22:24:47.653085+0000","last_active":"2024-03-11T22:26:51.551693+0000","last_peered":"2024-03-11T22:26:51.551693+0000","last_clean":"2024-03-11T22:26:51.551693+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:26:51.551693+0000","last_undegraded":"2024-03-11T22:26:51.551693+0000","last_fullsized":"2024-03-11T22:26:51.551693+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:47.652970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:24:47.652970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":1142,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.582909+0000","last_change":"2024-03-11T22:26:37.632763+0000","last_active":"2024-03-11T22:26:51.582909+0000","last_peered":"2024-03-11T22:26:51.582909+0000","last_clean":"2024-03-11T22:26:51.582909+0000","last_became_active":"2024-03-11T22:26:37.632412+0000","last_became_peered":"2024-03-11T22:26:37.632412+0000","last_unstale":"2024-03-11T22:26:51.582909+0000","last_undegraded":"2024-03-11T22:26:51.582909+0000","last_fullsized":"2024-03-11T22:26:51.582909+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":19,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:42.773024+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:24:42.773024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":790,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173697+0000","last_change":"2024-03-11T22:24:54.991281+0000","last_active":"2024-03-11T22:26:55.173697+0000","last_peered":"2024-03-11T22:26:55.173697+0000","last_clean":"2024-03-11T22:26:55.173697+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:26:55.173697+0000","last_undegraded":"2024-03-11T22:26:55.173697+0000","last_fullsized":"2024-03-11T22:26:55.173697+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:54.991081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:54.991081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":22,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551429+0000","last_change":"2024-03-11T22:26:45.685997+0000","last_active":"2024-03-11T22:26:51.551429+0000","last_peered":"2024-03-11T22:26:51.551429+0000","last_clean":"2024-03-11T22:26:51.551429+0000","last_became_active":"2024-03-11T22:26:45.685687+0000","last_became_peered":"2024-03-11T22:26:45.685687+0000","last_unstale":"2024-03-11T22:26:51.551429+0000","last_undegraded":"2024-03-11T22:26:51.551429+0000","last_fullsized":"2024-03-11T22:26:51.551429+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:50.638451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":811,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.830044+0000","last_change":"2024-03-11T22:24:48.727176+0000","last_active":"2024-03-11T22:26:50.830044+0000","last_peered":"2024-03-11T22:26:50.830044+0000","last_clean":"2024-03-11T22:26:50.830044+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:26:50.830044+0000","last_undegraded":"2024-03-11T22:26:50.830044+0000","last_fullsized":"2024-03-11T22:26:50.830044+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:48.727061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:48.727061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"273'17","reported_seq":834,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.548960+0000","last_change":"2024-03-11T22:24:41.396924+0000","last_active":"2024-03-11T22:26:51.548960+0000","last_peered":"2024-03-11T22:26:51.548960+0000","last_clean":"2024-03-11T22:26:51.548960+0000","last_became_active":"2024-03-11T22:08:35.057208+0000","last_became_peered":"2024-03-11T22:08:35.057208+0000","last_unstale":"2024-03-11T22:26:51.548960+0000","last_undegraded":"2024-03-11T22:26:51.548960+0000","last_fullsized":"2024-03-11T22:26:51.548960+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'17","last_scrub_stamp":"2024-03-11T22:24:41.396754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.396754+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":836,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829222+0000","last_change":"2024-03-11T22:26:06.037315+0000","last_active":"2024-03-11T22:26:50.829222+0000","last_peered":"2024-03-11T22:26:50.829222+0000","last_clean":"2024-03-11T22:26:50.829222+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:26:50.829222+0000","last_undegraded":"2024-03-11T22:26:50.829222+0000","last_fullsized":"2024-03-11T22:26:50.829222+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:06.037180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:26:06.037180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:21.409787+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":806,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.548649+0000","last_change":"2024-03-11T22:23:58.411257+0000","last_active":"2024-03-11T22:26:51.548649+0000","last_peered":"2024-03-11T22:26:51.548649+0000","last_clean":"2024-03-11T22:26:51.548649+0000","last_became_active":"2024-03-11T22:08:33.300342+0000","last_became_peered":"2024-03-11T22:08:33.300342+0000","last_unstale":"2024-03-11T22:26:51.548649+0000","last_undegraded":"2024-03-11T22:26:51.548649+0000","last_fullsized":"2024-03-11T22:26:51.548649+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:58.411052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.411052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":748,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721474+0000","last_change":"2024-03-11T22:24:38.834963+0000","last_active":"2024-03-11T22:26:54.721474+0000","last_peered":"2024-03-11T22:26:54.721474+0000","last_clean":"2024-03-11T22:26:54.721474+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:26:54.721474+0000","last_undegraded":"2024-03-11T22:26:54.721474+0000","last_fullsized":"2024-03-11T22:26:54.721474+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:38.834828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:24:38.834828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":1157,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.582872+0000","last_change":"2024-03-11T22:26:32.769183+0000","last_active":"2024-03-11T22:26:51.582872+0000","last_peered":"2024-03-11T22:26:51.582872+0000","last_clean":"2024-03-11T22:26:51.582872+0000","last_became_active":"2024-03-11T22:26:29.590669+0000","last_became_peered":"2024-03-11T22:26:29.590669+0000","last_unstale":"2024-03-11T22:26:51.582872+0000","last_undegraded":"2024-03-11T22:26:51.582872+0000","last_fullsized":"2024-03-11T22:26:51.582872+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:32.769096+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:32.769096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:44.408874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":22,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551990+0000","last_change":"2024-03-11T22:26:45.689140+0000","last_active":"2024-03-11T22:26:51.551990+0000","last_peered":"2024-03-11T22:26:51.551990+0000","last_clean":"2024-03-11T22:26:51.551990+0000","last_became_active":"2024-03-11T22:26:45.688545+0000","last_became_peered":"2024-03-11T22:26:45.688545+0000","last_unstale":"2024-03-11T22:26:51.551990+0000","last_undegraded":"2024-03-11T22:26:51.551990+0000","last_fullsized":"2024-03-11T22:26:51.551990+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:58.891414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":789,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583269+0000","last_change":"2024-03-11T22:24:01.031042+0000","last_active":"2024-03-11T22:26:51.583269+0000","last_peered":"2024-03-11T22:26:51.583269+0000","last_clean":"2024-03-11T22:26:51.583269+0000","last_became_active":"2024-03-11T22:08:37.074106+0000","last_became_peered":"2024-03-11T22:08:37.074106+0000","last_unstale":"2024-03-11T22:26:51.583269+0000","last_undegraded":"2024-03-11T22:26:51.583269+0000","last_fullsized":"2024-03-11T22:26:51.583269+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:01.030925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:01.030925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"273'40","reported_seq":1087,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.549168+0000","last_change":"2024-03-11T22:23:57.429937+0000","last_active":"2024-03-11T22:26:51.549168+0000","last_peered":"2024-03-11T22:26:51.549168+0000","last_clean":"2024-03-11T22:26:51.549168+0000","last_became_active":"2024-03-11T22:08:35.060969+0000","last_became_peered":"2024-03-11T22:08:35.060969+0000","last_unstale":"2024-03-11T22:26:51.549168+0000","last_undegraded":"2024-03-11T22:26:51.549168+0000","last_fullsized":"2024-03-11T22:26:51.549168+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'40","last_scrub_stamp":"2024-03-11T22:23:57.429825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:57.429825+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":564,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583178+0000","last_change":"2024-03-11T22:26:44.809895+0000","last_active":"2024-03-11T22:26:51.583178+0000","last_peered":"2024-03-11T22:26:51.583178+0000","last_clean":"2024-03-11T22:26:51.583178+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:26:51.583178+0000","last_undegraded":"2024-03-11T22:26:51.583178+0000","last_fullsized":"2024-03-11T22:26:51.583178+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.809803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.809803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:52.696010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":822,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173316+0000","last_change":"2024-03-11T22:26:31.988512+0000","last_active":"2024-03-11T22:26:55.173316+0000","last_peered":"2024-03-11T22:26:55.173316+0000","last_clean":"2024-03-11T22:26:55.173316+0000","last_became_active":"2024-03-11T22:08:32.029206+0000","last_became_peered":"2024-03-11T22:08:32.029206+0000","last_unstale":"2024-03-11T22:26:55.173316+0000","last_undegraded":"2024-03-11T22:26:55.173316+0000","last_fullsized":"2024-03-11T22:26:55.173316+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:26:31.988426+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:26:31.988426+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:33.647740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":788,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552918+0000","last_change":"2024-03-11T22:25:58.161115+0000","last_active":"2024-03-11T22:26:51.552918+0000","last_peered":"2024-03-11T22:26:51.552918+0000","last_clean":"2024-03-11T22:26:51.552918+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:26:51.552918+0000","last_undegraded":"2024-03-11T22:26:51.552918+0000","last_fullsized":"2024-03-11T22:26:51.552918+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:58.160972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:25:58.160972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:22.705703+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":1138,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.172668+0000","last_change":"2024-03-11T22:26:34.036467+0000","last_active":"2024-03-11T22:26:55.172668+0000","last_peered":"2024-03-11T22:26:55.172668+0000","last_clean":"2024-03-11T22:26:55.172668+0000","last_became_active":"2024-03-11T22:26:33.607145+0000","last_became_peered":"2024-03-11T22:26:33.607145+0000","last_unstale":"2024-03-11T22:26:55.172668+0000","last_undegraded":"2024-03-11T22:26:55.172668+0000","last_fullsized":"2024-03-11T22:26:55.172668+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:34.036387+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:34.036387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:54.518449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"436'109","reported_seq":138,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.684497+0000","last_change":"2024-03-11T22:26:45.834759+0000","last_active":"2024-03-11T22:26:55.684497+0000","last_peered":"2024-03-11T22:26:55.684497+0000","last_clean":"2024-03-11T22:26:55.684497+0000","last_became_active":"2024-03-11T22:26:45.834448+0000","last_became_peered":"2024-03-11T22:26:45.834448+0000","last_unstale":"2024-03-11T22:26:55.684497+0000","last_undegraded":"2024-03-11T22:26:55.684497+0000","last_fullsized":"2024-03-11T22:26:55.684497+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:57.363090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"273'15","reported_seq":834,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583412+0000","last_change":"2024-03-11T22:26:12.661812+0000","last_active":"2024-03-11T22:26:51.583412+0000","last_peered":"2024-03-11T22:26:51.583412+0000","last_clean":"2024-03-11T22:26:51.583412+0000","last_became_active":"2024-03-11T22:08:54.610256+0000","last_became_peered":"2024-03-11T22:08:54.610256+0000","last_unstale":"2024-03-11T22:26:51.583412+0000","last_undegraded":"2024-03-11T22:26:51.583412+0000","last_fullsized":"2024-03-11T22:26:51.583412+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:26:12.661479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:12.661479+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:19.570114+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":802,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173571+0000","last_change":"2024-03-11T22:24:58.853739+0000","last_active":"2024-03-11T22:26:55.173571+0000","last_peered":"2024-03-11T22:26:55.173571+0000","last_clean":"2024-03-11T22:26:55.173571+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:26:55.173571+0000","last_undegraded":"2024-03-11T22:26:55.173571+0000","last_fullsized":"2024-03-11T22:26:55.173571+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:58.853630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:58.853630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":819,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829431+0000","last_change":"2024-03-11T22:26:42.009815+0000","last_active":"2024-03-11T22:26:50.829431+0000","last_peered":"2024-03-11T22:26:50.829431+0000","last_clean":"2024-03-11T22:26:50.829431+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:26:50.829431+0000","last_undegraded":"2024-03-11T22:26:50.829431+0000","last_fullsized":"2024-03-11T22:26:50.829431+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:42.009705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:26:42.009705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:01.285782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":797,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583034+0000","last_change":"2024-03-11T22:24:53.948958+0000","last_active":"2024-03-11T22:26:51.583034+0000","last_peered":"2024-03-11T22:26:51.583034+0000","last_clean":"2024-03-11T22:26:51.583034+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:26:51.583034+0000","last_undegraded":"2024-03-11T22:26:51.583034+0000","last_fullsized":"2024-03-11T22:26:51.583034+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:53.948853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.948853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"436'115","reported_seq":145,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.699464+0000","last_change":"2024-03-11T22:26:45.835375+0000","last_active":"2024-03-11T22:26:55.699464+0000","last_peered":"2024-03-11T22:26:55.699464+0000","last_clean":"2024-03-11T22:26:55.699464+0000","last_became_active":"2024-03-11T22:26:45.834812+0000","last_became_peered":"2024-03-11T22:26:45.834812+0000","last_unstale":"2024-03-11T22:26:55.699464+0000","last_undegraded":"2024-03-11T22:26:55.699464+0000","last_fullsized":"2024-03-11T22:26:55.699464+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:44.806694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7536640,"num_objects":115,"num_object_clones":0,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":7360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"273'35","reported_seq":921,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.720886+0000","last_change":"2024-03-11T22:24:46.752170+0000","last_active":"2024-03-11T22:26:54.720886+0000","last_peered":"2024-03-11T22:26:54.720886+0000","last_clean":"2024-03-11T22:26:54.720886+0000","last_became_active":"2024-03-11T22:20:29.418041+0000","last_became_peered":"2024-03-11T22:20:29.418041+0000","last_unstale":"2024-03-11T22:26:54.720886+0000","last_undegraded":"2024-03-11T22:26:54.720886+0000","last_fullsized":"2024-03-11T22:26:54.720886+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'35","last_scrub_stamp":"2024-03-11T22:24:46.751969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.751969+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":753,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.174130+0000","last_change":"2024-03-11T22:26:04.099295+0000","last_active":"2024-03-11T22:26:55.174130+0000","last_peered":"2024-03-11T22:26:55.174130+0000","last_clean":"2024-03-11T22:26:55.174130+0000","last_became_active":"2024-03-11T22:20:29.413664+0000","last_became_peered":"2024-03-11T22:20:29.413664+0000","last_unstale":"2024-03-11T22:26:55.174130+0000","last_undegraded":"2024-03-11T22:26:55.174130+0000","last_fullsized":"2024-03-11T22:26:55.174130+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:04.099165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:04.099165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:33.591259+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":811,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829622+0000","last_change":"2024-03-11T22:26:09.051839+0000","last_active":"2024-03-11T22:26:50.829622+0000","last_peered":"2024-03-11T22:26:50.829622+0000","last_clean":"2024-03-11T22:26:50.829622+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:26:50.829622+0000","last_undegraded":"2024-03-11T22:26:50.829622+0000","last_fullsized":"2024-03-11T22:26:50.829622+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:09.051546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:26:09.051546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:14.339623+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":769,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829354+0000","last_change":"2024-03-11T22:26:02.011753+0000","last_active":"2024-03-11T22:26:50.829354+0000","last_peered":"2024-03-11T22:26:50.829354+0000","last_clean":"2024-03-11T22:26:50.829354+0000","last_became_active":"2024-03-11T22:20:29.415565+0000","last_became_peered":"2024-03-11T22:20:29.415565+0000","last_unstale":"2024-03-11T22:26:50.829354+0000","last_undegraded":"2024-03-11T22:26:50.829354+0000","last_fullsized":"2024-03-11T22:26:50.829354+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:02.011534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:20.931122+0000","last_clean_scrub_stamp":"2024-03-11T22:26:02.011534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:30.707282+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":23,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583438+0000","last_change":"2024-03-11T22:26:45.685196+0000","last_active":"2024-03-11T22:26:51.583438+0000","last_peered":"2024-03-11T22:26:51.583438+0000","last_clean":"2024-03-11T22:26:51.583438+0000","last_became_active":"2024-03-11T22:26:45.684886+0000","last_became_peered":"2024-03-11T22:26:45.684886+0000","last_unstale":"2024-03-11T22:26:51.583438+0000","last_undegraded":"2024-03-11T22:26:51.583438+0000","last_fullsized":"2024-03-11T22:26:51.583438+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:55.577692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":811,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829065+0000","last_change":"2024-03-11T22:24:56.704048+0000","last_active":"2024-03-11T22:26:50.829065+0000","last_peered":"2024-03-11T22:26:50.829065+0000","last_clean":"2024-03-11T22:26:50.829065+0000","last_became_active":"2024-03-11T22:08:37.070090+0000","last_became_peered":"2024-03-11T22:08:37.070090+0000","last_unstale":"2024-03-11T22:26:50.829065+0000","last_undegraded":"2024-03-11T22:26:50.829065+0000","last_fullsized":"2024-03-11T22:26:50.829065+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:56.703933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:24:56.703933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"273'18","reported_seq":626,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173004+0000","last_change":"2024-03-11T22:26:46.174985+0000","last_active":"2024-03-11T22:26:55.173004+0000","last_peered":"2024-03-11T22:26:55.173004+0000","last_clean":"2024-03-11T22:26:55.173004+0000","last_became_active":"2024-03-11T22:14:52.247400+0000","last_became_peered":"2024-03-11T22:14:52.247400+0000","last_unstale":"2024-03-11T22:26:55.173004+0000","last_undegraded":"2024-03-11T22:26:55.173004+0000","last_fullsized":"2024-03-11T22:26:55.173004+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:26:46.174869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:46.174869+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:48.333901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":579,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.549456+0000","last_change":"2024-03-11T22:24:53.435632+0000","last_active":"2024-03-11T22:26:51.549456+0000","last_peered":"2024-03-11T22:26:51.549456+0000","last_clean":"2024-03-11T22:26:51.549456+0000","last_became_active":"2024-03-11T22:20:29.416729+0000","last_became_peered":"2024-03-11T22:20:29.416729+0000","last_unstale":"2024-03-11T22:26:51.549456+0000","last_undegraded":"2024-03-11T22:26:51.549456+0000","last_fullsized":"2024-03-11T22:26:51.549456+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:53.435516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.435516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":798,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552471+0000","last_change":"2024-03-11T22:26:35.021997+0000","last_active":"2024-03-11T22:26:51.552471+0000","last_peered":"2024-03-11T22:26:51.552471+0000","last_clean":"2024-03-11T22:26:51.552471+0000","last_became_active":"2024-03-11T22:20:29.438413+0000","last_became_peered":"2024-03-11T22:20:29.438413+0000","last_unstale":"2024-03-11T22:26:51.552471+0000","last_undegraded":"2024-03-11T22:26:51.552471+0000","last_fullsized":"2024-03-11T22:26:51.552471+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:26:35.021902+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:26:35.021902+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:58.885618+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"436'123","reported_seq":153,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.708747+0000","last_change":"2024-03-11T22:26:45.683751+0000","last_active":"2024-03-11T22:26:55.708747+0000","last_peered":"2024-03-11T22:26:55.708747+0000","last_clean":"2024-03-11T22:26:55.708747+0000","last_became_active":"2024-03-11T22:26:45.683460+0000","last_became_peered":"2024-03-11T22:26:45.683460+0000","last_unstale":"2024-03-11T22:26:55.708747+0000","last_undegraded":"2024-03-11T22:26:55.708747+0000","last_fullsized":"2024-03-11T22:26:55.708747+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:49.655910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":123,"num_object_clones":0,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":7872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"310'11","reported_seq":991,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.720695+0000","last_change":"2024-03-11T22:23:56.945205+0000","last_active":"2024-03-11T22:26:54.720695+0000","last_peered":"2024-03-11T22:26:54.720695+0000","last_clean":"2024-03-11T22:26:54.720695+0000","last_became_active":"2024-03-11T22:20:29.418355+0000","last_became_peered":"2024-03-11T22:20:29.418355+0000","last_unstale":"2024-03-11T22:26:54.720695+0000","last_undegraded":"2024-03-11T22:26:54.720695+0000","last_fullsized":"2024-03-11T22:26:54.720695+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:23:56.945075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:56.945075+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"273'34","reported_seq":900,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.172695+0000","last_change":"2024-03-11T22:26:45.212688+0000","last_active":"2024-03-11T22:26:55.172695+0000","last_peered":"2024-03-11T22:26:55.172695+0000","last_clean":"2024-03-11T22:26:55.172695+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:26:55.172695+0000","last_undegraded":"2024-03-11T22:26:55.172695+0000","last_fullsized":"2024-03-11T22:26:55.172695+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:26:45.212591+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:26:45.212591+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:09.388275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"432'4","reported_seq":599,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583696+0000","last_change":"2024-03-11T22:26:50.847434+0000","last_active":"2024-03-11T22:26:51.583696+0000","last_peered":"2024-03-11T22:26:51.583696+0000","last_clean":"2024-03-11T22:26:51.583696+0000","last_became_active":"2024-03-11T22:20:29.415226+0000","last_became_peered":"2024-03-11T22:20:29.415226+0000","last_unstale":"2024-03-11T22:26:51.583696+0000","last_undegraded":"2024-03-11T22:26:51.583696+0000","last_fullsized":"2024-03-11T22:26:51.583696+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'4","last_scrub_stamp":"2024-03-11T22:26:50.847349+0000","last_deep_scrub":"432'4","last_deep_scrub_stamp":"2024-03-11T22:26:50.847349+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.847349+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:17.115592+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":790,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.172257+0000","last_change":"2024-03-11T22:26:49.194654+0000","last_active":"2024-03-11T22:26:55.172257+0000","last_peered":"2024-03-11T22:26:55.172257+0000","last_clean":"2024-03-11T22:26:55.172257+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:26:55.172257+0000","last_undegraded":"2024-03-11T22:26:55.172257+0000","last_fullsized":"2024-03-11T22:26:55.172257+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:49.194569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:49.194569+0000","last_clean_scrub_stamp":"2024-03-11T22:26:49.194569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:04.923121+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"273'12","reported_seq":829,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.553255+0000","last_change":"2024-03-11T22:25:52.052477+0000","last_active":"2024-03-11T22:26:51.553255+0000","last_peered":"2024-03-11T22:26:51.553255+0000","last_clean":"2024-03-11T22:26:51.553255+0000","last_became_active":"2024-03-11T22:20:29.441599+0000","last_became_peered":"2024-03-11T22:20:29.441599+0000","last_unstale":"2024-03-11T22:26:51.553255+0000","last_undegraded":"2024-03-11T22:26:51.553255+0000","last_fullsized":"2024-03-11T22:26:51.553255+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'12","last_scrub_stamp":"2024-03-11T22:25:52.052338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:52.052338+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:06.836803+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":809,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.720034+0000","last_change":"2024-03-11T22:23:48.806202+0000","last_active":"2024-03-11T22:26:54.720034+0000","last_peered":"2024-03-11T22:26:54.720034+0000","last_clean":"2024-03-11T22:26:54.720034+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:26:54.720034+0000","last_undegraded":"2024-03-11T22:26:54.720034+0000","last_fullsized":"2024-03-11T22:26:54.720034+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.805960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.805960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":793,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173234+0000","last_change":"2024-03-11T22:23:59.713751+0000","last_active":"2024-03-11T22:26:55.173234+0000","last_peered":"2024-03-11T22:26:55.173234+0000","last_clean":"2024-03-11T22:26:55.173234+0000","last_became_active":"2024-03-11T22:08:37.077473+0000","last_became_peered":"2024-03-11T22:08:37.077473+0000","last_unstale":"2024-03-11T22:26:55.173234+0000","last_undegraded":"2024-03-11T22:26:55.173234+0000","last_fullsized":"2024-03-11T22:26:55.173234+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:59.713626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:23:59.713626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":739,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721662+0000","last_change":"2024-03-11T22:23:46.876978+0000","last_active":"2024-03-11T22:26:54.721662+0000","last_peered":"2024-03-11T22:26:54.721662+0000","last_clean":"2024-03-11T22:26:54.721662+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:26:54.721662+0000","last_undegraded":"2024-03-11T22:26:54.721662+0000","last_fullsized":"2024-03-11T22:26:54.721662+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:46.876749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.876749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"273'19","reported_seq":853,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583727+0000","last_change":"2024-03-11T22:23:53.926458+0000","last_active":"2024-03-11T22:26:51.583727+0000","last_peered":"2024-03-11T22:26:51.583727+0000","last_clean":"2024-03-11T22:26:51.583727+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:26:51.583727+0000","last_undegraded":"2024-03-11T22:26:51.583727+0000","last_fullsized":"2024-03-11T22:26:51.583727+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_clean_scrub_stamp":"2024-03-11T22:23:53.926316+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":580,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.720619+0000","last_change":"2024-03-11T22:26:43.803872+0000","last_active":"2024-03-11T22:26:54.720619+0000","last_peered":"2024-03-11T22:26:54.720619+0000","last_clean":"2024-03-11T22:26:54.720619+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:26:54.720619+0000","last_undegraded":"2024-03-11T22:26:54.720619+0000","last_fullsized":"2024-03-11T22:26:54.720619+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:43.803749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:43.803749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:59.989139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":791,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.553170+0000","last_change":"2024-03-11T22:24:51.687892+0000","last_active":"2024-03-11T22:26:51.553170+0000","last_peered":"2024-03-11T22:26:51.553170+0000","last_clean":"2024-03-11T22:26:51.553170+0000","last_became_active":"2024-03-11T22:20:29.433005+0000","last_became_peered":"2024-03-11T22:20:29.433005+0000","last_unstale":"2024-03-11T22:26:51.553170+0000","last_undegraded":"2024-03-11T22:26:51.553170+0000","last_fullsized":"2024-03-11T22:26:51.553170+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:51.687767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:51.687767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":814,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.830027+0000","last_change":"2024-03-11T22:26:12.930124+0000","last_active":"2024-03-11T22:26:50.830027+0000","last_peered":"2024-03-11T22:26:50.830027+0000","last_clean":"2024-03-11T22:26:50.830027+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:26:50.830027+0000","last_undegraded":"2024-03-11T22:26:50.830027+0000","last_fullsized":"2024-03-11T22:26:50.830027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:12.930004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:26:12.930004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:33.373509+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"273'19","reported_seq":834,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.174169+0000","last_change":"2024-03-11T22:26:12.149477+0000","last_active":"2024-03-11T22:26:55.174169+0000","last_peered":"2024-03-11T22:26:55.174169+0000","last_clean":"2024-03-11T22:26:55.174169+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:26:55.174169+0000","last_undegraded":"2024-03-11T22:26:55.174169+0000","last_fullsized":"2024-03-11T22:26:55.174169+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:26:12.149360+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:26:12.149360+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:27.308834+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":808,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.582788+0000","last_change":"2024-03-11T22:23:50.030718+0000","last_active":"2024-03-11T22:26:51.582788+0000","last_peered":"2024-03-11T22:26:51.582788+0000","last_clean":"2024-03-11T22:26:51.582788+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:26:51.582788+0000","last_undegraded":"2024-03-11T22:26:51.582788+0000","last_fullsized":"2024-03-11T22:26:51.582788+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:50.030272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:50.030272+0000","last_clean_scrub_stamp":"2024-03-11T22:23:50.030272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'11","reported_seq":1042,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.553104+0000","last_change":"2024-03-11T22:24:53.646061+0000","last_active":"2024-03-11T22:26:51.553104+0000","last_peered":"2024-03-11T22:26:51.553104+0000","last_clean":"2024-03-11T22:26:51.553104+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:26:51.553104+0000","last_undegraded":"2024-03-11T22:26:51.553104+0000","last_fullsized":"2024-03-11T22:26:51.553104+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:24:53.645906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.645906+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":576,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829918+0000","last_change":"2024-03-11T22:25:58.161426+0000","last_active":"2024-03-11T22:26:50.829918+0000","last_peered":"2024-03-11T22:26:50.829918+0000","last_clean":"2024-03-11T22:26:50.829918+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:26:50.829918+0000","last_undegraded":"2024-03-11T22:26:50.829918+0000","last_fullsized":"2024-03-11T22:26:50.829918+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:58.161144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:48.490757+0000","last_clean_scrub_stamp":"2024-03-11T22:25:58.161144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:23.873459+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"273'20","reported_seq":831,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552973+0000","last_change":"2024-03-11T22:25:56.196465+0000","last_active":"2024-03-11T22:26:51.552973+0000","last_peered":"2024-03-11T22:26:51.552973+0000","last_clean":"2024-03-11T22:26:51.552973+0000","last_became_active":"2024-03-11T22:16:15.815814+0000","last_became_peered":"2024-03-11T22:16:15.815814+0000","last_unstale":"2024-03-11T22:26:51.552973+0000","last_undegraded":"2024-03-11T22:26:51.552973+0000","last_fullsized":"2024-03-11T22:26:51.552973+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:25:56.196035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:56.196035+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:26:56.306335+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":601,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.034951+0000","last_change":"2024-03-11T22:26:51.034990+0000","last_active":"2024-03-11T22:26:51.034951+0000","last_peered":"2024-03-11T22:26:51.034951+0000","last_clean":"2024-03-11T22:26:51.034951+0000","last_became_active":"2024-03-11T22:20:29.415019+0000","last_became_peered":"2024-03-11T22:20:29.415019+0000","last_unstale":"2024-03-11T22:26:51.034951+0000","last_undegraded":"2024-03-11T22:26:51.034951+0000","last_fullsized":"2024-03-11T22:26:51.034951+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:51.034868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:51.034868+0000","last_clean_scrub_stamp":"2024-03-11T22:26:51.034868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:58.249445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":1022,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173714+0000","last_change":"2024-03-11T22:23:53.776480+0000","last_active":"2024-03-11T22:26:55.173714+0000","last_peered":"2024-03-11T22:26:55.173714+0000","last_clean":"2024-03-11T22:26:55.173714+0000","last_became_active":"2024-03-11T22:08:37.076321+0000","last_became_peered":"2024-03-11T22:08:37.076321+0000","last_unstale":"2024-03-11T22:26:55.173714+0000","last_undegraded":"2024-03-11T22:26:55.173714+0000","last_fullsized":"2024-03-11T22:26:55.173714+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:23:53.776342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:53.776342+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":788,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583534+0000","last_change":"2024-03-11T22:24:45.964526+0000","last_active":"2024-03-11T22:26:51.583534+0000","last_peered":"2024-03-11T22:26:51.583534+0000","last_clean":"2024-03-11T22:26:51.583534+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:26:51.583534+0000","last_undegraded":"2024-03-11T22:26:51.583534+0000","last_fullsized":"2024-03-11T22:26:51.583534+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:45.964401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:45.964401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"273'19","reported_seq":626,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721127+0000","last_change":"2024-03-11T22:26:11.820643+0000","last_active":"2024-03-11T22:26:54.721127+0000","last_peered":"2024-03-11T22:26:54.721127+0000","last_clean":"2024-03-11T22:26:54.721127+0000","last_became_active":"2024-03-11T22:14:52.239810+0000","last_became_peered":"2024-03-11T22:14:52.239810+0000","last_unstale":"2024-03-11T22:26:54.721127+0000","last_undegraded":"2024-03-11T22:26:54.721127+0000","last_fullsized":"2024-03-11T22:26:54.721127+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:26:11.820506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:11.820506+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:22.134595+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":605,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721103+0000","last_change":"2024-03-11T22:26:38.840815+0000","last_active":"2024-03-11T22:26:54.721103+0000","last_peered":"2024-03-11T22:26:54.721103+0000","last_clean":"2024-03-11T22:26:54.721103+0000","last_became_active":"2024-03-11T22:20:29.420867+0000","last_became_peered":"2024-03-11T22:20:29.420867+0000","last_unstale":"2024-03-11T22:26:54.721103+0000","last_undegraded":"2024-03-11T22:26:54.721103+0000","last_fullsized":"2024-03-11T22:26:54.721103+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:26:38.840685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:38.840685+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:04.949659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":796,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551018+0000","last_change":"2024-03-11T22:25:55.622694+0000","last_active":"2024-03-11T22:26:51.551018+0000","last_peered":"2024-03-11T22:26:51.551018+0000","last_clean":"2024-03-11T22:26:51.551018+0000","last_became_active":"2024-03-11T22:08:37.074987+0000","last_became_peered":"2024-03-11T22:08:37.074987+0000","last_unstale":"2024-03-11T22:26:51.551018+0000","last_undegraded":"2024-03-11T22:26:51.551018+0000","last_fullsized":"2024-03-11T22:26:51.551018+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:55.622561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:25:55.622561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:17.810087+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":767,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551745+0000","last_change":"2024-03-11T22:23:50.723665+0000","last_active":"2024-03-11T22:26:51.551745+0000","last_peered":"2024-03-11T22:26:51.551745+0000","last_clean":"2024-03-11T22:26:51.551745+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:26:51.551745+0000","last_undegraded":"2024-03-11T22:26:51.551745+0000","last_fullsized":"2024-03-11T22:26:51.551745+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:50.723528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:50.723528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":829,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551114+0000","last_change":"2024-03-11T22:25:53.595372+0000","last_active":"2024-03-11T22:26:51.551114+0000","last_peered":"2024-03-11T22:26:51.551114+0000","last_clean":"2024-03-11T22:26:51.551114+0000","last_became_active":"2024-03-11T22:08:35.052989+0000","last_became_peered":"2024-03-11T22:08:35.052989+0000","last_unstale":"2024-03-11T22:26:51.551114+0000","last_undegraded":"2024-03-11T22:26:51.551114+0000","last_fullsized":"2024-03-11T22:26:51.551114+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:25:53.595233+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:25:53.595233+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:04.135015+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":730,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552395+0000","last_change":"2024-03-11T22:24:41.612477+0000","last_active":"2024-03-11T22:26:51.552395+0000","last_peered":"2024-03-11T22:26:51.552395+0000","last_clean":"2024-03-11T22:26:51.552395+0000","last_became_active":"2024-03-11T22:20:29.422747+0000","last_became_peered":"2024-03-11T22:20:29.422747+0000","last_unstale":"2024-03-11T22:26:51.552395+0000","last_undegraded":"2024-03-11T22:26:51.552395+0000","last_fullsized":"2024-03-11T22:26:51.552395+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:41.612363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.612363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":779,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552372+0000","last_change":"2024-03-11T22:24:49.679683+0000","last_active":"2024-03-11T22:26:51.552372+0000","last_peered":"2024-03-11T22:26:51.552372+0000","last_clean":"2024-03-11T22:26:51.552372+0000","last_became_active":"2024-03-11T22:08:37.070728+0000","last_became_peered":"2024-03-11T22:08:37.070728+0000","last_unstale":"2024-03-11T22:26:51.552372+0000","last_undegraded":"2024-03-11T22:26:51.552372+0000","last_fullsized":"2024-03-11T22:26:51.552372+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:49.679560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:24:49.679560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":784,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.582657+0000","last_change":"2024-03-11T22:25:53.844991+0000","last_active":"2024-03-11T22:26:51.582657+0000","last_peered":"2024-03-11T22:26:51.582657+0000","last_clean":"2024-03-11T22:26:51.582657+0000","last_became_active":"2024-03-11T22:08:39.081502+0000","last_became_peered":"2024-03-11T22:08:39.081502+0000","last_unstale":"2024-03-11T22:26:51.582657+0000","last_undegraded":"2024-03-11T22:26:51.582657+0000","last_fullsized":"2024-03-11T22:26:51.582657+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:53.844874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:09.057970+0000","last_clean_scrub_stamp":"2024-03-11T22:25:53.844874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:17.803514+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"274'20","reported_seq":843,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551992+0000","last_change":"2024-03-11T22:23:52.653463+0000","last_active":"2024-03-11T22:26:51.551992+0000","last_peered":"2024-03-11T22:26:51.551992+0000","last_clean":"2024-03-11T22:26:51.551992+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:26:51.551992+0000","last_undegraded":"2024-03-11T22:26:51.551992+0000","last_fullsized":"2024-03-11T22:26:51.551992+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'20","last_scrub_stamp":"2024-03-11T22:23:52.653309+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:23:52.653309+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":813,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173401+0000","last_change":"2024-03-11T22:24:41.895990+0000","last_active":"2024-03-11T22:26:55.173401+0000","last_peered":"2024-03-11T22:26:55.173401+0000","last_clean":"2024-03-11T22:26:55.173401+0000","last_became_active":"2024-03-11T22:20:29.416792+0000","last_became_peered":"2024-03-11T22:20:29.416792+0000","last_unstale":"2024-03-11T22:26:55.173401+0000","last_undegraded":"2024-03-11T22:26:55.173401+0000","last_fullsized":"2024-03-11T22:26:55.173401+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:41.895742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.895742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":579,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.720086+0000","last_change":"2024-03-11T22:26:44.807269+0000","last_active":"2024-03-11T22:26:54.720086+0000","last_peered":"2024-03-11T22:26:54.720086+0000","last_clean":"2024-03-11T22:26:54.720086+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:26:54.720086+0000","last_undegraded":"2024-03-11T22:26:54.720086+0000","last_fullsized":"2024-03-11T22:26:54.720086+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.807187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.807187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:13.805182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":775,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552496+0000","last_change":"2024-03-11T22:24:59.829935+0000","last_active":"2024-03-11T22:26:51.552496+0000","last_peered":"2024-03-11T22:26:51.552496+0000","last_clean":"2024-03-11T22:26:51.552496+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:26:51.552496+0000","last_undegraded":"2024-03-11T22:26:51.552496+0000","last_fullsized":"2024-03-11T22:26:51.552496+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:59.829811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:59.829811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"273'8","reported_seq":815,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551529+0000","last_change":"2024-03-11T22:26:50.833235+0000","last_active":"2024-03-11T22:26:51.551529+0000","last_peered":"2024-03-11T22:26:51.551529+0000","last_clean":"2024-03-11T22:26:51.551529+0000","last_became_active":"2024-03-11T22:08:54.610276+0000","last_became_peered":"2024-03-11T22:08:54.610276+0000","last_unstale":"2024-03-11T22:26:51.551529+0000","last_undegraded":"2024-03-11T22:26:51.551529+0000","last_fullsized":"2024-03-11T22:26:51.551529+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'8","last_scrub_stamp":"2024-03-11T22:26:50.833149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.833149+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:51.899617+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":796,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583710+0000","last_change":"2024-03-11T22:26:05.787798+0000","last_active":"2024-03-11T22:26:51.583710+0000","last_peered":"2024-03-11T22:26:51.583710+0000","last_clean":"2024-03-11T22:26:51.583710+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:26:51.583710+0000","last_undegraded":"2024-03-11T22:26:51.583710+0000","last_fullsized":"2024-03-11T22:26:51.583710+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:05.787543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:05.787543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:12.284499+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":387,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.720204+0000","last_change":"2024-03-11T22:24:37.763829+0000","last_active":"2024-03-11T22:26:54.720204+0000","last_peered":"2024-03-11T22:26:54.720204+0000","last_clean":"2024-03-11T22:26:54.720204+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:26:54.720204+0000","last_undegraded":"2024-03-11T22:26:54.720204+0000","last_fullsized":"2024-03-11T22:26:54.720204+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.763709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.763709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":748,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829568+0000","last_change":"2024-03-11T22:24:46.776804+0000","last_active":"2024-03-11T22:26:50.829568+0000","last_peered":"2024-03-11T22:26:50.829568+0000","last_clean":"2024-03-11T22:26:50.829568+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:26:50.829568+0000","last_undegraded":"2024-03-11T22:26:50.829568+0000","last_fullsized":"2024-03-11T22:26:50.829568+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:46.776600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.776600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"273'19","reported_seq":845,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721830+0000","last_change":"2024-03-11T22:26:40.797006+0000","last_active":"2024-03-11T22:26:54.721830+0000","last_peered":"2024-03-11T22:26:54.721830+0000","last_clean":"2024-03-11T22:26:54.721830+0000","last_became_active":"2024-03-11T22:08:35.058457+0000","last_became_peered":"2024-03-11T22:08:35.058457+0000","last_unstale":"2024-03-11T22:26:54.721830+0000","last_undegraded":"2024-03-11T22:26:54.721830+0000","last_fullsized":"2024-03-11T22:26:54.721830+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:26:40.796814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:40.796814+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:49.418743+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":787,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551610+0000","last_change":"2024-03-11T22:24:38.611323+0000","last_active":"2024-03-11T22:26:51.551610+0000","last_peered":"2024-03-11T22:26:51.551610+0000","last_clean":"2024-03-11T22:26:51.551610+0000","last_became_active":"2024-03-11T22:08:33.037866+0000","last_became_peered":"2024-03-11T22:08:33.037866+0000","last_unstale":"2024-03-11T22:26:51.551610+0000","last_undegraded":"2024-03-11T22:26:51.551610+0000","last_fullsized":"2024-03-11T22:26:51.551610+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:38.610924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:38.610924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":798,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583812+0000","last_change":"2024-03-11T22:26:43.777178+0000","last_active":"2024-03-11T22:26:51.583812+0000","last_peered":"2024-03-11T22:26:51.583812+0000","last_clean":"2024-03-11T22:26:51.583812+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:26:51.583812+0000","last_undegraded":"2024-03-11T22:26:51.583812+0000","last_fullsized":"2024-03-11T22:26:51.583812+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:43.777100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:43.777100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:46.467843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":792,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.549377+0000","last_change":"2024-03-11T22:25:05.513523+0000","last_active":"2024-03-11T22:26:51.549377+0000","last_peered":"2024-03-11T22:26:51.549377+0000","last_clean":"2024-03-11T22:26:51.549377+0000","last_became_active":"2024-03-11T22:08:39.075034+0000","last_became_peered":"2024-03-11T22:08:39.075034+0000","last_unstale":"2024-03-11T22:26:51.549377+0000","last_undegraded":"2024-03-11T22:26:51.549377+0000","last_fullsized":"2024-03-11T22:26:51.549377+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:05.513210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.513210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"273'20","reported_seq":860,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583685+0000","last_change":"2024-03-11T22:25:49.869260+0000","last_active":"2024-03-11T22:26:51.583685+0000","last_peered":"2024-03-11T22:26:51.583685+0000","last_clean":"2024-03-11T22:26:51.583685+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:26:51.583685+0000","last_undegraded":"2024-03-11T22:26:51.583685+0000","last_fullsized":"2024-03-11T22:26:51.583685+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:25:49.869107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:49.869107+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":806,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.548493+0000","last_change":"2024-03-11T22:26:03.627581+0000","last_active":"2024-03-11T22:26:51.548493+0000","last_peered":"2024-03-11T22:26:51.548493+0000","last_clean":"2024-03-11T22:26:51.548493+0000","last_became_active":"2024-03-11T22:08:33.030327+0000","last_became_peered":"2024-03-11T22:08:33.030327+0000","last_unstale":"2024-03-11T22:26:51.548493+0000","last_undegraded":"2024-03-11T22:26:51.548493+0000","last_fullsized":"2024-03-11T22:26:51.548493+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:26:03.627489+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:26:03.627489+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:24.143197+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'11","reported_seq":1072,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721710+0000","last_change":"2024-03-11T22:26:39.795881+0000","last_active":"2024-03-11T22:26:54.721710+0000","last_peered":"2024-03-11T22:26:54.721710+0000","last_clean":"2024-03-11T22:26:54.721710+0000","last_became_active":"2024-03-11T22:20:29.415781+0000","last_became_peered":"2024-03-11T22:20:29.415781+0000","last_unstale":"2024-03-11T22:26:54.721710+0000","last_undegraded":"2024-03-11T22:26:54.721710+0000","last_fullsized":"2024-03-11T22:26:54.721710+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-11T22:26:39.795787+0000","last_deep_scrub":"315'11","last_deep_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_clean_scrub_stamp":"2024-03-11T22:26:39.795787+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:52.004194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":742,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173362+0000","last_change":"2024-03-11T22:23:51.821013+0000","last_active":"2024-03-11T22:26:55.173362+0000","last_peered":"2024-03-11T22:26:55.173362+0000","last_clean":"2024-03-11T22:26:55.173362+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:26:55.173362+0000","last_undegraded":"2024-03-11T22:26:55.173362+0000","last_fullsized":"2024-03-11T22:26:55.173362+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:23:51.820865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.820865+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"273'18","reported_seq":815,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552595+0000","last_change":"2024-03-11T22:25:01.629462+0000","last_active":"2024-03-11T22:26:51.552595+0000","last_peered":"2024-03-11T22:26:51.552595+0000","last_clean":"2024-03-11T22:26:51.552595+0000","last_became_active":"2024-03-11T22:08:35.066082+0000","last_became_peered":"2024-03-11T22:08:35.066082+0000","last_unstale":"2024-03-11T22:26:51.552595+0000","last_undegraded":"2024-03-11T22:26:51.552595+0000","last_fullsized":"2024-03-11T22:26:51.552595+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:25:01.629321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:01.629321+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":563,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.550923+0000","last_change":"2024-03-11T22:25:04.394810+0000","last_active":"2024-03-11T22:26:51.550923+0000","last_peered":"2024-03-11T22:26:51.550923+0000","last_clean":"2024-03-11T22:26:51.550923+0000","last_became_active":"2024-03-11T22:14:52.245199+0000","last_became_peered":"2024-03-11T22:14:52.245199+0000","last_unstale":"2024-03-11T22:26:51.550923+0000","last_undegraded":"2024-03-11T22:26:51.550923+0000","last_fullsized":"2024-03-11T22:26:51.550923+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:04.394612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:25:04.394612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":545,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.553293+0000","last_change":"2024-03-11T22:25:05.656524+0000","last_active":"2024-03-11T22:26:51.553293+0000","last_peered":"2024-03-11T22:26:51.553293+0000","last_clean":"2024-03-11T22:26:51.553293+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:26:51.553293+0000","last_undegraded":"2024-03-11T22:26:51.553293+0000","last_fullsized":"2024-03-11T22:26:51.553293+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:05.656208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.656208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":776,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.553004+0000","last_change":"2024-03-11T22:26:37.062133+0000","last_active":"2024-03-11T22:26:51.553004+0000","last_peered":"2024-03-11T22:26:51.553004+0000","last_clean":"2024-03-11T22:26:51.553004+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:26:51.553004+0000","last_undegraded":"2024-03-11T22:26:51.553004+0000","last_fullsized":"2024-03-11T22:26:51.553004+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:37.062022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:37.062022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:04.112077+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"273'27","reported_seq":865,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-11T22:26:50.829707+0000","last_change":"2024-03-11T22:26:10.023949+0000","last_active":"2024-03-11T22:26:50.829707+0000","last_peered":"2024-03-11T22:26:50.829707+0000","last_clean":"2024-03-11T22:26:50.829707+0000","last_became_active":"2024-03-11T22:08:35.055499+0000","last_became_peered":"2024-03-11T22:08:35.055499+0000","last_unstale":"2024-03-11T22:26:50.829707+0000","last_undegraded":"2024-03-11T22:26:50.829707+0000","last_fullsized":"2024-03-11T22:26:50.829707+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:26:10.023796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:10.023796+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:33.040699+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":808,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583350+0000","last_change":"2024-03-11T22:24:55.760207+0000","last_active":"2024-03-11T22:26:51.583350+0000","last_peered":"2024-03-11T22:26:51.583350+0000","last_clean":"2024-03-11T22:26:51.583350+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:26:51.583350+0000","last_undegraded":"2024-03-11T22:26:51.583350+0000","last_fullsized":"2024-03-11T22:26:51.583350+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:55.760084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:24:55.760084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":796,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.549230+0000","last_change":"2024-03-11T22:24:37.481790+0000","last_active":"2024-03-11T22:26:51.549230+0000","last_peered":"2024-03-11T22:26:51.549230+0000","last_clean":"2024-03-11T22:26:51.549230+0000","last_became_active":"2024-03-11T22:08:37.077536+0000","last_became_peered":"2024-03-11T22:08:37.077536+0000","last_unstale":"2024-03-11T22:26:51.549230+0000","last_undegraded":"2024-03-11T22:26:51.549230+0000","last_fullsized":"2024-03-11T22:26:51.549230+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.481668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.481668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":577,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.721634+0000","last_change":"2024-03-11T22:25:59.830054+0000","last_active":"2024-03-11T22:26:54.721634+0000","last_peered":"2024-03-11T22:26:54.721634+0000","last_clean":"2024-03-11T22:26:54.721634+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:26:54.721634+0000","last_undegraded":"2024-03-11T22:26:54.721634+0000","last_fullsized":"2024-03-11T22:26:54.721634+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:59.829804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:07.097049+0000","last_clean_scrub_stamp":"2024-03-11T22:25:59.829804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:10.479827+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"273'9","reported_seq":622,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552101+0000","last_change":"2024-03-11T22:23:58.648966+0000","last_active":"2024-03-11T22:26:51.552101+0000","last_peered":"2024-03-11T22:26:51.552101+0000","last_clean":"2024-03-11T22:26:51.552101+0000","last_became_active":"2024-03-11T22:14:52.238411+0000","last_became_peered":"2024-03-11T22:14:52.238411+0000","last_unstale":"2024-03-11T22:26:51.552101+0000","last_undegraded":"2024-03-11T22:26:51.552101+0000","last_fullsized":"2024-03-11T22:26:51.552101+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9","last_scrub_stamp":"2024-03-11T22:23:58.648385+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.648385+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":801,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173502+0000","last_change":"2024-03-11T22:24:40.905435+0000","last_active":"2024-03-11T22:26:55.173502+0000","last_peered":"2024-03-11T22:26:55.173502+0000","last_clean":"2024-03-11T22:26:55.173502+0000","last_became_active":"2024-03-11T22:08:33.040282+0000","last_became_peered":"2024-03-11T22:08:33.040282+0000","last_unstale":"2024-03-11T22:26:55.173502+0000","last_undegraded":"2024-03-11T22:26:55.173502+0000","last_fullsized":"2024-03-11T22:26:55.173502+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:40.905314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:24:40.905314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":565,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.550784+0000","last_change":"2024-03-11T22:26:09.575351+0000","last_active":"2024-03-11T22:26:51.550784+0000","last_peered":"2024-03-11T22:26:51.550784+0000","last_clean":"2024-03-11T22:26:51.550784+0000","last_became_active":"2024-03-11T22:14:52.239391+0000","last_became_peered":"2024-03-11T22:14:52.239391+0000","last_unstale":"2024-03-11T22:26:51.550784+0000","last_undegraded":"2024-03-11T22:26:51.550784+0000","last_fullsized":"2024-03-11T22:26:51.550784+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:09.575187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:09.575187+0000","last_clean_scrub_stamp":"2024-03-11T22:26:09.575187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:29.155939+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":789,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.172368+0000","last_change":"2024-03-11T22:25:58.198243+0000","last_active":"2024-03-11T22:26:55.172368+0000","last_peered":"2024-03-11T22:26:55.172368+0000","last_clean":"2024-03-11T22:26:55.172368+0000","last_became_active":"2024-03-11T22:08:39.075573+0000","last_became_peered":"2024-03-11T22:08:39.075573+0000","last_unstale":"2024-03-11T22:26:55.172368+0000","last_undegraded":"2024-03-11T22:26:55.172368+0000","last_fullsized":"2024-03-11T22:26:55.172368+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:58.198115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:58.198115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:19.864910+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"273'15","reported_seq":597,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.551860+0000","last_change":"2024-03-11T22:25:03.685762+0000","last_active":"2024-03-11T22:26:51.551860+0000","last_peered":"2024-03-11T22:26:51.551860+0000","last_clean":"2024-03-11T22:26:51.551860+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:26:51.551860+0000","last_undegraded":"2024-03-11T22:26:51.551860+0000","last_fullsized":"2024-03-11T22:26:51.551860+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:25:03.685629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:03.685629+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":801,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173693+0000","last_change":"2024-03-11T22:23:58.700286+0000","last_active":"2024-03-11T22:26:55.173693+0000","last_peered":"2024-03-11T22:26:55.173693+0000","last_clean":"2024-03-11T22:26:55.173693+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:26:55.173693+0000","last_undegraded":"2024-03-11T22:26:55.173693+0000","last_fullsized":"2024-03-11T22:26:55.173693+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:58.700176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.700176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":805,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.172695+0000","last_change":"2024-03-11T22:24:49.941039+0000","last_active":"2024-03-11T22:26:55.172695+0000","last_peered":"2024-03-11T22:26:55.172695+0000","last_clean":"2024-03-11T22:26:55.172695+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:26:55.172695+0000","last_undegraded":"2024-03-11T22:26:55.172695+0000","last_fullsized":"2024-03-11T22:26:55.172695+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:49.940903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:24:49.940903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":784,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583031+0000","last_change":"2024-03-11T22:26:37.738661+0000","last_active":"2024-03-11T22:26:51.583031+0000","last_peered":"2024-03-11T22:26:51.583031+0000","last_clean":"2024-03-11T22:26:51.583031+0000","last_became_active":"2024-03-11T22:08:39.082720+0000","last_became_peered":"2024-03-11T22:08:39.082720+0000","last_unstale":"2024-03-11T22:26:51.583031+0000","last_undegraded":"2024-03-11T22:26:51.583031+0000","last_fullsized":"2024-03-11T22:26:51.583031+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:37.738562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:37.738562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:02.568391+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"273'27","reported_seq":826,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552101+0000","last_change":"2024-03-11T22:26:38.945645+0000","last_active":"2024-03-11T22:26:51.552101+0000","last_peered":"2024-03-11T22:26:51.552101+0000","last_clean":"2024-03-11T22:26:51.552101+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:26:51.552101+0000","last_undegraded":"2024-03-11T22:26:51.552101+0000","last_fullsized":"2024-03-11T22:26:51.552101+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:26:38.945505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:38.945505+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:51.610570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":803,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552658+0000","last_change":"2024-03-11T22:26:39.902683+0000","last_active":"2024-03-11T22:26:51.552658+0000","last_peered":"2024-03-11T22:26:51.552658+0000","last_clean":"2024-03-11T22:26:51.552658+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:26:51.552658+0000","last_undegraded":"2024-03-11T22:26:51.552658+0000","last_fullsized":"2024-03-11T22:26:51.552658+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:26:39.902588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:39.902588+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:53.907172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":578,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.720164+0000","last_change":"2024-03-11T22:25:54.762641+0000","last_active":"2024-03-11T22:26:54.720164+0000","last_peered":"2024-03-11T22:26:54.720164+0000","last_clean":"2024-03-11T22:26:54.720164+0000","last_became_active":"2024-03-11T22:14:52.247689+0000","last_became_peered":"2024-03-11T22:14:52.247689+0000","last_unstale":"2024-03-11T22:26:54.720164+0000","last_undegraded":"2024-03-11T22:26:54.720164+0000","last_fullsized":"2024-03-11T22:26:54.720164+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:54.762527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:25:54.762527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:10.802001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":787,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.552379+0000","last_change":"2024-03-11T22:26:34.021098+0000","last_active":"2024-03-11T22:26:51.552379+0000","last_peered":"2024-03-11T22:26:51.552379+0000","last_clean":"2024-03-11T22:26:51.552379+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:26:51.552379+0000","last_undegraded":"2024-03-11T22:26:51.552379+0000","last_fullsized":"2024-03-11T22:26:51.552379+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:34.020992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:34.020992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:58.307326+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"273'19","reported_seq":615,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.720287+0000","last_change":"2024-03-11T22:24:39.755038+0000","last_active":"2024-03-11T22:26:54.720287+0000","last_peered":"2024-03-11T22:26:54.720287+0000","last_clean":"2024-03-11T22:26:54.720287+0000","last_became_active":"2024-03-11T22:14:52.247411+0000","last_became_peered":"2024-03-11T22:14:52.247411+0000","last_unstale":"2024-03-11T22:26:54.720287+0000","last_undegraded":"2024-03-11T22:26:54.720287+0000","last_fullsized":"2024-03-11T22:26:54.720287+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:24:39.754894+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.754894+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":797,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-11T22:26:51.583501+0000","last_change":"2024-03-11T22:24:00.015418+0000","last_active":"2024-03-11T22:26:51.583501+0000","last_peered":"2024-03-11T22:26:51.583501+0000","last_clean":"2024-03-11T22:26:51.583501+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:26:51.583501+0000","last_undegraded":"2024-03-11T22:26:51.583501+0000","last_fullsized":"2024-03-11T22:26:51.583501+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:00.015301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.015301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":524,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:54.720320+0000","last_change":"2024-03-11T22:26:06.778513+0000","last_active":"2024-03-11T22:26:54.720320+0000","last_peered":"2024-03-11T22:26:54.720320+0000","last_clean":"2024-03-11T22:26:54.720320+0000","last_became_active":"2024-03-11T22:20:29.418227+0000","last_became_peered":"2024-03-11T22:20:29.418227+0000","last_unstale":"2024-03-11T22:26:54.720320+0000","last_undegraded":"2024-03-11T22:26:54.720320+0000","last_fullsized":"2024-03-11T22:26:54.720320+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:06.778392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:06.778392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:18.033688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":567,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-03-11T22:26:55.173102+0000","last_change":"2024-03-11T22:26:10.104750+0000","last_active":"2024-03-11T22:26:55.173102+0000","last_peered":"2024-03-11T22:26:55.173102+0000","last_clean":"2024-03-11T22:26:55.173102+0000","last_became_active":"2024-03-11T22:14:52.247273+0000","last_became_peered":"2024-03-11T22:14:52.247273+0000","last_unstale":"2024-03-11T22:26:55.173102+0000","last_undegraded":"2024-03-11T22:26:55.173102+0000","last_fullsized":"2024-03-11T22:26:55.173102+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:10.104642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:10.104642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:23.800893+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":37814272,"num_objects":577,"num_object_clones":0,"num_object_copies":1731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":577,"num_write_kb":36928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":115671040,"data_stored":115671040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":577,"ondisk_log_size":577,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1063,"num_read_kb":1058,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":12,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":1430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":733,"ondisk_log_size":733,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005281,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":684412,"kb_used_data":7660,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93060740,"statfs":{"total":95995035648,"available":95294197760,"internally_reserved":0,"allocated":7843840,"data_stored":7466782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":692976075},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332326,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":523560,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":93221592,"statfs":{"total":95995035648,"available":95458910208,"internally_reserved":0,"allocated":942080,"data_stored":585312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":535165382},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575103,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581880,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93163272,"statfs":{"total":95995035648,"available":95399190528,"internally_reserved":0,"allocated":954368,"data_stored":559873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738629,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":699856,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93045296,"statfs":{"total":95995035648,"available":95278383104,"internally_reserved":0,"allocated":921600,"data_stored":559199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411578,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501564,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93243588,"statfs":{"total":95995035648,"available":95481434112,"internally_reserved":0,"allocated":1155072,"data_stored":814033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149940,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":624100,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121052,"statfs":{"total":95995035648,"available":95355957248,"internally_reserved":0,"allocated":1523712,"data_stored":1142393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":637532615},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921009,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":567996,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":567038,"kb_avail":93177156,"statfs":{"total":95995035648,"available":95413407744,"internally_reserved":0,"allocated":966656,"data_stored":585043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":580647366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659370,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":662060,"kb_used_data":4768,"kb_used_omap":1,"kb_used_meta":657278,"kb_avail":93083092,"statfs":{"total":95995035648,"available":95317086208,"internally_reserved":0,"allocated":4882432,"data_stored":4517289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":673053126},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14221312,"data_stored":14221312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28901376,"data_stored":28901376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":55967744,"data_stored":55967744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16580608,"data_stored":16580608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:26:59.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:59.402 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:59.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:59.504 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:26:59.609 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:59.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:59.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:26:59.711 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:26:59.752 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:26:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:59.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:26:59.854 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:26:59.870 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:26:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:26:59.887 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:26:59.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:26:59.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:00.009 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:26:59 smithi045 ceph-mon[39715]: pgmap v888: 158 pgs: 1 peering, 157 active+clean; 228 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 909 op/s; 80 KiB/s, 0 objects/s recovering 2024-03-11T22:27:00.010 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:26:59 smithi045 ceph-mon[30178]: pgmap v888: 158 pgs: 1 peering, 157 active+clean; 228 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 909 op/s; 80 KiB/s, 0 objects/s recovering 2024-03-11T22:27:00.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:00.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:00.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:26:59 smithi144 ceph-mon[33984]: pgmap v888: 158 pgs: 1 peering, 157 active+clean; 228 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 909 op/s; 80 KiB/s, 0 objects/s recovering 2024-03-11T22:27:00.134 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:00.171 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 5 16 13877 13861 173.231 148.562 0.0061446 0.00576464 2024-03-11T22:27:00.174 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:00.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:00.276 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:00.309 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:00.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:00.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:00.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:00.451 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:00.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:00.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:00.463 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:27:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:27:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15783.0:14548 11.6 11:64f15a4e:::benchmark_data_smithi045_196393_object14547:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-11T22:27:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:27:00.449565+0000", 2024-03-11T22:27:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0035314130000000002, 2024-03-11T22:27:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035667239999999998, 2024-03-11T22:27:00.463 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15783", 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4086262919", 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 14548 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.449565+0000", 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.464 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.449565+0000", 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.449566+0000", 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.449e-06 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.449587+0000", 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1531000000000001e-05 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.465 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:27:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.449600+0000", 2024-03-11T22:27:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.223e-05 2024-03-11T22:27:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:27:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.449608+0000", 2024-03-11T22:27:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0190000000000006e-06 2024-03-11T22:27:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:27:00.466 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.449881+0000", 2024-03-11T22:27:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00027316500000000002 2024-03-11T22:27:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:27:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.450336+0000", 2024-03-11T22:27:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00045484399999999998 2024-03-11T22:27:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 1,2", 2024-03-11T22:27:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.450505+0000", 2024-03-11T22:27:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000169331 2024-03-11T22:27:00.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:27:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452227+0000", 2024-03-11T22:27:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017215729999999999 2024-03-11T22:27:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:27:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:27:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:27:00.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15783.0:14546 11.b e439/426 11:d56a954f:::benchmark_data_smithi045_196393_object14545:head v 439'884, mlcod=439'881)", 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:27:00.450335+0000", 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.002761295, 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028848599999999999, 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.450335+0000", 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:27:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.450335+0000", 2024-03-11T22:27:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:27:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:27:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.450335+0000", 2024-03-11T22:27:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8899999999999999e-07 2024-03-11T22:27:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:27:00.470 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452299+0000", 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019638120000000001 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452313+0000", 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3560000000000001e-05 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452322+0000", 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0629999999999995e-06 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.471 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:27:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452560+0000", 2024-03-11T22:27:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023838 2024-03-11T22:27:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:27:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452580+0000", 2024-03-11T22:27:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9459000000000001e-05 2024-03-11T22:27:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:27:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:27:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:27:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.15783.0:14555 11.2 e439/426 11:4f15ce33:::benchmark_data_smithi045_196393_object14554:head v 439'941, mlcod=439'938)", 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:27:00.452303+0000", 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.00079322099999999999, 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00097722099999999991, 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452303+0000", 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:27:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452303+0000", 2024-03-11T22:27:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:27:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:27:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452304+0000", 2024-03-11T22:27:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0400000000000005e-07 2024-03-11T22:27:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:27:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452327+0000", 2024-03-11T22:27:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2938e-05 2024-03-11T22:27:00.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:27:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452338+0000", 2024-03-11T22:27:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1107999999999999e-05 2024-03-11T22:27:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:27:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452345+0000", 2024-03-11T22:27:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1280000000000002e-06 2024-03-11T22:27:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:27:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452801+0000", 2024-03-11T22:27:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00045668299999999999 2024-03-11T22:27:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:27:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.452820+0000", 2024-03-11T22:27:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8552000000000001e-05 2024-03-11T22:27:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:27:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:27:00.476 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15783.0:14562 11.6 11.53e39706 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:27:00.453288+0000", 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4294967295.9998074, 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7664999999999998e-05, 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15783", 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4086262919", 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 14562 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.477 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:27:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:27:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.453288+0000", 2024-03-11T22:27:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:27:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:27:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.453288+0000", 2024-03-11T22:27:00.478 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:27:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:27:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.453290+0000", 2024-03-11T22:27:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5120000000000001e-06 2024-03-11T22:27:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:27:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.453313+0000", 2024-03-11T22:27:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2850999999999999e-05 2024-03-11T22:27:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.479 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.453328+0000", 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5620999999999998e-05 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:00.453337+0000", 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4500000000000004e-06 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 4 2024-03-11T22:27:00.480 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:27:00.553 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:00.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:00.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:00.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:00.739 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:00.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:00.841 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:00.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:27:00.875 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:00.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:00.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:00.977 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:01.010 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:01.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:01.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:01.112 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:01.143 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:01.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:01.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:01.171 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 6 16 16213 16197 168.672 146 0.00893506 0.00591989 2024-03-11T22:27:01.245 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:01.276 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:01.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:01.378 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:01.419 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:01.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:01.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:01.521 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:01.567 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:01.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:01.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:01.669 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:01.709 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:01.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:01.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:01.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:01 smithi045 ceph-mon[30178]: pgmap v889: 158 pgs: 1 peering, 157 active+clean; 228 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 893 B/s rd, 40 MiB/s wr, 634 op/s; 56 KiB/s, 0 objects/s recovering 2024-03-11T22:27:01.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:27:01.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:01.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:01 smithi045 ceph-mon[30178]: osdmap e440: 8 total, 8 up, 8 in 2024-03-11T22:27:01.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:01 smithi045 ceph-mon[39715]: pgmap v889: 158 pgs: 1 peering, 157 active+clean; 228 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 893 B/s rd, 40 MiB/s wr, 634 op/s; 56 KiB/s, 0 objects/s recovering 2024-03-11T22:27:01.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:27:01.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:01.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:01 smithi045 ceph-mon[39715]: osdmap e440: 8 total, 8 up, 8 in 2024-03-11T22:27:01.811 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:01.850 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:01.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:01.952 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:01.986 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:01.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:01.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:02.088 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:01 smithi144 ceph-mon[33984]: pgmap v889: 158 pgs: 1 peering, 157 active+clean; 228 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 893 B/s rd, 40 MiB/s wr, 634 op/s; 56 KiB/s, 0 objects/s recovering 2024-03-11T22:27:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:27:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:01 smithi144 ceph-mon[33984]: osdmap e440: 8 total, 8 up, 8 in 2024-03-11T22:27:02.135 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:02.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:02.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:02.171 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 7 16 18657 18641 166.394 152.75 0.00707318 0.00600288 2024-03-11T22:27:02.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:02.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:02.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:02.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:02.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:02.426 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:02.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:02.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:02.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:02.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:02.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:02.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:02.613 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:27:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:27:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:27:02.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T22:27:02.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:02.758 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:02.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:02.860 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:02.909 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:02.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:03.012 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:03.051 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:03.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:03.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:03.171 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 8 16 20718 20702 161.695 128.812 0.00435839 0.00617665 2024-03-11T22:27:03.190 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:03.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:03.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:03.331 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:03.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:03.400 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:27:03.433 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:03.463 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:03.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:03.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:03.565 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:03.596 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:03.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:03.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:03.698 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:03.731 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:03.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:03.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:03.833 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:03.881 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:03.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:03.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:03 smithi045 ceph-mon[30178]: pgmap v891: 158 pgs: 158 active+clean; 993 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 155 MiB/s wr, 2.49k op/s; 53 KiB/s, 0 objects/s recovering 2024-03-11T22:27:03.926 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:03 smithi045 ceph-mon[39715]: pgmap v891: 158 pgs: 158 active+clean; 993 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 155 MiB/s wr, 2.49k op/s; 53 KiB/s, 0 objects/s recovering 2024-03-11T22:27:03.983 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:04.018 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:04.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:04.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:03 smithi144 ceph-mon[33984]: pgmap v891: 158 pgs: 158 active+clean; 993 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 155 MiB/s wr, 2.49k op/s; 53 KiB/s, 0 objects/s recovering 2024-03-11T22:27:04.120 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:04.153 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:04.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:04.171 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 9 16 22653 22637 157.167 120.938 0.00958808 0.00635606 2024-03-11T22:27:04.254 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:04.291 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:04.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:04.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:04.294 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:27:04.294 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:27:04.294 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:27:04.294 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd out 1 2024-03-11T22:27:04.393 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:04.423 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:04.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:04.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:04.569 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:04.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:04.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:04.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:04.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:04.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:04.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:04.779 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:27:04.818 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:05.019 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:05.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:05.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:05.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:04 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/574626430' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:27:05.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:04 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:27:05.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:04 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/574626430' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:27:05.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:04 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:27:05.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:04 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/574626430' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:27:05.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:04 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:27:05.121 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:05.160 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:05.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:05.171 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 10 16 24670 24654 154.056 126.062 0.00710718 0.00648414 2024-03-11T22:27:05.261 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:05.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:05.285 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:05.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:05.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:05.424 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:05.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:05.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:05.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:05.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:05.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:05.657 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:05.681 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:05.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:05.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:05.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:05.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:05.807 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:05.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:05.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:05.934 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:05.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:06.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:06.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:05 smithi045 ceph-mon[30178]: pgmap v892: 158 pgs: 158 active+clean; 1.0 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 124 MiB/s wr, 1.99k op/s; 41 KiB/s, 0 objects/s recovering 2024-03-11T22:27:06.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:05 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:27:06.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:05 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:27:06.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:05 smithi045 ceph-mon[30178]: osdmap e441: 8 total, 8 up, 8 in 2024-03-11T22:27:06.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:05 smithi045 ceph-mon[39715]: pgmap v892: 158 pgs: 158 active+clean; 1.0 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 124 MiB/s wr, 1.99k op/s; 41 KiB/s, 0 objects/s recovering 2024-03-11T22:27:06.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:05 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:27:06.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:05 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:27:06.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:05 smithi045 ceph-mon[39715]: osdmap e441: 8 total, 8 up, 8 in 2024-03-11T22:27:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:06.077 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:06.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:06.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:05 smithi144 ceph-mon[33984]: pgmap v892: 158 pgs: 158 active+clean; 1.0 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 124 MiB/s wr, 1.99k op/s; 41 KiB/s, 0 objects/s recovering 2024-03-11T22:27:06.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:05 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:27:06.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:05 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:27:06.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:05 smithi144 ceph-mon[33984]: osdmap e441: 8 total, 8 up, 8 in 2024-03-11T22:27:06.171 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 11 15 25957 25942 147.369 80.5 0.00558654 0.00678012 2024-03-11T22:27:06.178 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:06.210 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:06.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:06.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:06.312 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:06.350 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:06.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:06.452 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:06.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:06.487 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:06.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:06.589 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:06.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:06.627 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:06.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:06.728 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:06.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:06.760 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:06.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:06.861 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:06.910 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:06.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:07.012 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:07.035 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:07.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:07.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:06 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:27:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:06 smithi144 ceph-mon[33984]: osdmap e442: 8 total, 8 up, 8 in 2024-03-11T22:27:07.099 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:27:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:27:07.137 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:07.167 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:07.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:07.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:07.171 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 12 16 27833 27817 144.853 117.188 0.00523873 0.00689685 2024-03-11T22:27:07.270 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:07.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:06 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:27:07.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:06 smithi045 ceph-mon[39715]: osdmap e442: 8 total, 8 up, 8 in 2024-03-11T22:27:07.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:06 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:27:07.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:06 smithi045 ceph-mon[30178]: osdmap e442: 8 total, 8 up, 8 in 2024-03-11T22:27:07.311 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:07.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:07.413 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:07.445 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:07.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:07.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:07.547 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:07.580 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:07.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:07.681 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:07.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:07.714 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:07.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:07.815 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:07.843 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:07.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:07.944 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:07.979 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:07.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:07.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:08.081 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:08.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:07 smithi144 ceph-mon[33984]: pgmap v894: 158 pgs: 158 active+clean; 1.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-11T22:27:08.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:08.110 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:08.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:08.171 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 13 16 29477 29461 141.614 102.75 0.00240718 0.00701362 2024-03-11T22:27:08.212 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:08.251 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:08.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:08.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:07 smithi045 ceph-mon[39715]: pgmap v894: 158 pgs: 158 active+clean; 1.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-11T22:27:08.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:08.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:07 smithi045 ceph-mon[30178]: pgmap v894: 158 pgs: 158 active+clean; 1.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-11T22:27:08.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:08.353 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:08.388 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:08.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:08.490 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:08.526 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:08.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:08.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:08.628 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:08.654 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:08.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:08.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:08.796 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:08.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:08.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:08.898 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:08.932 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:08.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:09.034 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:09.067 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:09.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:09.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:08 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:27:09.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:08 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:27:09.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:08 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:27:09.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:09.172 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 14 16 31060 31044 138.565 98.9375 0.00599407 0.00721031 2024-03-11T22:27:09.207 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:09.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:09.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:08 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:27:09.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:08 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:27:09.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:08 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:27:09.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:08 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:27:09.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:08 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:27:09.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:08 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:27:09.309 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:09.342 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:09.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:09.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:09.444 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:09.486 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:09.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:09.587 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:09.619 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:09.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:09.721 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:09.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:09.761 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:09.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:09.862 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:09.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:09.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:09.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:09.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:10.033 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:10.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:10.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:09 smithi144 ceph-mon[33984]: pgmap v896: 158 pgs: 158 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 154 MiB/s wr, 2.46k op/s 2024-03-11T22:27:10.136 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:10.164 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:10.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:10.172 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 15 16 32512 32496 135.377 90.75 0.00303897 0.00736997 2024-03-11T22:27:10.267 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:10.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:09 smithi045 ceph-mon[39715]: pgmap v896: 158 pgs: 158 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 154 MiB/s wr, 2.46k op/s 2024-03-11T22:27:10.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:09 smithi045 ceph-mon[30178]: pgmap v896: 158 pgs: 158 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 154 MiB/s wr, 2.46k op/s 2024-03-11T22:27:10.299 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:10.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:10.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:10.433 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:10.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:10.534 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:10.568 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:10.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:10.670 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:10.701 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:10.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:10.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:10.843 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:10.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:10.945 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:10.977 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:10.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:10.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:11.079 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:11.109 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:11.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:11.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:11.172 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 16 16 34261 34245 133.747 109.312 0.0138432 0.00747035 2024-03-11T22:27:11.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:11.247 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:11.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:11.349 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:11.389 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:11.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:11.490 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:11.528 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:11.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:11.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:11.630 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:11.668 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:11.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:11.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:11.770 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:11.800 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:11.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:11.902 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:11.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:11.926 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:11.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:12.027 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:12.072 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:12.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:12.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:11 smithi144 ceph-mon[33984]: pgmap v897: 158 pgs: 158 active+clean; 1.6 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:27:12.172 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 17 16 35991 35975 132.24 108.125 0.0064429 0.00755697 2024-03-11T22:27:12.175 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:12.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:12.216 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:12.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:12.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:11 smithi045 ceph-mon[39715]: pgmap v897: 158 pgs: 158 active+clean; 1.6 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:27:12.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:11 smithi045 ceph-mon[30178]: pgmap v897: 158 pgs: 158 active+clean; 1.6 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-11T22:27:12.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:12.359 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:12.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:12.461 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:12.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:12.505 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:12.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:12.606 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:12.640 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:12.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:12.741 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:12.783 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:12.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:12.884 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:12.917 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:12.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:12.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:12.938 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:27:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:27:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:27:12.996 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:27:13.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:13.077 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:13.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:13.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:13.172 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 18 16 37858 37842 131.375 116.688 0.0096857 0.00760643 2024-03-11T22:27:13.180 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:13.212 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:13.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:13.314 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:13.348 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:13.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:13.449 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:13.479 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:13.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:13.581 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:13.614 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:13.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:13.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:13.716 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:13.749 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:27:13.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:13.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:13.791 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:27:13.835 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:27:13.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:13 smithi045 ceph-mon[30178]: pgmap v898: 158 pgs: 158 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-11T22:27:13.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:13 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/554476409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:27:13.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:13 smithi045 ceph-mon[39715]: pgmap v898: 158 pgs: 158 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-11T22:27:13.836 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:13 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/554476409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:27:13.865 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:13.877 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:27:13.895 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:13.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:13.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:13.917 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:27:13.962 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:27:13.997 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:14.007 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:27:14.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:14.030 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:14.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:14.034 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:27:14.057 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:27:14.061 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:27:14.061 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:27:14.061 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:27:14.061 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:27:14.061 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:27:14.061 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:27:14.061 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:27:14.061 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:27:14.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:13 smithi144 ceph-mon[33984]: pgmap v898: 158 pgs: 158 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-11T22:27:14.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:13 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/554476409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:27:14.131 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:14.172 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 19 16 39536 39520 129.978 104.875 0.0121944 0.00768661 2024-03-11T22:27:14.182 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:14.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:14.284 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:14.314 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:14.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:14.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:14.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:14.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:14.456 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:14.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:14.557 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:14.596 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:14.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:14.698 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:14.732 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:14.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:14.834 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:14.866 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:14.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:14.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:14.991 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:27:15.007 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:15.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:15.110 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:15.135 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:15.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:15.172 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:27:15.169193+0000 min lat: 0.001317 max lat: 0.264986 avg lat: 0.00776526 2024-03-11T22:27:15.173 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:27:15.173 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 20 16 41202 41186 128.685 104.125 0.0113887 0.00776526 2024-03-11T22:27:15.236 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:15.298 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:15.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:15.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:15.423 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:15.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:15.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:15.524 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:15.559 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:15.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:15.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:15.661 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:15.693 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:15.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:15.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:15.795 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:15.829 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:15.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:15.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:15.931 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:15.965 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:15.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:16.067 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:16.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:15 smithi144 ceph-mon[33984]: pgmap v899: 158 pgs: 158 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-11T22:27:16.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:15 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:27:16.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:15 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:27:16.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:15 smithi144 ceph-mon[33984]: osdmap e443: 8 total, 8 up, 8 in 2024-03-11T22:27:16.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:16.117 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:16.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:16.123 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:15 smithi045 ceph-mon[39715]: pgmap v899: 158 pgs: 158 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-11T22:27:16.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:15 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:27:16.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:15 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:27:16.124 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:15 smithi045 ceph-mon[39715]: osdmap e443: 8 total, 8 up, 8 in 2024-03-11T22:27:16.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:15 smithi045 ceph-mon[30178]: pgmap v899: 158 pgs: 158 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-11T22:27:16.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:15 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:27:16.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:15 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:27:16.124 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:15 smithi045 ceph-mon[30178]: osdmap e443: 8 total, 8 up, 8 in 2024-03-11T22:27:16.173 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 21 15 42899 42884 127.607 106.125 0.0111419 0.00783151 2024-03-11T22:27:16.218 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:16.255 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:16.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:16.356 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:16.393 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:16.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:16.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:16.495 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:16.527 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:16.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:16.629 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:16.661 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:16.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:16.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:16.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:16.804 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:16.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:16.905 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:16.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:16.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:17.054 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:17.084 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:17.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:17.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:17.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:16 smithi144 ceph-mon[33984]: osdmap e444: 8 total, 8 up, 8 in 2024-03-11T22:27:17.099 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:27:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:27:17.173 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 22 16 44567 44551 126.542 104.188 0.00863953 0.00789618 2024-03-11T22:27:17.187 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:17.215 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:17.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:17.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:17.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:17.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:16 smithi045 ceph-mon[30178]: osdmap e444: 8 total, 8 up, 8 in 2024-03-11T22:27:17.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:17.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:16 smithi045 ceph-mon[39715]: osdmap e444: 8 total, 8 up, 8 in 2024-03-11T22:27:17.316 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:17.361 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:17.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:17.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:17.463 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:17.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:17.507 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:17.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:17.608 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:17.647 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:17.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:17.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:17.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:17.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:17.788 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:17.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:17.889 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:17.922 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:17.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:17.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:18.001 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:27:18.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:18.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:17 smithi144 ceph-mon[33984]: pgmap v901: 158 pgs: 158 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-03-11T22:27:18.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:18.101 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:18.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:18.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:18.174 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 23 16 46186 46170 125.44 101.188 0.0113903 0.00796715 2024-03-11T22:27:18.203 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:18.227 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:17 smithi045 ceph-mon[30178]: pgmap v901: 158 pgs: 158 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-03-11T22:27:18.228 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:18.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:17 smithi045 ceph-mon[39715]: pgmap v901: 158 pgs: 158 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-03-11T22:27:18.228 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:18.236 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:18.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:18.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:18.338 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:18.366 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:18.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:18.454 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:27:18.454 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:27:18.468 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:18.496 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:18.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:18.598 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:18.632 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:18.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:18.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:18.782 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:18.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:18.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:18.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:18.930 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:18.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:18.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:19.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:19.072 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:19.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:19.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:19.174 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 24 16 47676 47660 124.093 93.125 0.0152159 0.00805281 2024-03-11T22:27:19.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:19.211 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:19.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:19.312 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:19.341 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:19.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:19.444 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:19.475 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:19.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:19.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:19.577 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:19.607 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:19.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:19.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:19.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:19.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:19.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:19.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:19.852 INFO:teuthology.orchestra.run.smithi045.stderr:marked out osd.1. 2024-03-11T22:27:19.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:19.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:19.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:19.969 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:19 smithi045 ceph-mon[39715]: pgmap v903: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-11T22:27:19.969 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:19 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/599035473' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-03-11T22:27:19.969 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:19 smithi045 ceph-mon[39715]: Client client.admin marked osd.1 out, while it was still marked up 2024-03-11T22:27:19.970 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a[30155]: 2024-03-11T22:27:19.701+0000 7f7c98d0b700 -1 mon.a@0(leader).osd e444 definitely_dead 0 2024-03-11T22:27:19.970 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:19 smithi045 ceph-mon[30178]: pgmap v903: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-11T22:27:19.970 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:19 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/599035473' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-03-11T22:27:19.970 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:19 smithi045 ceph-mon[30178]: Client client.admin marked osd.1 out, while it was still marked up 2024-03-11T22:27:19.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T22:27:20.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:20.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:20.038 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:20.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:20.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:19 smithi144 ceph-mon[33984]: pgmap v903: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-03-11T22:27:20.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:19 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/599035473' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-03-11T22:27:20.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:19 smithi144 ceph-mon[33984]: Client client.admin marked osd.1 out, while it was still marked up 2024-03-11T22:27:20.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:20.163 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:20.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:20.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:20.173 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 25 16 48713 48697 121.722 64.8125 0.00297209 0.00811306 2024-03-11T22:27:20.265 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:20.289 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:20.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:20.391 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:20.415 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:20.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:20.517 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:20.541 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:20.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:20.642 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:20.667 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:20.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:20.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:20.794 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:20.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:20.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:20.817 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:27:20.896 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:20.901 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:27:20.916 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:27:20.923 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:20.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:20.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:21.025 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:21.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:20 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/599035473' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-03-11T22:27:21.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:20 smithi045 ceph-mon[39715]: osdmap e445: 8 total, 8 up, 7 in 2024-03-11T22:27:21.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:20 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/599035473' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-03-11T22:27:21.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:20 smithi045 ceph-mon[30178]: osdmap e445: 8 total, 8 up, 7 in 2024-03-11T22:27:21.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:21.050 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:21.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:21.151 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:21.173 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 26 16 48713 48697 117.04 0 - 0.00811306 2024-03-11T22:27:21.174 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:21.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:21.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:21.276 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:21.300 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:21.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:21.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:21.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:20 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/599035473' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-03-11T22:27:21.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:20 smithi144 ceph-mon[33984]: osdmap e445: 8 total, 8 up, 7 in 2024-03-11T22:27:21.402 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:21.426 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:21.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:21.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:21.553 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:21.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:21.655 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:21.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:21.679 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:21.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:21.781 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:21.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:21.806 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:21.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:21.907 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:21.952 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:21.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:22.053 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:22.095 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:22.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:22.174 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 27 16 48826 48810 112.967 3.53125 0.00788647 0.00884451 2024-03-11T22:27:22.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:22.237 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:22.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:22.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:22.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:21 smithi045 ceph-mon[39715]: pgmap v905: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-11T22:27:22.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:21 smithi045 ceph-mon[39715]: osdmap e446: 8 total, 8 up, 7 in 2024-03-11T22:27:22.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:21 smithi045 ceph-mon[30178]: pgmap v905: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-11T22:27:22.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:21 smithi045 ceph-mon[30178]: osdmap e446: 8 total, 8 up, 7 in 2024-03-11T22:27:22.339 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:22.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:21 smithi144 ceph-mon[33984]: pgmap v905: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-11T22:27:22.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:21 smithi144 ceph-mon[33984]: osdmap e446: 8 total, 8 up, 7 in 2024-03-11T22:27:22.438 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:22.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:22.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:22.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:22.578 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:22.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:22.649 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:27:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:27:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:27:22.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:22.722 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:22.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:22.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:22.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:22.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:22.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:22.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:22.975 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:23.015 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:23.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:23.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:22 smithi045 ceph-mon[39715]: osdmap e447: 8 total, 8 up, 7 in 2024-03-11T22:27:23.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:22 smithi045 ceph-mon[39715]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 93'130 2024-03-11T22:27:23.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:22 smithi045 ceph-mon[30178]: osdmap e447: 8 total, 8 up, 7 in 2024-03-11T22:27:23.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:22 smithi045 ceph-mon[30178]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 93'130 2024-03-11T22:27:23.115 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:23.162 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:23.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:23.174 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 28 15 49748 49733 110.993 57.6875 0.0259929 0.00900282 2024-03-11T22:27:23.265 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:23.298 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:23.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:23.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:23.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:22 smithi144 ceph-mon[33984]: osdmap e447: 8 total, 8 up, 7 in 2024-03-11T22:27:23.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:22 smithi144 ceph-mon[33984]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 93'130 2024-03-11T22:27:23.400 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:23.436 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:23.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:23.538 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:23.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:23.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:23.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:23.696 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:23.729 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:23.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:23.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:23.878 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:23.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:23.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:23.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:24.032 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:24.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:24.134 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:24.169 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:24.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:24.174 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 29 16 50641 50625 109.088 55.75 0.0278148 0.00915761 2024-03-11T22:27:24.270 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:24.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:23 smithi045 ceph-mon[30178]: pgmap v908: 158 pgs: 5 remapped+peering, 26 peering, 1 unknown, 126 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.4 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 9.1 KiB/s, 9 objects/s recovering 2024-03-11T22:27:24.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:23 smithi045 ceph-mon[30178]: Health check failed: Reduced data availability: 5 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-11T22:27:24.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:23 smithi045 ceph-mon[30178]: osdmap e448: 8 total, 8 up, 7 in 2024-03-11T22:27:24.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:23 smithi045 ceph-mon[39715]: pgmap v908: 158 pgs: 5 remapped+peering, 26 peering, 1 unknown, 126 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.4 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 9.1 KiB/s, 9 objects/s recovering 2024-03-11T22:27:24.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:23 smithi045 ceph-mon[39715]: Health check failed: Reduced data availability: 5 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-11T22:27:24.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:23 smithi045 ceph-mon[39715]: osdmap e448: 8 total, 8 up, 7 in 2024-03-11T22:27:24.319 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:24.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:24.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:23 smithi144 ceph-mon[33984]: pgmap v908: 158 pgs: 5 remapped+peering, 26 peering, 1 unknown, 126 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.4 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 9.1 KiB/s, 9 objects/s recovering 2024-03-11T22:27:24.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:23 smithi144 ceph-mon[33984]: Health check failed: Reduced data availability: 5 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-11T22:27:24.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:23 smithi144 ceph-mon[33984]: osdmap e448: 8 total, 8 up, 7 in 2024-03-11T22:27:24.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:24.467 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:24.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:24.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:24.608 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:24.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:24.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:24.710 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:24.754 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:24.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:24.856 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:24.900 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:24.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:24.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:25.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:25.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:25.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:25.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:25.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:25.174 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 30 16 51529 51513 107.301 55.5 0.00933556 0.0093124 2024-03-11T22:27:25.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:24 smithi045 ceph-mon[30178]: osdmap e449: 8 total, 8 up, 7 in 2024-03-11T22:27:25.190 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:24 smithi045 ceph-mon[39715]: osdmap e449: 8 total, 8 up, 7 in 2024-03-11T22:27:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:25.193 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:25.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:25.294 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:25.325 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:25.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:25.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:24 smithi144 ceph-mon[33984]: osdmap e449: 8 total, 8 up, 7 in 2024-03-11T22:27:25.427 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:25.476 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:25.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:25.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:25.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:25.617 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:25.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:25.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:25.770 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:25.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:25.811 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:27:25.811 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:27:25.811 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-03-11T22:27:25.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:27:25.871 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:25.901 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:25.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:26.002 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:26.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:25 smithi045 ceph-mon[30178]: pgmap v911: 158 pgs: 1 active+recovering+undersized+remapped, 5 remapped+peering, 26 peering, 1 unknown, 125 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s; 3029/146823 objects misplaced (2.063%); 413 KiB/s, 23 objects/s recovering 2024-03-11T22:27:26.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:25 smithi045 ceph-mon[39715]: pgmap v911: 158 pgs: 1 active+recovering+undersized+remapped, 5 remapped+peering, 26 peering, 1 unknown, 125 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s; 3029/146823 objects misplaced (2.063%); 413 KiB/s, 23 objects/s recovering 2024-03-11T22:27:26.044 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:26.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:26.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:26.146 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:26.174 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 31 16 52392 52376 105.579 53.9375 0.0235694 0.00946686 2024-03-11T22:27:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:26.184 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:26.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:26.285 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:26.322 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:26.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:26.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:25 smithi144 ceph-mon[33984]: pgmap v911: 158 pgs: 1 active+recovering+undersized+remapped, 5 remapped+peering, 26 peering, 1 unknown, 125 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 86 MiB/s wr, 1.38k op/s; 3029/146823 objects misplaced (2.063%); 413 KiB/s, 23 objects/s recovering 2024-03-11T22:27:26.423 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:26.467 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:26.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:26.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:26.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:26.603 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:26.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:26.704 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:26.737 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:26.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:26.838 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:26.872 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:26.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:26.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:26.974 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:27.005 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:27.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:27.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:27.099 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:27:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:27:27.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:27.149 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:27.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:27.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:27.174 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 32 16 53205 53189 103.868 50.8125 0.0276498 0.00962055 2024-03-11T22:27:27.251 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:27.282 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:27.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:27.385 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:27.424 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:27.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:27.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:27.567 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:27.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:27.607 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:27:27.669 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:27.699 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:27.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:27.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:27.801 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:27.841 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:27.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:27.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:27.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:27 smithi144 ceph-mon[33984]: pgmap v912: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 3 remapped+peering, 150 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 61 MiB/s wr, 969 op/s; 6136/152763 objects degraded (4.017%); 8709/152763 objects misplaced (5.701%); 4.9 MiB/s, 98 objects/s recovering 2024-03-11T22:27:27.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:27 smithi144 ceph-mon[33984]: osdmap e450: 8 total, 8 up, 7 in 2024-03-11T22:27:27.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:27.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:27 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1928778624' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:27:27.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:27 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:27:27.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:27 smithi144 ceph-mon[33984]: Health check failed: Degraded data redundancy: 6136/152763 objects degraded (4.017%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:27.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:27 smithi144 ceph-mon[33984]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 8 pgs peering) 2024-03-11T22:27:27.943 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:27.981 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:27.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:27.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:28.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[39715]: pgmap v912: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 3 remapped+peering, 150 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 61 MiB/s wr, 969 op/s; 6136/152763 objects degraded (4.017%); 8709/152763 objects misplaced (5.701%); 4.9 MiB/s, 98 objects/s recovering 2024-03-11T22:27:28.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[39715]: osdmap e450: 8 total, 8 up, 7 in 2024-03-11T22:27:28.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:28.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1928778624' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:27:28.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:27:28.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[39715]: Health check failed: Degraded data redundancy: 6136/152763 objects degraded (4.017%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:28.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[39715]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 8 pgs peering) 2024-03-11T22:27:28.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[30178]: pgmap v912: 158 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 3 remapped+peering, 150 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 61 MiB/s wr, 969 op/s; 6136/152763 objects degraded (4.017%); 8709/152763 objects misplaced (5.701%); 4.9 MiB/s, 98 objects/s recovering 2024-03-11T22:27:28.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[30178]: osdmap e450: 8 total, 8 up, 7 in 2024-03-11T22:27:28.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:28.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1928778624' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:27:28.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:27:28.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[30178]: Health check failed: Degraded data redundancy: 6136/152763 objects degraded (4.017%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:28.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:27 smithi045 ceph-mon[30178]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 8 pgs peering) 2024-03-11T22:27:28.083 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:28.122 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:28.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:28.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:28.174 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 33 16 54136 54120 102.484 58.1875 0.0385627 0.0097495 2024-03-11T22:27:28.224 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:28.259 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:28.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:28.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:28.361 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:28.403 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:28.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:28.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:28.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:28.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:28.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:28.651 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:28.688 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:28.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:28.790 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:28.824 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:28.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:28.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:28 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:27:28.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:28 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:27:28.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:28 smithi144 ceph-mon[33984]: osdmap e451: 8 total, 8 up, 7 in 2024-03-11T22:27:28.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:28 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:27:28.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:28 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:27:28.925 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:28 smithi045 ceph-mon[39715]: osdmap e451: 8 total, 8 up, 7 in 2024-03-11T22:27:28.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:28 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:27:28.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:28 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:27:28.926 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:28 smithi045 ceph-mon[30178]: osdmap e451: 8 total, 8 up, 7 in 2024-03-11T22:27:28.926 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:28.955 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:28.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:29.057 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:29.086 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:29.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:29.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:29.106 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:27:29.175 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:27:29.175 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:27:29.175 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 34 16 55033 55017 101.118 56.0625 0.0180949 0.00988205 2024-03-11T22:27:29.187 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:29.232 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:29.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:29.333 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:29.369 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:29.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:29.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:29.473 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:29.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:29.516 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:29.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:29.617 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:29.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:29.654 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:29.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:29.755 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:29.795 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:29.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:29.897 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:29.930 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:29.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:30.032 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:30.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:29 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:27:30.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:29 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:27:30.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:29 smithi045 ceph-mon[39715]: pgmap v915: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 917 B/s rd, 42 MiB/s wr, 679 op/s; 6136/158190 objects degraded (3.879%); 17296/158190 objects misplaced (10.934%); 11 MiB/s, 178 objects/s recovering 2024-03-11T22:27:30.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:29 smithi045 ceph-mon[39715]: osdmap e452: 8 total, 8 up, 7 in 2024-03-11T22:27:30.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:29 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:27:30.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:29 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:27:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:29 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:27:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:29 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:27:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:29 smithi045 ceph-mon[30178]: pgmap v915: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 917 B/s rd, 42 MiB/s wr, 679 op/s; 6136/158190 objects degraded (3.879%); 17296/158190 objects misplaced (10.934%); 11 MiB/s, 178 objects/s recovering 2024-03-11T22:27:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:29 smithi045 ceph-mon[30178]: osdmap e452: 8 total, 8 up, 7 in 2024-03-11T22:27:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:29 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:27:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:29 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:27:30.066 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:30.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:30.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:30.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:29 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:27:30.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:29 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:27:30.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:29 smithi144 ceph-mon[33984]: pgmap v915: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 917 B/s rd, 42 MiB/s wr, 679 op/s; 6136/158190 objects degraded (3.879%); 17296/158190 objects misplaced (10.934%); 11 MiB/s, 178 objects/s recovering 2024-03-11T22:27:30.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:29 smithi144 ceph-mon[33984]: osdmap e452: 8 total, 8 up, 7 in 2024-03-11T22:27:30.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:29 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:27:30.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:29 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:27:30.168 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:30.174 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 35 16 55913 55897 99.8005 55 0.00661098 0.0100129 2024-03-11T22:27:30.209 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:30.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:30.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:30.311 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:30.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:30.341 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:30.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:30.442 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:30.473 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:30.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:30.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:30.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:30.617 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:30.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:30.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:30.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:30.755 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:30.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:30.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:30.770 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:27:30.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:30.885 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:30.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:30.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:30.987 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:31.029 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:31.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:31.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:30 smithi045 ceph-mon[30178]: osdmap e453: 8 total, 8 up, 7 in 2024-03-11T22:27:31.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:30 smithi045 ceph-mon[39715]: osdmap e453: 8 total, 8 up, 7 in 2024-03-11T22:27:31.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:30 smithi144 ceph-mon[33984]: osdmap e453: 8 total, 8 up, 7 in 2024-03-11T22:27:31.130 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:31.164 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:31.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:31.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:31.175 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 36 16 56759 56743 98.497 52.875 0.0225202 0.0101462 2024-03-11T22:27:31.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:31.309 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:31.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:31.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:31.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:31.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:31.553 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:31.588 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:31.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:31.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:31.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:31.732 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:31.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:31.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:31.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:31.864 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:31.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:31.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:31.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:32.007 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:32.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:32.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:31 smithi045 ceph-mon[30178]: pgmap v918: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 453 op/s; 6136/158190 objects degraded (3.879%); 17296/158190 objects misplaced (10.934%); 8.4 MiB/s, 134 objects/s recovering 2024-03-11T22:27:32.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:27:32.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:32.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:31 smithi045 ceph-mon[30178]: osdmap e454: 8 total, 8 up, 7 in 2024-03-11T22:27:32.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:31 smithi045 ceph-mon[39715]: pgmap v918: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 453 op/s; 6136/158190 objects degraded (3.879%); 17296/158190 objects misplaced (10.934%); 8.4 MiB/s, 134 objects/s recovering 2024-03-11T22:27:32.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:27:32.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:32.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:31 smithi045 ceph-mon[39715]: osdmap e454: 8 total, 8 up, 7 in 2024-03-11T22:27:32.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:31 smithi144 ceph-mon[33984]: pgmap v918: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 453 op/s; 6136/158190 objects degraded (3.879%); 17296/158190 objects misplaced (10.934%); 8.4 MiB/s, 134 objects/s recovering 2024-03-11T22:27:32.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:27:32.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:31 smithi144 ceph-mon[33984]: osdmap e454: 8 total, 8 up, 7 in 2024-03-11T22:27:32.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:32.151 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:32.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:32.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:32.175 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 37 15 57627 57612 97.3028 54.3125 0.00840088 0.0102665 2024-03-11T22:27:32.253 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:32.289 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:32.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:32.390 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:32.421 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:32.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:32.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:32.524 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:32.559 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:32.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:32.661 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:32.684 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:32.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:32.786 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:32.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:32.827 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:32.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:32.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:32.966 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:27:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:27:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:27:32.967 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:32 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:27:32.967 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:32 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:27:32.967 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:32 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:27:32.967 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:32 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:27:32.971 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:32.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:32.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:33.073 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:33.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:32 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:27:33.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:32 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:27:33.114 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:33.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:33.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:33.180 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 38 16 58454 58438 96.1007 51.625 0.0412307 0.0103971 2024-03-11T22:27:33.216 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:33.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:33.241 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:33.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:33.342 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:33.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:33.385 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:33.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:33.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:33.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:33.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:33.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:33.642 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:33.675 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:33.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:33.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:33.777 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:33 smithi045 ceph-mon[30178]: pgmap v920: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 56 MiB/s wr, 899 op/s; 6136/170520 objects degraded (3.598%); 16373/170520 objects misplaced (9.602%); 14 MiB/s, 224 objects/s recovering 2024-03-11T22:27:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:33 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 6136/170520 objects degraded (3.598%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:33.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:33 smithi045 ceph-mon[30178]: osdmap e455: 8 total, 8 up, 7 in 2024-03-11T22:27:33.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:33 smithi045 ceph-mon[39715]: pgmap v920: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 56 MiB/s wr, 899 op/s; 6136/170520 objects degraded (3.598%); 16373/170520 objects misplaced (9.602%); 14 MiB/s, 224 objects/s recovering 2024-03-11T22:27:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:33 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 6136/170520 objects degraded (3.598%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:33.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:33 smithi045 ceph-mon[39715]: osdmap e455: 8 total, 8 up, 7 in 2024-03-11T22:27:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:33.815 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:33.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:33.916 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:33.955 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:33.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:33.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:34.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:34.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:33 smithi144 ceph-mon[33984]: pgmap v920: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 56 MiB/s wr, 899 op/s; 6136/170520 objects degraded (3.598%); 16373/170520 objects misplaced (9.602%); 14 MiB/s, 224 objects/s recovering 2024-03-11T22:27:34.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:33 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 6136/170520 objects degraded (3.598%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:34.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:33 smithi144 ceph-mon[33984]: osdmap e455: 8 total, 8 up, 7 in 2024-03-11T22:27:34.104 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:34.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:34.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:34.175 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 39 16 59281 59265 94.9619 51.6875 0.0242701 0.0105233 2024-03-11T22:27:34.206 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:34.239 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:34.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:34.341 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:34.379 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:34.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:34.480 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:34.513 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:34.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:34.615 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:34.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:34.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:34.757 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:34.785 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:34.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:34.887 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:34.923 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:34.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:35.025 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:35.064 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:35.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:35.165 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:35.175 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:27:35.171737+0000 min lat: 0.001317 max lat: 2.21187 avg lat: 0.0106475 2024-03-11T22:27:35.175 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:27:35.175 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 40 15 60090 60075 93.8534 50.625 0.00480189 0.0106475 2024-03-11T22:27:35.198 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:35.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:35.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:35.299 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:35.340 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:35.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:35.441 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:35.478 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:35.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:35.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:35.619 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:35.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:35.721 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:35.756 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:35.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:35.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:35.858 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:35.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:35.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:35.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:35.900 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:35 smithi144 ceph-mon[33984]: pgmap v922: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 873 B/s rd, 47 MiB/s wr, 747 op/s; 6136/171318 objects degraded (3.582%); 16183/171318 objects misplaced (9.446%); 12 MiB/s, 189 objects/s recovering 2024-03-11T22:27:35.998 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:36.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:35 smithi045 ceph-mon[30178]: pgmap v922: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 873 B/s rd, 47 MiB/s wr, 747 op/s; 6136/171318 objects degraded (3.582%); 16183/171318 objects misplaced (9.446%); 12 MiB/s, 189 objects/s recovering 2024-03-11T22:27:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:35 smithi045 ceph-mon[39715]: pgmap v922: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 873 B/s rd, 47 MiB/s wr, 747 op/s; 6136/171318 objects degraded (3.582%); 16183/171318 objects misplaced (9.446%); 12 MiB/s, 189 objects/s recovering 2024-03-11T22:27:36.039 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:36.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:36.140 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:36.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:36.174 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:36.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:36.175 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 41 16 60918 60902 92.8249 51.6875 0.0212547 0.0107673 2024-03-11T22:27:36.275 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:36.309 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:36.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:36.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:36.449 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:36.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:36.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:36.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:36.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:36.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:36.681 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:36.721 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:36.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:36.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:36.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:36.875 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:36.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:36.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:36.977 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:37.013 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:37.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:37.099 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:27:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:27:37.116 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:37.152 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:37.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:37.175 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 42 16 61826 61810 91.9658 56.75 0.0207766 0.0108673 2024-03-11T22:27:37.253 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:37.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:37.289 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:37.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:37.390 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:37.430 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:37.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:37.532 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:37.565 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:37.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:37.667 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:37.705 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:37.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:37.807 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:37.849 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:37.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:37.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:37 smithi144 ceph-mon[33984]: pgmap v923: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 747 B/s rd, 62 MiB/s wr, 988 op/s; 6136/178479 objects degraded (3.438%); 15618/178479 objects misplaced (8.751%); 15 MiB/s, 245 objects/s recovering 2024-03-11T22:27:37.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:37 smithi144 ceph-mon[33984]: osdmap e456: 8 total, 8 up, 7 in 2024-03-11T22:27:37.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:37.950 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:37.988 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:37.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:38.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:37 smithi045 ceph-mon[39715]: pgmap v923: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 747 B/s rd, 62 MiB/s wr, 988 op/s; 6136/178479 objects degraded (3.438%); 15618/178479 objects misplaced (8.751%); 15 MiB/s, 245 objects/s recovering 2024-03-11T22:27:38.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:37 smithi045 ceph-mon[39715]: osdmap e456: 8 total, 8 up, 7 in 2024-03-11T22:27:38.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:38.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:37 smithi045 ceph-mon[30178]: pgmap v923: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 747 B/s rd, 62 MiB/s wr, 988 op/s; 6136/178479 objects degraded (3.438%); 15618/178479 objects misplaced (8.751%); 15 MiB/s, 245 objects/s recovering 2024-03-11T22:27:38.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:37 smithi045 ceph-mon[30178]: osdmap e456: 8 total, 8 up, 7 in 2024-03-11T22:27:38.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:38.090 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:38.124 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:38.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:38.175 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 43 16 62714 62698 91.1177 55.5 0.0134281 0.010969 2024-03-11T22:27:38.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:38.265 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:38.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:38.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:38.413 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:38.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:38.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:38.562 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:38.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:38.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:27:38.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:27:38.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:27:38.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:27:38.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:27:38.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:27:38.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:27:38.628 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:27:38.629 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:27:38.629 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:27:38.629 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:27:38.629 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:27:38.629 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:27:38.629 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:27:38.629 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:27:38.629 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:27:38.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:38.703 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:38.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:38.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:38 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3587911934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:27:38.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:38 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3587911934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:27:38.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:38.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:38 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3587911934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:27:38.852 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:38.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:38.954 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:38.991 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:38.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:39.092 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:39.117 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:39.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:39.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:39.175 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 44 16 63488 63472 90.1462 48.375 0.0332427 0.0110868 2024-03-11T22:27:39.219 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:39.254 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:39.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:39.356 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:39.382 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:39.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:39.484 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:39.511 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:39.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:39.613 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:39.655 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:39.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:39.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:39.757 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:39.791 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:39.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:39.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:39.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:39 smithi144 ceph-mon[33984]: pgmap v925: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 55 MiB/s wr, 878 op/s; 6136/183828 objects degraded (3.338%); 15010/183828 objects misplaced (8.165%); 16 MiB/s, 254 objects/s recovering 2024-03-11T22:27:39.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:39 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 6136/183828 objects degraded (3.338%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:39.894 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:39.932 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:39.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:40.034 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:40.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:39 smithi045 ceph-mon[39715]: pgmap v925: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 55 MiB/s wr, 878 op/s; 6136/183828 objects degraded (3.338%); 15010/183828 objects misplaced (8.165%); 16 MiB/s, 254 objects/s recovering 2024-03-11T22:27:40.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:39 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 6136/183828 objects degraded (3.338%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:40.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:39 smithi045 ceph-mon[30178]: pgmap v925: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.4 KiB/s rd, 55 MiB/s wr, 878 op/s; 6136/183828 objects degraded (3.338%); 15010/183828 objects misplaced (8.165%); 16 MiB/s, 254 objects/s recovering 2024-03-11T22:27:40.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:39 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 6136/183828 objects degraded (3.338%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:40.079 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:40.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:40.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:40.176 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 45 16 64203 64187 89.136 44.6875 0.0270553 0.0112115 2024-03-11T22:27:40.181 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:40.226 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:40.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:40.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:40.328 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:40.364 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:40.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:40.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:40.465 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:40.510 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:40.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:40.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:40.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:40.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:40.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:40.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:40.800 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:40.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:40.903 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:40.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:40.944 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:40.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:41.045 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:41.089 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:41.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:41.176 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 46 16 65073 65057 88.3802 54.375 0.00972251 0.0113073 2024-03-11T22:27:41.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:41.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:41.238 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:41.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:41.314 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T22:27:41.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:41.407 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:41.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:41.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:41.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:41.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:41.552 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:41.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:41.654 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:41.698 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:41.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:41.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:41.800 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:41.836 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:41.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:41.937 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:41.972 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:41.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:42.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:41 smithi045 ceph-mon[39715]: pgmap v926: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 35 MiB/s wr, 555 op/s; 6136/183828 objects degraded (3.338%); 15010/183828 objects misplaced (8.165%); 11 MiB/s, 170 objects/s recovering 2024-03-11T22:27:42.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:41 smithi045 ceph-mon[30178]: pgmap v926: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 35 MiB/s wr, 555 op/s; 6136/183828 objects degraded (3.338%); 15010/183828 objects misplaced (8.165%); 11 MiB/s, 170 objects/s recovering 2024-03-11T22:27:42.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:42.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:41 smithi144 ceph-mon[33984]: pgmap v926: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 35 MiB/s wr, 555 op/s; 6136/183828 objects degraded (3.338%); 15010/183828 objects misplaced (8.165%); 11 MiB/s, 170 objects/s recovering 2024-03-11T22:27:42.115 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:42.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:42.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:42.176 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 47 16 65812 65796 87.4824 46.1875 0.0101771 0.0114248 2024-03-11T22:27:42.217 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:42.257 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:42.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:42.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:42.359 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:42.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:42.401 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:42.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:42.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:42.546 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:42.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:42.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:42.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:42.674 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:42.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:42.775 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:42.785 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:27:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:27:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:27:42.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:42.814 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:42.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:42.915 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:42.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:42.953 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:42.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:43.055 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:43.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:43.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:43.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:43.176 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 48 16 66478 66462 86.527 41.625 0.102999 0.0115364 2024-03-11T22:27:43.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:43.209 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:43.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:43.311 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:43.348 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:43.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:43.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:43.450 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:43.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:43.476 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:43.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:43.577 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:43.619 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:43.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:43.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:43.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:43.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:43.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:43.785 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:43 smithi045 ceph-mon[30178]: pgmap v927: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 844 op/s; 6136/195375 objects degraded (3.141%); 13991/195375 objects misplaced (7.161%); 15 MiB/s, 242 objects/s recovering 2024-03-11T22:27:43.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:43 smithi045 ceph-mon[39715]: pgmap v927: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 844 op/s; 6136/195375 objects degraded (3.141%); 13991/195375 objects misplaced (7.161%); 15 MiB/s, 242 objects/s recovering 2024-03-11T22:27:43.846 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:43.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:43.893 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:43.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:43.994 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:44.025 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:44.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:44.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:44.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:43 smithi144 ceph-mon[33984]: pgmap v927: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 844 op/s; 6136/195375 objects degraded (3.141%); 13991/195375 objects misplaced (7.161%); 15 MiB/s, 242 objects/s recovering 2024-03-11T22:27:44.127 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:44.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:44.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:44.177 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 49 16 67252 67236 85.7484 48.375 0.0064554 0.011655 2024-03-11T22:27:44.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:44.326 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:44.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:44.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:44.428 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:44.465 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:44.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:44.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:44.567 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:44.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:44.603 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:44.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:44.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:44.743 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:44.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:44.845 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:44.881 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:44.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:44.983 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:45.020 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:45.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:45.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:45.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:44 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:27:45.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:44 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:27:45.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:44 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:27:45.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:44 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:27:45.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:44 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:27:45.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:44 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:27:45.123 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:45.150 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:45.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:45.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:45.176 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 50 16 68089 68073 85.0796 52.3125 0.0184439 0.0117472 2024-03-11T22:27:45.252 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:45.294 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:45.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:45.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:45.396 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:45.434 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:45.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:45.536 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:45.571 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:45.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:45.673 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:45.706 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:45.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:45.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:45.846 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:45.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:45.948 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:45.985 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:45.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:46.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:45 smithi045 ceph-mon[39715]: pgmap v928: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 827 op/s; 6136/196116 objects degraded (3.129%); 13784/196116 objects misplaced (7.028%); 15 MiB/s, 239 objects/s recovering 2024-03-11T22:27:46.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:45 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 6136/196116 objects degraded (3.129%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:46.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:45 smithi045 ceph-mon[39715]: osdmap e457: 8 total, 8 up, 7 in 2024-03-11T22:27:46.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:45 smithi045 ceph-mon[30178]: pgmap v928: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 827 op/s; 6136/196116 objects degraded (3.129%); 13784/196116 objects misplaced (7.028%); 15 MiB/s, 239 objects/s recovering 2024-03-11T22:27:46.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:45 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 6136/196116 objects degraded (3.129%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:46.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:45 smithi045 ceph-mon[30178]: osdmap e457: 8 total, 8 up, 7 in 2024-03-11T22:27:46.086 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:46.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:45 smithi144 ceph-mon[33984]: pgmap v928: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 827 op/s; 6136/196116 objects degraded (3.129%); 13784/196116 objects misplaced (7.028%); 15 MiB/s, 239 objects/s recovering 2024-03-11T22:27:46.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:45 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 6136/196116 objects degraded (3.129%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:46.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:45 smithi144 ceph-mon[33984]: osdmap e457: 8 total, 8 up, 7 in 2024-03-11T22:27:46.119 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:46.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:46.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:46.176 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 51 15 69055 69040 84.5963 60.4375 0.0103614 0.0118143 2024-03-11T22:27:46.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:46.280 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:46.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:46.381 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:46.417 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:46.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:46.519 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:46.551 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:46.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:46.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:46.654 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:46.690 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:46.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:46.791 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:46.824 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:46.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:46.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:46.878 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:46.879 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:46 smithi045 ceph-mon[30178]: osdmap e458: 8 total, 8 up, 7 in 2024-03-11T22:27:46.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:46.879 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:46 smithi045 ceph-mon[39715]: osdmap e458: 8 total, 8 up, 7 in 2024-03-11T22:27:46.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:46.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:46.972 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:46.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:47.073 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:47.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:27:47.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:46 smithi144 ceph-mon[33984]: osdmap e458: 8 total, 8 up, 7 in 2024-03-11T22:27:47.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:27:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:27:47.109 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:47.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:47.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:47.176 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 52 15 69921 69906 84.0103 54.125 0.0219454 0.0118968 2024-03-11T22:27:47.211 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:47.244 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:47.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:47.345 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:47.384 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:47.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:47.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:47.524 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:47.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:47.626 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:47.659 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:47.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:47.718 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:27:47.723 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:27:47.761 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:47.834 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:47.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:47.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:47.935 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:47 smithi045 ceph-mon[39715]: pgmap v930: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 51 MiB/s wr, 824 op/s; 6136/202920 objects degraded (3.024%); 13142/202920 objects misplaced (6.476%); 16 MiB/s, 250 objects/s recovering 2024-03-11T22:27:47.935 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:47.935 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:47 smithi045 ceph-mon[30178]: pgmap v930: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 51 MiB/s wr, 824 op/s; 6136/202920 objects degraded (3.024%); 13142/202920 objects misplaced (6.476%); 16 MiB/s, 250 objects/s recovering 2024-03-11T22:27:47.935 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:47.937 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:47.985 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:47.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:48.086 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:48.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:47 smithi144 ceph-mon[33984]: pgmap v930: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 51 MiB/s wr, 824 op/s; 6136/202920 objects degraded (3.024%); 13142/202920 objects misplaced (6.476%); 16 MiB/s, 250 objects/s recovering 2024-03-11T22:27:48.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:48.120 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:48.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:48.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:48.138 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:27:48.143 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:27:48.176 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 53 16 70689 70673 83.3297 47.9375 0.0241329 0.0119937 2024-03-11T22:27:48.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:48.264 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:48.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:48.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:48.412 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:48.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:48.514 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:48.550 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:48.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:48.651 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:48.692 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:48.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:48.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:48.794 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:48.833 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:48.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:48.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:48.883 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:27:48.936 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:48.969 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:48.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:49.071 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:49.109 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:49.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:49.179 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 54 16 71554 71538 82.7864 54.0625 0.0124624 0.0120739 2024-03-11T22:27:49.210 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:49.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:49.246 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:49.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:49.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:49.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:49.392 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:49.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:49.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:49.539 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:49.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:49.641 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:49.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:49.688 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:49.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:49.789 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:49.826 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:49.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:49.927 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:49.962 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:49.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:50.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:49 smithi045 ceph-mon[30178]: from='client.25402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:27:50.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:49 smithi045 ceph-mon[30178]: pgmap v932: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 6136/208389 objects degraded (2.944%); 12530/208389 objects misplaced (6.013%); 19 MiB/s, 309 objects/s recovering 2024-03-11T22:27:50.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:49 smithi045 ceph-mon[39715]: from='client.25402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:27:50.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:49 smithi045 ceph-mon[39715]: pgmap v932: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 6136/208389 objects degraded (2.944%); 12530/208389 objects misplaced (6.013%); 19 MiB/s, 309 objects/s recovering 2024-03-11T22:27:50.064 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:50.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:50.098 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:50.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:50.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:49 smithi144 ceph-mon[33984]: from='client.25402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:27:50.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:49 smithi144 ceph-mon[33984]: pgmap v932: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 6136/208389 objects degraded (2.944%); 12530/208389 objects misplaced (6.013%); 19 MiB/s, 309 objects/s recovering 2024-03-11T22:27:50.177 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 55 16 72456 72440 82.3061 56.375 0.0113636 0.0121441 2024-03-11T22:27:50.199 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:50.240 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:50.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:50.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:50.342 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:50.375 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:50.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:50.477 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:50.519 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:50.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:50.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:50.621 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:50.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:50.659 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:50.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:50.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:50.800 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:50.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:50.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:50.901 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:50.941 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:50.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:51.042 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:51.081 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:51.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:51.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:51.178 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 56 16 73379 73363 81.8665 57.6875 0.0166909 0.0122086 2024-03-11T22:27:51.184 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:51.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:51.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:51.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:51.323 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:51.354 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:51.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:51.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:51.490 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:51.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:51.592 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:51.625 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:51.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:51.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:51.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:51.772 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:51.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:51.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:51.894 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:51 smithi144 ceph-mon[33984]: pgmap v933: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 34 MiB/s wr, 543 op/s; 6136/208389 objects degraded (2.944%); 12530/208389 objects misplaced (6.013%); 11 MiB/s, 182 objects/s recovering 2024-03-11T22:27:51.895 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:51 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 6136/208389 objects degraded (2.944%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:51.918 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:51.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:51.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:52.020 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:52.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:51 smithi045 ceph-mon[30178]: pgmap v933: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 34 MiB/s wr, 543 op/s; 6136/208389 objects degraded (2.944%); 12530/208389 objects misplaced (6.013%); 11 MiB/s, 182 objects/s recovering 2024-03-11T22:27:52.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:51 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 6136/208389 objects degraded (2.944%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:52.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:51 smithi045 ceph-mon[39715]: pgmap v933: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 34 MiB/s wr, 543 op/s; 6136/208389 objects degraded (2.944%); 12530/208389 objects misplaced (6.013%); 11 MiB/s, 182 objects/s recovering 2024-03-11T22:27:52.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:51 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 6136/208389 objects degraded (2.944%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:52.056 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:52.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:52.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:52.158 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:52.177 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 57 16 74279 74263 81.417 56.25 0.0232639 0.0122769 2024-03-11T22:27:52.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:52.197 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:52.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:52.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:52.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:52.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:52.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:52.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:52.625 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:52.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:52.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:52.761 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:52.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:52.785 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:27:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:27:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:27:52.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:52.907 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:52.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:53.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:53.051 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:53.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:53.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:53.178 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 58 16 75194 75178 80.9992 57.1875 0.00854697 0.0123389 2024-03-11T22:27:53.194 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:53.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:53.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:53.334 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:53.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:53.435 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:53.470 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:53.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:53.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:53.620 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:53.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:53.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:53.722 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:53.754 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:53.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:53.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:53.856 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:53.907 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:53.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:54.009 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:54.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:53 smithi045 ceph-mon[39715]: pgmap v934: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s; 6136/220542 objects degraded (2.782%); 11330/220542 objects misplaced (5.137%); 19 MiB/s, 306 objects/s recovering 2024-03-11T22:27:54.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:53 smithi045 ceph-mon[30178]: pgmap v934: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s; 6136/220542 objects degraded (2.782%); 11330/220542 objects misplaced (5.137%); 19 MiB/s, 306 objects/s recovering 2024-03-11T22:27:54.042 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:54.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:54.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:53 smithi144 ceph-mon[33984]: pgmap v934: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s; 6136/220542 objects degraded (2.782%); 11330/220542 objects misplaced (5.137%); 19 MiB/s, 306 objects/s recovering 2024-03-11T22:27:54.144 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:54.178 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 59 16 76074 76058 80.5582 55 0.00575713 0.0124064 2024-03-11T22:27:54.183 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:54.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:54.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:54.253 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":931,"stamp":"2024-03-11T22:27:46.575692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4420493118,"num_objects":67640,"num_object_clones":0,"num_object_copies":202920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6136,"num_objects_misplaced":13142,"num_objects_unfound":0,"num_objects_dirty":67640,"num_whiteouts":0,"num_read":1235,"num_read_kb":990,"num_write":68169,"num_write_kb":4316406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5354,"num_bytes_recovered":336717389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68075,"ondisk_log_size":68075,"up":474,"acting":467,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16367352,"kb_used_data":11709364,"kb_used_omap":10,"kb_used_meta":4657909,"kb_avail":639848712,"statfs":{"total":671965249536,"available":655205081088,"internally_reserved":0,"allocated":11990388736,"data_stored":11987116302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11154,"internal_metadata":4769698926},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,4,8,11,2,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":417071104,"num_objects":6364,"num_object_clones":0,"num_object_copies":19092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1868,"num_objects_unfound":0,"num_objects_dirty":6364,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6364,"num_write_kb":407296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1868,"num_bytes_recovered":122421248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.111182"},"pg_stats":[{"pgid":"11.3","version":"457'4333","reported_seq":7990,"reported_epoch":457,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-11T22:27:45.868579+0000","last_change":"2024-03-11T22:27:21.988299+0000","last_active":"2024-03-11T22:27:45.868579+0000","last_peered":"2024-03-11T22:27:45.868579+0000","last_clean":"2024-03-11T22:27:19.842692+0000","last_became_active":"2024-03-11T22:27:21.973450+0000","last_became_peered":"2024-03-11T22:27:21.973450+0000","last_unstale":"2024-03-11T22:27:45.868579+0000","last_undegraded":"2024-03-11T22:27:45.868579+0000","last_fullsized":"2024-03-11T22:27:21.851836+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4333,"log_dups_size":0,"ondisk_log_size":4333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283967488,"num_objects":4333,"num_object_clones":0,"num_object_copies":12999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1259,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4333,"num_write_kb":277312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1792,"num_bytes_recovered":117440512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7],"avail_no_missing":["6","1","7"],"object_location_counts":[{"shards":"1,6,7","objects":1259},{"shards":"6,7","objects":3074}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"294'25","reported_seq":646,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.587635+0000","last_change":"2024-03-11T22:24:39.621975+0000","last_active":"2024-03-11T22:27:36.587635+0000","last_peered":"2024-03-11T22:27:36.587635+0000","last_clean":"2024-03-11T22:27:36.587635+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:27:36.587635+0000","last_undegraded":"2024-03-11T22:27:36.587635+0000","last_fullsized":"2024-03-11T22:27:36.587635+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'25","last_scrub_stamp":"2024-03-11T22:24:39.621707+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.621707+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":834,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.595090+0000","last_change":"2024-03-11T22:24:37.604756+0000","last_active":"2024-03-11T22:27:36.595090+0000","last_peered":"2024-03-11T22:27:36.595090+0000","last_clean":"2024-03-11T22:27:36.595090+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:27:36.595090+0000","last_undegraded":"2024-03-11T22:27:36.595090+0000","last_fullsized":"2024-03-11T22:27:36.595090+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.601240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.601240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":1109,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.625867+0000","last_change":"2024-03-11T22:26:11.940151+0000","last_active":"2024-03-11T22:27:44.625867+0000","last_peered":"2024-03-11T22:27:44.625867+0000","last_clean":"2024-03-11T22:27:44.625867+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:27:44.625867+0000","last_undegraded":"2024-03-11T22:27:44.625867+0000","last_fullsized":"2024-03-11T22:27:44.625867+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:26:11.939962+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:26:11.939962+0000","last_clean_scrub_stamp":"2024-03-11T22:26:11.939962+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":36,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.625343+0000","last_change":"2024-03-11T22:27:20.873168+0000","last_active":"2024-03-11T22:27:44.625343+0000","last_peered":"2024-03-11T22:27:44.625343+0000","last_clean":"2024-03-11T22:27:44.625343+0000","last_became_active":"2024-03-11T22:27:20.872375+0000","last_became_peered":"2024-03-11T22:27:20.872375+0000","last_unstale":"2024-03-11T22:27:44.625343+0000","last_undegraded":"2024-03-11T22:27:44.625343+0000","last_fullsized":"2024-03-11T22:27:44.625343+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:49.601908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:49.601908+0000","last_clean_scrub_stamp":"2024-03-11T22:26:49.601908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:08.033135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":1187,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.588016+0000","last_change":"2024-03-11T22:26:43.904135+0000","last_active":"2024-03-11T22:27:36.588016+0000","last_peered":"2024-03-11T22:27:36.588016+0000","last_clean":"2024-03-11T22:27:36.588016+0000","last_became_active":"2024-03-11T22:20:32.293356+0000","last_became_peered":"2024-03-11T22:20:32.293356+0000","last_unstale":"2024-03-11T22:27:36.588016+0000","last_undegraded":"2024-03-11T22:27:36.588016+0000","last_fullsized":"2024-03-11T22:27:36.588016+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:43.904050+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:43.904050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:11.455590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"457'4216","reported_seq":4288,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:46.151542+0000","last_change":"2024-03-11T22:26:45.687974+0000","last_active":"2024-03-11T22:27:46.151542+0000","last_peered":"2024-03-11T22:27:46.151542+0000","last_clean":"2024-03-11T22:27:46.151542+0000","last_became_active":"2024-03-11T22:26:45.687642+0000","last_became_peered":"2024-03-11T22:26:45.687642+0000","last_unstale":"2024-03-11T22:27:46.151542+0000","last_undegraded":"2024-03-11T22:27:46.151542+0000","last_fullsized":"2024-03-11T22:27:46.151542+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4216,"log_dups_size":0,"ondisk_log_size":4216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:01.841667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276299776,"num_objects":4216,"num_object_clones":0,"num_object_copies":12648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4216,"num_write_kb":269824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"353'39","reported_seq":727,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.625914+0000","last_change":"2024-03-11T22:27:20.898203+0000","last_active":"2024-03-11T22:27:44.625914+0000","last_peered":"2024-03-11T22:27:44.625914+0000","last_clean":"2024-03-11T22:27:44.625914+0000","last_became_active":"2024-03-11T22:27:20.868794+0000","last_became_peered":"2024-03-11T22:27:20.868794+0000","last_unstale":"2024-03-11T22:27:44.625914+0000","last_undegraded":"2024-03-11T22:27:44.625914+0000","last_fullsized":"2024-03-11T22:27:44.625914+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"353'39","last_scrub_stamp":"2024-03-11T22:24:58.663290+0000","last_deep_scrub":"353'39","last_deep_scrub_stamp":"2024-03-11T22:23:05.461729+0000","last_clean_scrub_stamp":"2024-03-11T22:24:58.663290+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":866,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.623853+0000","last_change":"2024-03-11T22:27:20.866710+0000","last_active":"2024-03-11T22:27:44.623853+0000","last_peered":"2024-03-11T22:27:44.623853+0000","last_clean":"2024-03-11T22:27:44.623853+0000","last_became_active":"2024-03-11T22:27:20.864176+0000","last_became_peered":"2024-03-11T22:27:20.864176+0000","last_unstale":"2024-03-11T22:27:44.623853+0000","last_undegraded":"2024-03-11T22:27:44.623853+0000","last_fullsized":"2024-03-11T22:27:44.623853+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:51.794140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.794140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":32,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.580191+0000","last_change":"2024-03-11T22:27:20.880016+0000","last_active":"2024-03-11T22:27:36.580191+0000","last_peered":"2024-03-11T22:27:36.580191+0000","last_clean":"2024-03-11T22:27:36.580191+0000","last_became_active":"2024-03-11T22:27:20.879555+0000","last_became_peered":"2024-03-11T22:27:20.879555+0000","last_unstale":"2024-03-11T22:27:36.580191+0000","last_undegraded":"2024-03-11T22:27:36.580191+0000","last_fullsized":"2024-03-11T22:27:36.580191+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:50.571821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.571821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:18.198635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":857,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.625822+0000","last_change":"2024-03-11T22:25:51.099786+0000","last_active":"2024-03-11T22:27:44.625822+0000","last_peered":"2024-03-11T22:27:44.625822+0000","last_clean":"2024-03-11T22:27:44.625822+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:27:44.625822+0000","last_undegraded":"2024-03-11T22:27:44.625822+0000","last_fullsized":"2024-03-11T22:27:44.625822+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:51.099663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:51.099663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":1165,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631371+0000","last_change":"2024-03-11T22:26:03.097798+0000","last_active":"2024-03-11T22:27:44.631371+0000","last_peered":"2024-03-11T22:27:44.631371+0000","last_clean":"2024-03-11T22:27:44.631371+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:27:44.631371+0000","last_undegraded":"2024-03-11T22:27:44.631371+0000","last_fullsized":"2024-03-11T22:27:44.631371+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:03.097636+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:03.097636+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"457'4370","reported_seq":4441,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:45.853271+0000","last_change":"2024-03-11T22:26:45.687410+0000","last_active":"2024-03-11T22:27:45.853271+0000","last_peered":"2024-03-11T22:27:45.853271+0000","last_clean":"2024-03-11T22:27:45.853271+0000","last_became_active":"2024-03-11T22:26:45.687157+0000","last_became_peered":"2024-03-11T22:26:45.687157+0000","last_unstale":"2024-03-11T22:27:45.853271+0000","last_undegraded":"2024-03-11T22:27:45.853271+0000","last_fullsized":"2024-03-11T22:27:45.853271+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:54.836187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286392320,"num_objects":4370,"num_object_clones":0,"num_object_copies":13110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4370,"num_write_kb":279680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"273'19","reported_seq":686,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624388+0000","last_change":"2024-03-11T22:27:20.930132+0000","last_active":"2024-03-11T22:27:44.624388+0000","last_peered":"2024-03-11T22:27:44.624388+0000","last_clean":"2024-03-11T22:27:44.624388+0000","last_became_active":"2024-03-11T22:27:20.877036+0000","last_became_peered":"2024-03-11T22:27:20.877036+0000","last_unstale":"2024-03-11T22:27:44.624388+0000","last_undegraded":"2024-03-11T22:27:44.624388+0000","last_fullsized":"2024-03-11T22:27:44.624388+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:24:57.716144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:57.716144+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":628,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.622447+0000","last_change":"2024-03-11T22:24:57.690108+0000","last_active":"2024-03-11T22:27:44.622447+0000","last_peered":"2024-03-11T22:27:44.622447+0000","last_clean":"2024-03-11T22:27:44.622447+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:27:44.622447+0000","last_undegraded":"2024-03-11T22:27:44.622447+0000","last_fullsized":"2024-03-11T22:27:44.622447+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:57.689696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:57.689696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":836,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632706+0000","last_change":"2024-03-11T22:24:59.851270+0000","last_active":"2024-03-11T22:27:44.632706+0000","last_peered":"2024-03-11T22:27:44.632706+0000","last_clean":"2024-03-11T22:27:44.632706+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:27:44.632706+0000","last_undegraded":"2024-03-11T22:27:44.632706+0000","last_fullsized":"2024-03-11T22:27:44.632706+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:59.851040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:59.851040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":822,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.585895+0000","last_change":"2024-03-11T22:26:48.845265+0000","last_active":"2024-03-11T22:27:36.585895+0000","last_peered":"2024-03-11T22:27:36.585895+0000","last_clean":"2024-03-11T22:27:36.585895+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:27:36.585895+0000","last_undegraded":"2024-03-11T22:27:36.585895+0000","last_fullsized":"2024-03-11T22:27:36.585895+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:48.845177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:48.845177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:06.252440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"456'4127","reported_seq":5772,"reported_epoch":456,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-11T22:27:41.798506+0000","last_change":"2024-03-11T22:27:21.994582+0000","last_active":"2024-03-11T22:27:41.798506+0000","last_peered":"2024-03-11T22:27:41.798506+0000","last_clean":"2024-03-11T22:27:19.830519+0000","last_became_active":"2024-03-11T22:27:21.973476+0000","last_became_peered":"2024-03-11T22:27:21.973476+0000","last_unstale":"2024-03-11T22:27:41.798506+0000","last_undegraded":"2024-03-11T22:27:41.798506+0000","last_fullsized":"2024-03-11T22:27:21.854488+0000","mapping_epoch":446,"log_start":"436'100","ondisk_log_start":"436'100","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4027,"log_dups_size":0,"ondisk_log_size":4027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270467072,"num_objects":4127,"num_object_clones":0,"num_object_copies":12381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2281,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":264128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":786,"num_bytes_recovered":51511296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":2281},{"shards":"3,7","objects":1846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"273'34","reported_seq":557,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.587561+0000","last_change":"2024-03-11T22:27:20.924529+0000","last_active":"2024-03-11T22:27:36.587561+0000","last_peered":"2024-03-11T22:27:36.587561+0000","last_clean":"2024-03-11T22:27:36.587561+0000","last_became_active":"2024-03-11T22:27:20.883588+0000","last_became_peered":"2024-03-11T22:27:20.883588+0000","last_unstale":"2024-03-11T22:27:36.587561+0000","last_undegraded":"2024-03-11T22:27:36.587561+0000","last_fullsized":"2024-03-11T22:27:36.587561+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:26:05.664197+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:26:05.664197+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":852,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.587171+0000","last_change":"2024-03-11T22:27:20.887593+0000","last_active":"2024-03-11T22:27:36.587171+0000","last_peered":"2024-03-11T22:27:36.587171+0000","last_clean":"2024-03-11T22:27:36.587171+0000","last_became_active":"2024-03-11T22:27:20.882473+0000","last_became_peered":"2024-03-11T22:27:20.882473+0000","last_unstale":"2024-03-11T22:27:36.587171+0000","last_undegraded":"2024-03-11T22:27:36.587171+0000","last_fullsized":"2024-03-11T22:27:36.587171+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:06.616932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:25:06.616932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":783,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632472+0000","last_change":"2024-03-11T22:24:00.708144+0000","last_active":"2024-03-11T22:27:44.632472+0000","last_peered":"2024-03-11T22:27:44.632472+0000","last_clean":"2024-03-11T22:27:44.632472+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:27:44.632472+0000","last_undegraded":"2024-03-11T22:27:44.632472+0000","last_fullsized":"2024-03-11T22:27:44.632472+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:24:00.708018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.708018+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":838,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632180+0000","last_change":"2024-03-11T22:27:20.869366+0000","last_active":"2024-03-11T22:27:44.632180+0000","last_peered":"2024-03-11T22:27:44.632180+0000","last_clean":"2024-03-11T22:27:44.632180+0000","last_became_active":"2024-03-11T22:27:20.868494+0000","last_became_peered":"2024-03-11T22:27:20.868494+0000","last_unstale":"2024-03-11T22:27:44.632180+0000","last_undegraded":"2024-03-11T22:27:44.632180+0000","last_fullsized":"2024-03-11T22:27:44.632180+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:36.095984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:26:36.095984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:50.401200+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":1166,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631393+0000","last_change":"2024-03-11T22:24:48.925640+0000","last_active":"2024-03-11T22:27:44.631393+0000","last_peered":"2024-03-11T22:27:44.631393+0000","last_clean":"2024-03-11T22:27:44.631393+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:27:44.631393+0000","last_undegraded":"2024-03-11T22:27:44.631393+0000","last_fullsized":"2024-03-11T22:27:44.631393+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:48.925477+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:48.925477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"457'4319","reported_seq":4393,"reported_epoch":457,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T22:27:46.162853+0000","last_change":"2024-03-11T22:27:22.043381+0000","last_active":"2024-03-11T22:27:46.162853+0000","last_peered":"2024-03-11T22:27:46.162853+0000","last_clean":"2024-03-11T22:27:19.836182+0000","last_became_active":"2024-03-11T22:27:22.034607+0000","last_became_peered":"2024-03-11T22:27:22.034607+0000","last_unstale":"2024-03-11T22:27:46.162853+0000","last_undegraded":"2024-03-11T22:27:21.875724+0000","last_fullsized":"2024-03-11T22:27:21.849667+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4319,"log_dups_size":0,"ondisk_log_size":4319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283049984,"num_objects":4319,"num_object_clones":0,"num_object_copies":12957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4319,"num_write_kb":276416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7],"avail_no_missing":["2","1","7"],"object_location_counts":[{"shards":"1,2,7","objects":3026},{"shards":"2,7","objects":1293}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":619,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.588108+0000","last_change":"2024-03-11T22:26:09.040467+0000","last_active":"2024-03-11T22:27:36.588108+0000","last_peered":"2024-03-11T22:27:36.588108+0000","last_clean":"2024-03-11T22:27:36.588108+0000","last_became_active":"2024-03-11T22:20:29.438896+0000","last_became_peered":"2024-03-11T22:20:29.438896+0000","last_unstale":"2024-03-11T22:27:36.588108+0000","last_undegraded":"2024-03-11T22:27:36.588108+0000","last_fullsized":"2024-03-11T22:27:36.588108+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:09.040331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:09.040331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"273'30","reported_seq":932,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.587271+0000","last_change":"2024-03-11T22:26:05.020921+0000","last_active":"2024-03-11T22:27:36.587271+0000","last_peered":"2024-03-11T22:27:36.587271+0000","last_clean":"2024-03-11T22:27:36.587271+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:27:36.587271+0000","last_undegraded":"2024-03-11T22:27:36.587271+0000","last_fullsized":"2024-03-11T22:27:36.587271+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'30","last_scrub_stamp":"2024-03-11T22:26:05.020726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:05.020726+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":846,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.586796+0000","last_change":"2024-03-11T22:23:57.680282+0000","last_active":"2024-03-11T22:27:36.586796+0000","last_peered":"2024-03-11T22:27:36.586796+0000","last_clean":"2024-03-11T22:27:36.586796+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:27:36.586796+0000","last_undegraded":"2024-03-11T22:27:36.586796+0000","last_fullsized":"2024-03-11T22:27:36.586796+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:57.680156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:57.680156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":601,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.585960+0000","last_change":"2024-03-11T22:25:51.005350+0000","last_active":"2024-03-11T22:27:36.585960+0000","last_peered":"2024-03-11T22:27:36.585960+0000","last_clean":"2024-03-11T22:27:36.585960+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:27:36.585960+0000","last_undegraded":"2024-03-11T22:27:36.585960+0000","last_fullsized":"2024-03-11T22:27:36.585960+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:51.005219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:51.005219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":1138,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.625973+0000","last_change":"2024-03-11T22:26:49.699105+0000","last_active":"2024-03-11T22:27:44.625973+0000","last_peered":"2024-03-11T22:27:44.625973+0000","last_clean":"2024-03-11T22:27:44.625973+0000","last_became_active":"2024-03-11T22:26:49.698651+0000","last_became_peered":"2024-03-11T22:26:49.698651+0000","last_unstale":"2024-03-11T22:27:44.625973+0000","last_undegraded":"2024-03-11T22:27:44.625973+0000","last_fullsized":"2024-03-11T22:27:44.625973+0000","mapping_epoch":175,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:25:05.773963+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:25:05.773963+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.773963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"456'4063","reported_seq":4135,"reported_epoch":456,"state":"active+recovery_wait+remapped","last_fresh":"2024-03-11T22:27:41.672348+0000","last_change":"2024-03-11T22:27:22.049323+0000","last_active":"2024-03-11T22:27:41.672348+0000","last_peered":"2024-03-11T22:27:41.672348+0000","last_clean":"2024-03-11T22:27:19.836638+0000","last_became_active":"2024-03-11T22:27:22.038956+0000","last_became_peered":"2024-03-11T22:27:22.038956+0000","last_unstale":"2024-03-11T22:27:41.672348+0000","last_undegraded":"2024-03-11T22:27:41.672348+0000","last_fullsized":"2024-03-11T22:27:41.672348+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4063,"log_dups_size":0,"ondisk_log_size":4063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266272768,"num_objects":4063,"num_object_clones":0,"num_object_copies":12189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3040,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4063,"num_write_kb":260032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[1,4,7],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":1023}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"446'9","reported_seq":1076,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.587412+0000","last_change":"2024-03-11T22:27:20.928860+0000","last_active":"2024-03-11T22:27:36.587412+0000","last_peered":"2024-03-11T22:27:36.587412+0000","last_clean":"2024-03-11T22:27:36.587412+0000","last_became_active":"2024-03-11T22:27:20.883233+0000","last_became_peered":"2024-03-11T22:27:20.883233+0000","last_unstale":"2024-03-11T22:27:36.587412+0000","last_undegraded":"2024-03-11T22:27:36.587412+0000","last_fullsized":"2024-03-11T22:27:36.587412+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:26:13.061151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:13.061151+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"273'28","reported_seq":664,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631351+0000","last_change":"2024-03-11T22:24:44.976788+0000","last_active":"2024-03-11T22:27:44.631351+0000","last_peered":"2024-03-11T22:27:44.631351+0000","last_clean":"2024-03-11T22:27:44.631351+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:27:44.631351+0000","last_undegraded":"2024-03-11T22:27:44.631351+0000","last_fullsized":"2024-03-11T22:27:44.631351+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:24:44.976524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:44.976524+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":853,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.594809+0000","last_change":"2024-03-11T22:27:20.924693+0000","last_active":"2024-03-11T22:27:36.594809+0000","last_peered":"2024-03-11T22:27:36.594809+0000","last_clean":"2024-03-11T22:27:36.594809+0000","last_became_active":"2024-03-11T22:27:20.881986+0000","last_became_peered":"2024-03-11T22:27:20.881986+0000","last_unstale":"2024-03-11T22:27:36.594809+0000","last_undegraded":"2024-03-11T22:27:36.594809+0000","last_fullsized":"2024-03-11T22:27:36.594809+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:25:51.793162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:25:51.793162+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":791,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.626942+0000","last_change":"2024-03-11T22:23:59.089317+0000","last_active":"2024-03-11T22:27:44.626942+0000","last_peered":"2024-03-11T22:27:44.626942+0000","last_clean":"2024-03-11T22:27:44.626942+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:27:44.626942+0000","last_undegraded":"2024-03-11T22:27:44.626942+0000","last_fullsized":"2024-03-11T22:27:44.626942+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:59.089227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:23:59.089227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":1200,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.585712+0000","last_change":"2024-03-11T22:26:53.558080+0000","last_active":"2024-03-11T22:27:36.585712+0000","last_peered":"2024-03-11T22:27:36.585712+0000","last_clean":"2024-03-11T22:27:36.585712+0000","last_became_active":"2024-03-11T22:26:53.557811+0000","last_became_peered":"2024-03-11T22:26:53.557811+0000","last_unstale":"2024-03-11T22:27:36.585712+0000","last_undegraded":"2024-03-11T22:27:36.585712+0000","last_fullsized":"2024-03-11T22:27:36.585712+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:45.837466+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:45.837466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:49.115156+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"457'4346","reported_seq":4418,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:46.162646+0000","last_change":"2024-03-11T22:26:45.682819+0000","last_active":"2024-03-11T22:27:46.162646+0000","last_peered":"2024-03-11T22:27:46.162646+0000","last_clean":"2024-03-11T22:27:46.162646+0000","last_became_active":"2024-03-11T22:26:45.682468+0000","last_became_peered":"2024-03-11T22:26:45.682468+0000","last_unstale":"2024-03-11T22:27:46.162646+0000","last_undegraded":"2024-03-11T22:27:46.162646+0000","last_fullsized":"2024-03-11T22:27:46.162646+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4346,"log_dups_size":0,"ondisk_log_size":4346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:11.927997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284819456,"num_objects":4346,"num_object_clones":0,"num_object_copies":13038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4346,"num_write_kb":278144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"378'9","reported_seq":1074,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.579119+0000","last_change":"2024-03-11T22:24:54.792194+0000","last_active":"2024-03-11T22:27:36.579119+0000","last_peered":"2024-03-11T22:27:36.579119+0000","last_clean":"2024-03-11T22:27:36.579119+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:27:36.579119+0000","last_undegraded":"2024-03-11T22:27:36.579119+0000","last_fullsized":"2024-03-11T22:27:36.579119+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9","last_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_deep_scrub":"378'9","last_deep_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_clean_scrub_stamp":"2024-03-11T22:24:54.792049+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"273'20","reported_seq":535,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631694+0000","last_change":"2024-03-11T22:27:20.899321+0000","last_active":"2024-03-11T22:27:44.631694+0000","last_peered":"2024-03-11T22:27:44.631694+0000","last_clean":"2024-03-11T22:27:44.631694+0000","last_became_active":"2024-03-11T22:27:20.872444+0000","last_became_peered":"2024-03-11T22:27:20.872444+0000","last_unstale":"2024-03-11T22:27:44.631694+0000","last_undegraded":"2024-03-11T22:27:44.631694+0000","last_fullsized":"2024-03-11T22:27:44.631694+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:24:47.479158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:47.479158+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":845,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632395+0000","last_change":"2024-03-11T22:23:42.943711+0000","last_active":"2024-03-11T22:27:44.632395+0000","last_peered":"2024-03-11T22:27:44.632395+0000","last_clean":"2024-03-11T22:27:44.632395+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:27:44.632395+0000","last_undegraded":"2024-03-11T22:27:44.632395+0000","last_fullsized":"2024-03-11T22:27:44.632395+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:42.943546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:42.943546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":831,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.577674+0000","last_change":"2024-03-11T22:25:59.788866+0000","last_active":"2024-03-11T22:27:36.577674+0000","last_peered":"2024-03-11T22:27:36.577674+0000","last_clean":"2024-03-11T22:27:36.577674+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:27:36.577674+0000","last_undegraded":"2024-03-11T22:27:36.577674+0000","last_fullsized":"2024-03-11T22:27:36.577674+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:59.788747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:59.788747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":1117,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624495+0000","last_change":"2024-03-11T22:26:41.661797+0000","last_active":"2024-03-11T22:27:44.624495+0000","last_peered":"2024-03-11T22:27:44.624495+0000","last_clean":"2024-03-11T22:27:44.624495+0000","last_became_active":"2024-03-11T22:26:41.661375+0000","last_became_peered":"2024-03-11T22:26:41.661375+0000","last_unstale":"2024-03-11T22:27:44.624495+0000","last_undegraded":"2024-03-11T22:27:44.624495+0000","last_fullsized":"2024-03-11T22:27:44.624495+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":423,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:25:57.758735+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:25:57.758735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"457'4344","reported_seq":6302,"reported_epoch":457,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-11T22:27:46.180951+0000","last_change":"2024-03-11T22:27:22.010049+0000","last_active":"2024-03-11T22:27:46.180951+0000","last_peered":"2024-03-11T22:27:46.180951+0000","last_clean":"2024-03-11T22:27:19.822140+0000","last_became_active":"2024-03-11T22:27:21.976749+0000","last_became_peered":"2024-03-11T22:27:21.976749+0000","last_unstale":"2024-03-11T22:27:46.180951+0000","last_undegraded":"2024-03-11T22:27:46.180951+0000","last_fullsized":"2024-03-11T22:27:21.849785+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4344,"log_dups_size":0,"ondisk_log_size":4344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284688384,"num_objects":4344,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2102,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4344,"num_write_kb":278016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":941,"num_bytes_recovered":61669376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4],"avail_no_missing":["2","1","4"],"object_location_counts":[{"shards":"1,2,4","objects":2102},{"shards":"2,4","objects":2242}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":853,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.623792+0000","last_change":"2024-03-11T22:24:00.941070+0000","last_active":"2024-03-11T22:27:44.623792+0000","last_peered":"2024-03-11T22:27:44.623792+0000","last_clean":"2024-03-11T22:27:44.623792+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:27:44.623792+0000","last_undegraded":"2024-03-11T22:27:44.623792+0000","last_fullsized":"2024-03-11T22:27:44.623792+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:00.940803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.940803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"429'37","reported_seq":898,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.578815+0000","last_change":"2024-03-11T22:26:47.889297+0000","last_active":"2024-03-11T22:27:36.578815+0000","last_peered":"2024-03-11T22:27:36.578815+0000","last_clean":"2024-03-11T22:27:36.578815+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:27:36.578815+0000","last_undegraded":"2024-03-11T22:27:36.578815+0000","last_fullsized":"2024-03-11T22:27:36.578815+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'37","last_scrub_stamp":"2024-03-11T22:26:47.889156+0000","last_deep_scrub":"429'37","last_deep_scrub_stamp":"2024-03-11T22:26:47.889156+0000","last_clean_scrub_stamp":"2024-03-11T22:26:47.889156+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:07.369409+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.634134+0000","last_change":"2024-03-11T22:27:20.874227+0000","last_active":"2024-03-11T22:27:44.634134+0000","last_peered":"2024-03-11T22:27:44.634134+0000","last_clean":"2024-03-11T22:27:44.634134+0000","last_became_active":"2024-03-11T22:27:20.872924+0000","last_became_peered":"2024-03-11T22:27:20.872924+0000","last_unstale":"2024-03-11T22:27:44.634134+0000","last_undegraded":"2024-03-11T22:27:44.634134+0000","last_fullsized":"2024-03-11T22:27:44.634134+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:39.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:04.706082+0000","last_clean_scrub_stamp":"2024-03-11T22:26:39.629202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:58.791449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":34,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624167+0000","last_change":"2024-03-11T22:27:20.869446+0000","last_active":"2024-03-11T22:27:44.624167+0000","last_peered":"2024-03-11T22:27:44.624167+0000","last_clean":"2024-03-11T22:27:44.624167+0000","last_became_active":"2024-03-11T22:27:20.868965+0000","last_became_peered":"2024-03-11T22:27:20.868965+0000","last_unstale":"2024-03-11T22:27:44.624167+0000","last_undegraded":"2024-03-11T22:27:44.624167+0000","last_fullsized":"2024-03-11T22:27:44.624167+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.604198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.604198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:02.380557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":1092,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624646+0000","last_change":"2024-03-11T22:26:45.688875+0000","last_active":"2024-03-11T22:27:44.624646+0000","last_peered":"2024-03-11T22:27:44.624646+0000","last_clean":"2024-03-11T22:27:44.624646+0000","last_became_active":"2024-03-11T22:26:45.688609+0000","last_became_peered":"2024-03-11T22:26:45.688609+0000","last_unstale":"2024-03-11T22:27:44.624646+0000","last_undegraded":"2024-03-11T22:27:44.624646+0000","last_fullsized":"2024-03-11T22:27:44.624646+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:37.856736+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:37.856736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:04.208465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"456'4105","reported_seq":4174,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:41.799112+0000","last_change":"2024-03-11T22:26:45.685167+0000","last_active":"2024-03-11T22:27:41.799112+0000","last_peered":"2024-03-11T22:27:41.799112+0000","last_clean":"2024-03-11T22:27:41.799112+0000","last_became_active":"2024-03-11T22:26:45.684876+0000","last_became_peered":"2024-03-11T22:26:45.684876+0000","last_unstale":"2024-03-11T22:27:41.799112+0000","last_undegraded":"2024-03-11T22:27:41.799112+0000","last_fullsized":"2024-03-11T22:27:41.799112+0000","mapping_epoch":426,"log_start":"436'100","ondisk_log_start":"436'100","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4005,"log_dups_size":0,"ondisk_log_size":4005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:50.023912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269025280,"num_objects":4105,"num_object_clones":0,"num_object_copies":12315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4105,"num_write_kb":262720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":842,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632795+0000","last_change":"2024-03-11T22:27:20.872033+0000","last_active":"2024-03-11T22:27:44.632795+0000","last_peered":"2024-03-11T22:27:44.632795+0000","last_clean":"2024-03-11T22:27:44.632795+0000","last_became_active":"2024-03-11T22:27:20.871652+0000","last_became_peered":"2024-03-11T22:27:20.871652+0000","last_unstale":"2024-03-11T22:27:44.632795+0000","last_undegraded":"2024-03-11T22:27:44.632795+0000","last_fullsized":"2024-03-11T22:27:44.632795+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:39.906243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.906243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"273'24","reported_seq":902,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.588148+0000","last_change":"2024-03-11T22:27:20.908383+0000","last_active":"2024-03-11T22:27:36.588148+0000","last_peered":"2024-03-11T22:27:36.588148+0000","last_clean":"2024-03-11T22:27:36.588148+0000","last_became_active":"2024-03-11T22:27:20.877244+0000","last_became_peered":"2024-03-11T22:27:20.877244+0000","last_unstale":"2024-03-11T22:27:36.588148+0000","last_undegraded":"2024-03-11T22:27:36.588148+0000","last_fullsized":"2024-03-11T22:27:36.588148+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'24","last_scrub_stamp":"2024-03-11T22:26:35.660739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:35.660739+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:57.798000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":232,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:40.935695+0000","last_change":"2024-03-11T22:27:20.925116+0000","last_active":"2024-03-11T22:27:40.935695+0000","last_peered":"2024-03-11T22:27:40.935695+0000","last_clean":"2024-03-11T22:27:40.935695+0000","last_became_active":"2024-03-11T22:27:20.878589+0000","last_became_peered":"2024-03-11T22:27:20.878589+0000","last_unstale":"2024-03-11T22:27:40.935695+0000","last_undegraded":"2024-03-11T22:27:40.935695+0000","last_fullsized":"2024-03-11T22:27:40.935695+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:26:35.669961+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:26:35.669961+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:42.694606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":111,"num_read_kb":106,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":843,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.578032+0000","last_change":"2024-03-11T22:24:46.870653+0000","last_active":"2024-03-11T22:27:36.578032+0000","last_peered":"2024-03-11T22:27:36.578032+0000","last_clean":"2024-03-11T22:27:36.578032+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:27:36.578032+0000","last_undegraded":"2024-03-11T22:27:36.578032+0000","last_fullsized":"2024-03-11T22:27:36.578032+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:46.870527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.870527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":34,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631949+0000","last_change":"2024-03-11T22:27:20.872657+0000","last_active":"2024-03-11T22:27:44.631949+0000","last_peered":"2024-03-11T22:27:44.631949+0000","last_clean":"2024-03-11T22:27:44.631949+0000","last_became_active":"2024-03-11T22:27:20.872294+0000","last_became_peered":"2024-03-11T22:27:20.872294+0000","last_unstale":"2024-03-11T22:27:44.631949+0000","last_undegraded":"2024-03-11T22:27:44.631949+0000","last_fullsized":"2024-03-11T22:27:44.631949+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:33.609412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:22.750560+0000","last_clean_scrub_stamp":"2024-03-11T22:26:33.609412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:57.101037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":360,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.626529+0000","last_change":"2024-03-11T22:27:23.894604+0000","last_active":"2024-03-11T22:27:44.626529+0000","last_peered":"2024-03-11T22:27:44.626529+0000","last_clean":"2024-03-11T22:27:44.626529+0000","last_became_active":"2024-03-11T22:27:23.894238+0000","last_became_peered":"2024-03-11T22:27:23.894238+0000","last_unstale":"2024-03-11T22:27:44.626529+0000","last_undegraded":"2024-03-11T22:27:44.626529+0000","last_fullsized":"2024-03-11T22:27:44.626529+0000","mapping_epoch":448,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:45.428590+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:24:45.428590+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:45.631416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"456'4033","reported_seq":4102,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:41.683247+0000","last_change":"2024-03-11T22:26:45.687046+0000","last_active":"2024-03-11T22:27:41.683247+0000","last_peered":"2024-03-11T22:27:41.683247+0000","last_clean":"2024-03-11T22:27:41.683247+0000","last_became_active":"2024-03-11T22:26:45.686711+0000","last_became_peered":"2024-03-11T22:26:45.686711+0000","last_unstale":"2024-03-11T22:27:41.683247+0000","last_undegraded":"2024-03-11T22:27:41.683247+0000","last_fullsized":"2024-03-11T22:27:41.683247+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4033,"log_dups_size":0,"ondisk_log_size":4033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:51.239711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264306688,"num_objects":4033,"num_object_clones":0,"num_object_copies":12099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4033,"num_write_kb":258112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":820,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.635516+0000","last_change":"2024-03-11T22:26:42.960839+0000","last_active":"2024-03-11T22:27:44.635516+0000","last_peered":"2024-03-11T22:27:44.635516+0000","last_clean":"2024-03-11T22:27:44.635516+0000","last_became_active":"2024-03-11T22:20:29.420497+0000","last_became_peered":"2024-03-11T22:20:29.420497+0000","last_unstale":"2024-03-11T22:27:44.635516+0000","last_undegraded":"2024-03-11T22:27:44.635516+0000","last_fullsized":"2024-03-11T22:27:44.635516+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:42.960739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:26:42.960739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:02.582899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"273'28","reported_seq":1128,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624781+0000","last_change":"2024-03-11T22:24:42.729950+0000","last_active":"2024-03-11T22:27:44.624781+0000","last_peered":"2024-03-11T22:27:44.624781+0000","last_clean":"2024-03-11T22:27:44.624781+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:27:44.624781+0000","last_undegraded":"2024-03-11T22:27:44.624781+0000","last_fullsized":"2024-03-11T22:27:44.624781+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'28","last_scrub_stamp":"2024-03-11T22:24:42.729799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:42.729799+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":835,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.585769+0000","last_change":"2024-03-11T22:24:47.653085+0000","last_active":"2024-03-11T22:27:36.585769+0000","last_peered":"2024-03-11T22:27:36.585769+0000","last_clean":"2024-03-11T22:27:36.585769+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:27:36.585769+0000","last_undegraded":"2024-03-11T22:27:36.585769+0000","last_fullsized":"2024-03-11T22:27:36.585769+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:47.652970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:24:47.652970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":1190,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.577800+0000","last_change":"2024-03-11T22:26:37.632763+0000","last_active":"2024-03-11T22:27:36.577800+0000","last_peered":"2024-03-11T22:27:36.577800+0000","last_clean":"2024-03-11T22:27:36.577800+0000","last_became_active":"2024-03-11T22:26:37.632412+0000","last_became_peered":"2024-03-11T22:26:37.632412+0000","last_unstale":"2024-03-11T22:27:36.577800+0000","last_undegraded":"2024-03-11T22:27:36.577800+0000","last_fullsized":"2024-03-11T22:27:36.577800+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":19,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:24:42.773024+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:13:35.093204+0000","last_clean_scrub_stamp":"2024-03-11T22:24:42.773024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":832,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632430+0000","last_change":"2024-03-11T22:24:54.991281+0000","last_active":"2024-03-11T22:27:44.632430+0000","last_peered":"2024-03-11T22:27:44.632430+0000","last_clean":"2024-03-11T22:27:44.632430+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:27:44.632430+0000","last_undegraded":"2024-03-11T22:27:44.632430+0000","last_fullsized":"2024-03-11T22:27:44.632430+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:54.991081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:54.991081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"456'4055","reported_seq":4124,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:41.789532+0000","last_change":"2024-03-11T22:26:45.685997+0000","last_active":"2024-03-11T22:27:41.789532+0000","last_peered":"2024-03-11T22:27:41.789532+0000","last_clean":"2024-03-11T22:27:41.789532+0000","last_became_active":"2024-03-11T22:26:45.685687+0000","last_became_peered":"2024-03-11T22:26:45.685687+0000","last_unstale":"2024-03-11T22:27:41.789532+0000","last_undegraded":"2024-03-11T22:27:41.789532+0000","last_fullsized":"2024-03-11T22:27:41.789532+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4055,"log_dups_size":0,"ondisk_log_size":4055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:47.331803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265748480,"num_objects":4055,"num_object_clones":0,"num_object_copies":12165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4055,"num_write_kb":259520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":861,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.625448+0000","last_change":"2024-03-11T22:24:48.727176+0000","last_active":"2024-03-11T22:27:44.625448+0000","last_peered":"2024-03-11T22:27:44.625448+0000","last_clean":"2024-03-11T22:27:44.625448+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:27:44.625448+0000","last_undegraded":"2024-03-11T22:27:44.625448+0000","last_fullsized":"2024-03-11T22:27:44.625448+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:48.727061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:48.727061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"273'17","reported_seq":538,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632600+0000","last_change":"2024-03-11T22:27:20.884637+0000","last_active":"2024-03-11T22:27:44.632600+0000","last_peered":"2024-03-11T22:27:44.632600+0000","last_clean":"2024-03-11T22:27:44.632600+0000","last_became_active":"2024-03-11T22:27:20.871655+0000","last_became_peered":"2024-03-11T22:27:20.871655+0000","last_unstale":"2024-03-11T22:27:44.632600+0000","last_undegraded":"2024-03-11T22:27:44.632600+0000","last_fullsized":"2024-03-11T22:27:44.632600+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'17","last_scrub_stamp":"2024-03-11T22:24:41.396754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.396754+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":887,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.622539+0000","last_change":"2024-03-11T22:26:06.037315+0000","last_active":"2024-03-11T22:27:44.622539+0000","last_peered":"2024-03-11T22:27:44.622539+0000","last_clean":"2024-03-11T22:27:44.622539+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:27:44.622539+0000","last_undegraded":"2024-03-11T22:27:44.622539+0000","last_fullsized":"2024-03-11T22:27:44.622539+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:06.037180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:26:06.037180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":34,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.620976+0000","last_change":"2024-03-11T22:27:20.873366+0000","last_active":"2024-03-11T22:27:44.620976+0000","last_peered":"2024-03-11T22:27:44.620976+0000","last_clean":"2024-03-11T22:27:44.620976+0000","last_became_active":"2024-03-11T22:27:20.873061+0000","last_became_peered":"2024-03-11T22:27:20.873061+0000","last_unstale":"2024-03-11T22:27:44.620976+0000","last_undegraded":"2024-03-11T22:27:44.620976+0000","last_fullsized":"2024-03-11T22:27:44.620976+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:58.411052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.411052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":791,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624782+0000","last_change":"2024-03-11T22:24:38.834963+0000","last_active":"2024-03-11T22:27:44.624782+0000","last_peered":"2024-03-11T22:27:44.624782+0000","last_clean":"2024-03-11T22:27:44.624782+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:27:44.624782+0000","last_undegraded":"2024-03-11T22:27:44.624782+0000","last_fullsized":"2024-03-11T22:27:44.624782+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:38.834828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:24:38.834828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":1205,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.580808+0000","last_change":"2024-03-11T22:26:32.769183+0000","last_active":"2024-03-11T22:27:36.580808+0000","last_peered":"2024-03-11T22:27:36.580808+0000","last_clean":"2024-03-11T22:27:36.580808+0000","last_became_active":"2024-03-11T22:26:29.590669+0000","last_became_peered":"2024-03-11T22:26:29.590669+0000","last_unstale":"2024-03-11T22:27:36.580808+0000","last_undegraded":"2024-03-11T22:27:36.580808+0000","last_fullsized":"2024-03-11T22:27:36.580808+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:32.769096+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:32.769096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:37.973881+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"456'4064","reported_seq":5752,"reported_epoch":456,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-11T22:27:41.817248+0000","last_change":"2024-03-11T22:27:22.033296+0000","last_active":"2024-03-11T22:27:41.817248+0000","last_peered":"2024-03-11T22:27:41.817248+0000","last_clean":"2024-03-11T22:27:19.835342+0000","last_became_active":"2024-03-11T22:27:22.000407+0000","last_became_peered":"2024-03-11T22:27:22.000407+0000","last_unstale":"2024-03-11T22:27:41.817248+0000","last_undegraded":"2024-03-11T22:27:41.817248+0000","last_fullsized":"2024-03-11T22:27:21.853227+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266338304,"num_objects":4064,"num_object_clones":0,"num_object_copies":12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2210,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":260096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":809,"num_bytes_recovered":53018624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0],"avail_no_missing":["3","0","1"],"object_location_counts":[{"shards":"0,1,3","objects":2210},{"shards":"0,3","objects":1854}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":841,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.581255+0000","last_change":"2024-03-11T22:27:20.881514+0000","last_active":"2024-03-11T22:27:36.581255+0000","last_peered":"2024-03-11T22:27:36.581255+0000","last_clean":"2024-03-11T22:27:36.581255+0000","last_became_active":"2024-03-11T22:27:20.881183+0000","last_became_peered":"2024-03-11T22:27:20.881183+0000","last_unstale":"2024-03-11T22:27:36.581255+0000","last_undegraded":"2024-03-11T22:27:36.581255+0000","last_fullsized":"2024-03-11T22:27:36.581255+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:01.030925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:01.030925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"446'41","reported_seq":709,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.634440+0000","last_change":"2024-03-11T22:27:20.905196+0000","last_active":"2024-03-11T22:27:44.634440+0000","last_peered":"2024-03-11T22:27:44.634440+0000","last_clean":"2024-03-11T22:27:44.634440+0000","last_became_active":"2024-03-11T22:27:20.872714+0000","last_became_peered":"2024-03-11T22:27:20.872714+0000","last_unstale":"2024-03-11T22:27:44.634440+0000","last_undegraded":"2024-03-11T22:27:44.634440+0000","last_fullsized":"2024-03-11T22:27:44.634440+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'40","last_scrub_stamp":"2024-03-11T22:23:57.429825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:23:57.429825+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":610,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.578743+0000","last_change":"2024-03-11T22:26:44.809895+0000","last_active":"2024-03-11T22:27:36.578743+0000","last_peered":"2024-03-11T22:27:36.578743+0000","last_clean":"2024-03-11T22:27:36.578743+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:27:36.578743+0000","last_undegraded":"2024-03-11T22:27:36.578743+0000","last_fullsized":"2024-03-11T22:27:36.578743+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.809803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.809803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:52.696010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":875,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631593+0000","last_change":"2024-03-11T22:27:20.908011+0000","last_active":"2024-03-11T22:27:44.631593+0000","last_peered":"2024-03-11T22:27:44.631593+0000","last_clean":"2024-03-11T22:27:44.631593+0000","last_became_active":"2024-03-11T22:27:20.878785+0000","last_became_peered":"2024-03-11T22:27:20.878785+0000","last_unstale":"2024-03-11T22:27:44.631593+0000","last_undegraded":"2024-03-11T22:27:44.631593+0000","last_fullsized":"2024-03-11T22:27:44.631593+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:26:31.988426+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:26:31.988426+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:57.604543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":834,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.587391+0000","last_change":"2024-03-11T22:25:58.161115+0000","last_active":"2024-03-11T22:27:36.587391+0000","last_peered":"2024-03-11T22:27:36.587391+0000","last_clean":"2024-03-11T22:27:36.587391+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:27:36.587391+0000","last_undegraded":"2024-03-11T22:27:36.587391+0000","last_fullsized":"2024-03-11T22:27:36.587391+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:58.160972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:25:58.160972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":1181,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631307+0000","last_change":"2024-03-11T22:26:34.036467+0000","last_active":"2024-03-11T22:27:44.631307+0000","last_peered":"2024-03-11T22:27:44.631307+0000","last_clean":"2024-03-11T22:27:44.631307+0000","last_became_active":"2024-03-11T22:26:33.607145+0000","last_became_peered":"2024-03-11T22:26:33.607145+0000","last_unstale":"2024-03-11T22:27:44.631307+0000","last_undegraded":"2024-03-11T22:27:44.631307+0000","last_fullsized":"2024-03-11T22:27:44.631307+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:34.036387+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:34.036387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"457'4256","reported_seq":4327,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:45.862094+0000","last_change":"2024-03-11T22:26:45.834759+0000","last_active":"2024-03-11T22:27:45.862094+0000","last_peered":"2024-03-11T22:27:45.862094+0000","last_clean":"2024-03-11T22:27:45.862094+0000","last_became_active":"2024-03-11T22:26:45.834448+0000","last_became_peered":"2024-03-11T22:26:45.834448+0000","last_unstale":"2024-03-11T22:27:45.862094+0000","last_undegraded":"2024-03-11T22:27:45.862094+0000","last_fullsized":"2024-03-11T22:27:45.862094+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4256,"log_dups_size":0,"ondisk_log_size":4256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:57.363090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278921216,"num_objects":4256,"num_object_clones":0,"num_object_copies":12768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4256,"num_write_kb":272384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"273'15","reported_seq":895,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.579009+0000","last_change":"2024-03-11T22:27:20.996860+0000","last_active":"2024-03-11T22:27:36.579009+0000","last_peered":"2024-03-11T22:27:36.579009+0000","last_clean":"2024-03-11T22:27:36.579009+0000","last_became_active":"2024-03-11T22:27:20.883375+0000","last_became_peered":"2024-03-11T22:27:20.883375+0000","last_unstale":"2024-03-11T22:27:36.579009+0000","last_undegraded":"2024-03-11T22:27:36.579009+0000","last_fullsized":"2024-03-11T22:27:36.579009+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:26:12.661479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:12.661479+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:39.262593+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":844,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632060+0000","last_change":"2024-03-11T22:24:58.853739+0000","last_active":"2024-03-11T22:27:44.632060+0000","last_peered":"2024-03-11T22:27:44.632060+0000","last_clean":"2024-03-11T22:27:44.632060+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:27:44.632060+0000","last_undegraded":"2024-03-11T22:27:44.632060+0000","last_fullsized":"2024-03-11T22:27:44.632060+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:58.853630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:58.853630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":869,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.623376+0000","last_change":"2024-03-11T22:26:42.009815+0000","last_active":"2024-03-11T22:27:44.623376+0000","last_peered":"2024-03-11T22:27:44.623376+0000","last_clean":"2024-03-11T22:27:44.623376+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:27:44.623376+0000","last_undegraded":"2024-03-11T22:27:44.623376+0000","last_fullsized":"2024-03-11T22:27:44.623376+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:42.009705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:26:42.009705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:01.285782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":842,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.578244+0000","last_change":"2024-03-11T22:24:53.948958+0000","last_active":"2024-03-11T22:27:36.578244+0000","last_peered":"2024-03-11T22:27:36.578244+0000","last_clean":"2024-03-11T22:27:36.578244+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:27:36.578244+0000","last_undegraded":"2024-03-11T22:27:36.578244+0000","last_fullsized":"2024-03-11T22:27:36.578244+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:53.948853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.948853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"457'4308","reported_seq":4380,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:45.714018+0000","last_change":"2024-03-11T22:26:45.835375+0000","last_active":"2024-03-11T22:27:45.714018+0000","last_peered":"2024-03-11T22:27:45.714018+0000","last_clean":"2024-03-11T22:27:45.714018+0000","last_became_active":"2024-03-11T22:26:45.834812+0000","last_became_peered":"2024-03-11T22:26:45.834812+0000","last_unstale":"2024-03-11T22:27:45.714018+0000","last_undegraded":"2024-03-11T22:27:45.714018+0000","last_fullsized":"2024-03-11T22:27:45.714018+0000","mapping_epoch":426,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4308,"log_dups_size":0,"ondisk_log_size":4308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282329088,"num_objects":4308,"num_object_clones":0,"num_object_copies":12924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4308,"num_write_kb":275712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"273'35","reported_seq":989,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.623647+0000","last_change":"2024-03-11T22:27:20.887968+0000","last_active":"2024-03-11T22:27:44.623647+0000","last_peered":"2024-03-11T22:27:44.623647+0000","last_clean":"2024-03-11T22:27:44.623647+0000","last_became_active":"2024-03-11T22:27:20.864911+0000","last_became_peered":"2024-03-11T22:27:20.864911+0000","last_unstale":"2024-03-11T22:27:44.623647+0000","last_undegraded":"2024-03-11T22:27:44.623647+0000","last_fullsized":"2024-03-11T22:27:44.623647+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'35","last_scrub_stamp":"2024-03-11T22:24:46.751969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.751969+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":795,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632547+0000","last_change":"2024-03-11T22:26:04.099295+0000","last_active":"2024-03-11T22:27:44.632547+0000","last_peered":"2024-03-11T22:27:44.632547+0000","last_clean":"2024-03-11T22:27:44.632547+0000","last_became_active":"2024-03-11T22:20:29.413664+0000","last_became_peered":"2024-03-11T22:20:29.413664+0000","last_unstale":"2024-03-11T22:27:44.632547+0000","last_undegraded":"2024-03-11T22:27:44.632547+0000","last_fullsized":"2024-03-11T22:27:44.632547+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:04.099165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:04.099165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":861,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624672+0000","last_change":"2024-03-11T22:26:09.051839+0000","last_active":"2024-03-11T22:27:44.624672+0000","last_peered":"2024-03-11T22:27:44.624672+0000","last_clean":"2024-03-11T22:27:44.624672+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:27:44.624672+0000","last_undegraded":"2024-03-11T22:27:44.624672+0000","last_fullsized":"2024-03-11T22:27:44.624672+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:09.051546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:26:09.051546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":819,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.621070+0000","last_change":"2024-03-11T22:26:02.011753+0000","last_active":"2024-03-11T22:27:44.621070+0000","last_peered":"2024-03-11T22:27:44.621070+0000","last_clean":"2024-03-11T22:27:44.621070+0000","last_became_active":"2024-03-11T22:20:29.415565+0000","last_became_peered":"2024-03-11T22:20:29.415565+0000","last_unstale":"2024-03-11T22:27:44.621070+0000","last_undegraded":"2024-03-11T22:27:44.621070+0000","last_fullsized":"2024-03-11T22:27:44.621070+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:02.011534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:20.931122+0000","last_clean_scrub_stamp":"2024-03-11T22:26:02.011534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.6","version":"456'4119","reported_seq":5799,"reported_epoch":456,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-11T22:27:42.839842+0000","last_change":"2024-03-11T22:27:22.034856+0000","last_active":"2024-03-11T22:27:42.839842+0000","last_peered":"2024-03-11T22:27:42.839842+0000","last_clean":"2024-03-11T22:27:19.840028+0000","last_became_active":"2024-03-11T22:27:21.987891+0000","last_became_peered":"2024-03-11T22:27:21.987891+0000","last_unstale":"2024-03-11T22:27:42.839842+0000","last_undegraded":"2024-03-11T22:27:42.839842+0000","last_fullsized":"2024-03-11T22:27:21.852019+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4119,"log_dups_size":0,"ondisk_log_size":4119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269942784,"num_objects":4119,"num_object_clones":0,"num_object_copies":12357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2250,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4119,"num_write_kb":263616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":804,"num_bytes_recovered":52690944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":2250},{"shards":"0,2","objects":1869}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":867,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.620798+0000","last_change":"2024-03-11T22:27:20.869051+0000","last_active":"2024-03-11T22:27:44.620798+0000","last_peered":"2024-03-11T22:27:44.620798+0000","last_clean":"2024-03-11T22:27:44.620798+0000","last_became_active":"2024-03-11T22:27:20.867959+0000","last_became_peered":"2024-03-11T22:27:20.867959+0000","last_unstale":"2024-03-11T22:27:44.620798+0000","last_undegraded":"2024-03-11T22:27:44.620798+0000","last_fullsized":"2024-03-11T22:27:44.620798+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:56.703933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:24:56.703933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"273'18","reported_seq":315,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.588365+0000","last_change":"2024-03-11T22:27:21.001629+0000","last_active":"2024-03-11T22:27:36.588365+0000","last_peered":"2024-03-11T22:27:36.588365+0000","last_clean":"2024-03-11T22:27:36.588365+0000","last_became_active":"2024-03-11T22:27:20.897821+0000","last_became_peered":"2024-03-11T22:27:20.897821+0000","last_unstale":"2024-03-11T22:27:36.588365+0000","last_undegraded":"2024-03-11T22:27:36.588365+0000","last_fullsized":"2024-03-11T22:27:36.588365+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:26:46.174869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:46.174869+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:51.731276+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.585003+0000","last_change":"2024-03-11T22:27:20.883415+0000","last_active":"2024-03-11T22:27:36.585003+0000","last_peered":"2024-03-11T22:27:36.585003+0000","last_clean":"2024-03-11T22:27:36.585003+0000","last_became_active":"2024-03-11T22:27:20.881506+0000","last_became_peered":"2024-03-11T22:27:20.881506+0000","last_unstale":"2024-03-11T22:27:36.585003+0000","last_undegraded":"2024-03-11T22:27:36.585003+0000","last_fullsized":"2024-03-11T22:27:36.585003+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:53.435516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.435516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":854,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.601724+0000","last_change":"2024-03-11T22:27:20.998157+0000","last_active":"2024-03-11T22:27:36.601724+0000","last_peered":"2024-03-11T22:27:36.601724+0000","last_clean":"2024-03-11T22:27:36.601724+0000","last_became_active":"2024-03-11T22:27:20.887445+0000","last_became_peered":"2024-03-11T22:27:20.887445+0000","last_unstale":"2024-03-11T22:27:36.601724+0000","last_undegraded":"2024-03-11T22:27:36.601724+0000","last_fullsized":"2024-03-11T22:27:36.601724+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:26:35.021902+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:26:35.021902+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:41.925208+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"457'4384","reported_seq":4459,"reported_epoch":457,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T22:27:45.710395+0000","last_change":"2024-03-11T22:27:22.049266+0000","last_active":"2024-03-11T22:27:45.710395+0000","last_peered":"2024-03-11T22:27:45.710395+0000","last_clean":"2024-03-11T22:27:19.836535+0000","last_became_active":"2024-03-11T22:27:22.042479+0000","last_became_peered":"2024-03-11T22:27:22.042479+0000","last_unstale":"2024-03-11T22:27:45.710395+0000","last_undegraded":"2024-03-11T22:27:21.870576+0000","last_fullsized":"2024-03-11T22:27:21.851240+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":426,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:44.667009+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.667009+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287309824,"num_objects":4384,"num_object_clones":0,"num_object_copies":13152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4384,"num_write_kb":280576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4],"avail_no_missing":["5","1","4"],"object_location_counts":[{"shards":"1,4,5","objects":3110},{"shards":"4,5","objects":1274}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"310'11","reported_seq":1043,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624186+0000","last_change":"2024-03-11T22:23:56.945205+0000","last_active":"2024-03-11T22:27:44.624186+0000","last_peered":"2024-03-11T22:27:44.624186+0000","last_clean":"2024-03-11T22:27:44.624186+0000","last_became_active":"2024-03-11T22:20:29.418355+0000","last_became_peered":"2024-03-11T22:20:29.418355+0000","last_unstale":"2024-03-11T22:27:44.624186+0000","last_undegraded":"2024-03-11T22:27:44.624186+0000","last_fullsized":"2024-03-11T22:27:44.624186+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:23:56.945075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:56.945075+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"273'34","reported_seq":942,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631259+0000","last_change":"2024-03-11T22:26:45.212688+0000","last_active":"2024-03-11T22:27:44.631259+0000","last_peered":"2024-03-11T22:27:44.631259+0000","last_clean":"2024-03-11T22:27:44.631259+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:27:44.631259+0000","last_undegraded":"2024-03-11T22:27:44.631259+0000","last_fullsized":"2024-03-11T22:27:44.631259+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:26:45.212591+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:26:45.212591+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:09.388275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"432'4","reported_seq":644,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.582017+0000","last_change":"2024-03-11T22:26:50.847434+0000","last_active":"2024-03-11T22:27:36.582017+0000","last_peered":"2024-03-11T22:27:36.582017+0000","last_clean":"2024-03-11T22:27:36.582017+0000","last_became_active":"2024-03-11T22:20:29.415226+0000","last_became_peered":"2024-03-11T22:20:29.415226+0000","last_unstale":"2024-03-11T22:27:36.582017+0000","last_undegraded":"2024-03-11T22:27:36.582017+0000","last_fullsized":"2024-03-11T22:27:36.582017+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'4","last_scrub_stamp":"2024-03-11T22:26:50.847349+0000","last_deep_scrub":"432'4","last_deep_scrub_stamp":"2024-03-11T22:26:50.847349+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.847349+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:17.115592+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":832,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631807+0000","last_change":"2024-03-11T22:26:49.194654+0000","last_active":"2024-03-11T22:27:44.631807+0000","last_peered":"2024-03-11T22:27:44.631807+0000","last_clean":"2024-03-11T22:27:44.631807+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:27:44.631807+0000","last_undegraded":"2024-03-11T22:27:44.631807+0000","last_fullsized":"2024-03-11T22:27:44.631807+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:49.194569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:49.194569+0000","last_clean_scrub_stamp":"2024-03-11T22:26:49.194569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:04.923121+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"273'12","reported_seq":889,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.587841+0000","last_change":"2024-03-11T22:27:20.995891+0000","last_active":"2024-03-11T22:27:36.587841+0000","last_peered":"2024-03-11T22:27:36.587841+0000","last_clean":"2024-03-11T22:27:36.587841+0000","last_became_active":"2024-03-11T22:27:20.888929+0000","last_became_peered":"2024-03-11T22:27:20.888929+0000","last_unstale":"2024-03-11T22:27:36.587841+0000","last_undegraded":"2024-03-11T22:27:36.587841+0000","last_fullsized":"2024-03-11T22:27:36.587841+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'12","last_scrub_stamp":"2024-03-11T22:25:52.052338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:52.052338+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":852,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.623488+0000","last_change":"2024-03-11T22:23:48.806202+0000","last_active":"2024-03-11T22:27:44.623488+0000","last_peered":"2024-03-11T22:27:44.623488+0000","last_clean":"2024-03-11T22:27:44.623488+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:27:44.623488+0000","last_undegraded":"2024-03-11T22:27:44.623488+0000","last_fullsized":"2024-03-11T22:27:44.623488+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:48.805960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:23:48.805960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":841,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632280+0000","last_change":"2024-03-11T22:27:20.878741+0000","last_active":"2024-03-11T22:27:44.632280+0000","last_peered":"2024-03-11T22:27:44.632280+0000","last_clean":"2024-03-11T22:27:44.632280+0000","last_became_active":"2024-03-11T22:27:20.877784+0000","last_became_peered":"2024-03-11T22:27:20.877784+0000","last_unstale":"2024-03-11T22:27:44.632280+0000","last_undegraded":"2024-03-11T22:27:44.632280+0000","last_fullsized":"2024-03-11T22:27:44.632280+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:59.713626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:23:59.713626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":782,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624963+0000","last_change":"2024-03-11T22:23:46.876978+0000","last_active":"2024-03-11T22:27:44.624963+0000","last_peered":"2024-03-11T22:27:44.624963+0000","last_clean":"2024-03-11T22:27:44.624963+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:27:44.624963+0000","last_undegraded":"2024-03-11T22:27:44.624963+0000","last_fullsized":"2024-03-11T22:27:44.624963+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:46.876749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:23:46.876749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"273'19","reported_seq":898,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.595021+0000","last_change":"2024-03-11T22:23:53.926458+0000","last_active":"2024-03-11T22:27:36.595021+0000","last_peered":"2024-03-11T22:27:36.595021+0000","last_clean":"2024-03-11T22:27:36.595021+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:27:36.595021+0000","last_undegraded":"2024-03-11T22:27:36.595021+0000","last_fullsized":"2024-03-11T22:27:36.595021+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_clean_scrub_stamp":"2024-03-11T22:23:53.926316+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":622,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.623903+0000","last_change":"2024-03-11T22:26:43.803872+0000","last_active":"2024-03-11T22:27:44.623903+0000","last_peered":"2024-03-11T22:27:44.623903+0000","last_clean":"2024-03-11T22:27:44.623903+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:27:44.623903+0000","last_undegraded":"2024-03-11T22:27:44.623903+0000","last_fullsized":"2024-03-11T22:27:44.623903+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:43.803749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:43.803749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:59.989139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":843,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.587741+0000","last_change":"2024-03-11T22:27:20.886983+0000","last_active":"2024-03-11T22:27:36.587741+0000","last_peered":"2024-03-11T22:27:36.587741+0000","last_clean":"2024-03-11T22:27:36.587741+0000","last_became_active":"2024-03-11T22:27:20.883218+0000","last_became_peered":"2024-03-11T22:27:20.883218+0000","last_unstale":"2024-03-11T22:27:36.587741+0000","last_undegraded":"2024-03-11T22:27:36.587741+0000","last_fullsized":"2024-03-11T22:27:36.587741+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:51.687767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:51.687767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":865,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.634525+0000","last_change":"2024-03-11T22:26:12.930124+0000","last_active":"2024-03-11T22:27:44.634525+0000","last_peered":"2024-03-11T22:27:44.634525+0000","last_clean":"2024-03-11T22:27:44.634525+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:27:44.634525+0000","last_undegraded":"2024-03-11T22:27:44.634525+0000","last_fullsized":"2024-03-11T22:27:44.634525+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:12.930004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:26:12.930004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"273'19","reported_seq":876,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632589+0000","last_change":"2024-03-11T22:26:12.149477+0000","last_active":"2024-03-11T22:27:44.632589+0000","last_peered":"2024-03-11T22:27:44.632589+0000","last_clean":"2024-03-11T22:27:44.632589+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:27:44.632589+0000","last_undegraded":"2024-03-11T22:27:44.632589+0000","last_fullsized":"2024-03-11T22:27:44.632589+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:26:12.149360+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:26:12.149360+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":854,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.580714+0000","last_change":"2024-03-11T22:23:50.030718+0000","last_active":"2024-03-11T22:27:36.580714+0000","last_peered":"2024-03-11T22:27:36.580714+0000","last_clean":"2024-03-11T22:27:36.580714+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:27:36.580714+0000","last_undegraded":"2024-03-11T22:27:36.580714+0000","last_fullsized":"2024-03-11T22:27:36.580714+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:50.030272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:50.030272+0000","last_clean_scrub_stamp":"2024-03-11T22:23:50.030272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"310'11","reported_seq":1099,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.586906+0000","last_change":"2024-03-11T22:24:53.646061+0000","last_active":"2024-03-11T22:27:36.586906+0000","last_peered":"2024-03-11T22:27:36.586906+0000","last_clean":"2024-03-11T22:27:36.586906+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:27:36.586906+0000","last_undegraded":"2024-03-11T22:27:36.586906+0000","last_fullsized":"2024-03-11T22:27:36.586906+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:24:53.645906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:53.645906+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":627,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.625670+0000","last_change":"2024-03-11T22:25:58.161426+0000","last_active":"2024-03-11T22:27:44.625670+0000","last_peered":"2024-03-11T22:27:44.625670+0000","last_clean":"2024-03-11T22:27:44.625670+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:27:44.625670+0000","last_undegraded":"2024-03-11T22:27:44.625670+0000","last_fullsized":"2024-03-11T22:27:44.625670+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:58.161144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:48.490757+0000","last_clean_scrub_stamp":"2024-03-11T22:25:58.161144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"273'20","reported_seq":896,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.586121+0000","last_change":"2024-03-11T22:27:20.997654+0000","last_active":"2024-03-11T22:27:36.586121+0000","last_peered":"2024-03-11T22:27:36.586121+0000","last_clean":"2024-03-11T22:27:36.586121+0000","last_became_active":"2024-03-11T22:27:20.888408+0000","last_became_peered":"2024-03-11T22:27:20.888408+0000","last_unstale":"2024-03-11T22:27:36.586121+0000","last_undegraded":"2024-03-11T22:27:36.586121+0000","last_fullsized":"2024-03-11T22:27:36.586121+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:25:56.196035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:56.196035+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":651,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624580+0000","last_change":"2024-03-11T22:26:51.034990+0000","last_active":"2024-03-11T22:27:44.624580+0000","last_peered":"2024-03-11T22:27:44.624580+0000","last_clean":"2024-03-11T22:27:44.624580+0000","last_became_active":"2024-03-11T22:20:29.415019+0000","last_became_peered":"2024-03-11T22:20:29.415019+0000","last_unstale":"2024-03-11T22:27:44.624580+0000","last_undegraded":"2024-03-11T22:27:44.624580+0000","last_fullsized":"2024-03-11T22:27:44.624580+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:51.034868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:51.034868+0000","last_clean_scrub_stamp":"2024-03-11T22:26:51.034868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:58.249445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"446'9","reported_seq":1085,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632296+0000","last_change":"2024-03-11T22:27:20.930350+0000","last_active":"2024-03-11T22:27:44.632296+0000","last_peered":"2024-03-11T22:27:44.632296+0000","last_clean":"2024-03-11T22:27:44.632296+0000","last_became_active":"2024-03-11T22:27:20.878071+0000","last_became_peered":"2024-03-11T22:27:20.878071+0000","last_unstale":"2024-03-11T22:27:44.632296+0000","last_undegraded":"2024-03-11T22:27:44.632296+0000","last_fullsized":"2024-03-11T22:27:44.632296+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:23:53.776342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:23:53.776342+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":833,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.588017+0000","last_change":"2024-03-11T22:24:45.964526+0000","last_active":"2024-03-11T22:27:36.588017+0000","last_peered":"2024-03-11T22:27:36.588017+0000","last_clean":"2024-03-11T22:27:36.588017+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:27:36.588017+0000","last_undegraded":"2024-03-11T22:27:36.588017+0000","last_fullsized":"2024-03-11T22:27:36.588017+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:45.964401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:45.964401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"273'19","reported_seq":686,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.626765+0000","last_change":"2024-03-11T22:27:20.880610+0000","last_active":"2024-03-11T22:27:44.626765+0000","last_peered":"2024-03-11T22:27:44.626765+0000","last_clean":"2024-03-11T22:27:44.626765+0000","last_became_active":"2024-03-11T22:27:20.864718+0000","last_became_peered":"2024-03-11T22:27:20.864718+0000","last_unstale":"2024-03-11T22:27:44.626765+0000","last_undegraded":"2024-03-11T22:27:44.626765+0000","last_fullsized":"2024-03-11T22:27:44.626765+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:26:11.820506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:11.820506+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":647,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624518+0000","last_change":"2024-03-11T22:26:38.840815+0000","last_active":"2024-03-11T22:27:44.624518+0000","last_peered":"2024-03-11T22:27:44.624518+0000","last_clean":"2024-03-11T22:27:44.624518+0000","last_became_active":"2024-03-11T22:20:29.420867+0000","last_became_peered":"2024-03-11T22:20:29.420867+0000","last_unstale":"2024-03-11T22:27:44.624518+0000","last_undegraded":"2024-03-11T22:27:44.624518+0000","last_fullsized":"2024-03-11T22:27:44.624518+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:26:38.840685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:38.840685+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:04.949659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":34,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631726+0000","last_change":"2024-03-11T22:27:20.864373+0000","last_active":"2024-03-11T22:27:44.631726+0000","last_peered":"2024-03-11T22:27:44.631726+0000","last_clean":"2024-03-11T22:27:44.631726+0000","last_became_active":"2024-03-11T22:27:20.863750+0000","last_became_peered":"2024-03-11T22:27:20.863750+0000","last_unstale":"2024-03-11T22:27:44.631726+0000","last_undegraded":"2024-03-11T22:27:44.631726+0000","last_fullsized":"2024-03-11T22:27:44.631726+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:55.622561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:25:55.622561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":814,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.585574+0000","last_change":"2024-03-11T22:23:50.723665+0000","last_active":"2024-03-11T22:27:36.585574+0000","last_peered":"2024-03-11T22:27:36.585574+0000","last_clean":"2024-03-11T22:27:36.585574+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:27:36.585574+0000","last_undegraded":"2024-03-11T22:27:36.585574+0000","last_fullsized":"2024-03-11T22:27:36.585574+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:50.723528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:50.723528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"273'19","reported_seq":532,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.623970+0000","last_change":"2024-03-11T22:27:20.922568+0000","last_active":"2024-03-11T22:27:44.623970+0000","last_peered":"2024-03-11T22:27:44.623970+0000","last_clean":"2024-03-11T22:27:44.623970+0000","last_became_active":"2024-03-11T22:27:20.878606+0000","last_became_peered":"2024-03-11T22:27:20.878606+0000","last_unstale":"2024-03-11T22:27:44.623970+0000","last_undegraded":"2024-03-11T22:27:44.623970+0000","last_fullsized":"2024-03-11T22:27:44.623970+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:25:53.595233+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:25:53.595233+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":777,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.587014+0000","last_change":"2024-03-11T22:24:41.612477+0000","last_active":"2024-03-11T22:27:36.587014+0000","last_peered":"2024-03-11T22:27:36.587014+0000","last_clean":"2024-03-11T22:27:36.587014+0000","last_became_active":"2024-03-11T22:20:29.422747+0000","last_became_peered":"2024-03-11T22:20:29.422747+0000","last_unstale":"2024-03-11T22:27:36.587014+0000","last_undegraded":"2024-03-11T22:27:36.587014+0000","last_fullsized":"2024-03-11T22:27:36.587014+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:41.612363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.612363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":832,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.601473+0000","last_change":"2024-03-11T22:27:20.897483+0000","last_active":"2024-03-11T22:27:36.601473+0000","last_peered":"2024-03-11T22:27:36.601473+0000","last_clean":"2024-03-11T22:27:36.601473+0000","last_became_active":"2024-03-11T22:27:20.896798+0000","last_became_peered":"2024-03-11T22:27:20.896798+0000","last_unstale":"2024-03-11T22:27:36.601473+0000","last_undegraded":"2024-03-11T22:27:36.601473+0000","last_fullsized":"2024-03-11T22:27:36.601473+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:49.679560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:24:49.679560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":836,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.577617+0000","last_change":"2024-03-11T22:27:20.871432+0000","last_active":"2024-03-11T22:27:36.577617+0000","last_peered":"2024-03-11T22:27:36.577617+0000","last_clean":"2024-03-11T22:27:36.577617+0000","last_became_active":"2024-03-11T22:27:20.870904+0000","last_became_peered":"2024-03-11T22:27:20.870904+0000","last_unstale":"2024-03-11T22:27:36.577617+0000","last_undegraded":"2024-03-11T22:27:36.577617+0000","last_fullsized":"2024-03-11T22:27:36.577617+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:53.844874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:09.057970+0000","last_clean_scrub_stamp":"2024-03-11T22:25:53.844874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"274'20","reported_seq":890,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.588187+0000","last_change":"2024-03-11T22:23:52.653463+0000","last_active":"2024-03-11T22:27:36.588187+0000","last_peered":"2024-03-11T22:27:36.588187+0000","last_clean":"2024-03-11T22:27:36.588187+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:27:36.588187+0000","last_undegraded":"2024-03-11T22:27:36.588187+0000","last_fullsized":"2024-03-11T22:27:36.588187+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'20","last_scrub_stamp":"2024-03-11T22:23:52.653309+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:23:52.653309+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":861,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632315+0000","last_change":"2024-03-11T22:27:20.878609+0000","last_active":"2024-03-11T22:27:44.632315+0000","last_peered":"2024-03-11T22:27:44.632315+0000","last_clean":"2024-03-11T22:27:44.632315+0000","last_became_active":"2024-03-11T22:27:20.878236+0000","last_became_peered":"2024-03-11T22:27:20.878236+0000","last_unstale":"2024-03-11T22:27:44.632315+0000","last_undegraded":"2024-03-11T22:27:44.632315+0000","last_fullsized":"2024-03-11T22:27:44.632315+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:41.895742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:41.895742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":621,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.623423+0000","last_change":"2024-03-11T22:26:44.807269+0000","last_active":"2024-03-11T22:27:44.623423+0000","last_peered":"2024-03-11T22:27:44.623423+0000","last_clean":"2024-03-11T22:27:44.623423+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:27:44.623423+0000","last_undegraded":"2024-03-11T22:27:44.623423+0000","last_fullsized":"2024-03-11T22:27:44.623423+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.807187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.807187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:13.805182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":822,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.601849+0000","last_change":"2024-03-11T22:24:59.829935+0000","last_active":"2024-03-11T22:27:36.601849+0000","last_peered":"2024-03-11T22:27:36.601849+0000","last_clean":"2024-03-11T22:27:36.601849+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:27:36.601849+0000","last_undegraded":"2024-03-11T22:27:36.601849+0000","last_fullsized":"2024-03-11T22:27:36.601849+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:59.829811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:59.829811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"273'8","reported_seq":874,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.587039+0000","last_change":"2024-03-11T22:27:20.904573+0000","last_active":"2024-03-11T22:27:36.587039+0000","last_peered":"2024-03-11T22:27:36.587039+0000","last_clean":"2024-03-11T22:27:36.587039+0000","last_became_active":"2024-03-11T22:27:20.881939+0000","last_became_peered":"2024-03-11T22:27:20.881939+0000","last_unstale":"2024-03-11T22:27:36.587039+0000","last_undegraded":"2024-03-11T22:27:36.587039+0000","last_fullsized":"2024-03-11T22:27:36.587039+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'8","last_scrub_stamp":"2024-03-11T22:26:50.833149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.833149+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:55.491755+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":841,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.582120+0000","last_change":"2024-03-11T22:26:05.787798+0000","last_active":"2024-03-11T22:27:36.582120+0000","last_peered":"2024-03-11T22:27:36.582120+0000","last_clean":"2024-03-11T22:27:36.582120+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:27:36.582120+0000","last_undegraded":"2024-03-11T22:27:36.582120+0000","last_fullsized":"2024-03-11T22:27:36.582120+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:05.787543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:05.787543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":430,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.623880+0000","last_change":"2024-03-11T22:24:37.763829+0000","last_active":"2024-03-11T22:27:44.623880+0000","last_peered":"2024-03-11T22:27:44.623880+0000","last_clean":"2024-03-11T22:27:44.623880+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:27:44.623880+0000","last_undegraded":"2024-03-11T22:27:44.623880+0000","last_fullsized":"2024-03-11T22:27:44.623880+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.763709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.763709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":799,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.623607+0000","last_change":"2024-03-11T22:24:46.776804+0000","last_active":"2024-03-11T22:27:44.623607+0000","last_peered":"2024-03-11T22:27:44.623607+0000","last_clean":"2024-03-11T22:27:44.623607+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:27:44.623607+0000","last_undegraded":"2024-03-11T22:27:44.623607+0000","last_fullsized":"2024-03-11T22:27:44.623607+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:46.776600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:46.776600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"273'19","reported_seq":906,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624591+0000","last_change":"2024-03-11T22:27:20.934069+0000","last_active":"2024-03-11T22:27:44.624591+0000","last_peered":"2024-03-11T22:27:44.624591+0000","last_clean":"2024-03-11T22:27:44.624591+0000","last_became_active":"2024-03-11T22:27:20.883534+0000","last_became_peered":"2024-03-11T22:27:20.883534+0000","last_unstale":"2024-03-11T22:27:44.624591+0000","last_undegraded":"2024-03-11T22:27:44.624591+0000","last_fullsized":"2024-03-11T22:27:44.624591+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:26:40.796814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:40.796814+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:50.812018+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":839,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.586946+0000","last_change":"2024-03-11T22:27:20.888296+0000","last_active":"2024-03-11T22:27:36.586946+0000","last_peered":"2024-03-11T22:27:36.586946+0000","last_clean":"2024-03-11T22:27:36.586946+0000","last_became_active":"2024-03-11T22:27:20.883847+0000","last_became_peered":"2024-03-11T22:27:20.883847+0000","last_unstale":"2024-03-11T22:27:36.586946+0000","last_undegraded":"2024-03-11T22:27:36.586946+0000","last_fullsized":"2024-03-11T22:27:36.586946+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:38.610924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:38.610924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":843,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.595169+0000","last_change":"2024-03-11T22:26:43.777178+0000","last_active":"2024-03-11T22:27:36.595169+0000","last_peered":"2024-03-11T22:27:36.595169+0000","last_clean":"2024-03-11T22:27:36.595169+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:27:36.595169+0000","last_undegraded":"2024-03-11T22:27:36.595169+0000","last_fullsized":"2024-03-11T22:27:36.595169+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:43.777100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:43.777100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:46.467843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":34,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632080+0000","last_change":"2024-03-11T22:27:20.868978+0000","last_active":"2024-03-11T22:27:44.632080+0000","last_peered":"2024-03-11T22:27:44.632080+0000","last_clean":"2024-03-11T22:27:44.632080+0000","last_became_active":"2024-03-11T22:27:20.868015+0000","last_became_peered":"2024-03-11T22:27:20.868015+0000","last_unstale":"2024-03-11T22:27:44.632080+0000","last_undegraded":"2024-03-11T22:27:44.632080+0000","last_fullsized":"2024-03-11T22:27:44.632080+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:05.513210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.513210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"273'20","reported_seq":905,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.594921+0000","last_change":"2024-03-11T22:25:49.869260+0000","last_active":"2024-03-11T22:27:36.594921+0000","last_peered":"2024-03-11T22:27:36.594921+0000","last_clean":"2024-03-11T22:27:36.594921+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:27:36.594921+0000","last_undegraded":"2024-03-11T22:27:36.594921+0000","last_fullsized":"2024-03-11T22:27:36.594921+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:25:49.869107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:49.869107+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":400,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.586044+0000","last_change":"2024-03-11T22:27:20.991977+0000","last_active":"2024-03-11T22:27:36.586044+0000","last_peered":"2024-03-11T22:27:36.586044+0000","last_clean":"2024-03-11T22:27:36.586044+0000","last_became_active":"2024-03-11T22:27:20.888749+0000","last_became_peered":"2024-03-11T22:27:20.888749+0000","last_unstale":"2024-03-11T22:27:36.586044+0000","last_undegraded":"2024-03-11T22:27:36.586044+0000","last_fullsized":"2024-03-11T22:27:36.586044+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:26:03.627489+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:26:03.627489+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"315'11","reported_seq":1125,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624393+0000","last_change":"2024-03-11T22:26:39.795881+0000","last_active":"2024-03-11T22:27:44.624393+0000","last_peered":"2024-03-11T22:27:44.624393+0000","last_clean":"2024-03-11T22:27:44.624393+0000","last_became_active":"2024-03-11T22:20:29.415781+0000","last_became_peered":"2024-03-11T22:20:29.415781+0000","last_unstale":"2024-03-11T22:27:44.624393+0000","last_undegraded":"2024-03-11T22:27:44.624393+0000","last_fullsized":"2024-03-11T22:27:44.624393+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-11T22:26:39.795787+0000","last_deep_scrub":"315'11","last_deep_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_clean_scrub_stamp":"2024-03-11T22:26:39.795787+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:52.004194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":784,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631781+0000","last_change":"2024-03-11T22:23:51.821013+0000","last_active":"2024-03-11T22:27:44.631781+0000","last_peered":"2024-03-11T22:27:44.631781+0000","last_clean":"2024-03-11T22:27:44.631781+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:27:44.631781+0000","last_undegraded":"2024-03-11T22:27:44.631781+0000","last_fullsized":"2024-03-11T22:27:44.631781+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:23:51.820865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:23:51.820865+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"273'18","reported_seq":879,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.587315+0000","last_change":"2024-03-11T22:27:20.918842+0000","last_active":"2024-03-11T22:27:36.587315+0000","last_peered":"2024-03-11T22:27:36.587315+0000","last_clean":"2024-03-11T22:27:36.587315+0000","last_became_active":"2024-03-11T22:27:20.887735+0000","last_became_peered":"2024-03-11T22:27:20.887735+0000","last_unstale":"2024-03-11T22:27:36.587315+0000","last_undegraded":"2024-03-11T22:27:36.587315+0000","last_fullsized":"2024-03-11T22:27:36.587315+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:25:01.629321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:01.629321+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":37,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.623465+0000","last_change":"2024-03-11T22:27:20.878202+0000","last_active":"2024-03-11T22:27:44.623465+0000","last_peered":"2024-03-11T22:27:44.623465+0000","last_clean":"2024-03-11T22:27:44.623465+0000","last_became_active":"2024-03-11T22:27:20.877889+0000","last_became_peered":"2024-03-11T22:27:20.877889+0000","last_unstale":"2024-03-11T22:27:44.623465+0000","last_undegraded":"2024-03-11T22:27:44.623465+0000","last_fullsized":"2024-03-11T22:27:44.623465+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:04.394612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:25:04.394612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":592,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.587525+0000","last_change":"2024-03-11T22:25:05.656524+0000","last_active":"2024-03-11T22:27:36.587525+0000","last_peered":"2024-03-11T22:27:36.587525+0000","last_clean":"2024-03-11T22:27:36.587525+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:27:36.587525+0000","last_undegraded":"2024-03-11T22:27:36.587525+0000","last_fullsized":"2024-03-11T22:27:36.587525+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:05.656208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:25:05.656208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":823,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.586640+0000","last_change":"2024-03-11T22:26:37.062133+0000","last_active":"2024-03-11T22:27:36.586640+0000","last_peered":"2024-03-11T22:27:36.586640+0000","last_clean":"2024-03-11T22:27:36.586640+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:27:36.586640+0000","last_undegraded":"2024-03-11T22:27:36.586640+0000","last_fullsized":"2024-03-11T22:27:36.586640+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:37.062022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:37.062022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:28:04.112077+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"273'27","reported_seq":937,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.633807+0000","last_change":"2024-03-11T22:27:20.923017+0000","last_active":"2024-03-11T22:27:44.633807+0000","last_peered":"2024-03-11T22:27:44.633807+0000","last_clean":"2024-03-11T22:27:44.633807+0000","last_became_active":"2024-03-11T22:27:20.873269+0000","last_became_peered":"2024-03-11T22:27:20.873269+0000","last_unstale":"2024-03-11T22:27:44.633807+0000","last_undegraded":"2024-03-11T22:27:44.633807+0000","last_fullsized":"2024-03-11T22:27:44.633807+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:26:10.023796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:10.023796+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":853,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.578897+0000","last_change":"2024-03-11T22:24:55.760207+0000","last_active":"2024-03-11T22:27:36.578897+0000","last_peered":"2024-03-11T22:27:36.578897+0000","last_clean":"2024-03-11T22:27:36.578897+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:27:36.578897+0000","last_undegraded":"2024-03-11T22:27:36.578897+0000","last_fullsized":"2024-03-11T22:27:36.578897+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:55.760084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:24:55.760084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":32,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.581355+0000","last_change":"2024-03-11T22:27:20.881360+0000","last_active":"2024-03-11T22:27:36.581355+0000","last_peered":"2024-03-11T22:27:36.581355+0000","last_clean":"2024-03-11T22:27:36.581355+0000","last_became_active":"2024-03-11T22:27:20.880995+0000","last_became_peered":"2024-03-11T22:27:20.880995+0000","last_unstale":"2024-03-11T22:27:36.581355+0000","last_undegraded":"2024-03-11T22:27:36.581355+0000","last_fullsized":"2024-03-11T22:27:36.581355+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:37.481668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:24:37.481668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":619,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624548+0000","last_change":"2024-03-11T22:25:59.830054+0000","last_active":"2024-03-11T22:27:44.624548+0000","last_peered":"2024-03-11T22:27:44.624548+0000","last_clean":"2024-03-11T22:27:44.624548+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:27:44.624548+0000","last_undegraded":"2024-03-11T22:27:44.624548+0000","last_fullsized":"2024-03-11T22:27:44.624548+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:59.829804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:07.097049+0000","last_clean_scrub_stamp":"2024-03-11T22:25:59.829804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"273'9","reported_seq":681,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.594975+0000","last_change":"2024-03-11T22:27:20.995917+0000","last_active":"2024-03-11T22:27:36.594975+0000","last_peered":"2024-03-11T22:27:36.594975+0000","last_clean":"2024-03-11T22:27:36.594975+0000","last_became_active":"2024-03-11T22:27:20.888918+0000","last_became_peered":"2024-03-11T22:27:20.888918+0000","last_unstale":"2024-03-11T22:27:36.594975+0000","last_undegraded":"2024-03-11T22:27:36.594975+0000","last_fullsized":"2024-03-11T22:27:36.594975+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9","last_scrub_stamp":"2024-03-11T22:23:58.648385+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.648385+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":849,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631953+0000","last_change":"2024-03-11T22:27:20.865276+0000","last_active":"2024-03-11T22:27:44.631953+0000","last_peered":"2024-03-11T22:27:44.631953+0000","last_clean":"2024-03-11T22:27:44.631953+0000","last_became_active":"2024-03-11T22:27:20.864924+0000","last_became_peered":"2024-03-11T22:27:20.864924+0000","last_unstale":"2024-03-11T22:27:44.631953+0000","last_undegraded":"2024-03-11T22:27:44.631953+0000","last_fullsized":"2024-03-11T22:27:44.631953+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:40.905314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:24:40.905314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":31,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.580441+0000","last_change":"2024-03-11T22:27:20.878099+0000","last_active":"2024-03-11T22:27:36.580441+0000","last_peered":"2024-03-11T22:27:36.580441+0000","last_clean":"2024-03-11T22:27:36.580441+0000","last_became_active":"2024-03-11T22:27:20.877382+0000","last_became_peered":"2024-03-11T22:27:20.877382+0000","last_unstale":"2024-03-11T22:27:36.580441+0000","last_undegraded":"2024-03-11T22:27:36.580441+0000","last_fullsized":"2024-03-11T22:27:36.580441+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:09.575187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:09.575187+0000","last_clean_scrub_stamp":"2024-03-11T22:26:09.575187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":837,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631469+0000","last_change":"2024-03-11T22:27:20.863695+0000","last_active":"2024-03-11T22:27:44.631469+0000","last_peered":"2024-03-11T22:27:44.631469+0000","last_clean":"2024-03-11T22:27:44.631469+0000","last_became_active":"2024-03-11T22:27:20.863100+0000","last_became_peered":"2024-03-11T22:27:20.863100+0000","last_unstale":"2024-03-11T22:27:44.631469+0000","last_undegraded":"2024-03-11T22:27:44.631469+0000","last_fullsized":"2024-03-11T22:27:44.631469+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:58.198115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:58.198115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"273'15","reported_seq":644,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.585843+0000","last_change":"2024-03-11T22:25:03.685762+0000","last_active":"2024-03-11T22:27:36.585843+0000","last_peered":"2024-03-11T22:27:36.585843+0000","last_clean":"2024-03-11T22:27:36.585843+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:27:36.585843+0000","last_undegraded":"2024-03-11T22:27:36.585843+0000","last_fullsized":"2024-03-11T22:27:36.585843+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:25:03.685629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:03.685629+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":843,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.632226+0000","last_change":"2024-03-11T22:23:58.700286+0000","last_active":"2024-03-11T22:27:44.632226+0000","last_peered":"2024-03-11T22:27:44.632226+0000","last_clean":"2024-03-11T22:27:44.632226+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:27:44.632226+0000","last_undegraded":"2024-03-11T22:27:44.632226+0000","last_fullsized":"2024-03-11T22:27:44.632226+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:23:58.700176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:23:58.700176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":847,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631802+0000","last_change":"2024-03-11T22:24:49.941039+0000","last_active":"2024-03-11T22:27:44.631802+0000","last_peered":"2024-03-11T22:27:44.631802+0000","last_clean":"2024-03-11T22:27:44.631802+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:27:44.631802+0000","last_undegraded":"2024-03-11T22:27:44.631802+0000","last_fullsized":"2024-03-11T22:27:44.631802+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:49.940903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:24:49.940903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":835,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.578014+0000","last_change":"2024-03-11T22:27:20.874206+0000","last_active":"2024-03-11T22:27:36.578014+0000","last_peered":"2024-03-11T22:27:36.578014+0000","last_clean":"2024-03-11T22:27:36.578014+0000","last_became_active":"2024-03-11T22:27:20.873785+0000","last_became_peered":"2024-03-11T22:27:20.873785+0000","last_unstale":"2024-03-11T22:27:36.578014+0000","last_undegraded":"2024-03-11T22:27:36.578014+0000","last_fullsized":"2024-03-11T22:27:36.578014+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:37.738562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:37.738562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:38.086606+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"273'27","reported_seq":872,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.588276+0000","last_change":"2024-03-11T22:26:38.945645+0000","last_active":"2024-03-11T22:27:36.588276+0000","last_peered":"2024-03-11T22:27:36.588276+0000","last_clean":"2024-03-11T22:27:36.588276+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:27:36.588276+0000","last_undegraded":"2024-03-11T22:27:36.588276+0000","last_fullsized":"2024-03-11T22:27:36.588276+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:26:38.945505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:38.945505+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:51.610570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":849,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.587126+0000","last_change":"2024-03-11T22:26:39.902683+0000","last_active":"2024-03-11T22:27:36.587126+0000","last_peered":"2024-03-11T22:27:36.587126+0000","last_clean":"2024-03-11T22:27:36.587126+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:27:36.587126+0000","last_undegraded":"2024-03-11T22:27:36.587126+0000","last_fullsized":"2024-03-11T22:27:36.587126+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:26:39.902588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:39.902588+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:53.907172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":627,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.623547+0000","last_change":"2024-03-11T22:27:20.870485+0000","last_active":"2024-03-11T22:27:44.623547+0000","last_peered":"2024-03-11T22:27:44.623547+0000","last_clean":"2024-03-11T22:27:44.623547+0000","last_became_active":"2024-03-11T22:27:20.869836+0000","last_became_peered":"2024-03-11T22:27:20.869836+0000","last_unstale":"2024-03-11T22:27:44.623547+0000","last_undegraded":"2024-03-11T22:27:44.623547+0000","last_fullsized":"2024-03-11T22:27:44.623547+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:54.762527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:25:54.762527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":833,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.601571+0000","last_change":"2024-03-11T22:26:34.021098+0000","last_active":"2024-03-11T22:27:36.601571+0000","last_peered":"2024-03-11T22:27:36.601571+0000","last_clean":"2024-03-11T22:27:36.601571+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:27:36.601571+0000","last_undegraded":"2024-03-11T22:27:36.601571+0000","last_fullsized":"2024-03-11T22:27:36.601571+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:34.020992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:34.020992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:27:58.307326+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"273'19","reported_seq":674,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.624523+0000","last_change":"2024-03-11T22:27:20.905472+0000","last_active":"2024-03-11T22:27:44.624523+0000","last_peered":"2024-03-11T22:27:44.624523+0000","last_clean":"2024-03-11T22:27:44.624523+0000","last_became_active":"2024-03-11T22:27:20.866014+0000","last_became_peered":"2024-03-11T22:27:20.866014+0000","last_unstale":"2024-03-11T22:27:44.624523+0000","last_undegraded":"2024-03-11T22:27:44.624523+0000","last_fullsized":"2024-03-11T22:27:44.624523+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:24:39.754894+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:24:39.754894+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":843,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-11T22:27:36.581745+0000","last_change":"2024-03-11T22:24:00.015418+0000","last_active":"2024-03-11T22:27:36.581745+0000","last_peered":"2024-03-11T22:27:36.581745+0000","last_clean":"2024-03-11T22:27:36.581745+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:27:36.581745+0000","last_undegraded":"2024-03-11T22:27:36.581745+0000","last_fullsized":"2024-03-11T22:27:36.581745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:00.015301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:24:00.015301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":312,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631251+0000","last_change":"2024-03-11T22:27:20.868736+0000","last_active":"2024-03-11T22:27:44.631251+0000","last_peered":"2024-03-11T22:27:44.631251+0000","last_clean":"2024-03-11T22:27:44.631251+0000","last_became_active":"2024-03-11T22:27:20.868196+0000","last_became_peered":"2024-03-11T22:27:20.868196+0000","last_unstale":"2024-03-11T22:27:44.631251+0000","last_undegraded":"2024-03-11T22:27:44.631251+0000","last_fullsized":"2024-03-11T22:27:44.631251+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:06.778392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:06.778392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":616,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-11T22:27:44.631456+0000","last_change":"2024-03-11T22:27:20.862218+0000","last_active":"2024-03-11T22:27:44.631456+0000","last_peered":"2024-03-11T22:27:44.631456+0000","last_clean":"2024-03-11T22:27:44.631456+0000","last_became_active":"2024-03-11T22:27:20.861869+0000","last_became_peered":"2024-03-11T22:27:20.861869+0000","last_unstale":"2024-03-11T22:27:44.631456+0000","last_undegraded":"2024-03-11T22:27:44.631456+0000","last_fullsized":"2024-03-11T22:27:44.631456+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:10.104642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:10.104642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4419878912,"num_objects":67442,"num_object_clones":0,"num_object_copies":202326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6136,"num_objects_misplaced":13142,"num_objects_unfound":0,"num_objects_dirty":67442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67442,"num_write_kb":4316288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5132,"num_bytes_recovered":336330752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13758234624,"data_stored":13758234624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67242,"ondisk_log_size":67242,"up":48,"acting":41,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":106,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":608258,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":9,"num_write":5,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":3768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":495616,"data_stored":13114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":734,"ondisk_log_size":734,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005291,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2474808,"kb_used_data":1753648,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":91270344,"statfs":{"total":95995035648,"available":93460832256,"internally_reserved":0,"allocated":1795735552,"data_stored":1795259439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":738458059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332336,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2912204,"kb_used_data":2331264,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":90832948,"statfs":{"total":95995035648,"available":93012938752,"internally_reserved":0,"allocated":2387214336,"data_stored":2386730434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575113,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738639,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2314416,"kb_used_data":1521064,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":91430736,"statfs":{"total":95995035648,"available":93625073664,"internally_reserved":0,"allocated":1557569536,"data_stored":1557100810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,1,1,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411589,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2156640,"kb_used_data":1629844,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91588512,"statfs":{"total":95995035648,"available":93786636288,"internally_reserved":0,"allocated":1668960256,"data_stored":1668520813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149950,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2541192,"kb_used_data":1829760,"kb_used_omap":1,"kb_used_meta":711422,"kb_avail":91203960,"statfs":{"total":95995035648,"available":93392855040,"internally_reserved":0,"allocated":1873674240,"data_stored":1873182836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":728496583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921019,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1994232,"kb_used_data":1354988,"kb_used_omap":1,"kb_used_meta":639230,"kb_avail":91750920,"statfs":{"total":95995035648,"available":93952942080,"internally_reserved":0,"allocated":1387507712,"data_stored":1387043054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":654571974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659380,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1973860,"kb_used_data":1288796,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":91771292,"statfs":{"total":95995035648,"available":93973803008,"internally_reserved":0,"allocated":1319727104,"data_stored":1319278916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":701495750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1570766848,"data_stored":1570766848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1666777088,"data_stored":1666777088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1690304512,"data_stored":1690304512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1872625664,"data_stored":1872625664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1409024000,"data_stored":1409024000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1328480256,"data_stored":1328480256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2400059392,"data_stored":2400059392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1820196864,"data_stored":1820196864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:27:54.261 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-11T22:27:54.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool set unique_pool_4 pg_num 26 2024-03-11T22:27:54.285 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:54.325 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:54.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:54.427 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:54.463 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:54.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:54.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:54.618 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:54.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:54.719 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:54.752 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:54.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:54.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:54.854 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:54.892 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:54.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:54.994 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:55.026 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:55.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:55.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:55.128 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:55.163 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:55.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:55.178 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:27:55.174539+0000 min lat: 0.001317 max lat: 2.21187 avg lat: 0.0124857 2024-03-11T22:27:55.178 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:27:55.178 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 60 16 76873 76857 80.0478 49.9375 0.0123726 0.0124857 2024-03-11T22:27:55.266 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:55.307 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:55.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:55.409 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:55.446 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:55.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:55.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:55.548 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:55.588 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:55.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:55.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:55.690 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:55.727 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:55.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:55.828 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:55.870 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:55.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:55.913 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:55 smithi045 ceph-mon[39715]: pgmap v935: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 854 op/s; 6136/221334 objects degraded (2.772%); 11117/221334 objects misplaced (5.023%); 17 MiB/s, 270 objects/s recovering 2024-03-11T22:27:55.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:55 smithi045 ceph-mon[30178]: pgmap v935: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 854 op/s; 6136/221334 objects degraded (2.772%); 11117/221334 objects misplaced (5.023%); 17 MiB/s, 270 objects/s recovering 2024-03-11T22:27:55.972 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:56.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:56.005 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:56.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:56.050 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:27:56.054 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:27:56.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:55 smithi144 ceph-mon[33984]: pgmap v935: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 854 op/s; 6136/221334 objects degraded (2.772%); 11117/221334 objects misplaced (5.023%); 17 MiB/s, 270 objects/s recovering 2024-03-11T22:27:56.107 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:56.142 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:56.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:56.179 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 61 16 77704 77688 79.5866 51.9375 0.0196012 0.0125583 2024-03-11T22:27:56.243 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:56.279 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:56.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:56.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:56.381 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:56.407 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:56.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:56.509 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:56.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:56.549 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:56.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:56.651 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:56.678 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:56.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:56.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:56.780 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:56.810 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:56.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:56.912 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:56.936 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:56.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:57.002 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:56 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:27:57.003 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:56 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:27:57.003 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:56 smithi045 ceph-mon[30178]: osdmap e459: 8 total, 8 up, 7 in 2024-03-11T22:27:57.003 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:56 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 6136/221334 objects degraded (2.772%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:57.003 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:56 smithi045 ceph-mon[30178]: osdmap e460: 8 total, 8 up, 7 in 2024-03-11T22:27:57.003 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:56 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:27:57.003 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:56 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:27:57.003 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:56 smithi045 ceph-mon[39715]: osdmap e459: 8 total, 8 up, 7 in 2024-03-11T22:27:57.003 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:56 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 6136/221334 objects degraded (2.772%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:57.003 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:56 smithi045 ceph-mon[39715]: osdmap e460: 8 total, 8 up, 7 in 2024-03-11T22:27:57.038 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:57.065 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:57.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:57.099 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:27:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:27:57.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:56 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:27:57.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:56 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:27:57.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:56 smithi144 ceph-mon[33984]: osdmap e459: 8 total, 8 up, 7 in 2024-03-11T22:27:57.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:56 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 6136/221334 objects degraded (2.772%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:27:57.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:56 smithi144 ceph-mon[33984]: osdmap e460: 8 total, 8 up, 7 in 2024-03-11T22:27:57.168 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:57.178 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 62 16 78318 78302 78.9217 38.375 0.00881638 0.0125921 2024-03-11T22:27:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:57.207 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:57.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:57.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:57.342 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:57.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:57.445 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:57.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:57.470 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:57.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:57.571 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:27:57.581 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:27:57.582 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:27:57.582 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-03-11T22:27:57.582 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:27:57.599 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:57.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:57.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:27:57.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:57.746 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:57.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:27:57.848 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:57.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:57.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:57.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:27:58.029 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:58.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:58.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:27:58.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:57 smithi045 ceph-mon[30178]: pgmap v937: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 873 op/s; 6136/228807 objects degraded (2.682%); 10222/228807 objects misplaced (4.468%); 18 MiB/s, 295 objects/s recovering 2024-03-11T22:27:58.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:58.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:57 smithi045 ceph-mon[30178]: osdmap e461: 8 total, 8 up, 7 in 2024-03-11T22:27:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:57 smithi045 ceph-mon[39715]: pgmap v937: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 873 op/s; 6136/228807 objects degraded (2.682%); 10222/228807 objects misplaced (4.468%); 18 MiB/s, 295 objects/s recovering 2024-03-11T22:27:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:57 smithi045 ceph-mon[39715]: osdmap e461: 8 total, 8 up, 7 in 2024-03-11T22:27:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:57 smithi144 ceph-mon[33984]: pgmap v937: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 873 op/s; 6136/228807 objects degraded (2.682%); 10222/228807 objects misplaced (4.468%); 18 MiB/s, 295 objects/s recovering 2024-03-11T22:27:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:27:58.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:57 smithi144 ceph-mon[33984]: osdmap e461: 8 total, 8 up, 7 in 2024-03-11T22:27:58.131 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:58.168 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:58.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:58.179 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 63 15 78783 78768 78.1313 29.125 0.0132764 0.012792 2024-03-11T22:27:58.270 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:58.303 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:58.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:58.405 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:58.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:58.447 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:58.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:58.527 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T22:27:58.548 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:58.580 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:58.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:58.682 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:27:58.703 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:27:58.727 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:58 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/798871166' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:27:58.727 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:58 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:27:58.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:58 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/798871166' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:27:58.727 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:58 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:27:58.765 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:58.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:58.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:27:58.868 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:27:58.900 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:58.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:27:59.001 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:59.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:59.037 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:59.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:59.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:58 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/798871166' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:27:59.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:58 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:27:59.138 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:59.169 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:59.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:59.178 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 64 16 79576 79560 77.6839 49.5 0.00685865 0.0128654 2024-03-11T22:27:59.271 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:59.303 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:59.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:59.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:59.405 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:27:59.440 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:59.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:59.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:27:59.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:27:59.589 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:59.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:27:59.690 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:59.719 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:59.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:59.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:59.821 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:27:59.855 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:27:59.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:59.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:27:59.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:27:59.994 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:27:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:27:59.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:00.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:59 smithi045 ceph-mon[30178]: pgmap v940: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 47 MiB/s wr, 759 op/s; 6136/234198 objects degraded (2.620%); 9428/234198 objects misplaced (4.026%); 20 MiB/s, 316 objects/s recovering 2024-03-11T22:28:00.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:59 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:28:00.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:59 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:28:00.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:27:59 smithi045 ceph-mon[30178]: osdmap e462: 8 total, 8 up, 7 in 2024-03-11T22:28:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:59 smithi045 ceph-mon[39715]: pgmap v940: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 47 MiB/s wr, 759 op/s; 6136/234198 objects degraded (2.620%); 9428/234198 objects misplaced (4.026%); 20 MiB/s, 316 objects/s recovering 2024-03-11T22:28:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:59 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:28:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:59 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:28:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:27:59 smithi045 ceph-mon[39715]: osdmap e462: 8 total, 8 up, 7 in 2024-03-11T22:28:00.097 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:00.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:59 smithi144 ceph-mon[33984]: pgmap v940: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 47 MiB/s wr, 759 op/s; 6136/234198 objects degraded (2.620%); 9428/234198 objects misplaced (4.026%); 20 MiB/s, 316 objects/s recovering 2024-03-11T22:28:00.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:59 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:28:00.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:59 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:28:00.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:27:59 smithi144 ceph-mon[33984]: osdmap e462: 8 total, 8 up, 7 in 2024-03-11T22:28:00.129 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:00.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:00.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:00.179 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 65 16 80410 80394 77.2906 52.125 0.0184777 0.012932 2024-03-11T22:28:00.231 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:00.266 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:00.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:00.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:00.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:00.409 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:00.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:00.510 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:00.546 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:00.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:00.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:00.648 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:00.682 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:00.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:00.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:00.828 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:00.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:00.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:00.974 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:00.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:01.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:00 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:28:01.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:00 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:28:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:00 smithi045 ceph-mon[39715]: osdmap e463: 8 total, 8 up, 7 in 2024-03-11T22:28:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:00 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:28:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:00 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:28:01.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:00 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:28:01.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:00 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:28:01.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:00 smithi045 ceph-mon[30178]: osdmap e463: 8 total, 8 up, 7 in 2024-03-11T22:28:01.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:00 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:28:01.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:00 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:28:01.077 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:01.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:00 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:28:01.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:00 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:28:01.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:00 smithi144 ceph-mon[33984]: osdmap e463: 8 total, 8 up, 7 in 2024-03-11T22:28:01.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:00 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:28:01.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:00 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:28:01.126 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:01.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:01.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:01.179 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 66 16 81258 81242 76.9225 53 0.0109689 0.0129945 2024-03-11T22:28:01.229 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:01.254 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:01.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:01.356 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:01.393 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:01.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:01.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:01.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:01.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:01.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:01.639 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:01.671 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:01.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:01.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:01.773 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:01.813 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:01.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:01.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:01.954 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:01.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:02.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:01 smithi045 ceph-mon[39715]: pgmap v943: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 450 op/s; 6136/234198 objects degraded (2.620%); 9428/234198 objects misplaced (4.026%); 12 MiB/s, 198 objects/s recovering 2024-03-11T22:28:02.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:01 smithi045 ceph-mon[39715]: osdmap e464: 8 total, 8 up, 7 in 2024-03-11T22:28:02.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:28:02.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:02.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:01 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 6136/234198 objects degraded (2.620%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:28:02.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:01 smithi045 ceph-mon[30178]: pgmap v943: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 450 op/s; 6136/234198 objects degraded (2.620%); 9428/234198 objects misplaced (4.026%); 12 MiB/s, 198 objects/s recovering 2024-03-11T22:28:02.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:01 smithi045 ceph-mon[30178]: osdmap e464: 8 total, 8 up, 7 in 2024-03-11T22:28:02.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:28:02.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:02.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:01 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 6136/234198 objects degraded (2.620%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:28:02.057 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:02.092 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:02.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:01 smithi144 ceph-mon[33984]: pgmap v943: 158 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 450 op/s; 6136/234198 objects degraded (2.620%); 9428/234198 objects misplaced (4.026%); 12 MiB/s, 198 objects/s recovering 2024-03-11T22:28:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:01 smithi144 ceph-mon[33984]: osdmap e464: 8 total, 8 up, 7 in 2024-03-11T22:28:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:28:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:01 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 6136/234198 objects degraded (2.620%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T22:28:02.142 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:28:02.179 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 67 16 82100 82084 76.5594 52.625 0.0163103 0.0130553 2024-03-11T22:28:02.194 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:02.231 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:02.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:02.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:02.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:02.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:02.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:02.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:02.538 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:02.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:02.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:02.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:02.678 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:02.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:02.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:02.780 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:02.810 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:02.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:02.895 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:28:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:28:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:28:02.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:02.954 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:02.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:03.055 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:03.089 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:03.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:03.180 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 68 16 82940 82924 76.2055 52.5 0.0227722 0.0131165 2024-03-11T22:28:03.191 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:03.215 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:03.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:03.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:03.317 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:03.360 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:03.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:03.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:03.510 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:03.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:03.612 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:03.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:03.644 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:03.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:03.745 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:03.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:03.779 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:03.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:03.879 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:03.914 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:03.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:04.015 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:04.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:03 smithi045 ceph-mon[39715]: pgmap v945: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 46 MiB/s wr, 738 op/s; 3110/244389 objects degraded (1.273%); 11237/244389 objects misplaced (4.598%); 19 MiB/s, 301 objects/s recovering 2024-03-11T22:28:04.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:03 smithi045 ceph-mon[30178]: pgmap v945: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 46 MiB/s wr, 738 op/s; 3110/244389 objects degraded (1.273%); 11237/244389 objects misplaced (4.598%); 19 MiB/s, 301 objects/s recovering 2024-03-11T22:28:04.048 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:04.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:04.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:04.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:03 smithi144 ceph-mon[33984]: pgmap v945: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 46 MiB/s wr, 738 op/s; 3110/244389 objects degraded (1.273%); 11237/244389 objects misplaced (4.598%); 19 MiB/s, 301 objects/s recovering 2024-03-11T22:28:04.150 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:04.180 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 69 16 83754 83738 75.8383 50.875 0.0101379 0.0131793 2024-03-11T22:28:04.200 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:04.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:04.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:04.302 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:04.339 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:04.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:04.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:04.440 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:04.485 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:04.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:04.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:04.587 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:04.619 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:04.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:04.720 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:04.754 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:04.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:04.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:04.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:04.887 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:04.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:04.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:05.028 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:05.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:05.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:05.130 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:05.163 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:05.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:05.181 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 70 16 84589 84573 75.5004 52.1875 0.00581781 0.0132382 2024-03-11T22:28:05.265 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:05.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:05.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:05.397 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:05.429 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:05.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:05.531 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:05.563 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:05.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:05.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:05.688 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:05.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:05.790 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:05.829 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:05.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:05.930 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:05.964 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:05.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:06.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:05 smithi045 ceph-mon[39715]: pgmap v946: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 38 MiB/s wr, 607 op/s; 3110/245118 objects degraded (1.269%); 11003/245118 objects misplaced (4.489%); 15 MiB/s, 241 objects/s recovering 2024-03-11T22:28:06.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:05 smithi045 ceph-mon[30178]: pgmap v946: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 38 MiB/s wr, 607 op/s; 3110/245118 objects degraded (1.269%); 11003/245118 objects misplaced (4.489%); 15 MiB/s, 241 objects/s recovering 2024-03-11T22:28:06.066 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:06.090 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:06.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:06.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:06.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:05 smithi144 ceph-mon[33984]: pgmap v946: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 38 MiB/s wr, 607 op/s; 3110/245118 objects degraded (1.269%); 11003/245118 objects misplaced (4.489%); 15 MiB/s, 241 objects/s recovering 2024-03-11T22:28:06.180 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 71 16 85326 85310 75.0858 46.0625 0.0288842 0.0133118 2024-03-11T22:28:06.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:06.239 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:06.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:06.341 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:06.381 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:06.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:06.483 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:06.515 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:06.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:06.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:06.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:06.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:06.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:06.798 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:06.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:06.900 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:06.933 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:06.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:07.035 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:07.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:06 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 3110/245118 objects degraded (1.269%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:28:07.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:06 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 3110/245118 objects degraded (1.269%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:28:07.067 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:07.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:07.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:06 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 3110/245118 objects degraded (1.269%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:28:07.099 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:28:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:28:07.169 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:07.180 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 72 16 86064 86048 74.6835 46.125 0.0157352 0.0133841 2024-03-11T22:28:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:07.204 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:07.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:07.305 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:07.330 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:07.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:07.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:07.487 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:07.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:07.589 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:07.622 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:07.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:07.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:07.724 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:07.769 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:07.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:07.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:07.871 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:07.912 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:07.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:07.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:08.014 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:08.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:07 smithi045 ceph-mon[30178]: pgmap v947: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 657 B/s rd, 48 MiB/s wr, 763 op/s; 3110/252036 objects degraded (1.234%); 10617/252036 objects misplaced (4.212%); 15 MiB/s, 235 objects/s recovering 2024-03-11T22:28:08.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:08.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:07 smithi045 ceph-mon[39715]: pgmap v947: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 657 B/s rd, 48 MiB/s wr, 763 op/s; 3110/252036 objects degraded (1.234%); 10617/252036 objects misplaced (4.212%); 15 MiB/s, 235 objects/s recovering 2024-03-11T22:28:08.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:08.044 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:08.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:08.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:08.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:07 smithi144 ceph-mon[33984]: pgmap v947: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 657 B/s rd, 48 MiB/s wr, 763 op/s; 3110/252036 objects degraded (1.234%); 10617/252036 objects misplaced (4.212%); 15 MiB/s, 235 objects/s recovering 2024-03-11T22:28:08.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:08.146 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:08.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:08.177 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:08.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:08.180 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 73 16 86862 86846 74.3436 49.875 0.0138085 0.0134458 2024-03-11T22:28:08.279 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:08.317 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:08.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:08.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:08.419 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:08.457 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:08.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:08.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:08.559 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:08.596 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:08.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:08.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:08.698 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:08.729 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:08.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:08.831 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:08.867 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:08.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:08.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:08.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:09.008 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:09.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:09.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:09.162 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:09.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:09.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:09.180 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 74 16 87679 87663 74.029 51.0625 0.018115 0.0135021 2024-03-11T22:28:09.264 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:09.302 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:09.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:09.403 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:09.549 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:09.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:09.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:09.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:09.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:09.711 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:09.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:09.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:09.857 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:09.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:09.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:09.959 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:09.996 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:09.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:09.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:10.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:09 smithi045 ceph-mon[39715]: pgmap v948: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 874 op/s; 3110/257199 objects degraded (1.209%); 9810/257199 objects misplaced (3.814%); 19 MiB/s, 301 objects/s recovering 2024-03-11T22:28:10.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:09 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:28:10.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:09 smithi045 ceph-mon[30178]: pgmap v948: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 874 op/s; 3110/257199 objects degraded (1.209%); 9810/257199 objects misplaced (3.814%); 19 MiB/s, 301 objects/s recovering 2024-03-11T22:28:10.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:09 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:28:10.098 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:10.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:09 smithi144 ceph-mon[33984]: pgmap v948: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 874 op/s; 3110/257199 objects degraded (1.209%); 9810/257199 objects misplaced (3.814%); 19 MiB/s, 301 objects/s recovering 2024-03-11T22:28:10.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:09 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:28:10.123 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:10.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:10.180 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 75 16 88373 88357 73.6202 43.375 0.0458683 0.0135776 2024-03-11T22:28:10.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:10.259 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:10.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:10.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:10.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:10.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:10.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:10.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:10.546 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:10.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:10.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:10.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:10.688 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:10.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:10.790 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:10.828 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:10.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:10.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:10.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:10.965 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:10.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:11.070 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:11.108 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:11.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:11.180 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 76 16 89257 89241 73.3785 55.25 0.00789354 0.0136216 2024-03-11T22:28:11.210 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:11.242 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:11.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:11.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:11.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:11.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:11.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:11.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:11.512 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:11.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:11.613 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:11.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:11.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:11.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:11.761 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:11.799 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:11.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:11.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:11.901 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:11.924 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:11.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:11.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:12.026 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:12.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:11 smithi045 ceph-mon[39715]: pgmap v949: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 48 MiB/s wr, 767 op/s; 3110/257199 objects degraded (1.209%); 9810/257199 objects misplaced (3.814%); 17 MiB/s, 264 objects/s recovering 2024-03-11T22:28:12.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:11 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 3110/257199 objects degraded (1.209%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:28:12.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:11 smithi045 ceph-mon[30178]: pgmap v949: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 48 MiB/s wr, 767 op/s; 3110/257199 objects degraded (1.209%); 9810/257199 objects misplaced (3.814%); 17 MiB/s, 264 objects/s recovering 2024-03-11T22:28:12.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:11 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 3110/257199 objects degraded (1.209%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:28:12.066 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:12.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:12.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:12.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:11 smithi144 ceph-mon[33984]: pgmap v949: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 48 MiB/s wr, 767 op/s; 3110/257199 objects degraded (1.209%); 9810/257199 objects misplaced (3.814%); 17 MiB/s, 264 objects/s recovering 2024-03-11T22:28:12.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:11 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 3110/257199 objects degraded (1.209%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:28:12.168 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:12.180 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 77 16 90055 90039 73.0732 49.875 0.019165 0.0136787 2024-03-11T22:28:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:12.208 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:12.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:12.309 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:12.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:12.352 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:12.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:12.453 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:12.493 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:12.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:12.595 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:12.630 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:12.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:12.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:12.732 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:12.763 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:12.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:12.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:12.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:12.904 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:12.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:12.928 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:28:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:28:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:28:13.005 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:13.047 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:13.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:13.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:13.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:13.181 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 78 16 90974 90958 72.8724 57.4375 0.0124569 0.013716 2024-03-11T22:28:13.187 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:13.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:13.289 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:13.324 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:13.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:13.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:13.426 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:13.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:13.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:13.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:13.614 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:13.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:13.716 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:13.741 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:13.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:13.843 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:13.880 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:13.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:13.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:13.982 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:14.029 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:14.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:14.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:13 smithi045 ceph-mon[30178]: pgmap v950: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 870 B/s rd, 60 MiB/s wr, 966 op/s; 3110/268278 objects degraded (1.159%); 8779/268278 objects misplaced (3.272%); 20 MiB/s, 312 objects/s recovering 2024-03-11T22:28:14.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:13 smithi045 ceph-mon[39715]: pgmap v950: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 870 B/s rd, 60 MiB/s wr, 966 op/s; 3110/268278 objects degraded (1.159%); 8779/268278 objects misplaced (3.272%); 20 MiB/s, 312 objects/s recovering 2024-03-11T22:28:14.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:13 smithi144 ceph-mon[33984]: pgmap v950: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 870 B/s rd, 60 MiB/s wr, 966 op/s; 3110/268278 objects degraded (1.159%); 8779/268278 objects misplaced (3.272%); 20 MiB/s, 312 objects/s recovering 2024-03-11T22:28:14.130 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:14.134 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:28:14.157 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:14.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:14.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:14.185 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 79 16 91745 91729 72.5588 48.1875 0.0140965 0.0137739 2024-03-11T22:28:14.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:14.303 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:14.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:14.405 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:14.438 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:14.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:14.541 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:14.566 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:14.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:14.667 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:14.700 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:14.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:14.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:14.845 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:14.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:14.946 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:14.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:14.988 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:14.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:15.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:15.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:15.141 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:15.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:15.182 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:2024-03-11T22:28:15.178588+0000 min lat: 0.001317 max lat: 2.21187 avg lat: 0.013824 2024-03-11T22:28:15.182 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T22:28:15.182 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 80 16 92570 92554 72.2963 51.5625 0.0228294 0.013824 2024-03-11T22:28:15.242 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:15.278 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:15.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:15.379 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:15.418 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:15.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:15.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:15.555 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:15.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:15.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:15.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:15.696 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:15.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:15.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:15.798 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:15.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:15.830 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:15.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:15.931 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:15.964 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:15.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:15.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:16.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:15 smithi045 ceph-mon[39715]: pgmap v951: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 685 op/s; 3110/269022 objects degraded (1.156%); 8471/269022 objects misplaced (3.149%); 14 MiB/s, 230 objects/s recovering 2024-03-11T22:28:16.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:28:16.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:28:16.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:28:16.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:15 smithi045 ceph-mon[30178]: pgmap v951: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 685 op/s; 3110/269022 objects degraded (1.156%); 8471/269022 objects misplaced (3.149%); 14 MiB/s, 230 objects/s recovering 2024-03-11T22:28:16.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:28:16.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:28:16.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:28:16.066 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:16.099 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:16.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:15 smithi144 ceph-mon[33984]: pgmap v951: 158 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 685 op/s; 3110/269022 objects degraded (1.156%); 8471/269022 objects misplaced (3.149%); 14 MiB/s, 230 objects/s recovering 2024-03-11T22:28:16.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:28:16.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:28:16.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:28:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:16.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:16.182 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 81 16 93369 93353 72.0202 49.9375 0.028882 0.0138799 2024-03-11T22:28:16.201 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:16.235 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:16.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:16.337 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:16.363 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:16.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:16.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:16.465 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:16.503 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:16.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:16.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:16.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:16.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:16.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:16.760 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:16.789 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:16.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:16.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:16.891 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:16.919 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:16.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:17.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:17.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:17.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:16 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 3110/269022 objects degraded (1.156%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:28:17.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:16 smithi045 ceph-mon[39715]: osdmap e465: 8 total, 8 up, 7 in 2024-03-11T22:28:17.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:17.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:16 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 3110/269022 objects degraded (1.156%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:28:17.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:16 smithi045 ceph-mon[30178]: osdmap e465: 8 total, 8 up, 7 in 2024-03-11T22:28:17.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:17.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:17.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:17.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:16 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 3110/269022 objects degraded (1.156%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:28:17.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:16 smithi144 ceph-mon[33984]: osdmap e465: 8 total, 8 up, 7 in 2024-03-11T22:28:17.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:28:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:28:17.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:17.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:17.179 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:17.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:17.182 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 82 16 93906 93890 71.5512 33.5625 0.00539515 0.0138902 2024-03-11T22:28:17.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:17.310 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:17.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:17.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:17.412 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:17.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:17.439 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:17.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:17.540 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:17.568 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:17.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:17.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:17.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:17.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:17.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:17.859 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:17.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:17.960 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:17.985 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:17.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:18.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:17 smithi045 ceph-mon[30178]: pgmap v952: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 53 MiB/s wr, 856 op/s; 3110/275943 objects degraded (1.127%); 8040/275943 objects misplaced (2.914%); 15 MiB/s, 246 objects/s recovering 2024-03-11T22:28:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:18.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:17 smithi045 ceph-mon[30178]: osdmap e466: 8 total, 8 up, 7 in 2024-03-11T22:28:18.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:17 smithi045 ceph-mon[39715]: pgmap v952: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 53 MiB/s wr, 856 op/s; 3110/275943 objects degraded (1.127%); 8040/275943 objects misplaced (2.914%); 15 MiB/s, 246 objects/s recovering 2024-03-11T22:28:18.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:18.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:17 smithi045 ceph-mon[39715]: osdmap e466: 8 total, 8 up, 7 in 2024-03-11T22:28:18.086 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:18.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:17 smithi144 ceph-mon[33984]: pgmap v952: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+remapped, 5 active+recovering+undersized+remapped, 151 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 53 MiB/s wr, 856 op/s; 3110/275943 objects degraded (1.127%); 8040/275943 objects misplaced (2.914%); 15 MiB/s, 246 objects/s recovering 2024-03-11T22:28:18.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:18.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:17 smithi144 ceph-mon[33984]: osdmap e466: 8 total, 8 up, 7 in 2024-03-11T22:28:18.123 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:18.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:18.182 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 83 16 94347 94331 71.0213 27.5625 0.0299632 0.0140741 2024-03-11T22:28:18.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:18.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:18.263 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:18.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:18.364 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:18.397 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:18.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:18.499 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:18.523 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:18.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:18.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:18.624 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:18.666 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:18.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:18.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:18.815 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:18.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:18.917 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:18.942 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:18.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:18.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:19.043 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:19.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:19.084 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:19.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:19.182 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 84 16 95160 95144 70.7806 50.8125 0.0105709 0.0141221 2024-03-11T22:28:19.185 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:19.220 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:19.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:19.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:19.322 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:19.356 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:19.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:19.458 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:19.497 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:19.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:19.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:19.634 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:19.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:19.736 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:19.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:19.779 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:19.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:19.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:19.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:19.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:19.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:20.021 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:20.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:19 smithi045 ceph-mon[30178]: pgmap v955: 158 pgs: 1 active+recovery_wait+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 3110/281232 objects degraded (1.106%); 6801/281232 objects misplaced (2.418%); 23 MiB/s, 375 objects/s recovering 2024-03-11T22:28:20.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:19 smithi045 ceph-mon[39715]: pgmap v955: 158 pgs: 1 active+recovery_wait+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 3110/281232 objects degraded (1.106%); 6801/281232 objects misplaced (2.418%); 23 MiB/s, 375 objects/s recovering 2024-03-11T22:28:20.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:20.057 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:20.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:20.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:19 smithi144 ceph-mon[33984]: pgmap v955: 158 pgs: 1 active+recovery_wait+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 3110/281232 objects degraded (1.106%); 6801/281232 objects misplaced (2.418%); 23 MiB/s, 375 objects/s recovering 2024-03-11T22:28:20.158 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:20.182 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 85 16 96039 96023 70.5942 54.9375 0.0159437 0.0141591 2024-03-11T22:28:20.196 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:20.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:20.298 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:20.335 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:20.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:20.436 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:20.460 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:20.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:20.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:20.562 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:20.591 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:20.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:20.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:20.693 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:20.812 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:20.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:20.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:20.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:20.940 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:20.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:20.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:21.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:21.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:21.081 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:21.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:21.182 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:21.183 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 86 16 96801 96785 70.3271 47.625 0.00458349 0.0141977 2024-03-11T22:28:21.212 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:21.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:21.314 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:21.340 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:21.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:21.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:21.443 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:21.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:21.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:21.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:21.572 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:21.599 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:21.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:21.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:21.701 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:21.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:21.739 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:21.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:21.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:21.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:21.866 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:21.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:21.968 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:21.993 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:21.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:22.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:21 smithi045 ceph-mon[30178]: pgmap v956: 158 pgs: 1 active+recovery_wait+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 34 MiB/s wr, 540 op/s; 3110/281232 objects degraded (1.106%); 6801/281232 objects misplaced (2.418%); 15 MiB/s, 247 objects/s recovering 2024-03-11T22:28:22.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:21 smithi045 ceph-mon[30178]: osdmap e467: 8 total, 8 up, 7 in 2024-03-11T22:28:22.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:21 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1858628501' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T22:28:22.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:21 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 3110/281232 objects degraded (1.106%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:28:22.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:21 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1858628501' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-11T22:28:22.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:21 smithi045 ceph-mon[30178]: osdmap e468: 8 total, 8 up, 7 in 2024-03-11T22:28:22.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:21 smithi045 ceph-mon[39715]: pgmap v956: 158 pgs: 1 active+recovery_wait+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 34 MiB/s wr, 540 op/s; 3110/281232 objects degraded (1.106%); 6801/281232 objects misplaced (2.418%); 15 MiB/s, 247 objects/s recovering 2024-03-11T22:28:22.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:21 smithi045 ceph-mon[39715]: osdmap e467: 8 total, 8 up, 7 in 2024-03-11T22:28:22.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:21 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1858628501' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T22:28:22.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:21 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 3110/281232 objects degraded (1.106%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:28:22.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:21 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1858628501' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-11T22:28:22.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:21 smithi045 ceph-mon[39715]: osdmap e468: 8 total, 8 up, 7 in 2024-03-11T22:28:22.094 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:22.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:21 smithi144 ceph-mon[33984]: pgmap v956: 158 pgs: 1 active+recovery_wait+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 34 MiB/s wr, 540 op/s; 3110/281232 objects degraded (1.106%); 6801/281232 objects misplaced (2.418%); 15 MiB/s, 247 objects/s recovering 2024-03-11T22:28:22.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:21 smithi144 ceph-mon[33984]: osdmap e467: 8 total, 8 up, 7 in 2024-03-11T22:28:22.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:21 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1858628501' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T22:28:22.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:21 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 3110/281232 objects degraded (1.106%), 1 pg degraded (PG_DEGRADED) 2024-03-11T22:28:22.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:21 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1858628501' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-11T22:28:22.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:21 smithi144 ceph-mon[33984]: osdmap e468: 8 total, 8 up, 7 in 2024-03-11T22:28:22.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:22.119 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:22.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:22.183 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 87 16 97002 96986 69.6632 12.5625 0.00991688 0.0142822 2024-03-11T22:28:22.219 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:22.244 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:22.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:22.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:22.372 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:22.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:22.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:22.473 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:22.498 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:22.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:22.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:22.600 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:22.637 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:22.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:22.738 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:22.806 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:22.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:22.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:22.835 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:28:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:28:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:28:22.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:22 smithi045 ceph-mon[39715]: 3.a scrub starts 2024-03-11T22:28:22.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:22 smithi045 ceph-mon[39715]: 3.a scrub ok 2024-03-11T22:28:22.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:22 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T22:28:22.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:22 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T22:28:22.835 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:22 smithi045 ceph-mon[39715]: osdmap e469: 8 total, 8 up, 7 in 2024-03-11T22:28:22.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:22 smithi045 ceph-mon[30178]: 3.a scrub starts 2024-03-11T22:28:22.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:22 smithi045 ceph-mon[30178]: 3.a scrub ok 2024-03-11T22:28:22.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:22 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T22:28:22.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:22 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T22:28:22.836 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:22 smithi045 ceph-mon[30178]: osdmap e469: 8 total, 8 up, 7 in 2024-03-11T22:28:22.907 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:22.933 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:22.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:23.034 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:23.057 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:23.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:23.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:23.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:22 smithi144 ceph-mon[33984]: 3.a scrub starts 2024-03-11T22:28:23.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:22 smithi144 ceph-mon[33984]: 3.a scrub ok 2024-03-11T22:28:23.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:22 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T22:28:23.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:22 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T22:28:23.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:22 smithi144 ceph-mon[33984]: osdmap e469: 8 total, 8 up, 7 in 2024-03-11T22:28:23.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:23.183 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 88 16 97014 96998 68.8801 0.75 0.0359123 0.0143741 2024-03-11T22:28:23.185 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:23.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:23.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:23.287 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:23.311 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:23.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:23.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:23.413 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:23.440 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:23.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:23.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:23.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:23.568 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:23.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:23.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:23.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:23.709 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:23.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:23.811 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:23.838 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:23.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:23.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:23.922 INFO:teuthology.orchestra.run.smithi045.stderr:set pool 11 pg_num to 26 2024-03-11T22:28:23.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:23.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:23.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:24.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:24.098 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:24.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:23 smithi144 ceph-mon[33984]: pgmap v959: 158 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 153 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 868 B/s rd, 52 MiB/s wr, 829 op/s; 8581/290595 objects misplaced (2.953%); 27 MiB/s, 435 objects/s recovering 2024-03-11T22:28:24.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:23 smithi144 ceph-mon[33984]: osdmap e470: 8 total, 8 up, 7 in 2024-03-11T22:28:24.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:24.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:24.183 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 89 16 97328 97312 68.3267 19.625 0.00305117 0.0145554 2024-03-11T22:28:24.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:24.230 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:24.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:24.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:24.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:23 smithi045 ceph-mon[39715]: pgmap v959: 158 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 153 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 868 B/s rd, 52 MiB/s wr, 829 op/s; 8581/290595 objects misplaced (2.953%); 27 MiB/s, 435 objects/s recovering 2024-03-11T22:28:24.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:23 smithi045 ceph-mon[39715]: osdmap e470: 8 total, 8 up, 7 in 2024-03-11T22:28:24.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:23 smithi045 ceph-mon[30178]: pgmap v959: 158 pgs: 1 active+recovering+remapped, 4 active+recovering+undersized+remapped, 153 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 868 B/s rd, 52 MiB/s wr, 829 op/s; 8581/290595 objects misplaced (2.953%); 27 MiB/s, 435 objects/s recovering 2024-03-11T22:28:24.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:23 smithi045 ceph-mon[30178]: osdmap e470: 8 total, 8 up, 7 in 2024-03-11T22:28:24.332 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:24.355 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:24.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:24.457 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:24.481 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:24.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:24.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:24.583 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:24.610 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:24.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:24.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:24.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:24.744 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:24.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:24.846 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:24.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:24.871 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:24.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:24.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:24.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:24.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:25.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:25.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T22:28:25.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:24 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-11T22:28:25.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:24 smithi144 ceph-mon[33984]: osdmap e471: 8 total, 8 up, 7 in 2024-03-11T22:28:25.127 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:25.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:25.183 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 90 16 97328 97312 67.5676 0 - 0.0145554 2024-03-11T22:28:25.229 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:25.256 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:25.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:25.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T22:28:25.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:24 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-11T22:28:25.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:24 smithi045 ceph-mon[39715]: osdmap e471: 8 total, 8 up, 7 in 2024-03-11T22:28:25.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T22:28:25.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:24 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-11T22:28:25.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:24 smithi045 ceph-mon[30178]: osdmap e471: 8 total, 8 up, 7 in 2024-03-11T22:28:25.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:25.386 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:25.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:25.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:25.516 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:25.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:25.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:25.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:25.648 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:25.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:25.750 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:25.775 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:25.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:25.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:25.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:25.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:25.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:26.007 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:26.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:26.032 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:26.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:26.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:25 smithi144 ceph-mon[33984]: 11.9 scrub starts 2024-03-11T22:28:26.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:25 smithi144 ceph-mon[33984]: pgmap v962: 168 pgs: 1 peering, 10 unknown, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 153 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 820 op/s; 8238/291063 objects misplaced (2.830%); 26 MiB/s, 418 objects/s recovering 2024-03-11T22:28:26.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:25 smithi144 ceph-mon[33984]: 1.3 deep-scrub starts 2024-03-11T22:28:26.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:25 smithi144 ceph-mon[33984]: 1.3 deep-scrub ok 2024-03-11T22:28:26.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:25 smithi144 ceph-mon[33984]: 11.9 scrub ok 2024-03-11T22:28:26.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:25 smithi144 ceph-mon[33984]: osdmap e472: 8 total, 8 up, 7 in 2024-03-11T22:28:26.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:26.160 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:26.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:26.183 INFO:tasks.radosbench.radosbench.0.smithi045.stdout: 91 16 97328 97312 66.8251 0 - 0.0145554 2024-03-11T22:28:26.262 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:26.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:25 smithi045 ceph-mon[39715]: 11.9 scrub starts 2024-03-11T22:28:26.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:25 smithi045 ceph-mon[39715]: pgmap v962: 168 pgs: 1 peering, 10 unknown, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 153 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 820 op/s; 8238/291063 objects misplaced (2.830%); 26 MiB/s, 418 objects/s recovering 2024-03-11T22:28:26.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:25 smithi045 ceph-mon[39715]: 1.3 deep-scrub starts 2024-03-11T22:28:26.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:25 smithi045 ceph-mon[39715]: 1.3 deep-scrub ok 2024-03-11T22:28:26.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:25 smithi045 ceph-mon[39715]: 11.9 scrub ok 2024-03-11T22:28:26.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:25 smithi045 ceph-mon[39715]: osdmap e472: 8 total, 8 up, 7 in 2024-03-11T22:28:26.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:25 smithi045 ceph-mon[30178]: 11.9 scrub starts 2024-03-11T22:28:26.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:25 smithi045 ceph-mon[30178]: pgmap v962: 168 pgs: 1 peering, 10 unknown, 1 active+recovering+remapped, 3 active+recovering+undersized+remapped, 153 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 820 op/s; 8238/291063 objects misplaced (2.830%); 26 MiB/s, 418 objects/s recovering 2024-03-11T22:28:26.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:25 smithi045 ceph-mon[30178]: 1.3 deep-scrub starts 2024-03-11T22:28:26.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:25 smithi045 ceph-mon[30178]: 1.3 deep-scrub ok 2024-03-11T22:28:26.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:25 smithi045 ceph-mon[30178]: 11.9 scrub ok 2024-03-11T22:28:26.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:25 smithi045 ceph-mon[30178]: osdmap e472: 8 total, 8 up, 7 in 2024-03-11T22:28:26.287 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:26.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:26.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:26.389 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:26.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:26.414 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:26.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:26.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:26.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:26.547 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:26.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:26.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:26.690 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:26.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:26.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:26.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:26.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:26.828 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:26.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:26.929 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:26.956 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:26.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:26.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:27.057 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:27.081 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:27.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:27.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:27.099 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:28:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:28:27.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:26 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:28:27.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:26 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:28:27.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:26 smithi144 ceph-mon[33984]: 11.8 scrub starts 2024-03-11T22:28:27.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:26 smithi144 ceph-mon[33984]: 11.1 scrub starts 2024-03-11T22:28:27.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:26 smithi144 ceph-mon[33984]: 11.8 scrub ok 2024-03-11T22:28:27.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:26 smithi144 ceph-mon[33984]: 11.1 scrub ok 2024-03-11T22:28:27.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:26 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-03-11T22:28:27.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:26 smithi144 ceph-mon[33984]: osdmap e473: 8 total, 8 up, 7 in 2024-03-11T22:28:27.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:26 smithi144 ceph-mon[33984]: 11.12 starting backfill to osd.7 from (0'0,0'0] MAX to 470'5949 2024-03-11T22:28:27.183 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:27.184 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Total time run: 91.5905 2024-03-11T22:28:27.184 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Total writes made: 97328 2024-03-11T22:28:27.184 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Write size: 65536 2024-03-11T22:28:27.184 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Object size: 65536 2024-03-11T22:28:27.184 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Bandwidth (MB/sec): 66.4152 2024-03-11T22:28:27.184 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev Bandwidth: 40.1963 2024-03-11T22:28:27.184 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max bandwidth (MB/sec): 192.125 2024-03-11T22:28:27.184 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min bandwidth (MB/sec): 0 2024-03-11T22:28:27.184 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Average IOPS: 1062 2024-03-11T22:28:27.184 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev IOPS: 643.149 2024-03-11T22:28:27.185 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max IOPS: 3074 2024-03-11T22:28:27.185 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min IOPS: 0 2024-03-11T22:28:27.185 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Average Latency(s): 0.0150507 2024-03-11T22:28:27.185 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Stddev Latency(s): 0.0560649 2024-03-11T22:28:27.185 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Max latency(s): 3.93171 2024-03-11T22:28:27.185 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Min latency(s): 0.001317 2024-03-11T22:28:27.191 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T22:28:27.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:27.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:27.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:27.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:28:27.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:28:27.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[30178]: 11.8 scrub starts 2024-03-11T22:28:27.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[30178]: 11.1 scrub starts 2024-03-11T22:28:27.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[30178]: 11.8 scrub ok 2024-03-11T22:28:27.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[30178]: 11.1 scrub ok 2024-03-11T22:28:27.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-03-11T22:28:27.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[30178]: osdmap e473: 8 total, 8 up, 7 in 2024-03-11T22:28:27.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[30178]: 11.12 starting backfill to osd.7 from (0'0,0'0] MAX to 470'5949 2024-03-11T22:28:27.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:28:27.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:28:27.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[39715]: 11.8 scrub starts 2024-03-11T22:28:27.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[39715]: 11.1 scrub starts 2024-03-11T22:28:27.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[39715]: 11.8 scrub ok 2024-03-11T22:28:27.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[39715]: 11.1 scrub ok 2024-03-11T22:28:27.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-03-11T22:28:27.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[39715]: osdmap e473: 8 total, 8 up, 7 in 2024-03-11T22:28:27.262 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:26 smithi045 ceph-mon[39715]: 11.12 starting backfill to osd.7 from (0'0,0'0] MAX to 470'5949 2024-03-11T22:28:27.275 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:28:27.322 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:27.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:27.348 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:27.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:27.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:27.501 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:27.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:27.603 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:27.630 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:27.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:27.731 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:27.770 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:27.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:27.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:27.872 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:27.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:27.896 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:27.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:27.997 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:28.031 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:28.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:28.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:28.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.0 scrub starts 2024-03-11T22:28:28.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: pgmap v965: 168 pgs: 5 remapped+peering, 1 active+clean+scrubbing, 1 peering, 5 unknown, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 5760/282903 objects misplaced (2.036%); 23 MiB/s, 363 objects/s recovering 2024-03-11T22:28:28.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 468'6132 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 470'6136 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6044 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.14 starting backfill to osd.4 from (0'0,0'0] MAX to 470'6086 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6060 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 470'5983 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 470'6124 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.17 starting backfill to osd.7 from (0'0,0'0] MAX to 468'6132 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.15 starting backfill to osd.4 from (0'0,0'0] MAX to 470'6044 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6128 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 470'6060 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.13 starting backfill to osd.4 from (0'0,0'0] MAX to 470'6136 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6124 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.16 starting backfill to osd.5 from (0'0,0'0] MAX to 470'5983 2024-03-11T22:28:28.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6019 2024-03-11T22:28:28.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.18 starting backfill to osd.4 from (0'0,0'0] MAX to 470'6019 2024-03-11T22:28:28.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 470'6128 2024-03-11T22:28:28.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 470'5983 2024-03-11T22:28:28.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.3 scrub starts 2024-03-11T22:28:28.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:28.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.0 scrub ok 2024-03-11T22:28:28.100 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[39715]: 11.3 scrub ok 2024-03-11T22:28:28.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.0 scrub starts 2024-03-11T22:28:28.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: pgmap v965: 168 pgs: 5 remapped+peering, 1 active+clean+scrubbing, 1 peering, 5 unknown, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 5760/282903 objects misplaced (2.036%); 23 MiB/s, 363 objects/s recovering 2024-03-11T22:28:28.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 468'6132 2024-03-11T22:28:28.100 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 470'6136 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6044 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.14 starting backfill to osd.4 from (0'0,0'0] MAX to 470'6086 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6060 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 470'5983 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 470'6124 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.17 starting backfill to osd.7 from (0'0,0'0] MAX to 468'6132 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.15 starting backfill to osd.4 from (0'0,0'0] MAX to 470'6044 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6128 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 470'6060 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.13 starting backfill to osd.4 from (0'0,0'0] MAX to 470'6136 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6124 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.16 starting backfill to osd.5 from (0'0,0'0] MAX to 470'5983 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6019 2024-03-11T22:28:28.101 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.18 starting backfill to osd.4 from (0'0,0'0] MAX to 470'6019 2024-03-11T22:28:28.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 470'6128 2024-03-11T22:28:28.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 470'5983 2024-03-11T22:28:28.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.3 scrub starts 2024-03-11T22:28:28.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:28.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.0 scrub ok 2024-03-11T22:28:28.102 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:27 smithi045 ceph-mon[30178]: 11.3 scrub ok 2024-03-11T22:28:28.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.0 scrub starts 2024-03-11T22:28:28.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: pgmap v965: 168 pgs: 5 remapped+peering, 1 active+clean+scrubbing, 1 peering, 5 unknown, 1 active+recovering+remapped, 2 active+recovering+undersized+remapped, 153 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 5760/282903 objects misplaced (2.036%); 23 MiB/s, 363 objects/s recovering 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.17 starting backfill to osd.6 from (0'0,0'0] MAX to 468'6132 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.13 starting backfill to osd.0 from (0'0,0'0] MAX to 470'6136 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.15 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6044 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.14 starting backfill to osd.4 from (0'0,0'0] MAX to 470'6086 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.19 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6060 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.16 starting backfill to osd.3 from (0'0,0'0] MAX to 470'5983 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.10 starting backfill to osd.0 from (0'0,0'0] MAX to 470'6124 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.17 starting backfill to osd.7 from (0'0,0'0] MAX to 468'6132 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.15 starting backfill to osd.4 from (0'0,0'0] MAX to 470'6044 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.11 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6128 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.19 starting backfill to osd.7 from (0'0,0'0] MAX to 470'6060 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.13 starting backfill to osd.4 from (0'0,0'0] MAX to 470'6136 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.10 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6124 2024-03-11T22:28:28.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.16 starting backfill to osd.5 from (0'0,0'0] MAX to 470'5983 2024-03-11T22:28:28.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.18 starting backfill to osd.2 from (0'0,0'0] MAX to 470'6019 2024-03-11T22:28:28.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.18 starting backfill to osd.4 from (0'0,0'0] MAX to 470'6019 2024-03-11T22:28:28.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.11 starting backfill to osd.7 from (0'0,0'0] MAX to 470'6128 2024-03-11T22:28:28.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.16 starting backfill to osd.7 from (0'0,0'0] MAX to 470'5983 2024-03-11T22:28:28.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.3 scrub starts 2024-03-11T22:28:28.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:28.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.0 scrub ok 2024-03-11T22:28:28.104 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:27 smithi144 ceph-mon[33984]: 11.3 scrub ok 2024-03-11T22:28:28.133 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:28.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:28.168 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:28.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:28.269 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:28.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:28.386 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:28:28.386 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:28:28.386 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:28:28.386 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:28:28.386 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:28:28.386 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:28:28.386 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:28:28.387 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:28:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:28:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:28:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:28:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:28:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:28:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:28:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:28:28.387 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:28:28.416 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:28.448 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:28.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:28.550 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:28.590 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:28.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:28.692 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:28.734 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:28.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:28.836 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:28.871 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:28.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:28.973 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:29.012 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:29.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:28 smithi144 ceph-mon[33984]: 11.4 scrub starts 2024-03-11T22:28:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:28 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/265950426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:28:29.114 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:29.147 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:29.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:29.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:29.249 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:29.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:28 smithi045 ceph-mon[39715]: 11.4 scrub starts 2024-03-11T22:28:29.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:28 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/265950426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:28:29.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:28 smithi045 ceph-mon[30178]: 11.4 scrub starts 2024-03-11T22:28:29.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:28 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/265950426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:28:29.287 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:29.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:29.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:29.389 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:29.425 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:29.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:29.527 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:29.551 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:29.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:29.652 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:29.680 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:29.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:29.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:29.782 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:29.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:29.818 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:29.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:29.919 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:29.945 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:29.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:29.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:30.046 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:30.080 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:30.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:30.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:29 smithi144 ceph-mon[33984]: 11.4 scrub ok 2024-03-11T22:28:30.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:29 smithi144 ceph-mon[33984]: pgmap v967: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+recovering+remapped, 5 remapped+peering, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 154 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 38057/292158 objects misplaced (13.026%); 2.9 MiB/s, 47 objects/s recovering 2024-03-11T22:28:30.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:29 smithi144 ceph-mon[33984]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:28:30.141 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:29 smithi045 ceph-mon[39715]: 11.4 scrub ok 2024-03-11T22:28:30.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:29 smithi045 ceph-mon[39715]: pgmap v967: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+recovering+remapped, 5 remapped+peering, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 154 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 38057/292158 objects misplaced (13.026%); 2.9 MiB/s, 47 objects/s recovering 2024-03-11T22:28:30.142 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:29 smithi045 ceph-mon[39715]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:28:30.142 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:29 smithi045 ceph-mon[30178]: 11.4 scrub ok 2024-03-11T22:28:30.142 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:29 smithi045 ceph-mon[30178]: pgmap v967: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+recovering+remapped, 5 remapped+peering, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 154 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 38057/292158 objects misplaced (13.026%); 2.9 MiB/s, 47 objects/s recovering 2024-03-11T22:28:30.142 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:29 smithi045 ceph-mon[30178]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:28:30.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:30.220 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:30.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:30.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:30.322 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:30.353 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:30.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:30.455 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:30.488 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:30.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:30.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:30.589 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:30.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:30.635 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:30.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:30.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:30.778 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:30.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:30.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:30.880 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:30.919 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:30.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:30.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:31.022 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:31.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:31.046 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:31.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:31.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:31.189 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:31.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:31.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:31.290 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:31.329 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:31.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:31.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:31.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:31.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:31.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:31.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:31.608 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:31.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:31.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:31.710 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:31.739 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:31.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:31.840 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:31.866 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:31.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:31.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:31.967 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:32.000 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:32.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:32.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:31 smithi144 ceph-mon[33984]: pgmap v968: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+recovering+remapped, 5 remapped+peering, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 154 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 38057/292158 objects misplaced (13.026%); 2.4 MiB/s, 38 objects/s recovering 2024-03-11T22:28:32.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:28:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:31 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-11T22:28:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:31 smithi144 ceph-mon[33984]: osdmap e474: 8 total, 8 up, 7 in 2024-03-11T22:28:32.102 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:32.131 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:32.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:32.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:32.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:32.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:32.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:32.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:31 smithi045 ceph-mon[30178]: pgmap v968: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+recovering+remapped, 5 remapped+peering, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 154 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 38057/292158 objects misplaced (13.026%); 2.4 MiB/s, 38 objects/s recovering 2024-03-11T22:28:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:28:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:31 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-11T22:28:32.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:31 smithi045 ceph-mon[30178]: osdmap e474: 8 total, 8 up, 7 in 2024-03-11T22:28:32.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:31 smithi045 ceph-mon[39715]: pgmap v968: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 1 active+recovering+remapped, 5 remapped+peering, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 154 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 38057/292158 objects misplaced (13.026%); 2.4 MiB/s, 38 objects/s recovering 2024-03-11T22:28:32.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:28:32.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:32.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:31 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-11T22:28:32.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:31 smithi045 ceph-mon[39715]: osdmap e474: 8 total, 8 up, 7 in 2024-03-11T22:28:32.373 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:32.401 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:32.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:32.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:32.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:32.541 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:32.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:32.643 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:32.673 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:32.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:32.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:32.775 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:32.808 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:32.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:32.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:32.950 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:32.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:32.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:32.994 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:28:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:28:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:28:33.053 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:33.085 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:33.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:33.187 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:33.226 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:33.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:33.328 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:33.364 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:33.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:33.465 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:33.498 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:33.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:33.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:33.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:33.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:33.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:33.793 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:33.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:33.894 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:33.926 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:33.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:34.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:34.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:33 smithi045 ceph-mon[39715]: pgmap v970: 168 pgs: 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1365/280545 objects degraded (0.487%); 58594/280545 objects misplaced (20.886%) 2024-03-11T22:28:34.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:33 smithi045 ceph-mon[30178]: pgmap v970: 168 pgs: 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1365/280545 objects degraded (0.487%); 58594/280545 objects misplaced (20.886%) 2024-03-11T22:28:34.063 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:34.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:34.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:34.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:33 smithi144 ceph-mon[33984]: pgmap v970: 168 pgs: 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1365/280545 objects degraded (0.487%); 58594/280545 objects misplaced (20.886%) 2024-03-11T22:28:34.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:34.210 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:34.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:34.312 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:34.348 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:34.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:34.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:34.450 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:34.481 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:34.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:34.582 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:34.605 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:34.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:34.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:34.707 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:34.742 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:34.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:34.844 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:34.879 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:34.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:34.981 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:35.015 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:35.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:35.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:34 smithi144 ceph-mon[33984]: 3.10 scrub starts 2024-03-11T22:28:35.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:34 smithi144 ceph-mon[33984]: 3.10 scrub ok 2024-03-11T22:28:35.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:34 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:28:35.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:34 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:28:35.117 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:35.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:34 smithi045 ceph-mon[30178]: 3.10 scrub starts 2024-03-11T22:28:35.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:34 smithi045 ceph-mon[30178]: 3.10 scrub ok 2024-03-11T22:28:35.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:34 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:28:35.122 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:34 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:28:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:34 smithi045 ceph-mon[39715]: 3.10 scrub starts 2024-03-11T22:28:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:34 smithi045 ceph-mon[39715]: 3.10 scrub ok 2024-03-11T22:28:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:34 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:28:35.122 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:34 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:28:35.149 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:35.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:35.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:35.251 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:35.277 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:35.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:35.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:35.379 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:35.416 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:35.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:35.504 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:28:35.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:35.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:35.590 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:35.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:35.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:35.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:35.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:35.835 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:35.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:35 smithi045 ceph-mon[30178]: pgmap v971: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 1365/278481 objects degraded (0.490%); 56259/278481 objects misplaced (20.202%) 2024-03-11T22:28:35.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:35 smithi045 ceph-mon[30178]: 6.3 scrub starts 2024-03-11T22:28:35.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:35 smithi045 ceph-mon[30178]: 6.3 scrub ok 2024-03-11T22:28:35.850 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:35 smithi045 ceph-mon[30178]: osdmap e475: 8 total, 8 up, 7 in 2024-03-11T22:28:35.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:35.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:35.976 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:36.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:36.011 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:36.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:36.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:35 smithi144 ceph-mon[33984]: pgmap v971: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 1365/278481 objects degraded (0.490%); 56259/278481 objects misplaced (20.202%) 2024-03-11T22:28:36.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:35 smithi144 ceph-mon[33984]: 6.3 scrub starts 2024-03-11T22:28:36.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:35 smithi144 ceph-mon[33984]: 6.3 scrub ok 2024-03-11T22:28:36.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:35 smithi144 ceph-mon[33984]: osdmap e475: 8 total, 8 up, 7 in 2024-03-11T22:28:36.112 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:36.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:36.154 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:36.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:36.255 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:35 smithi045 ceph-mon[39715]: pgmap v971: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 1365/278481 objects degraded (0.490%); 56259/278481 objects misplaced (20.202%) 2024-03-11T22:28:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:35 smithi045 ceph-mon[39715]: 6.3 scrub starts 2024-03-11T22:28:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:35 smithi045 ceph-mon[39715]: 6.3 scrub ok 2024-03-11T22:28:36.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:35 smithi045 ceph-mon[39715]: osdmap e475: 8 total, 8 up, 7 in 2024-03-11T22:28:36.291 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:36.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:36.393 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:36.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:36.437 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:36.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:36.537 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:36.566 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:36.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:36.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:36.668 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:36.723 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:36.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:36.825 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:36.858 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:36.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:36.959 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:36.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:36.999 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:36.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:37.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:36 smithi144 ceph-mon[33984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:28:37.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:36 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 1365/278481 objects degraded (0.490%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:28:37.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:36 smithi144 ceph-mon[33984]: osdmap e476: 8 total, 8 up, 7 in 2024-03-11T22:28:37.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:28:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:28:37.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:37.166 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:37.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:37.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:37.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:36 smithi045 ceph-mon[30178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:28:37.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:36 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 1365/278481 objects degraded (0.490%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:28:37.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:36 smithi045 ceph-mon[30178]: osdmap e476: 8 total, 8 up, 7 in 2024-03-11T22:28:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:36 smithi045 ceph-mon[39715]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:28:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:36 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 1365/278481 objects degraded (0.490%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:28:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:36 smithi045 ceph-mon[39715]: osdmap e476: 8 total, 8 up, 7 in 2024-03-11T22:28:37.317 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:37.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:37.419 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:37.449 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:37.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:37.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:37.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:37.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:37.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:37.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:37.696 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:37.730 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:37.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:37.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:37.832 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:37.836 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:28:37.836 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:28:37.870 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:37.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:37.972 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:38.008 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:38.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:38.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:37 smithi045 ceph-mon[30178]: pgmap v973: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 767 B/s rd, 0 op/s; 1233/269424 objects degraded (0.458%); 54464/269424 objects misplaced (20.215%); 4.5 MiB/s, 72 objects/s recovering 2024-03-11T22:28:38.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:38.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:37 smithi045 ceph-mon[39715]: pgmap v973: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 767 B/s rd, 0 op/s; 1233/269424 objects degraded (0.458%); 54464/269424 objects misplaced (20.215%); 4.5 MiB/s, 72 objects/s recovering 2024-03-11T22:28:38.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:38.110 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:38.154 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:38.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:38.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:38.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:38.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:38.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:37 smithi144 ceph-mon[33984]: pgmap v973: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 767 B/s rd, 0 op/s; 1233/269424 objects degraded (0.458%); 54464/269424 objects misplaced (20.215%); 4.5 MiB/s, 72 objects/s recovering 2024-03-11T22:28:38.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:38.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:38.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:38.441 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:38.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:38.508 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:28:38.542 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:38.579 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:38.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:38.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:38.680 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:38.711 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:38.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:38.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:38.813 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:38.857 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:38.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:38.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:38.959 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:39.010 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T22:28:39.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:39.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:39.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:39.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:39.151 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:38 smithi045 ceph-mon[30178]: 5.2 scrub starts 2024-03-11T22:28:39.151 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:38 smithi045 ceph-mon[30178]: 5.2 scrub ok 2024-03-11T22:28:39.151 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:38 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:28:39.151 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:38 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:28:39.151 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:38 smithi045 ceph-mon[30178]: osdmap e477: 8 total, 8 up, 7 in 2024-03-11T22:28:39.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:38 smithi045 ceph-mon[39715]: 5.2 scrub starts 2024-03-11T22:28:39.151 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:38 smithi045 ceph-mon[39715]: 5.2 scrub ok 2024-03-11T22:28:39.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:38 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:28:39.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:38 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:28:39.152 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:38 smithi045 ceph-mon[39715]: osdmap e477: 8 total, 8 up, 7 in 2024-03-11T22:28:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:39.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:39.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:39.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:39.312 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:39.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:39.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:39.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:38 smithi144 ceph-mon[33984]: 5.2 scrub starts 2024-03-11T22:28:39.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:38 smithi144 ceph-mon[33984]: 5.2 scrub ok 2024-03-11T22:28:39.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:38 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:28:39.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:38 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:28:39.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:38 smithi144 ceph-mon[33984]: osdmap e477: 8 total, 8 up, 7 in 2024-03-11T22:28:39.415 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:39.470 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:39.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:39.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:39.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:39.613 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:39.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:39.714 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:39.751 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:39.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:39.853 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:39.890 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:39.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:39.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:39.992 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:40.005 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:39 smithi045 ceph-mon[39715]: pgmap v976: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s; 1233/261429 objects degraded (0.472%); 51178/261429 objects misplaced (19.576%); 21 MiB/s, 334 objects/s recovering 2024-03-11T22:28:40.005 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:39 smithi045 ceph-mon[30178]: pgmap v976: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s; 1233/261429 objects degraded (0.472%); 51178/261429 objects misplaced (19.576%); 21 MiB/s, 334 objects/s recovering 2024-03-11T22:28:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:40.034 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:40.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:40.135 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:40.169 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:28:40.169 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:28:40.169 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-03-11T22:28:40.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd in 1 2024-03-11T22:28:40.175 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:40.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:40.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:40.278 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:40.307 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:40.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:40.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:40.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:39 smithi144 ceph-mon[33984]: pgmap v976: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s; 1233/261429 objects degraded (0.472%); 51178/261429 objects misplaced (19.576%); 21 MiB/s, 334 objects/s recovering 2024-03-11T22:28:40.409 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:40.446 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:40.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:40.548 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:40.582 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:40.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:40.684 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:40.717 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:40.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:40.819 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:40.848 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:40.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:40.950 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:40.980 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:40.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:40.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:41.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:40 smithi045 ceph-mon[39715]: osdmap e478: 8 total, 8 up, 7 in 2024-03-11T22:28:41.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:40 smithi045 ceph-mon[30178]: osdmap e478: 8 total, 8 up, 7 in 2024-03-11T22:28:41.081 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:41.121 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:41.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:41.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:41.265 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:41.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:41.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:41.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:40 smithi144 ceph-mon[33984]: osdmap e478: 8 total, 8 up, 7 in 2024-03-11T22:28:41.367 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:41.402 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:41.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:41.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:41.498 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:28:41.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:41.572 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:41.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:41.615 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:28:41.674 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:41.718 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:41.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:41.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:41.820 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:41.851 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:41.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:41.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:41.953 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:41.994 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:41.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:42.096 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:42.102 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:28:42.124 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:42.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:42.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:42.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[30178]: pgmap v978: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 907 B/s rd, 0 B/s wr, 1.01k op/s; 1233/261429 objects degraded (0.472%); 51178/261429 objects misplaced (19.576%); 15 MiB/s, 247 objects/s recovering 2024-03-11T22:28:42.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[30178]: osdmap e479: 8 total, 8 up, 7 in 2024-03-11T22:28:42.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3216273185' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:28:42.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:28:42.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 1233/261429 objects degraded (0.472%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:28:42.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:28:42.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:28:42.214 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[30178]: osdmap e480: 8 total, 8 up, 7 in 2024-03-11T22:28:42.214 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[39715]: pgmap v978: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 907 B/s rd, 0 B/s wr, 1.01k op/s; 1233/261429 objects degraded (0.472%); 51178/261429 objects misplaced (19.576%); 15 MiB/s, 247 objects/s recovering 2024-03-11T22:28:42.214 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[39715]: osdmap e479: 8 total, 8 up, 7 in 2024-03-11T22:28:42.214 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3216273185' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:28:42.214 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:28:42.214 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 1233/261429 objects degraded (0.472%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:28:42.214 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:28:42.215 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:28:42.215 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:41 smithi045 ceph-mon[39715]: osdmap e480: 8 total, 8 up, 7 in 2024-03-11T22:28:42.226 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:42.260 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:42.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:42.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:42.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:41 smithi144 ceph-mon[33984]: pgmap v978: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 155 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 907 B/s rd, 0 B/s wr, 1.01k op/s; 1233/261429 objects degraded (0.472%); 51178/261429 objects misplaced (19.576%); 15 MiB/s, 247 objects/s recovering 2024-03-11T22:28:42.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:41 smithi144 ceph-mon[33984]: osdmap e479: 8 total, 8 up, 7 in 2024-03-11T22:28:42.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:41 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3216273185' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:28:42.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:41 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:28:42.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:41 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 1233/261429 objects degraded (0.472%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-11T22:28:42.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:41 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:28:42.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:41 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:28:42.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:41 smithi144 ceph-mon[33984]: osdmap e480: 8 total, 8 up, 7 in 2024-03-11T22:28:42.363 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:42.398 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:42.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:42.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:42.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:42.538 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:42.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:42.639 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:42.673 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:42.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:42.775 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:42.802 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:42.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:42.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:42.878 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:28:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:28:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:28:42.904 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:42.938 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:42.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:43.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:43.090 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:43.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:43.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:43.193 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:43.227 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:43.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:43.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:43.329 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:43.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:43.368 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:43.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:43.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:43.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:43.508 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:43.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:43.609 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:43.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:43.653 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:43.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:43.754 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:43.789 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:43.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:43.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:43.892 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:43.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:43.926 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:43.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:44.027 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:44.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:44.066 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:44.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:44.170 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:44.172 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:43 smithi045 ceph-mon[39715]: pgmap v981: 168 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+remapped+backfilling, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 155 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 62 KiB/s rd, 0 B/s wr, 1.22k op/s; 46504/248088 objects misplaced (18.745%); 21 MiB/s, 342 objects/s recovering 2024-03-11T22:28:44.172 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:43 smithi045 ceph-mon[30178]: pgmap v981: 168 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+remapped+backfilling, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 155 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 62 KiB/s rd, 0 B/s wr, 1.22k op/s; 46504/248088 objects misplaced (18.745%); 21 MiB/s, 342 objects/s recovering 2024-03-11T22:28:44.200 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:44.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:44.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:44.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:43 smithi144 ceph-mon[33984]: pgmap v981: 168 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+remapped+backfilling, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 155 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 62 KiB/s rd, 0 B/s wr, 1.22k op/s; 46504/248088 objects misplaced (18.745%); 21 MiB/s, 342 objects/s recovering 2024-03-11T22:28:44.351 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:44.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:44.452 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:44.489 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:44.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:44.590 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:44.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:44.625 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:44.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:44.726 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:44.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:44.762 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:44.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:44.820 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T22:28:44.863 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:44.890 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:44.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:44.894 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:28:44.955 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:28:44.992 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:45.010 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:28:45.010 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:44 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2403364610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:28:45.011 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:44 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2403364610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:28:45.021 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:45.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:45.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:45.057 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:28:45.097 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:28:45.123 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:45.141 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:28:45.160 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:45.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:45.184 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:28:45.227 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:28:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:28:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:28:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:28:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:28:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:28:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:28:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:28:45.240 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:28:45.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:45.305 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:45.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:45.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:45.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:44 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2403364610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:28:45.407 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:45.443 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:45.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:45.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:45.588 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:45.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:45.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:45.691 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:45.726 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:45.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:45.827 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:45.863 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:45.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:45.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:46.004 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:46.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:46.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:45 smithi045 ceph-mon[30178]: pgmap v982: 168 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 155 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 59 KiB/s rd, 0 B/s wr, 955 op/s; 45237/245964 objects misplaced (18.392%); 11 MiB/s, 180 objects/s recovering 2024-03-11T22:28:46.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:45 smithi045 ceph-mon[39715]: pgmap v982: 168 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 155 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 59 KiB/s rd, 0 B/s wr, 955 op/s; 45237/245964 objects misplaced (18.392%); 11 MiB/s, 180 objects/s recovering 2024-03-11T22:28:46.106 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:46.143 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:46.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:46.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:46.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:46.286 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:46.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:46.349 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:45 smithi144 ceph-mon[33984]: pgmap v982: 168 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 155 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 59 KiB/s rd, 0 B/s wr, 955 op/s; 45237/245964 objects misplaced (18.392%); 11 MiB/s, 180 objects/s recovering 2024-03-11T22:28:46.387 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:46.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:46.427 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:46.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:46.528 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:46.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:46.563 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:46.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:46.664 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:46.701 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:46.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:46.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:46.803 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:46.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:46.843 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:46.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:46.944 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:46.979 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:46.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:47.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:47.099 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:28:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:28:47.118 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:47.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:47.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:47.220 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:47.254 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:47.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:47.355 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:47.392 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:47.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:47.494 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:47.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:47.518 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:47.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:47.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:47.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:47.663 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:47.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:47.764 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:47.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:47 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:28:47.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:47 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:47.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:47 smithi045 ceph-mon[30178]: pgmap v983: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 156 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 66 KiB/s rd, 0 B/s wr, 1.38k op/s; 43159/236253 objects misplaced (18.268%); 16 MiB/s, 255 objects/s recovering 2024-03-11T22:28:47.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:47 smithi045 ceph-mon[30178]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-11T22:28:47.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:47.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:47 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:28:47.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:47 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:47.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:47 smithi045 ceph-mon[39715]: pgmap v983: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 156 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 66 KiB/s rd, 0 B/s wr, 1.38k op/s; 43159/236253 objects misplaced (18.268%); 16 MiB/s, 255 objects/s recovering 2024-03-11T22:28:47.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:47 smithi045 ceph-mon[39715]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-11T22:28:47.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:47.802 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:47.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:47.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:47.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:47 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:28:47.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:47 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:28:47.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:47 smithi144 ceph-mon[33984]: pgmap v983: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 156 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 66 KiB/s rd, 0 B/s wr, 1.38k op/s; 43159/236253 objects misplaced (18.268%); 16 MiB/s, 255 objects/s recovering 2024-03-11T22:28:47.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:47 smithi144 ceph-mon[33984]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-11T22:28:47.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:47.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:47.941 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:47.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:47.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:48.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:48.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:48.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:48.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:48.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:48.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:48.209 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:48.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:48.310 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:48.348 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:48.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:48.449 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:48.487 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:48.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:48.589 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:48.629 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:48.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:48.731 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:48.782 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:48.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:48.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:48.884 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:48.922 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:48.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:48.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:49.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:49.048 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:49.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:49.150 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:49.187 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:49.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:49.288 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:49.331 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:49.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:49.432 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:49.472 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:49.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:49.574 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:49.613 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:49.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:49.715 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:49.751 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:49.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:49.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:49 smithi144 ceph-mon[33984]: pgmap v984: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 156 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.49k op/s; 41135/227838 objects misplaced (18.054%); 14 MiB/s, 227 objects/s recovering 2024-03-11T22:28:49.853 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:49.891 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:49.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:49.993 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:50.031 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:50.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:50.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:50.035 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:49 smithi045 ceph-mon[30178]: pgmap v984: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 156 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.49k op/s; 41135/227838 objects misplaced (18.054%); 14 MiB/s, 227 objects/s recovering 2024-03-11T22:28:50.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:49 smithi045 ceph-mon[39715]: pgmap v984: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 156 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.49k op/s; 41135/227838 objects misplaced (18.054%); 14 MiB/s, 227 objects/s recovering 2024-03-11T22:28:50.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:50.158 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:50.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:50.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:50.297 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:50.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:50.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:50.399 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:50.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:50.434 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:50.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:50.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:50.585 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:50.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:50.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:50.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:50.724 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:50.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:50.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:50.826 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:50.859 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:50.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:50.961 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:50.996 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:50.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:51.098 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:51.129 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:51.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:51.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:51.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:51.277 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:51.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:51.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:51.423 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:51.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:51.525 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:51.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:51.563 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:51.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:51.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:51.707 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:51.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:51.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:51.809 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:51.847 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:51.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:51.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:51.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:51 smithi144 ceph-mon[33984]: pgmap v985: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 156 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 43 KiB/s rd, 0 B/s wr, 1.16k op/s; 41135/227838 objects misplaced (18.054%); 8.5 MiB/s, 135 objects/s recovering 2024-03-11T22:28:51.949 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:51.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:51.982 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:51.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:52.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:51 smithi045 ceph-mon[39715]: pgmap v985: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 156 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 43 KiB/s rd, 0 B/s wr, 1.16k op/s; 41135/227838 objects misplaced (18.054%); 8.5 MiB/s, 135 objects/s recovering 2024-03-11T22:28:52.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:51 smithi045 ceph-mon[30178]: pgmap v985: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 156 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 43 KiB/s rd, 0 B/s wr, 1.16k op/s; 41135/227838 objects misplaced (18.054%); 8.5 MiB/s, 135 objects/s recovering 2024-03-11T22:28:52.084 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:52.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:52.116 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:52.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:52.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:52.255 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:52.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:52.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:52.325 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:28:52.325 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-03-11T22:28:52.325 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-03-11T22:28:52.325 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-03-11T22:28:52.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.325 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:27622 9.0 9:0d226f4d:::benchmark_data_smithi045_141553_object27621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-03-11T22:28:52.325 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:19:05.232839+0000", 2024-03-11T22:28:52.325 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 587.08395492800003, 2024-03-11T22:28:52.326 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.209376971, 2024-03-11T22:28:52.326 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.326 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.326 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.326 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:28:52.326 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:28:52.326 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27622 2024-03-11T22:28:52.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.327 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.327 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.327 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.327 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.232839+0000", 2024-03-11T22:28:52.327 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.328 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.328 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.232839+0000", 2024-03-11T22:28:52.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.232840+0000", 2024-03-11T22:28:52.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1699999999999998e-07 2024-03-11T22:28:52.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.232857+0000", 2024-03-11T22:28:52.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7618e-05 2024-03-11T22:28:52.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.331 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.232865+0000", 2024-03-11T22:28:52.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.926e-06 2024-03-11T22:28:52.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.232871+0000", 2024-03-11T22:28:52.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7880000000000003e-06 2024-03-11T22:28:52.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235294+0000", 2024-03-11T22:28:52.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024229109999999998 2024-03-11T22:28:52.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235441+0000", 2024-03-11T22:28:52.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000147331 2024-03-11T22:28:52.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:28:52.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235598+0000", 2024-03-11T22:28:52.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015707399999999999 2024-03-11T22:28:52.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237487+0000", 2024-03-11T22:28:52.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001888982 2024-03-11T22:28:52.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237718+0000", 2024-03-11T22:28:52.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023094200000000001 2024-03-11T22:28:52.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238514+0000", 2024-03-11T22:28:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00079607400000000002 2024-03-11T22:28:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238567+0000", 2024-03-11T22:28:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2692000000000003e-05 2024-03-11T22:28:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.442216+0000", 2024-03-11T22:28:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20364911599999999 2024-03-11T22:28:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.336 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:27629 9.e 9:7c277282:::benchmark_data_smithi045_141553_object27628:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:19:05.235674+0000", 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 587.08111981599995, 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20701077900000001, 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 27629 2024-03-11T22:28:52.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235674+0000", 2024-03-11T22:28:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235674+0000", 2024-03-11T22:28:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235676+0000", 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4640000000000001e-06 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235698+0000", 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2189e-05 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235711+0000", 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2806000000000001e-05 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235719+0000", 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4230000000000005e-06 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235781+0000", 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1409999999999996e-05 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.235956+0000", 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017530599999999999 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4", 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.236094+0000", 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013804999999999999 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237827+0000", 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001733355 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.442524+0000", 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20469699899999999 2024-03-11T22:28:52.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.442574+0000", 2024-03-11T22:28:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0099999999999998e-05 2024-03-11T22:28:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.442635+0000", 2024-03-11T22:28:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0624000000000001e-05 2024-03-11T22:28:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.442685+0000", 2024-03-11T22:28:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0052999999999997e-05 2024-03-11T22:28:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25142.0:27633 9.5 e282/273 9:aa73ac9a:::benchmark_data_smithi045_141553_object27632:head v 282'1754, mlcod=282'1751)", 2024-03-11T22:28:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:19:05.237622+0000", 2024-03-11T22:28:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 587.07917253999995, 2024-03-11T22:28:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.207793806, 2024-03-11T22:28:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.343 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.344 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.344 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.344 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237622+0000", 2024-03-11T22:28:52.344 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.345 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.345 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237622+0000", 2024-03-11T22:28:52.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.345 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237622+0000", 2024-03-11T22:28:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7899999999999997e-07 2024-03-11T22:28:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237644+0000", 2024-03-11T22:28:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1651e-05 2024-03-11T22:28:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.346 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237655+0000", 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0794e-05 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237662+0000", 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7939999999999997e-06 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237733+0000", 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.114e-05 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.237750+0000", 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7611000000000001e-05 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.445132+0000", 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.207381707 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.445415+0000", 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028332999999999999 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.348 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.349 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.349 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25142.0:27632 9.6 e282/273 9:630e0520:::benchmark_data_smithi045_141553_object27631:head v 282'1686, mlcod=282'1683)", 2024-03-11T22:28:52.349 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:19:05.238284+0000", 2024-03-11T22:28:52.349 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 587.07851061600002, 2024-03-11T22:28:52.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20882996100000001, 2024-03-11T22:28:52.349 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.350 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.350 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238284+0000", 2024-03-11T22:28:52.350 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238284+0000", 2024-03-11T22:28:52.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238284+0000", 2024-03-11T22:28:52.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5100000000000001e-07 2024-03-11T22:28:52.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238399+0000", 2024-03-11T22:28:52.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114759 2024-03-11T22:28:52.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238407+0000", 2024-03-11T22:28:52.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0830000000000004e-06 2024-03-11T22:28:52.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238413+0000", 2024-03-11T22:28:52.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3550000000000001e-06 2024-03-11T22:28:52.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.354 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.354 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238535+0000", 2024-03-11T22:28:52.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000121909 2024-03-11T22:28:52.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.355 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.355 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.355 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238553+0000", 2024-03-11T22:28:52.355 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8008000000000001e-05 2024-03-11T22:28:52.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.355 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.447080+0000", 2024-03-11T22:28:52.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.208527037 2024-03-11T22:28:52.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.447113+0000", 2024-03-11T22:28:52.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3458999999999999e-05 2024-03-11T22:28:52.357 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.357 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.357 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25142.0:27627 9.a e282/273 9:5a94e6de:::benchmark_data_smithi045_141553_object27626:head v 282'1723, mlcod=282'1719)", 2024-03-11T22:28:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:19:05.238431+0000", 2024-03-11T22:28:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 587.07836285300004, 2024-03-11T22:28:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20935579900000001, 2024-03-11T22:28:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238431+0000", 2024-03-11T22:28:52.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238431+0000", 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238432+0000", 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8599999999999999e-07 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.359 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238463+0000", 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1668999999999998e-05 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238471+0000", 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6340000000000001e-06 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.238495+0000", 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3975e-05 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.360 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.440891+0000", 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20239650200000001 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.440918+0000", 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6743000000000001e-05 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.447743+0000", 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068248400000000004 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:19:05.447787+0000", 2024-03-11T22:28:52.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.405e-05 2024-03-11T22:28:52.362 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.362 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.362 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.362 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:146759 9.0 9:0e3b4301:::benchmark_data_smithi045_141553_object22639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:28:52.362 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:46.168781+0000", 2024-03-11T22:28:52.362 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 486.14801311299999, 2024-03-11T22:28:52.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25229241000000002, 2024-03-11T22:28:52.362 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.362 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 146759 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.168781+0000", 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.168781+0000", 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.168783+0000", 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.654e-06 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.168789+0000", 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8510000000000004e-06 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.168803+0000", 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4766999999999999e-05 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.168812+0000", 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1090000000000004e-06 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.168884+0000", 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1663999999999997e-05 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.169017+0000", 2024-03-11T22:28:52.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000133387 2024-03-11T22:28:52.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:28:52.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.169141+0000", 2024-03-11T22:28:52.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012384399999999999 2024-03-11T22:28:52.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.174788+0000", 2024-03-11T22:28:52.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0056466700000000003 2024-03-11T22:28:52.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.367 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.175217+0000", 2024-03-11T22:28:52.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00042867799999999999 2024-03-11T22:28:52.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.421005+0000", 2024-03-11T22:28:52.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24578830099999999 2024-03-11T22:28:52.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.421045+0000", 2024-03-11T22:28:52.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9727000000000001e-05 2024-03-11T22:28:52.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.421073+0000", 2024-03-11T22:28:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8758000000000001e-05 2024-03-11T22:28:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:146778 9.0 9:0d2c086d:::benchmark_data_smithi045_141553_object22658:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-03-11T22:28:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:20:46.190282+0000", 2024-03-11T22:28:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 486.12651231500001, 2024-03-11T22:28:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23117253099999999, 2024-03-11T22:28:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.369 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:28:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:28:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 146778 2024-03-11T22:28:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.190282+0000", 2024-03-11T22:28:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.190282+0000", 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.190283+0000", 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1880000000000001e-06 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.190288+0000", 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.7509999999999999e-06 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.190298+0000", 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8369999999999992e-06 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.190306+0000", 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9840000000000001e-06 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.190372+0000", 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6283000000000002e-05 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.190504+0000", 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013227000000000001 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.190638+0000", 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000133707 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.193472+0000", 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028344849999999999 2024-03-11T22:28:52.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.193601+0000", 2024-03-11T22:28:52.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012863999999999999 2024-03-11T22:28:52.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.421353+0000", 2024-03-11T22:28:52.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.227751705 2024-03-11T22:28:52.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.421412+0000", 2024-03-11T22:28:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9750999999999998e-05 2024-03-11T22:28:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:20:46.421454+0000", 2024-03-11T22:28:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.193e-05 2024-03-11T22:28:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.375 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:183781 9.4 9:26751673:::benchmark_data_smithi045_141553_object59661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:24.331110+0000", 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 447.98568421900001, 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24024922700000001, 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 183781 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.331110+0000", 2024-03-11T22:28:52.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.331110+0000", 2024-03-11T22:28:52.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.331111+0000", 2024-03-11T22:28:52.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3600000000000003e-07 2024-03-11T22:28:52.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.378 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.331113+0000", 2024-03-11T22:28:52.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8940000000000002e-06 2024-03-11T22:28:52.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.378 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.378 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.331121+0000", 2024-03-11T22:28:52.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7880000000000007e-06 2024-03-11T22:28:52.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.378 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.331127+0000", 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7219999999999996e-06 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.331396+0000", 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00026953399999999998 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.332864+0000", 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001467463 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.332971+0000", 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000106717 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.367047+0000", 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.034076453 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.367097+0000", 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0087999999999999e-05 2024-03-11T22:28:52.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571245+0000", 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20414790599999999 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571306+0000", 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0857999999999999e-05 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571359+0000", 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3168000000000003e-05 2024-03-11T22:28:52.381 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.382 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.382 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.382 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:183785 9.4 9:2ce798a8:::benchmark_data_smithi045_141553_object59665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-03-11T22:28:52.382 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:24.331554+0000", 2024-03-11T22:28:52.382 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 447.98524003799997, 2024-03-11T22:28:52.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.24078059900000001, 2024-03-11T22:28:52.382 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.382 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.382 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.382 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:28:52.382 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 183785 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.331554+0000", 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.331554+0000", 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.331555+0000", 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0070000000000001e-06 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.331559+0000", 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.233e-06 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.331570+0000", 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0186e-05 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.331578+0000", 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4069999999999993e-06 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.338705+0000", 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0071272000000000002 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.338826+0000", 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000120949 2024-03-11T22:28:52.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.338959+0000", 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013273899999999999 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.367087+0000", 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.028128266999999998 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.372233+0000", 2024-03-11T22:28:52.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0051458759999999997 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571559+0000", 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19932562600000001 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.572290+0000", 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00073099600000000003 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.572335+0000", 2024-03-11T22:28:52.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5113e-05 2024-03-11T22:28:52.388 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.388 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.388 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.389 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25142.0:183788 9.5 e322/310 9:a1cd4e28:::benchmark_data_smithi045_141553_object59668:head v 322'11764, mlcod=322'11762)", 2024-03-11T22:28:52.389 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:24.338162+0000", 2024-03-11T22:28:52.390 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 447.978631957, 2024-03-11T22:28:52.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.23340393100000001, 2024-03-11T22:28:52.390 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.390 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.390 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.390 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.390 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.338162+0000", 2024-03-11T22:28:52.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.338162+0000", 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.338163+0000", 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7100000000000001e-07 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.338169+0000", 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7470000000000002e-06 2024-03-11T22:28:52.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.338179+0000", 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0475000000000001e-05 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.338186+0000", 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7139999999999996e-06 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.338981+0000", 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00079481599999999995 2024-03-11T22:28:52.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.366060+0000", 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.027079503000000001 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571240+0000", 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.205179322 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571566+0000", 2024-03-11T22:28:52.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00032658299999999997 2024-03-11T22:28:52.394 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.394 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.394 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.394 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:52.396 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:183792 9.4 9:21b0e205:::benchmark_data_smithi045_141553_object59672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-03-11T22:28:52.396 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:24.364705+0000", 2024-03-11T22:28:52.396 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 447.95208912800001, 2024-03-11T22:28:52.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.207858456, 2024-03-11T22:28:52.397 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.397 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.397 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.397 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:28:52.398 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:28:52.398 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 183792 2024-03-11T22:28:52.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.398 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.398 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.398 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.364705+0000", 2024-03-11T22:28:52.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.399 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.399 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.364705+0000", 2024-03-11T22:28:52.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.364706+0000", 2024-03-11T22:28:52.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.327e-06 2024-03-11T22:28:52.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.364710+0000", 2024-03-11T22:28:52.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3409999999999999e-06 2024-03-11T22:28:52.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.364718+0000", 2024-03-11T22:28:52.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8000000000000004e-06 2024-03-11T22:28:52.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.364725+0000", 2024-03-11T22:28:52.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.618e-06 2024-03-11T22:28:52.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.403 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.403 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.367225+0000", 2024-03-11T22:28:52.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002499944 2024-03-11T22:28:52.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.404 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.404 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.367346+0000", 2024-03-11T22:28:52.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012115 2024-03-11T22:28:52.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.404 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-03-11T22:28:52.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.367467+0000", 2024-03-11T22:28:52.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012040300000000001 2024-03-11T22:28:52.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571371+0000", 2024-03-11T22:28:52.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20390465199999999 2024-03-11T22:28:52.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.572391+0000", 2024-03-11T22:28:52.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001019903 2024-03-11T22:28:52.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.572470+0000", 2024-03-11T22:28:52.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8553999999999994e-05 2024-03-11T22:28:52.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.572523+0000", 2024-03-11T22:28:52.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3650999999999998e-05 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.572563+0000", 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0113e-05 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:183796 9.4 9:289059e7:::benchmark_data_smithi045_141553_object59676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:24.364759+0000", 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 447.95203492899998, 2024-03-11T22:28:52.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.207935552, 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 183796 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.364759+0000", 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.364759+0000", 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.364760+0000", 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0899999999999997e-07 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.364762+0000", 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1689999999999999e-06 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.364768+0000", 2024-03-11T22:28:52.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1500000000000004e-06 2024-03-11T22:28:52.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.364773+0000", 2024-03-11T22:28:52.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2809999999999999e-06 2024-03-11T22:28:52.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.369778+0000", 2024-03-11T22:28:52.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0050048820000000004 2024-03-11T22:28:52.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.369904+0000", 2024-03-11T22:28:52.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012626599999999999 2024-03-11T22:28:52.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 3,4", 2024-03-11T22:28:52.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.370021+0000", 2024-03-11T22:28:52.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000116482 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.572349+0000", 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20232861399999999 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.572431+0000", 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2021999999999995e-05 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.572605+0000", 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017384200000000001 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.572658+0000", 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3041999999999998e-05 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.572695+0000", 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6492999999999998e-05 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.413 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.414 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25142.0:183797 9.5 e322/310 9:ad3fd206:::benchmark_data_smithi045_141553_object59677:head v 322'11765, mlcod=322'11762)", 2024-03-11T22:28:52.414 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:24.369888+0000", 2024-03-11T22:28:52.414 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 447.94690564799998, 2024-03-11T22:28:52.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21192298600000001, 2024-03-11T22:28:52.414 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.414 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.414 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.369888+0000", 2024-03-11T22:28:52.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.369888+0000", 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.369889+0000", 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3399999999999999e-07 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.369896+0000", 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5270000000000004e-06 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.369906+0000", 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9399999999999997e-06 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.369913+0000", 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3690000000000001e-06 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.370369+0000", 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00045565999999999998 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.370391+0000", 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2702999999999999e-05 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.581756+0000", 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.211364947 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.581811+0000", 2024-03-11T22:28:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5206e-05 2024-03-11T22:28:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_repop(client.25142.0:183801 9.6 e322/273 9:6eb676d8:::benchmark_data_smithi045_141553_object59681:head v 322'11130, mlcod=322'11127)", 2024-03-11T22:28:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:24.372809+0000", 2024-03-11T22:28:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 447.943984769, 2024-03-11T22:28:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21106256800000001, 2024-03-11T22:28:52.418 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.372809+0000", 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.372809+0000", 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.372810+0000", 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2799999999999995e-07 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.372815+0000", 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1510000000000003e-06 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571244+0000", 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.198428569 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571269+0000", 2024-03-11T22:28:52.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4800999999999999e-05 2024-03-11T22:28:52.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571425+0000", 2024-03-11T22:28:52.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015614599999999999 2024-03-11T22:28:52.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571449+0000", 2024-03-11T22:28:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4379000000000002e-05 2024-03-11T22:28:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.583644+0000", 2024-03-11T22:28:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012194861 2024-03-11T22:28:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.583872+0000", 2024-03-11T22:28:52.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00022783300000000001 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:183806 9.d 9:bacb5766:::benchmark_data_smithi045_141553_object59686:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:24.372979+0000", 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 447.94381471899999, 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21342491599999999, 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:28:52.423 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 183806 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.372979+0000", 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.372979+0000", 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.372980+0000", 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0899999999999999e-06 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.372985+0000", 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8960000000000002e-06 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571251+0000", 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.198266005 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571268+0000", 2024-03-11T22:28:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6546999999999999e-05 2024-03-11T22:28:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571492+0000", 2024-03-11T22:28:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000223926 2024-03-11T22:28:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571839+0000", 2024-03-11T22:28:52.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00034725400000000002 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 4,5,7", 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.571983+0000", 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014382800000000001 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.578894+0000", 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0069106100000000002 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.578938+0000", 2024-03-11T22:28:52.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4350000000000001e-05 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.579053+0000", 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114974 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.586234+0000", 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0071815560000000004 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.586286+0000", 2024-03-11T22:28:52.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1198e-05 2024-03-11T22:28:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:24.586404+0000", 2024-03-11T22:28:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000118682 2024-03-11T22:28:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:220292 9.0 9:077ac908:::benchmark_data_smithi045_141553_object96172:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-11T22:28:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:59.691612+0000", 2024-03-11T22:28:52.429 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.62518216900003, 2024-03-11T22:28:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22885145200000001, 2024-03-11T22:28:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:28:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:28:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 220292 2024-03-11T22:28:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691612+0000", 2024-03-11T22:28:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691612+0000", 2024-03-11T22:28:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691619+0000", 2024-03-11T22:28:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4270000000000002e-06 2024-03-11T22:28:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691623+0000", 2024-03-11T22:28:52.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.602e-06 2024-03-11T22:28:52.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691633+0000", 2024-03-11T22:28:52.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0383999999999999e-05 2024-03-11T22:28:52.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691641+0000", 2024-03-11T22:28:52.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3730000000000004e-06 2024-03-11T22:28:52.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691696+0000", 2024-03-11T22:28:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5492999999999999e-05 2024-03-11T22:28:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691808+0000", 2024-03-11T22:28:52.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000111746 2024-03-11T22:28:52.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.435 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:28:52.435 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691935+0000", 2024-03-11T22:28:52.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012748999999999999 2024-03-11T22:28:52.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.435 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.699018+0000", 2024-03-11T22:28:52.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0070826250000000004 2024-03-11T22:28:52.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.436 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.699143+0000", 2024-03-11T22:28:52.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012491999999999999 2024-03-11T22:28:52.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.920335+0000", 2024-03-11T22:28:52.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22119229900000001 2024-03-11T22:28:52.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.920407+0000", 2024-03-11T22:28:52.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2183999999999999e-05 2024-03-11T22:28:52.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.920463+0000", 2024-03-11T22:28:52.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5909000000000003e-05 2024-03-11T22:28:52.438 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.438 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.438 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.439 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:220294 9.0 9:0ab89818:::benchmark_data_smithi045_141553_object96174:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-11T22:28:52.439 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:59.691667+0000", 2024-03-11T22:28:52.439 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.62512719699998, 2024-03-11T22:28:52.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22893835300000001, 2024-03-11T22:28:52.439 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.439 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.439 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.440 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:28:52.440 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:28:52.440 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 220294 2024-03-11T22:28:52.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.440 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.440 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691667+0000", 2024-03-11T22:28:52.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.441 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.441 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691667+0000", 2024-03-11T22:28:52.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.441 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691667+0000", 2024-03-11T22:28:52.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3900000000000005e-07 2024-03-11T22:28:52.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691670+0000", 2024-03-11T22:28:52.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9620000000000002e-06 2024-03-11T22:28:52.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691678+0000", 2024-03-11T22:28:52.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9640000000000003e-06 2024-03-11T22:28:52.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.691685+0000", 2024-03-11T22:28:52.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6209999999999998e-06 2024-03-11T22:28:52.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.696129+0000", 2024-03-11T22:28:52.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.004443621 2024-03-11T22:28:52.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.696277+0000", 2024-03-11T22:28:52.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014789 2024-03-11T22:28:52.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:28:52.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.696385+0000", 2024-03-11T22:28:52.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000108566 2024-03-11T22:28:52.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.446 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.446 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.699338+0000", 2024-03-11T22:28:52.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029529230000000001 2024-03-11T22:28:52.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.446 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.446 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.701131+0000", 2024-03-11T22:28:52.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001792674 2024-03-11T22:28:52.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.920514+0000", 2024-03-11T22:28:52.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21938300299999999 2024-03-11T22:28:52.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.920567+0000", 2024-03-11T22:28:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2895000000000001e-05 2024-03-11T22:28:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.920605+0000", 2024-03-11T22:28:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8695000000000001e-05 2024-03-11T22:28:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.448 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.25142.0:220322 9.0 9:0b10f4dc:::benchmark_data_smithi045_141553_object96202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:21:59.710637+0000", 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 412.60615725399998, 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21012619599999999, 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.25142", 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4289170454", 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 220322 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.449 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.710637+0000", 2024-03-11T22:28:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.710637+0000", 2024-03-11T22:28:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.710638+0000", 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0389999999999999e-06 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.710642+0000", 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.493e-06 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.710658+0000", 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.552e-05 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.710674+0000", 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5914999999999999e-05 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.710752+0000", 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8240999999999998e-05 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.710907+0000", 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015494099999999999 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2,6", 2024-03-11T22:28:52.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.711017+0000", 2024-03-11T22:28:52.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010950699999999999 2024-03-11T22:28:52.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.714533+0000", 2024-03-11T22:28:52.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035165740000000002 2024-03-11T22:28:52.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.720799+0000", 2024-03-11T22:28:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0062659539999999998 2024-03-11T22:28:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.920652+0000", 2024-03-11T22:28:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19985289000000001 2024-03-11T22:28:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.454 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.920726+0000", 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4352999999999998e-05 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:21:59.920763+0000", 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6769e-05 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15783.0:63617 11.e 11:76490e4b:::benchmark_data_smithi045_196393_object63616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:27:39.308377+0000", 2024-03-11T22:28:52.455 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 73.008417140000006, 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.22047223699999999, 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15783", 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4086262919", 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 63617 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.308377+0000", 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.308377+0000", 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.308379+0000", 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1840000000000002e-06 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.308404+0000", 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4765000000000001e-05 2024-03-11T22:28:52.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.308419+0000", 2024-03-11T22:28:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4644000000000001e-05 2024-03-11T22:28:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.308427+0000", 2024-03-11T22:28:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2120000000000003e-06 2024-03-11T22:28:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.314518+0000", 2024-03-11T22:28:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0060907160000000004 2024-03-11T22:28:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.314935+0000", 2024-03-11T22:28:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000417651 2024-03-11T22:28:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4,7", 2024-03-11T22:28:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.315455+0000", 2024-03-11T22:28:52.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000519514 2024-03-11T22:28:52.460 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.460 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.460 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.320599+0000", 2024-03-11T22:28:52.460 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0051441760000000003 2024-03-11T22:28:52.460 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.461 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.461 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.320626+0000", 2024-03-11T22:28:52.461 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6741999999999999e-05 2024-03-11T22:28:52.461 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.461 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.461 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.461 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.328269+0000", 2024-03-11T22:28:52.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0076436949999999998 2024-03-11T22:28:52.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.462 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.462 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.462 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.526349+0000", 2024-03-11T22:28:52.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.198079744 2024-03-11T22:28:52.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.463 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.463 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.526528+0000", 2024-03-11T22:28:52.463 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017933900000000001 2024-03-11T22:28:52.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.464 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.464 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.464 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.528849+0000", 2024-03-11T22:28:52.464 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023208550000000001 2024-03-11T22:28:52.464 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.464 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.465 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.465 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.465 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.15783.0:63622 11.e 11:72ea0d6f:::benchmark_data_smithi045_196393_object63621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-03-11T22:28:52.465 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-03-11T22:27:39.311679+0000", 2024-03-11T22:28:52.465 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 73.005114664999994, 2024-03-11T22:28:52.466 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.21921431699999999, 2024-03-11T22:28:52.466 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-03-11T22:28:52.466 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T22:28:52.466 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-03-11T22:28:52.466 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.15783", 2024-03-11T22:28:52.466 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.45:0/4086262919", 2024-03-11T22:28:52.467 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 63622 2024-03-11T22:28:52.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.467 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-03-11T22:28:52.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-03-11T22:28:52.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.311679+0000", 2024-03-11T22:28:52.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-03-11T22:28:52.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.311679+0000", 2024-03-11T22:28:52.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-03-11T22:28:52.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-03-11T22:28:52.470 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.311681+0000", 2024-03-11T22:28:52.470 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7460000000000001e-06 2024-03-11T22:28:52.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.471 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-03-11T22:28:52.471 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.311704+0000", 2024-03-11T22:28:52.471 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2365e-05 2024-03-11T22:28:52.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.471 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-03-11T22:28:52.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.311718+0000", 2024-03-11T22:28:52.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4884e-05 2024-03-11T22:28:52.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.472 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-03-11T22:28:52.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.311727+0000", 2024-03-11T22:28:52.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9949999999999994e-06 2024-03-11T22:28:52.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-03-11T22:28:52.473 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.317609+0000", 2024-03-11T22:28:52.473 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058820299999999999 2024-03-11T22:28:52.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.474 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-03-11T22:28:52.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.317846+0000", 2024-03-11T22:28:52.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023614400000000001 2024-03-11T22:28:52.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 0,4,7", 2024-03-11T22:28:52.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.317978+0000", 2024-03-11T22:28:52.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013187999999999999 2024-03-11T22:28:52.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-03-11T22:28:52.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.320769+0000", 2024-03-11T22:28:52.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027917609999999998 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.321274+0000", 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00050460200000000005 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.328321+0000", 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0070473410000000004 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-03-11T22:28:52.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.528914+0000", 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20059317600000001 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.528975+0000", 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0127000000000002e-05 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-03-11T22:27:39.530894+0000", 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001919266 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.477 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.478 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:28:52.478 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:28:52.478 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:28:52.506 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:52.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:52.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:52.608 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:52.635 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:52.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:52.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:52.779 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:52.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:52.785 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:28:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:28:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:28:52.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:52 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:28:52.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:52 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:28:52.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:52 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:28:52.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:52 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:28:52.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:52 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:28:52.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:52 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:28:52.881 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:52.912 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:52.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:52.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:53.015 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:53.057 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:53.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:53.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:53.159 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:53.193 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:53.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:53.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:53.332 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:53.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:53.434 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:53.465 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:53.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:53.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:53.567 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:53.600 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:53.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:53.703 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:53.740 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:53.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:53.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:53.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:53 smithi144 ceph-mon[33984]: pgmap v986: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 156 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 27 KiB/s rd, 0 B/s wr, 1.28k op/s; 38179/212853 objects misplaced (17.937%); 10 MiB/s, 167 objects/s recovering 2024-03-11T22:28:53.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:53 smithi144 ceph-mon[33984]: osdmap e481: 8 total, 8 up, 7 in 2024-03-11T22:28:53.874 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:53.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:53.976 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:54.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:54.014 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:54.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:54.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:53 smithi045 ceph-mon[39715]: pgmap v986: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 156 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 27 KiB/s rd, 0 B/s wr, 1.28k op/s; 38179/212853 objects misplaced (17.937%); 10 MiB/s, 167 objects/s recovering 2024-03-11T22:28:54.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:53 smithi045 ceph-mon[39715]: osdmap e481: 8 total, 8 up, 7 in 2024-03-11T22:28:54.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:53 smithi045 ceph-mon[30178]: pgmap v986: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 4 active+remapped+backfill_wait, 156 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 27 KiB/s rd, 0 B/s wr, 1.28k op/s; 38179/212853 objects misplaced (17.937%); 10 MiB/s, 167 objects/s recovering 2024-03-11T22:28:54.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:53 smithi045 ceph-mon[30178]: osdmap e481: 8 total, 8 up, 7 in 2024-03-11T22:28:54.115 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:54.149 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:54.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:54.250 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:54.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:54.287 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:54.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:54.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:54.430 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:54.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:54.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:54.532 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:54.594 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:54.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:54.695 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:54.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:54.725 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:54.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:54.826 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:54.852 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:54.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:54.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:54.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:54.996 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:54.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:54.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:55.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:28:55.098 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:55.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:55.130 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:55.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:55.231 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:55.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:55.267 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:55.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:55.368 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:55.407 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:55.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:55.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:55.509 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:55.544 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:55.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:55.645 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:55.681 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:55.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:55.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:55.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:55.829 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:55.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:55.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:55 smithi144 ceph-mon[33984]: pgmap v988: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 37102/210747 objects misplaced (17.605%); 9.7 MiB/s, 154 objects/s recovering 2024-03-11T22:28:55.931 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:55.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:55.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:55.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:56.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:55 smithi045 ceph-mon[39715]: pgmap v988: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 37102/210747 objects misplaced (17.605%); 9.7 MiB/s, 154 objects/s recovering 2024-03-11T22:28:56.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:55 smithi045 ceph-mon[30178]: pgmap v988: 168 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 37102/210747 objects misplaced (17.605%); 9.7 MiB/s, 154 objects/s recovering 2024-03-11T22:28:56.069 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:56.095 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:56.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:56.196 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:56.232 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:56.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:56.333 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:56.365 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:56.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:56.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:56.467 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:56.500 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:56.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:56.601 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:56.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:56.652 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:56.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:56.753 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:56.789 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:56.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:56.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:56.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:56.931 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:56.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:56.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:57.033 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:57.067 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:57.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:57.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:57.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:28:57.215 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:57.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:57.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:28:57.317 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:57.348 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:57.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:57.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:28:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:28:57.450 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:57.490 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:57.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:57.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:57.592 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:57.638 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:57.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:57.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:57.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:57.781 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:57.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:57.883 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:57.914 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:57 smithi045 ceph-mon[39715]: pgmap v989: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 38703/202134 objects misplaced (19.147%); 12 MiB/s, 184 objects/s recovering 2024-03-11T22:28:57.914 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:57 smithi045 ceph-mon[39715]: osdmap e482: 8 total, 8 up, 7 in 2024-03-11T22:28:57.915 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:57 smithi045 ceph-mon[30178]: pgmap v989: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 38703/202134 objects misplaced (19.147%); 12 MiB/s, 184 objects/s recovering 2024-03-11T22:28:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:57 smithi045 ceph-mon[30178]: osdmap e482: 8 total, 8 up, 7 in 2024-03-11T22:28:57.915 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:57.924 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:57.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:57.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:58.026 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:28:58.062 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:58.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:28:58.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:57 smithi144 ceph-mon[33984]: pgmap v989: 168 pgs: 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 1 active+recovering+remapped, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 38703/202134 objects misplaced (19.147%); 12 MiB/s, 184 objects/s recovering 2024-03-11T22:28:58.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:57 smithi144 ceph-mon[33984]: osdmap e482: 8 total, 8 up, 7 in 2024-03-11T22:28:58.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:28:58.164 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:58.198 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:58.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:58.299 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:58.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:58.337 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:58.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:58.438 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:58.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:58.475 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:58.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:58.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:58.606 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:58.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:58.707 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:58.744 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:58.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:58.846 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:58.879 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:58.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:58.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:28:58.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:28:59.019 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:59.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:59.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:28:59.121 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:28:59.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:59.159 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:59.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:28:59.261 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:59.296 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:59.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:59.397 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:59.439 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:59.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:59.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:59.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:28:59.579 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:59.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:59.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:28:59.681 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:28:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:59.710 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:59.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:28:59.811 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:28:59.846 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:28:59.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:59.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:28:59.948 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:28:59.988 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:28:59.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:28:59.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:00.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:28:59 smithi045 ceph-mon[39715]: pgmap v991: 168 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 35898/194568 objects misplaced (18.450%); 19 MiB/s, 296 objects/s recovering 2024-03-11T22:29:00.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:28:59 smithi045 ceph-mon[30178]: pgmap v991: 168 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 35898/194568 objects misplaced (18.450%); 19 MiB/s, 296 objects/s recovering 2024-03-11T22:29:00.090 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:00.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:28:59 smithi144 ceph-mon[33984]: pgmap v991: 168 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 35898/194568 objects misplaced (18.450%); 19 MiB/s, 296 objects/s recovering 2024-03-11T22:29:00.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:00.129 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:00.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:00.230 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:00.267 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:00.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:00.368 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:00.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:00.397 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:00.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:00.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:00.537 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:00.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:00.638 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:00.671 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:00.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:00.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:00.773 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:00.814 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:00.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:00.916 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:00.947 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:00.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:00.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:01.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:01.085 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:01.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:01.187 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:01.218 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:01.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:01.319 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:01.343 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:01.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:01.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:01.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:01.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:01.573 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:01.603 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:01.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:01.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:01.733 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:01.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:01.838 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:01.877 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:01.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:01.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:01 smithi045 ceph-mon[30178]: pgmap v992: 168 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 762 op/s; 35898/194568 objects misplaced (18.450%); 12 MiB/s, 196 objects/s recovering 2024-03-11T22:29:01.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:01 smithi045 ceph-mon[30178]: osdmap e483: 8 total, 8 up, 7 in 2024-03-11T22:29:01.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:01.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:01 smithi045 ceph-mon[30178]: osdmap e484: 8 total, 8 up, 7 in 2024-03-11T22:29:01.969 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:01 smithi045 ceph-mon[39715]: pgmap v992: 168 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 762 op/s; 35898/194568 objects misplaced (18.450%); 12 MiB/s, 196 objects/s recovering 2024-03-11T22:29:01.969 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:01 smithi045 ceph-mon[39715]: osdmap e483: 8 total, 8 up, 7 in 2024-03-11T22:29:01.969 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:01.969 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:01 smithi045 ceph-mon[39715]: osdmap e484: 8 total, 8 up, 7 in 2024-03-11T22:29:01.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:02.013 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:02.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:02.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:02.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:01 smithi144 ceph-mon[33984]: pgmap v992: 168 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 762 op/s; 35898/194568 objects misplaced (18.450%); 12 MiB/s, 196 objects/s recovering 2024-03-11T22:29:02.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:01 smithi144 ceph-mon[33984]: osdmap e483: 8 total, 8 up, 7 in 2024-03-11T22:29:02.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:02.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:01 smithi144 ceph-mon[33984]: osdmap e484: 8 total, 8 up, 7 in 2024-03-11T22:29:02.114 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:02.156 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:02.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:02.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:02.258 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:02.296 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:02.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:02.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:02.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:02.438 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:02.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:02.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:02.540 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:02.584 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:02.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:02.683 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:02 smithi045 ceph-mon[30178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:02.683 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:02 smithi045 ceph-mon[30178]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-11T22:29:02.684 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:29:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:29:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:29:02.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:02 smithi045 ceph-mon[39715]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:02.684 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:02 smithi045 ceph-mon[39715]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-11T22:29:02.686 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:02.697 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:29:02.715 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:02.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:02.741 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T22:29:02.765 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:29:02.817 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:02.914 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:02.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:03.015 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:03.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:03.061 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:03.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:03.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:02 smithi144 ceph-mon[33984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:03.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:02 smithi144 ceph-mon[33984]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-11T22:29:03.162 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:03.213 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:03.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:03.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:03.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:03.371 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:03.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:03.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:03.512 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:03 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a[30155]: 2024-03-11T22:29:03.243+0000 7f7c98d0b700 -1 mon.a@0(leader).osd e484 definitely_dead 0 2024-03-11T22:29:03.523 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:03.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:03.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:03.625 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:03.663 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:03.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:03.688 INFO:teuthology.orchestra.run.smithi045.stderr:marked in osd.1. 2024-03-11T22:29:03.764 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:03.769 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:03 smithi045 ceph-mon[39715]: pgmap v995: 168 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s; 31776/182196 objects misplaced (17.441%); 22 MiB/s, 346 objects/s recovering 2024-03-11T22:29:03.769 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:03 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2605901432' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-11T22:29:03.769 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:03 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-11T22:29:03.769 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:03 smithi045 ceph-mon[30178]: pgmap v995: 168 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s; 31776/182196 objects misplaced (17.441%); 22 MiB/s, 346 objects/s recovering 2024-03-11T22:29:03.770 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:03 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2605901432' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-11T22:29:03.770 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:03 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-11T22:29:03.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:03.795 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:03.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:03.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:03.929 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:03.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:03.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:04.031 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:04.061 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:04.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:04.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:04.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:03 smithi144 ceph-mon[33984]: pgmap v995: 168 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 156 active+clean; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s; 31776/182196 objects misplaced (17.441%); 22 MiB/s, 346 objects/s recovering 2024-03-11T22:29:04.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:03 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2605901432' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-11T22:29:04.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:03 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-03-11T22:29:04.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:04.197 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:04.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:04.299 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:04.329 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:04.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:04.431 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:04.458 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:04.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:04.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:04.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:04.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:04.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:04.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:04.695 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:04.729 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:04.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:04.831 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:04.863 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:04.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:04.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:04.965 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:04.996 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:04.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:05.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:04 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:29:05.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:04 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:29:05.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:04 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-03-11T22:29:05.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:04 smithi045 ceph-mon[39715]: osdmap e485: 8 total, 8 up, 8 in 2024-03-11T22:29:05.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:04 smithi045 ceph-mon[39715]: osdmap e486: 8 total, 8 up, 8 in 2024-03-11T22:29:05.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:04 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:29:05.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:04 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:29:05.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:04 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-03-11T22:29:05.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:04 smithi045 ceph-mon[30178]: osdmap e485: 8 total, 8 up, 8 in 2024-03-11T22:29:05.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:04 smithi045 ceph-mon[30178]: osdmap e486: 8 total, 8 up, 8 in 2024-03-11T22:29:05.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:05.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:04 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:29:05.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:04 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:29:05.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:04 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-03-11T22:29:05.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:04 smithi144 ceph-mon[33984]: osdmap e485: 8 total, 8 up, 8 in 2024-03-11T22:29:05.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:04 smithi144 ceph-mon[33984]: osdmap e486: 8 total, 8 up, 8 in 2024-03-11T22:29:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:05.123 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:05.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:05.224 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:05.255 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:05.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:05.357 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:05.389 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:05.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:05.491 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:05.516 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:05.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:05.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:05.618 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:05.649 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:05.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:05.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:05.751 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:05.776 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:05.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:05.877 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:05.913 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:05.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:05.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:05.961 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[30178]: pgmap v997: 168 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 156 active+clean; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 780 op/s; 28818/180534 objects misplaced (15.963%); 27 MiB/s, 424 objects/s recovering 2024-03-11T22:29:05.961 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[30178]: 11.14 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7291 2024-03-11T22:29:05.961 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[30178]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7280 2024-03-11T22:29:05.961 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[30178]: 11.14 continuing backfill to osd.4 from (442'1700,484'7291] MIN to 484'7291 2024-03-11T22:29:05.961 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[30178]: 11.15 continuing backfill to osd.2 from (440'1200,484'7280] 11:abd6885d:::benchmark_data_smithi045_196393_object79528:head to 484'7280 2024-03-11T22:29:05.961 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[30178]: 11.15 continuing backfill to osd.4 from (440'1200,484'7280] 11:abd6885d:::benchmark_data_smithi045_196393_object79528:head to 484'7280 2024-03-11T22:29:05.961 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[30178]: 1.3 starting backfill to osd.1 from (0'0,0'0] MAX to 93'130 2024-03-11T22:29:05.961 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[30178]: osdmap e487: 8 total, 8 up, 8 in 2024-03-11T22:29:05.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[39715]: pgmap v997: 168 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 156 active+clean; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 780 op/s; 28818/180534 objects misplaced (15.963%); 27 MiB/s, 424 objects/s recovering 2024-03-11T22:29:05.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[39715]: 11.14 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7291 2024-03-11T22:29:05.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[39715]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7280 2024-03-11T22:29:05.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[39715]: 11.14 continuing backfill to osd.4 from (442'1700,484'7291] MIN to 484'7291 2024-03-11T22:29:05.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[39715]: 11.15 continuing backfill to osd.2 from (440'1200,484'7280] 11:abd6885d:::benchmark_data_smithi045_196393_object79528:head to 484'7280 2024-03-11T22:29:05.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[39715]: 11.15 continuing backfill to osd.4 from (440'1200,484'7280] 11:abd6885d:::benchmark_data_smithi045_196393_object79528:head to 484'7280 2024-03-11T22:29:05.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[39715]: 1.3 starting backfill to osd.1 from (0'0,0'0] MAX to 93'130 2024-03-11T22:29:05.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:05 smithi045 ceph-mon[39715]: osdmap e487: 8 total, 8 up, 8 in 2024-03-11T22:29:06.001 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:05 smithi144 ceph-mon[33984]: pgmap v997: 168 pgs: 1 active+recovering+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 156 active+clean; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 780 op/s; 28818/180534 objects misplaced (15.963%); 27 MiB/s, 424 objects/s recovering 2024-03-11T22:29:06.001 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:05 smithi144 ceph-mon[33984]: 11.14 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7291 2024-03-11T22:29:06.001 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:05 smithi144 ceph-mon[33984]: 11.15 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7280 2024-03-11T22:29:06.001 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:05 smithi144 ceph-mon[33984]: 11.14 continuing backfill to osd.4 from (442'1700,484'7291] MIN to 484'7291 2024-03-11T22:29:06.001 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:05 smithi144 ceph-mon[33984]: 11.15 continuing backfill to osd.2 from (440'1200,484'7280] 11:abd6885d:::benchmark_data_smithi045_196393_object79528:head to 484'7280 2024-03-11T22:29:06.001 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:05 smithi144 ceph-mon[33984]: 11.15 continuing backfill to osd.4 from (440'1200,484'7280] 11:abd6885d:::benchmark_data_smithi045_196393_object79528:head to 484'7280 2024-03-11T22:29:06.001 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:05 smithi144 ceph-mon[33984]: 1.3 starting backfill to osd.1 from (0'0,0'0] MAX to 93'130 2024-03-11T22:29:06.001 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:05 smithi144 ceph-mon[33984]: osdmap e487: 8 total, 8 up, 8 in 2024-03-11T22:29:06.016 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:06.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:06.053 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:06.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:06.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:06.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:06.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:06.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:06.297 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:06.333 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:06.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:06.434 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:06.476 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:06.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:06.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:06.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:06.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:06.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:06.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:06.763 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:06.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:06.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:06.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[39715]: 11.0 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7328 2024-03-11T22:29:06.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[39715]: 11.b starting backfill to osd.1 from (0'0,0'0] MAX to 484'8461 2024-03-11T22:29:06.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[39715]: 11.c starting backfill to osd.1 from (0'0,0'0] MAX to 484'8563 2024-03-11T22:29:06.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[39715]: 11.3 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7347 2024-03-11T22:29:06.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[39715]: 11.7 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7402 2024-03-11T22:29:06.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[39715]: 11.6 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7178 2024-03-11T22:29:06.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[39715]: 11.f starting backfill to osd.1 from (0'0,0'0] MAX to 484'8509 2024-03-11T22:29:06.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[39715]: osdmap e488: 8 total, 8 up, 8 in 2024-03-11T22:29:06.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[30178]: 11.0 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7328 2024-03-11T22:29:06.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[30178]: 11.b starting backfill to osd.1 from (0'0,0'0] MAX to 484'8461 2024-03-11T22:29:06.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[30178]: 11.c starting backfill to osd.1 from (0'0,0'0] MAX to 484'8563 2024-03-11T22:29:06.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[30178]: 11.3 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7347 2024-03-11T22:29:06.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[30178]: 11.7 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7402 2024-03-11T22:29:06.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[30178]: 11.6 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7178 2024-03-11T22:29:06.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[30178]: 11.f starting backfill to osd.1 from (0'0,0'0] MAX to 484'8509 2024-03-11T22:29:06.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:06 smithi045 ceph-mon[30178]: osdmap e488: 8 total, 8 up, 8 in 2024-03-11T22:29:06.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:06.905 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:06.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:06.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:07.005 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-03-11T22:29:07.007 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:07.059 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:07.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:07.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:29:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:29:07.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:06 smithi144 ceph-mon[33984]: 11.0 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7328 2024-03-11T22:29:07.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:06 smithi144 ceph-mon[33984]: 11.b starting backfill to osd.1 from (0'0,0'0] MAX to 484'8461 2024-03-11T22:29:07.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:06 smithi144 ceph-mon[33984]: 11.c starting backfill to osd.1 from (0'0,0'0] MAX to 484'8563 2024-03-11T22:29:07.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:06 smithi144 ceph-mon[33984]: 11.3 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7347 2024-03-11T22:29:07.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:06 smithi144 ceph-mon[33984]: 11.7 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7402 2024-03-11T22:29:07.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:06 smithi144 ceph-mon[33984]: 11.6 starting backfill to osd.1 from (0'0,0'0] MAX to 484'7178 2024-03-11T22:29:07.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:06 smithi144 ceph-mon[33984]: 11.f starting backfill to osd.1 from (0'0,0'0] MAX to 484'8509 2024-03-11T22:29:07.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:06 smithi144 ceph-mon[33984]: osdmap e488: 8 total, 8 up, 8 in 2024-03-11T22:29:07.161 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:07.203 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:07.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:07.305 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:07.356 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:07.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:07.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:07.496 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:07.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:07.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:07.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:07.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:07.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:07.741 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:07.784 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:07.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:07.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:07.886 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:07.934 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:29:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:07.939 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:07.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:08.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:07 smithi045 ceph-mon[39715]: pgmap v1000: 168 pgs: 1 remapped+peering, 1 active+recovering+remapped, 1 peering, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 1 active+remapped, 153 active+clean; 3.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 4.7 KiB/s rd, 30 KiB/s wr, 656 op/s; 33068/177276 objects misplaced (18.653%); 60 MiB/s, 969 objects/s recovering 2024-03-11T22:29:08.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:07 smithi045 ceph-mon[39715]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:29:08.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:08.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:07 smithi045 ceph-mon[30178]: pgmap v1000: 168 pgs: 1 remapped+peering, 1 active+recovering+remapped, 1 peering, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 1 active+remapped, 153 active+clean; 3.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 4.7 KiB/s rd, 30 KiB/s wr, 656 op/s; 33068/177276 objects misplaced (18.653%); 60 MiB/s, 969 objects/s recovering 2024-03-11T22:29:08.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:07 smithi045 ceph-mon[30178]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:29:08.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:08.041 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:08.084 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:08.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:08.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:08.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:07 smithi144 ceph-mon[33984]: pgmap v1000: 168 pgs: 1 remapped+peering, 1 active+recovering+remapped, 1 peering, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 1 active+remapped, 153 active+clean; 3.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 4.7 KiB/s rd, 30 KiB/s wr, 656 op/s; 33068/177276 objects misplaced (18.653%); 60 MiB/s, 969 objects/s recovering 2024-03-11T22:29:08.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:07 smithi144 ceph-mon[33984]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-11T22:29:08.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:08.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:08.227 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:08.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:08.329 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:08.370 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:08.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:08.472 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:08.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:08.511 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:08.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:08.613 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:08.660 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:08.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:08.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:08.762 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:08.799 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:08.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:08.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:08.901 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:08.944 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:08.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:09.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:08 smithi045 ceph-mon[39715]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:09.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:08 smithi045 ceph-mon[30178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:09.045 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:09.079 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:09.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:09.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:09.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:08 smithi144 ceph-mon[33984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 9 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:09.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:09.224 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:09.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:09.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:09.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:09.365 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:09.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:09.467 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:09.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:09.510 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:09.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:09.611 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:09.645 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:09.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:09.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:09.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:09.796 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:09.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:09.862 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:09 smithi045 ceph-mon[39715]: pgmap v1002: 168 pgs: 1 remapped+peering, 4 active+remapped+backfilling, 9 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 1 active+remapped, 151 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 3.5 KiB/s rd, 30 KiB/s wr, 568 op/s; 39734/172371 objects misplaced (23.051%); 55 MiB/s, 908 objects/s recovering 2024-03-11T22:29:09.863 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:09 smithi045 ceph-mon[30178]: pgmap v1002: 168 pgs: 1 remapped+peering, 4 active+remapped+backfilling, 9 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 1 active+remapped, 151 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 3.5 KiB/s rd, 30 KiB/s wr, 568 op/s; 39734/172371 objects misplaced (23.051%); 55 MiB/s, 908 objects/s recovering 2024-03-11T22:29:09.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:09.939 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:09.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:10.040 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:10.080 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:10.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:10.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:09 smithi144 ceph-mon[33984]: pgmap v1002: 168 pgs: 1 remapped+peering, 4 active+remapped+backfilling, 9 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 1 active+remapped, 151 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 3.5 KiB/s rd, 30 KiB/s wr, 568 op/s; 39734/172371 objects misplaced (23.051%); 55 MiB/s, 908 objects/s recovering 2024-03-11T22:29:10.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:10.220 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:10.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:10.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:10.322 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:10.360 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:10.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:10.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:10.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:10.504 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:10.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:10.605 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:10.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:10.640 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:10.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:10.741 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:10.783 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:10.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:10.884 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:10.920 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:10.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:10.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:11.022 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:11.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:10 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4053693827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:29:11.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:10 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2338369098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:29:11.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:10 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4053693827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:29:11.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:10 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2338369098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:29:11.057 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:11.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:11.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:11.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:10 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4053693827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:29:11.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:10 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2338369098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:29:11.159 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:11.195 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:11.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:11.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:11.297 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:11.335 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:11.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:11.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:11.437 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:11.477 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:11.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:11.497 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:29:11.497 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:29:11.497 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:29:11.497 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:29:11.497 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:29:11.497 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:29:11.498 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:29:11.498 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:29:11.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:29:11.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:29:11.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:29:11.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:29:11.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:29:11.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:29:11.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:29:11.498 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:29:11.578 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:11.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:11.624 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:11.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:11.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:11.765 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:11.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:11.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:11.910 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:11.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:12.005 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:29:12.005 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:29:12.006 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-11T22:29:12.006 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2024-03-11T22:29:12.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-11T22:29:12.012 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:12.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:11 smithi045 ceph-mon[39715]: pgmap v1003: 168 pgs: 4 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 1 active+remapped, 151 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2.8 KiB/s rd, 24 KiB/s wr, 457 op/s; 39734/172371 objects misplaced (23.051%); 44 MiB/s, 729 objects/s recovering 2024-03-11T22:29:12.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:11 smithi045 ceph-mon[39715]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:29:12.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:11 smithi045 ceph-mon[39715]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-11T22:29:12.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:11 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4053693827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:29:12.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:11 smithi045 ceph-mon[39715]: osdmap e489: 8 total, 8 up, 8 in 2024-03-11T22:29:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:11 smithi045 ceph-mon[39715]: osdmap e490: 8 total, 8 up, 8 in 2024-03-11T22:29:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:11 smithi045 ceph-mon[30178]: pgmap v1003: 168 pgs: 4 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 1 active+remapped, 151 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2.8 KiB/s rd, 24 KiB/s wr, 457 op/s; 39734/172371 objects misplaced (23.051%); 44 MiB/s, 729 objects/s recovering 2024-03-11T22:29:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:11 smithi045 ceph-mon[30178]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:29:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:11 smithi045 ceph-mon[30178]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-11T22:29:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:11 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4053693827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:29:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:11 smithi045 ceph-mon[30178]: osdmap e489: 8 total, 8 up, 8 in 2024-03-11T22:29:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:11 smithi045 ceph-mon[30178]: osdmap e490: 8 total, 8 up, 8 in 2024-03-11T22:29:12.079 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:12.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:12.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:12.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:11 smithi144 ceph-mon[33984]: pgmap v1003: 168 pgs: 4 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 1 active+remapped, 151 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 2.8 KiB/s rd, 24 KiB/s wr, 457 op/s; 39734/172371 objects misplaced (23.051%); 44 MiB/s, 729 objects/s recovering 2024-03-11T22:29:12.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:11 smithi144 ceph-mon[33984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:29:12.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:11 smithi144 ceph-mon[33984]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-11T22:29:12.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:11 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4053693827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T22:29:12.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:11 smithi144 ceph-mon[33984]: osdmap e489: 8 total, 8 up, 8 in 2024-03-11T22:29:12.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:11 smithi144 ceph-mon[33984]: osdmap e490: 8 total, 8 up, 8 in 2024-03-11T22:29:12.181 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:12.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:12.217 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:12.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:12.318 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:12.356 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:12.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:12.458 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:12.488 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:12.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:12.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:12.590 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:12.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:12.626 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:12.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:12.727 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:12.763 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:12.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:12.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:12.903 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:12.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:13.006 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:13.036 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:29:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:29:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:29:13.040 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:13.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:13.143 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:13.179 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:13.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:13.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:13.282 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:13.312 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:13.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:13.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:13.414 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:13.442 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:13.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:13.544 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:13.571 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:13.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:13.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:13.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:13.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:13.711 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:13.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:13.812 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:13.845 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:13.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:13.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:13.947 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:13.988 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:13.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:13.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:14.090 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:14.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:13 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:29:14.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:13 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:29:14.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:13 smithi144 ceph-mon[33984]: pgmap v1006: 168 pgs: 4 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 151 active+clean; 3.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 1.06k op/s; 33290/158142 objects misplaced (21.051%); 34 MiB/s, 555 objects/s recovering 2024-03-11T22:29:14.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:13 smithi144 ceph-mon[33984]: osdmap e491: 8 total, 8 up, 8 in 2024-03-11T22:29:14.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:13 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:29:14.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:13 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:29:14.119 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:14.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:14.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:14.263 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:14.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:14.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:13 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:29:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:13 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:29:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:13 smithi045 ceph-mon[39715]: pgmap v1006: 168 pgs: 4 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 151 active+clean; 3.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 1.06k op/s; 33290/158142 objects misplaced (21.051%); 34 MiB/s, 555 objects/s recovering 2024-03-11T22:29:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:13 smithi045 ceph-mon[39715]: osdmap e491: 8 total, 8 up, 8 in 2024-03-11T22:29:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:13 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:29:14.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:13 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:29:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:13 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:29:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:13 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:29:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:13 smithi045 ceph-mon[30178]: pgmap v1006: 168 pgs: 4 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 151 active+clean; 3.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 1.06k op/s; 33290/158142 objects misplaced (21.051%); 34 MiB/s, 555 objects/s recovering 2024-03-11T22:29:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:13 smithi045 ceph-mon[30178]: osdmap e491: 8 total, 8 up, 8 in 2024-03-11T22:29:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:13 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:29:14.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:13 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:29:14.365 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:14.396 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:14.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:14.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:14.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:14.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:14.539 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:14.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:14.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:14.678 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:14.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:14.780 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:14.827 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:14.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:14.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:14.973 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:14.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:14.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:15.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:14 smithi045 ceph-mon[39715]: osdmap e492: 8 total, 8 up, 8 in 2024-03-11T22:29:15.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:14 smithi045 ceph-mon[30178]: osdmap e492: 8 total, 8 up, 8 in 2024-03-11T22:29:15.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:15.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:14 smithi144 ceph-mon[33984]: osdmap e492: 8 total, 8 up, 8 in 2024-03-11T22:29:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:15.124 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:15.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:15.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:15.272 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:15.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:15.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:15.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:15.454 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:15.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:15.557 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:15.617 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is set 2024-03-11T22:29:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:15.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:15.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:15.740 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:15.779 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:15.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:15.880 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:15.924 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:15.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:15.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:16.026 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:16.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:15 smithi045 ceph-mon[39715]: pgmap v1009: 168 pgs: 4 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 151 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 32361/155835 objects misplaced (20.766%); 37 MiB/s, 594 objects/s recovering 2024-03-11T22:29:16.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:29:16.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:15 smithi045 ceph-mon[30178]: pgmap v1009: 168 pgs: 4 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 151 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 32361/155835 objects misplaced (20.766%); 37 MiB/s, 594 objects/s recovering 2024-03-11T22:29:16.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:29:16.070 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:16.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:16.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:15 smithi144 ceph-mon[33984]: pgmap v1009: 168 pgs: 4 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 151 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 32361/155835 objects misplaced (20.766%); 37 MiB/s, 594 objects/s recovering 2024-03-11T22:29:16.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:29:16.173 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:16.232 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:16.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:16.334 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:16.383 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:16.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:16.484 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:16.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:16.521 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:16.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:16.622 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:16.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:16.661 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:16.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:16.762 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:16.803 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:16.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:16.905 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:16.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:16.945 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:16.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:17.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:29:17.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:29:17.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:29:17.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:29:17.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:17.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:16 smithi045 ceph-mon[39715]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 10 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:17.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:29:17.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:29:17.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:29:17.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:29:17.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:17.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:16 smithi045 ceph-mon[30178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 10 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:17.045 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:17.085 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:17.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:17.099 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:29:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:29:17.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:29:17.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:29:17.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:29:17.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:29:17.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:17.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:16 smithi144 ceph-mon[33984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 10 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:17.188 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:17.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:17.227 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:17.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:17.328 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:17.379 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:17.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:17.481 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:17.515 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:17.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:17.533 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:29:17.542 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:29:17.542 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:29:17.616 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:17.651 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:17.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:17.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:17.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:17.800 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:17.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:17.902 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:17.947 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:17.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:17.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:18.049 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:18.090 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:18.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:18.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:17 smithi144 ceph-mon[33984]: pgmap v1010: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 152 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 902 B/s rd, 0 B/s wr, 1.38k op/s; 27629/148878 objects misplaced (18.558%); 48 MiB/s, 767 objects/s recovering 2024-03-11T22:29:18.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:18.111 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:29:18.172 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:17 smithi045 ceph-mon[30178]: pgmap v1010: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 152 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 902 B/s rd, 0 B/s wr, 1.38k op/s; 27629/148878 objects misplaced (18.558%); 48 MiB/s, 767 objects/s recovering 2024-03-11T22:29:18.172 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:18.172 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:17 smithi045 ceph-mon[39715]: pgmap v1010: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 152 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 902 B/s rd, 0 B/s wr, 1.38k op/s; 27629/148878 objects misplaced (18.558%); 48 MiB/s, 767 objects/s recovering 2024-03-11T22:29:18.172 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:18.191 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:18.243 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:18.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:18.344 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:18.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:18.385 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:18.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:18.486 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:18.517 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:18.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:18.562 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T22:29:18.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:18.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:18.688 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:18.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:18.789 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:18.831 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:18.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:18.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:18.933 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:18.966 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:18.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:19.067 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:19.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:19.094 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:19.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:19.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:29:19.195 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:19.220 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:19.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:19.321 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:19.345 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:19.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:19.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:19.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:19.481 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:19.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:19.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:19.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:19 smithi045 ceph-mon[39715]: osdmap e493: 8 total, 8 up, 8 in 2024-03-11T22:29:19.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:19 smithi045 ceph-mon[39715]: pgmap v1012: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 152 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 908 op/s; 25623/141798 objects misplaced (18.070%); 21 MiB/s, 329 objects/s recovering 2024-03-11T22:29:19.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:19 smithi045 ceph-mon[30178]: osdmap e493: 8 total, 8 up, 8 in 2024-03-11T22:29:19.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:19 smithi045 ceph-mon[30178]: pgmap v1012: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 152 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 908 op/s; 25623/141798 objects misplaced (18.070%); 21 MiB/s, 329 objects/s recovering 2024-03-11T22:29:19.583 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:19.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:19.612 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:19.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:19.713 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:19.745 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:19.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:19.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:19.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:19 smithi144 ceph-mon[33984]: osdmap e493: 8 total, 8 up, 8 in 2024-03-11T22:29:19.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:19 smithi144 ceph-mon[33984]: pgmap v1012: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 152 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 908 op/s; 25623/141798 objects misplaced (18.070%); 21 MiB/s, 329 objects/s recovering 2024-03-11T22:29:19.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:19.880 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:19.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:19.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:20.012 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:20.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:20.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:20.114 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:20.138 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:20.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:20.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:20.270 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:20.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:20.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:20.372 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:20.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:20.397 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:20.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:20.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:20.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:20.531 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:20.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:20.632 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:20.665 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:20.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:20.767 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:20.785 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:20 smithi045 ceph-mon[39715]: osdmap e494: 8 total, 8 up, 8 in 2024-03-11T22:29:20.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:20 smithi045 ceph-mon[30178]: osdmap e494: 8 total, 8 up, 8 in 2024-03-11T22:29:20.797 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:20.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:20.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:20 smithi144 ceph-mon[33984]: osdmap e494: 8 total, 8 up, 8 in 2024-03-11T22:29:20.898 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:20.932 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:20.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:21.033 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:21.064 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:21.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:21.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:21.166 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:21.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:21.200 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:21.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:21.201 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:29:21.209 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:29:21.301 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:21.349 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:21.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:21.451 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:21.487 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:21.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:21.588 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:21.627 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:21.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:21.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:21.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:21.769 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:21.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:21.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:21 smithi045 ceph-mon[39715]: osdmap e495: 8 total, 8 up, 8 in 2024-03-11T22:29:21.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:21 smithi045 ceph-mon[39715]: pgmap v1015: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 152 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 780 op/s; 25623/141798 objects misplaced (18.070%); 21 MiB/s, 329 objects/s recovering 2024-03-11T22:29:21.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:21 smithi045 ceph-mon[30178]: osdmap e495: 8 total, 8 up, 8 in 2024-03-11T22:29:21.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:21 smithi045 ceph-mon[30178]: pgmap v1015: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 152 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 780 op/s; 25623/141798 objects misplaced (18.070%); 21 MiB/s, 329 objects/s recovering 2024-03-11T22:29:21.849 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:21 smithi144 ceph-mon[33984]: osdmap e495: 8 total, 8 up, 8 in 2024-03-11T22:29:21.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:21 smithi144 ceph-mon[33984]: pgmap v1015: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 3 active+remapped+backfill_wait, 152 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 780 op/s; 25623/141798 objects misplaced (18.070%); 21 MiB/s, 329 objects/s recovering 2024-03-11T22:29:21.871 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:21.898 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:21.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:22.000 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:22.030 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:22.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:22.132 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:22.158 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:22.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:22.259 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:22.302 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:22.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:22.403 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:22.435 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:22.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:22.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:22.537 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:22.570 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:22.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:22.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:22.672 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:22.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:22.714 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:22.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:22.815 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:22.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:22.860 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:22.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:22.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:22.998 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:29:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:29:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:29:22.998 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:22 smithi045 ceph-mon[30178]: osdmap e496: 8 total, 8 up, 8 in 2024-03-11T22:29:22.998 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:22 smithi045 ceph-mon[39715]: osdmap e496: 8 total, 8 up, 8 in 2024-03-11T22:29:23.010 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:23.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:23.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:23.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:22 smithi144 ceph-mon[33984]: osdmap e496: 8 total, 8 up, 8 in 2024-03-11T22:29:23.113 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:23.146 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:23.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:23.248 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:23.296 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:23.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:23.398 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:23.426 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:23.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:23.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:23.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:23.580 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:23.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:23.682 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:23.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:23.727 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:23.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:23.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:23.865 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:23.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:23.967 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:23.998 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:23.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:24.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:23 smithi045 ceph-mon[39715]: pgmap v1017: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 155 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 802 op/s; 20214/131952 objects misplaced (15.319%); 55 MiB/s, 886 objects/s recovering 2024-03-11T22:29:24.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:23 smithi045 ceph-mon[30178]: pgmap v1017: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 155 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 802 op/s; 20214/131952 objects misplaced (15.319%); 55 MiB/s, 886 objects/s recovering 2024-03-11T22:29:24.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:24.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:23 smithi144 ceph-mon[33984]: pgmap v1017: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 155 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 802 op/s; 20214/131952 objects misplaced (15.319%); 55 MiB/s, 886 objects/s recovering 2024-03-11T22:29:24.143 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:24.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:24.246 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:24.280 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:24.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:24.382 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:24.417 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:24.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:24.501 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:29:24.518 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:24.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:24.569 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:24.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:24.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:24.725 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:24.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:24.827 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:24.859 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:24.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:24.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:24.961 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:24.998 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:24.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:24.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:25.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:25.137 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:25.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:25.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:25.239 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:25.262 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:25.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:25.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:25.364 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:25.395 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:25.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:25.496 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:25.526 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:25.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:25.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:25.628 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:25.662 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:25.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:25.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:25.764 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:25.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:25.796 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:25.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:25.892 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:25 smithi045 ceph-mon[39715]: pgmap v1018: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 155 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 640 op/s; 19536/130287 objects misplaced (14.995%); 38 MiB/s, 604 objects/s recovering 2024-03-11T22:29:25.893 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:25 smithi045 ceph-mon[39715]: osdmap e497: 8 total, 8 up, 8 in 2024-03-11T22:29:25.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:25 smithi045 ceph-mon[30178]: pgmap v1018: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 155 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 640 op/s; 19536/130287 objects misplaced (14.995%); 38 MiB/s, 604 objects/s recovering 2024-03-11T22:29:25.893 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:25 smithi045 ceph-mon[30178]: osdmap e497: 8 total, 8 up, 8 in 2024-03-11T22:29:25.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:25.938 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:25.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:26.044 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:26.077 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:26.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:26.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:26.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:25 smithi144 ceph-mon[33984]: pgmap v1018: 168 pgs: 3 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 155 active+clean; 2.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 640 op/s; 19536/130287 objects misplaced (14.995%); 38 MiB/s, 604 objects/s recovering 2024-03-11T22:29:26.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:25 smithi144 ceph-mon[33984]: osdmap e497: 8 total, 8 up, 8 in 2024-03-11T22:29:26.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:26.223 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:26.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:26.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:26.325 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:26.366 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:26.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:26.468 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:26.497 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:26.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:26.599 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:26.632 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:26.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:26.670 INFO:teuthology.orchestra.run.smithi045.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-11T22:29:26.734 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:26.766 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:26.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:26.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:26 smithi045 ceph-mon[39715]: osdmap e498: 8 total, 8 up, 8 in 2024-03-11T22:29:26.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:26 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/341261572' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:29:26.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:26 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:29:26.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:26 smithi045 ceph-mon[30178]: osdmap e498: 8 total, 8 up, 8 in 2024-03-11T22:29:26.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:26 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/341261572' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:29:26.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:26 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:29:26.868 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:26.899 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:26.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:27.001 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:27.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:27.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:27.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:27.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:26 smithi144 ceph-mon[33984]: osdmap e498: 8 total, 8 up, 8 in 2024-03-11T22:29:27.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:26 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/341261572' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:29:27.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:26 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:29:27.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:29:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:29:27.151 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:27.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:27.184 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:27.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:27.285 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:27.318 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:27.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:27.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:27.420 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:27.463 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:27.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:27.565 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:27.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:27.603 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:27.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:27.705 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:27.736 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:27.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:27.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:27.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:27.894 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:27.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:27.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:27.996 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:28.027 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:28.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:28.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:27 smithi045 ceph-mon[39715]: pgmap v1021: 168 pgs: 2 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 156 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 18039/123324 objects misplaced (14.627%); 50 MiB/s, 800 objects/s recovering 2024-03-11T22:29:28.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:27 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:29:28.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:27 smithi045 ceph-mon[39715]: osdmap e499: 8 total, 8 up, 8 in 2024-03-11T22:29:28.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:28.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:27 smithi045 ceph-mon[30178]: pgmap v1021: 168 pgs: 2 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 156 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 18039/123324 objects misplaced (14.627%); 50 MiB/s, 800 objects/s recovering 2024-03-11T22:29:28.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:27 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:29:28.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:27 smithi045 ceph-mon[30178]: osdmap e499: 8 total, 8 up, 8 in 2024-03-11T22:29:28.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:28.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:27 smithi144 ceph-mon[33984]: pgmap v1021: 168 pgs: 2 active+remapped+backfilling, 10 active+remapped+backfill_toofull, 156 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 18039/123324 objects misplaced (14.627%); 50 MiB/s, 800 objects/s recovering 2024-03-11T22:29:28.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:27 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:29:28.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:27 smithi144 ceph-mon[33984]: osdmap e499: 8 total, 8 up, 8 in 2024-03-11T22:29:28.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:28.129 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:28.161 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:28.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:28.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:28.263 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:28.309 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:28.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:28.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:28.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:28.459 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:28.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:28.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:28.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:28.651 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:28.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:28.752 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:28.789 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:28.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:28.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:28.931 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:28.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:28.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:29.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:29.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:28 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:29:29.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:28 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:29:29.040 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:28 smithi045 ceph-mon[30178]: osdmap e500: 8 total, 8 up, 8 in 2024-03-11T22:29:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:28 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:29:29.040 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:28 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:29:29.041 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:28 smithi045 ceph-mon[39715]: osdmap e500: 8 total, 8 up, 8 in 2024-03-11T22:29:29.077 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:29.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:28 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:29:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:28 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:29:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:28 smithi144 ceph-mon[33984]: osdmap e500: 8 total, 8 up, 8 in 2024-03-11T22:29:29.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:29.271 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:29.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:29.373 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:29.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:29.405 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:29.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:29.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:29.544 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:29.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:29.645 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:29.690 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:29.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:29.792 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:29.824 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:29.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:29.926 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:29.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:29.953 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:29.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:30.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:29 smithi045 ceph-mon[39715]: pgmap v1024: 168 pgs: 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 15167/116370 objects misplaced (13.033%); 33 MiB/s, 521 objects/s recovering 2024-03-11T22:29:30.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:29 smithi045 ceph-mon[39715]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:29 smithi045 ceph-mon[30178]: pgmap v1024: 168 pgs: 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 15167/116370 objects misplaced (13.033%); 33 MiB/s, 521 objects/s recovering 2024-03-11T22:29:30.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:29 smithi045 ceph-mon[30178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:30.055 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:30.089 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:30.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:30.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:30.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:29 smithi144 ceph-mon[33984]: pgmap v1024: 168 pgs: 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 15167/116370 objects misplaced (13.033%); 33 MiB/s, 521 objects/s recovering 2024-03-11T22:29:30.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:29 smithi144 ceph-mon[33984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:30.191 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:30.224 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:30.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:30.325 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:30.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:30.367 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:30.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:30.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:30.509 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:30.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:30.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:30.612 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:30.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:30.647 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:30.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:30.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:30.789 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:30.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:30.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:30.891 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:30.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:30.917 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:30.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:31.018 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:31.060 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:31.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:31.162 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:31.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:31.196 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:31.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:31.297 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:31.323 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:31.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:31.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:31.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:31.466 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:31.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:31.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:31.568 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:31.592 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:31.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:31.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:31.736 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:31.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:31.838 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:31.866 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:31.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:31.968 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:32.006 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:32.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:32.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:31 smithi045 ceph-mon[39715]: pgmap v1025: 168 pgs: 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 872 B/s rd, 0 B/s wr, 790 op/s; 15167/116370 objects misplaced (13.033%); 22 MiB/s, 355 objects/s recovering 2024-03-11T22:29:32.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:32.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:31 smithi045 ceph-mon[30178]: pgmap v1025: 168 pgs: 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 872 B/s rd, 0 B/s wr, 790 op/s; 15167/116370 objects misplaced (13.033%); 22 MiB/s, 355 objects/s recovering 2024-03-11T22:29:32.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:31 smithi144 ceph-mon[33984]: pgmap v1025: 168 pgs: 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 156 active+clean; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 872 B/s rd, 0 B/s wr, 790 op/s; 15167/116370 objects misplaced (13.033%); 22 MiB/s, 355 objects/s recovering 2024-03-11T22:29:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:32.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:32.148 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:32.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:32.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:32.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:32.289 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:32.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:32.391 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:32.418 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:32.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:32.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:32.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:32.560 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:32.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:32.662 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:32.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:32.686 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:32.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:32.787 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:32.837 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:32.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:32.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:32.898 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:29:32.898 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:29:32.898 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-11T22:29:32.898 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T22:29:32.898 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T22:29:32.899 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-11T22:29:32.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:32.941 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:29:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:29:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:29:32.968 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:32.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:32.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:33.070 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:33.102 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:33.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:33.204 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:33.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:33.238 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:33.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:33.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:33.381 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:33.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:33.482 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:33.514 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:33.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:33.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:33.616 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:33.674 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:33.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:33.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:33.776 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:33.811 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:33.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:33.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:33.955 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:33.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:33.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:34.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:33 smithi045 ceph-mon[39715]: pgmap v1026: 168 pgs: 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 156 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.30k op/s; 12643/103572 objects misplaced (12.207%); 30 MiB/s, 475 objects/s recovering 2024-03-11T22:29:34.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:33 smithi045 ceph-mon[30178]: pgmap v1026: 168 pgs: 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 156 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.30k op/s; 12643/103572 objects misplaced (12.207%); 30 MiB/s, 475 objects/s recovering 2024-03-11T22:29:34.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:34.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:34.081 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:34.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:34.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:33 smithi144 ceph-mon[33984]: pgmap v1026: 168 pgs: 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 156 active+clean; 2.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.30k op/s; 12643/103572 objects misplaced (12.207%); 30 MiB/s, 475 objects/s recovering 2024-03-11T22:29:34.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:34.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:34.222 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:34.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:34.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:34.360 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:34.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:34.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:34.502 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:34.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:34.603 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:34.632 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:34.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:34.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:34.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:34.780 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:34.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:34.882 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:34.913 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:34.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:35.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:35.052 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:35.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:35.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:35.154 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:35.190 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:35.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:35.291 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:35.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:35.327 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:35.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:35.428 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:35.476 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:35.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:35.578 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:35.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:35.611 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:35.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:35.712 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:35.743 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:35.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:35.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:35.886 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:35.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:35.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:35.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:36.023 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:36.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:36.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:36.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:35 smithi045 ceph-mon[39715]: pgmap v1027: 168 pgs: 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 156 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 10597/101331 objects misplaced (10.458%); 20 MiB/s, 313 objects/s recovering 2024-03-11T22:29:36.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:35 smithi045 ceph-mon[30178]: pgmap v1027: 168 pgs: 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 156 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 10597/101331 objects misplaced (10.458%); 20 MiB/s, 313 objects/s recovering 2024-03-11T22:29:36.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:35 smithi144 ceph-mon[33984]: pgmap v1027: 168 pgs: 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull, 156 active+clean; 2.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 10597/101331 objects misplaced (10.458%); 20 MiB/s, 313 objects/s recovering 2024-03-11T22:29:36.125 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:36.149 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:36.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:36.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:36.283 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:36.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:36.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:36.385 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:36.411 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:36.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:36.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:36.513 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:36.545 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:36.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:36.647 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:36.673 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:36.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:36.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:36.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:36.799 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:36.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:36.900 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:36.942 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:36.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:36.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:37.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:36 smithi045 ceph-mon[39715]: osdmap e501: 8 total, 8 up, 8 in 2024-03-11T22:29:37.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:36 smithi045 ceph-mon[39715]: osdmap e502: 8 total, 8 up, 8 in 2024-03-11T22:29:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:36 smithi045 ceph-mon[30178]: osdmap e501: 8 total, 8 up, 8 in 2024-03-11T22:29:37.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:36 smithi045 ceph-mon[30178]: osdmap e502: 8 total, 8 up, 8 in 2024-03-11T22:29:37.045 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:37.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:37.075 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:37.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:37.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:36 smithi144 ceph-mon[33984]: osdmap e501: 8 total, 8 up, 8 in 2024-03-11T22:29:37.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:36 smithi144 ceph-mon[33984]: osdmap e502: 8 total, 8 up, 8 in 2024-03-11T22:29:37.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:29:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:29:37.177 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:37.216 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:37.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:37.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:37.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:37.354 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:37.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:37.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:37.455 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:37.480 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:37.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:37.582 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:37.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:37.607 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:37.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:37.684 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:29:37.708 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:37.726 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:29:37.737 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:37.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:37.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:37.769 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:29:37.804 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:29:37.835 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:29:37.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:37.867 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:29:37.870 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:37.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:37.900 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:29:37.935 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:29:37.935 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[39715]: pgmap v1029: 168 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 156 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 580 B/s rd, 0 B/s wr, 906 op/s; 9699/94641 objects misplaced (10.248%); 17 MiB/s, 275 objects/s recovering 2024-03-11T22:29:37.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:29:37.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:29:37.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[39715]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:37.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:37.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[39715]: osdmap e503: 8 total, 8 up, 8 in 2024-03-11T22:29:37.936 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2882435078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:29:37.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[30178]: pgmap v1029: 168 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 156 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 580 B/s rd, 0 B/s wr, 906 op/s; 9699/94641 objects misplaced (10.248%); 17 MiB/s, 275 objects/s recovering 2024-03-11T22:29:37.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:29:37.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:29:37.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[30178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:37.936 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:37.937 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[30178]: osdmap e503: 8 total, 8 up, 8 in 2024-03-11T22:29:37.937 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:37 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2882435078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:29:37.937 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:29:37.937 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:29:37.937 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:29:37.937 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:29:37.937 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:29:37.937 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:29:37.937 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:29:37.938 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:29:37.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:38.004 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:38.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:38.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:38.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:37 smithi144 ceph-mon[33984]: pgmap v1029: 168 pgs: 5 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 156 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 580 B/s rd, 0 B/s wr, 906 op/s; 9699/94641 objects misplaced (10.248%); 17 MiB/s, 275 objects/s recovering 2024-03-11T22:29:38.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:37 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:29:38.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:37 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:29:38.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:37 smithi144 ceph-mon[33984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:38.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:38.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:37 smithi144 ceph-mon[33984]: osdmap e503: 8 total, 8 up, 8 in 2024-03-11T22:29:38.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:37 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2882435078' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:29:38.106 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:38.135 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:38.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:38.237 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:38.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:38.262 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:38.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:38.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:38.392 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:38.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:38.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:38.494 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:38.519 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:38.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:38.620 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:38.644 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:38.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:38.746 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:38.782 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:38.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:38.885 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:38.910 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:38.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:39.012 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:39.036 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:39.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:39.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:39.138 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:39.163 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:39.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:39.265 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:39.291 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:39.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:39.392 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:39.419 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:39.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:39.521 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:39.546 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:39.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:39.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:39.648 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:39.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:39.673 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:39.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:39.774 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:39.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:39.808 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:39.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:39.909 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:39.933 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:39.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:40.035 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:40.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:39 smithi045 ceph-mon[39715]: pgmap v1032: 168 pgs: 1 peering, 6 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 158 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 777 op/s; 6251/89583 objects misplaced (6.978%); 34 MiB/s, 547 objects/s recovering 2024-03-11T22:29:40.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:39 smithi045 ceph-mon[39715]: osdmap e504: 8 total, 8 up, 8 in 2024-03-11T22:29:40.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:39 smithi045 ceph-mon[30178]: pgmap v1032: 168 pgs: 1 peering, 6 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 158 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 777 op/s; 6251/89583 objects misplaced (6.978%); 34 MiB/s, 547 objects/s recovering 2024-03-11T22:29:40.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:39 smithi045 ceph-mon[30178]: osdmap e504: 8 total, 8 up, 8 in 2024-03-11T22:29:40.074 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:40.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:40.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:40.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:39 smithi144 ceph-mon[33984]: pgmap v1032: 168 pgs: 1 peering, 6 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 158 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 777 op/s; 6251/89583 objects misplaced (6.978%); 34 MiB/s, 547 objects/s recovering 2024-03-11T22:29:40.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:39 smithi144 ceph-mon[33984]: osdmap e504: 8 total, 8 up, 8 in 2024-03-11T22:29:40.176 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:40.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:40.210 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:40.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:40.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:40.353 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:40.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:40.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:40.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:40.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:40.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:40.600 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:40.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:40.635 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:40.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:40.736 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:40.771 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:40.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:40.873 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:40.910 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:40.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:41.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:41.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:40 smithi045 ceph-mon[39715]: osdmap e505: 8 total, 8 up, 8 in 2024-03-11T22:29:41.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:40 smithi045 ceph-mon[30178]: osdmap e505: 8 total, 8 up, 8 in 2024-03-11T22:29:41.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:41.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:41.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:41.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:40 smithi144 ceph-mon[33984]: osdmap e505: 8 total, 8 up, 8 in 2024-03-11T22:29:41.151 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:41.183 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:41.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:41.284 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:41.321 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:41.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:41.422 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:41.468 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:41.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:41.569 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:41.600 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:41.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:41.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:41.686 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:41 smithi045 ceph-mon[30178]: pgmap v1035: 168 pgs: 1 peering, 6 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 158 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 422 op/s; 6251/89583 objects misplaced (6.978%); 33 MiB/s, 535 objects/s recovering 2024-03-11T22:29:41.686 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:41 smithi045 ceph-mon[30178]: osdmap e506: 8 total, 8 up, 8 in 2024-03-11T22:29:41.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:41 smithi045 ceph-mon[39715]: pgmap v1035: 168 pgs: 1 peering, 6 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 158 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 422 op/s; 6251/89583 objects misplaced (6.978%); 33 MiB/s, 535 objects/s recovering 2024-03-11T22:29:41.687 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:41 smithi045 ceph-mon[39715]: osdmap e506: 8 total, 8 up, 8 in 2024-03-11T22:29:41.702 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:41.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:41.748 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:41.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:41.849 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:41.894 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:41.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:41.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:42.024 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:42.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:42.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:42.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:41 smithi144 ceph-mon[33984]: pgmap v1035: 168 pgs: 1 peering, 6 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 158 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 422 op/s; 6251/89583 objects misplaced (6.978%); 33 MiB/s, 535 objects/s recovering 2024-03-11T22:29:42.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:41 smithi144 ceph-mon[33984]: osdmap e506: 8 total, 8 up, 8 in 2024-03-11T22:29:42.126 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:42.160 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:42.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:42.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:42.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:42.287 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:42.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:42.388 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:42.421 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:42.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:42.522 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:42.557 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:42.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:42.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:42.705 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:42.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:42.806 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:42.830 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:42.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:42.931 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:42.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:42.955 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:42.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:43.036 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:29:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:29:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:29:43.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:43.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:43.083 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:43.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:43.184 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:43.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:43.210 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:43.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:43.311 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:43.335 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:43.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:43.437 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:43.465 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:43.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:43.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:43.567 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:43.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:43.592 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:43.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:43.693 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:43.719 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:43.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:43.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:43.821 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:43.852 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:43.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:43.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:43.953 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:43.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:43.990 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:43.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:44.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:43 smithi045 ceph-mon[30178]: pgmap v1037: 168 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 612 op/s; 3765/81723 objects misplaced (4.607%); 32 MiB/s, 513 objects/s recovering 2024-03-11T22:29:44.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:43 smithi045 ceph-mon[30178]: osdmap e507: 8 total, 8 up, 8 in 2024-03-11T22:29:44.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:43 smithi045 ceph-mon[39715]: pgmap v1037: 168 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 612 op/s; 3765/81723 objects misplaced (4.607%); 32 MiB/s, 513 objects/s recovering 2024-03-11T22:29:44.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:43 smithi045 ceph-mon[39715]: osdmap e507: 8 total, 8 up, 8 in 2024-03-11T22:29:44.091 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:44.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:43 smithi144 ceph-mon[33984]: pgmap v1037: 168 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 612 op/s; 3765/81723 objects misplaced (4.607%); 32 MiB/s, 513 objects/s recovering 2024-03-11T22:29:44.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:43 smithi144 ceph-mon[33984]: osdmap e507: 8 total, 8 up, 8 in 2024-03-11T22:29:44.120 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:44.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:44.222 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:44.263 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:44.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:44.365 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:44.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:44.400 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:44.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:44.501 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:44.540 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:44.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:44.641 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:44.680 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:44.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:44.781 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:44.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:44.813 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:44.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:44.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:44.939 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:44.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:45.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:44 smithi045 ceph-mon[39715]: osdmap e508: 8 total, 8 up, 8 in 2024-03-11T22:29:45.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:44 smithi045 ceph-mon[30178]: osdmap e508: 8 total, 8 up, 8 in 2024-03-11T22:29:45.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:45.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:45.077 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:45.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:45.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:44 smithi144 ceph-mon[33984]: osdmap e508: 8 total, 8 up, 8 in 2024-03-11T22:29:45.178 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:45.213 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:45.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:45.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:45.315 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:45.347 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:45.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:45.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:45.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:45.497 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:45.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:45.598 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:45.632 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:45.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:45.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:45.734 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:45.762 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:45.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:45.864 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:45.893 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:45.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:45.954 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:45 smithi045 ceph-mon[39715]: pgmap v1040: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 583 op/s; 3496/81030 objects misplaced (4.314%); 17 MiB/s, 278 objects/s recovering 2024-03-11T22:29:45.955 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:45 smithi045 ceph-mon[30178]: pgmap v1040: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 583 op/s; 3496/81030 objects misplaced (4.314%); 17 MiB/s, 278 objects/s recovering 2024-03-11T22:29:45.995 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:46.036 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:46.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:46.044 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:29:46.044 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:29:46.044 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-03-11T22:29:46.044 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-03-11T22:29:46.044 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:29:46.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:45 smithi144 ceph-mon[33984]: pgmap v1040: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 583 op/s; 3496/81030 objects misplaced (4.314%); 17 MiB/s, 278 objects/s recovering 2024-03-11T22:29:46.138 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:46.168 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:46.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:46.270 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:46.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:46.304 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:46.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:46.405 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:46.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:46.540 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:46.570 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:46.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:46.672 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:46.698 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:46.699 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:46 smithi045 ceph-mon[30178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:46.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:46.723 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:46.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:46.824 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:46.853 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:46.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:46.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:46.995 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:46.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:47.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:47.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:46 smithi045 ceph-mon[39715]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:47.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:47.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:29:47.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:46 smithi144 ceph-mon[33984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:29:47.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:29:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:29:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:47.130 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:47.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:47.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:47.266 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:47.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:47.367 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:47.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:29:47.404 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:47.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:47.506 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:47.537 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:47.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:47.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:47.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:47.685 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:47.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:47.709 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:29:47.711 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:47 smithi045 ceph-mon[39715]: pgmap v1041: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 900 op/s; 3470/73377 objects misplaced (4.729%); 14 MiB/s, 226 objects/s recovering 2024-03-11T22:29:47.711 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:47.711 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:47 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4139407' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:29:47.787 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:47.819 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:47.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:47.921 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:47.961 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:47.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:47.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:48.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:47 smithi045 ceph-mon[30178]: pgmap v1041: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 900 op/s; 3470/73377 objects misplaced (4.729%); 14 MiB/s, 226 objects/s recovering 2024-03-11T22:29:48.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:48.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4139407' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:29:48.062 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:48.099 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:48.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:48.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:47 smithi144 ceph-mon[33984]: pgmap v1041: 168 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 164 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 900 op/s; 3470/73377 objects misplaced (4.729%); 14 MiB/s, 226 objects/s recovering 2024-03-11T22:29:48.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:48.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:47 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4139407' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:29:48.200 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:48.239 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:48.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:48.342 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:48.354 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T22:29:48.369 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:48.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:48.419 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:29:48.433 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:29:48.471 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:48.533 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:48.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:48.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:48.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:48.688 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:48.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:48.789 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:48.816 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:48.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:48.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:48.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:48 smithi045 ceph-mon[30178]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:29:48.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:48 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4139407' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:29:48.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:48 smithi045 ceph-mon[30178]: osdmap e509: 8 total, 8 up, 8 in 2024-03-11T22:29:48.824 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:48 smithi045 ceph-mon[30178]: 11.2 scrub starts 2024-03-11T22:29:48.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:48 smithi045 ceph-mon[39715]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:29:48.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:48 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4139407' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:29:48.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:48 smithi045 ceph-mon[39715]: osdmap e509: 8 total, 8 up, 8 in 2024-03-11T22:29:48.826 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:48 smithi045 ceph-mon[39715]: 11.2 scrub starts 2024-03-11T22:29:48.918 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:48.952 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:48.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:49.054 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:49.091 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:49.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:49.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:49.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:48 smithi144 ceph-mon[33984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:29:49.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:48 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4139407' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:29:49.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:48 smithi144 ceph-mon[33984]: osdmap e509: 8 total, 8 up, 8 in 2024-03-11T22:29:49.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:48 smithi144 ceph-mon[33984]: 11.2 scrub starts 2024-03-11T22:29:49.192 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:49.226 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:49.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:49.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:49.328 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:49.369 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:49.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:49.471 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:49.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:49.506 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:49.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:49.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:49.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:49.648 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:49.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:49.749 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:49.783 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:49.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:49.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:49.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:49 smithi045 ceph-mon[39715]: 11.2 scrub ok 2024-03-11T22:29:49.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:49 smithi045 ceph-mon[39715]: pgmap v1043: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 919 op/s; 1893/65181 objects misplaced (2.904%); 15 MiB/s, 236 objects/s recovering 2024-03-11T22:29:49.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:49 smithi045 ceph-mon[39715]: 11.7 scrub starts 2024-03-11T22:29:49.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:49 smithi045 ceph-mon[39715]: 1.3 scrub starts 2024-03-11T22:29:49.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:49 smithi045 ceph-mon[39715]: 1.3 scrub ok 2024-03-11T22:29:49.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:49 smithi045 ceph-mon[30178]: 11.2 scrub ok 2024-03-11T22:29:49.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:49 smithi045 ceph-mon[30178]: pgmap v1043: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 919 op/s; 1893/65181 objects misplaced (2.904%); 15 MiB/s, 236 objects/s recovering 2024-03-11T22:29:49.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:49 smithi045 ceph-mon[30178]: 11.7 scrub starts 2024-03-11T22:29:49.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:49 smithi045 ceph-mon[30178]: 1.3 scrub starts 2024-03-11T22:29:49.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:49 smithi045 ceph-mon[30178]: 1.3 scrub ok 2024-03-11T22:29:49.885 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:49.918 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:49.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:49.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:50.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:50.063 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:50.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:50.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:49 smithi144 ceph-mon[33984]: 11.2 scrub ok 2024-03-11T22:29:50.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:49 smithi144 ceph-mon[33984]: pgmap v1043: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 919 op/s; 1893/65181 objects misplaced (2.904%); 15 MiB/s, 236 objects/s recovering 2024-03-11T22:29:50.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:49 smithi144 ceph-mon[33984]: 11.7 scrub starts 2024-03-11T22:29:50.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:49 smithi144 ceph-mon[33984]: 1.3 scrub starts 2024-03-11T22:29:50.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:49 smithi144 ceph-mon[33984]: 1.3 scrub ok 2024-03-11T22:29:50.164 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:50.201 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:50.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:50.303 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:50.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:50.338 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:50.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:50.439 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:50.473 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:50.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:50.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:50.575 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:50.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:50.610 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:50.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:50.711 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:50.740 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:50.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:50.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:50 smithi045 ceph-mon[39715]: 11.7 scrub ok 2024-03-11T22:29:50.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:50 smithi045 ceph-mon[39715]: 11.9 scrub starts 2024-03-11T22:29:50.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:50 smithi045 ceph-mon[39715]: 11.9 scrub ok 2024-03-11T22:29:50.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:50 smithi045 ceph-mon[39715]: 11.16 scrub starts 2024-03-11T22:29:50.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:50 smithi045 ceph-mon[39715]: 11.17 scrub starts 2024-03-11T22:29:50.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:50 smithi045 ceph-mon[30178]: 11.7 scrub ok 2024-03-11T22:29:50.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:50 smithi045 ceph-mon[30178]: 11.9 scrub starts 2024-03-11T22:29:50.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:50 smithi045 ceph-mon[30178]: 11.9 scrub ok 2024-03-11T22:29:50.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:50 smithi045 ceph-mon[30178]: 11.16 scrub starts 2024-03-11T22:29:50.788 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:50 smithi045 ceph-mon[30178]: 11.17 scrub starts 2024-03-11T22:29:50.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:50.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:50.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:50.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:50.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:51.035 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:51.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:51.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:51.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:50 smithi144 ceph-mon[33984]: 11.7 scrub ok 2024-03-11T22:29:51.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:50 smithi144 ceph-mon[33984]: 11.9 scrub starts 2024-03-11T22:29:51.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:50 smithi144 ceph-mon[33984]: 11.9 scrub ok 2024-03-11T22:29:51.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:50 smithi144 ceph-mon[33984]: 11.16 scrub starts 2024-03-11T22:29:51.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:50 smithi144 ceph-mon[33984]: 11.17 scrub starts 2024-03-11T22:29:51.137 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:51.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:51.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:51.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:51.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:51.310 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:51.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:51.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:51.412 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:51.445 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:51.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:51.543 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:29:51.546 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:51.575 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:51.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:51.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:51.677 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:51.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:51.726 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:51.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:51.827 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:51.857 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:51.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:51.932 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:29:51.958 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:52.035 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:52.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:52.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[30178]: 11.18 scrub starts 2024-03-11T22:29:52.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[30178]: 11.18 scrub ok 2024-03-11T22:29:52.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[30178]: 11.16 scrub ok 2024-03-11T22:29:52.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[30178]: 11.19 scrub starts 2024-03-11T22:29:52.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[30178]: 11.17 scrub ok 2024-03-11T22:29:52.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[30178]: pgmap v1044: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 651 B/s rd, 0 B/s wr, 702 op/s; 1893/65181 objects misplaced (2.904%); 11 MiB/s, 180 objects/s recovering 2024-03-11T22:29:52.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[30178]: 6.11 scrub starts 2024-03-11T22:29:52.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[30178]: 11.19 scrub ok 2024-03-11T22:29:52.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[30178]: 6.11 scrub ok 2024-03-11T22:29:52.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[30178]: 11.11 scrub starts 2024-03-11T22:29:52.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[30178]: 11.14 scrub starts 2024-03-11T22:29:52.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[30178]: osdmap e510: 8 total, 8 up, 8 in 2024-03-11T22:29:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[39715]: 11.18 scrub starts 2024-03-11T22:29:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[39715]: 11.18 scrub ok 2024-03-11T22:29:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[39715]: 11.16 scrub ok 2024-03-11T22:29:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[39715]: 11.19 scrub starts 2024-03-11T22:29:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[39715]: 11.17 scrub ok 2024-03-11T22:29:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[39715]: pgmap v1044: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 651 B/s rd, 0 B/s wr, 702 op/s; 1893/65181 objects misplaced (2.904%); 11 MiB/s, 180 objects/s recovering 2024-03-11T22:29:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[39715]: 6.11 scrub starts 2024-03-11T22:29:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[39715]: 11.19 scrub ok 2024-03-11T22:29:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[39715]: 6.11 scrub ok 2024-03-11T22:29:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[39715]: 11.11 scrub starts 2024-03-11T22:29:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[39715]: 11.14 scrub starts 2024-03-11T22:29:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:51 smithi045 ceph-mon[39715]: osdmap e510: 8 total, 8 up, 8 in 2024-03-11T22:29:52.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:51 smithi144 ceph-mon[33984]: 11.18 scrub starts 2024-03-11T22:29:52.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:51 smithi144 ceph-mon[33984]: 11.18 scrub ok 2024-03-11T22:29:52.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:51 smithi144 ceph-mon[33984]: 11.16 scrub ok 2024-03-11T22:29:52.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:51 smithi144 ceph-mon[33984]: 11.19 scrub starts 2024-03-11T22:29:52.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:51 smithi144 ceph-mon[33984]: 11.17 scrub ok 2024-03-11T22:29:52.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:51 smithi144 ceph-mon[33984]: pgmap v1044: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 651 B/s rd, 0 B/s wr, 702 op/s; 1893/65181 objects misplaced (2.904%); 11 MiB/s, 180 objects/s recovering 2024-03-11T22:29:52.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:51 smithi144 ceph-mon[33984]: 6.11 scrub starts 2024-03-11T22:29:52.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:51 smithi144 ceph-mon[33984]: 11.19 scrub ok 2024-03-11T22:29:52.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:51 smithi144 ceph-mon[33984]: 6.11 scrub ok 2024-03-11T22:29:52.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:51 smithi144 ceph-mon[33984]: 11.11 scrub starts 2024-03-11T22:29:52.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:51 smithi144 ceph-mon[33984]: 11.14 scrub starts 2024-03-11T22:29:52.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:51 smithi144 ceph-mon[33984]: osdmap e510: 8 total, 8 up, 8 in 2024-03-11T22:29:52.136 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:52.171 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:52.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:52.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:52.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:52.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:52.421 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:52.457 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:52.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:52.558 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:52.587 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:52.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:52.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:52.686 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:29:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:29:52.689 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:52.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:52.722 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:52.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:52.823 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:52.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:52.860 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:52.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:52.962 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:52.990 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:52.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:52.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:53.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[30178]: 11.11 scrub ok 2024-03-11T22:29:53.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[30178]: 11.14 scrub ok 2024-03-11T22:29:53.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[30178]: 11.6 scrub starts 2024-03-11T22:29:53.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[30178]: 4.11 scrub starts 2024-03-11T22:29:53.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[30178]: 11.6 scrub ok 2024-03-11T22:29:53.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[30178]: 4.11 scrub ok 2024-03-11T22:29:53.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[30178]: 11.12 scrub starts 2024-03-11T22:29:53.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[30178]: 11.5 scrub starts 2024-03-11T22:29:53.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[30178]: 4.4 scrub starts 2024-03-11T22:29:53.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[30178]: 4.4 scrub ok 2024-03-11T22:29:53.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[39715]: 11.11 scrub ok 2024-03-11T22:29:53.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[39715]: 11.14 scrub ok 2024-03-11T22:29:53.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[39715]: 11.6 scrub starts 2024-03-11T22:29:53.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[39715]: 4.11 scrub starts 2024-03-11T22:29:53.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[39715]: 11.6 scrub ok 2024-03-11T22:29:53.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[39715]: 4.11 scrub ok 2024-03-11T22:29:53.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[39715]: 11.12 scrub starts 2024-03-11T22:29:53.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[39715]: 11.5 scrub starts 2024-03-11T22:29:53.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[39715]: 4.4 scrub starts 2024-03-11T22:29:53.001 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:52 smithi045 ceph-mon[39715]: 4.4 scrub ok 2024-03-11T22:29:53.092 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:53.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:52 smithi144 ceph-mon[33984]: 11.11 scrub ok 2024-03-11T22:29:53.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:52 smithi144 ceph-mon[33984]: 11.14 scrub ok 2024-03-11T22:29:53.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:52 smithi144 ceph-mon[33984]: 11.6 scrub starts 2024-03-11T22:29:53.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:52 smithi144 ceph-mon[33984]: 4.11 scrub starts 2024-03-11T22:29:53.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:52 smithi144 ceph-mon[33984]: 11.6 scrub ok 2024-03-11T22:29:53.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:52 smithi144 ceph-mon[33984]: 4.11 scrub ok 2024-03-11T22:29:53.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:52 smithi144 ceph-mon[33984]: 11.12 scrub starts 2024-03-11T22:29:53.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:52 smithi144 ceph-mon[33984]: 11.5 scrub starts 2024-03-11T22:29:53.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:52 smithi144 ceph-mon[33984]: 4.4 scrub starts 2024-03-11T22:29:53.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:52 smithi144 ceph-mon[33984]: 4.4 scrub ok 2024-03-11T22:29:53.132 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:53.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:53.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:53.234 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:53.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:53.265 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:53.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:53.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:53.409 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:53.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:53.511 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:53.558 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:53.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:53.659 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:53.693 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:53.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:53.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:53.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:53.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:53.973 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:53.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:53.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:53.998 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[30178]: 11.12 scrub ok 2024-03-11T22:29:53.998 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[30178]: 11.5 scrub ok 2024-03-11T22:29:53.998 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[30178]: 6.15 scrub starts 2024-03-11T22:29:53.998 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[30178]: 6.15 scrub ok 2024-03-11T22:29:53.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[30178]: pgmap v1046: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 983 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1328/47763 objects misplaced (2.780%) 2024-03-11T22:29:53.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[30178]: 11.15 scrub starts 2024-03-11T22:29:53.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:29:53.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:29:53.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[30178]: 3.1d scrub starts 2024-03-11T22:29:53.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[30178]: 3.1d scrub ok 2024-03-11T22:29:53.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[39715]: 11.12 scrub ok 2024-03-11T22:29:53.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[39715]: 11.5 scrub ok 2024-03-11T22:29:53.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[39715]: 6.15 scrub starts 2024-03-11T22:29:53.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[39715]: 6.15 scrub ok 2024-03-11T22:29:54.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[39715]: pgmap v1046: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 983 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1328/47763 objects misplaced (2.780%) 2024-03-11T22:29:54.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[39715]: 11.15 scrub starts 2024-03-11T22:29:54.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:29:54.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:29:54.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[39715]: 3.1d scrub starts 2024-03-11T22:29:54.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:53 smithi045 ceph-mon[39715]: 3.1d scrub ok 2024-03-11T22:29:54.076 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:54.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:53 smithi144 ceph-mon[33984]: 11.12 scrub ok 2024-03-11T22:29:54.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:53 smithi144 ceph-mon[33984]: 11.5 scrub ok 2024-03-11T22:29:54.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:53 smithi144 ceph-mon[33984]: 6.15 scrub starts 2024-03-11T22:29:54.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:53 smithi144 ceph-mon[33984]: 6.15 scrub ok 2024-03-11T22:29:54.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:53 smithi144 ceph-mon[33984]: pgmap v1046: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 983 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1328/47763 objects misplaced (2.780%) 2024-03-11T22:29:54.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:53 smithi144 ceph-mon[33984]: 11.15 scrub starts 2024-03-11T22:29:54.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:53 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:29:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:53 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:29:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:53 smithi144 ceph-mon[33984]: 3.1d scrub starts 2024-03-11T22:29:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:53 smithi144 ceph-mon[33984]: 3.1d scrub ok 2024-03-11T22:29:54.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:54.108 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:54.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:54.209 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:54.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:54.248 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:54.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:54.349 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:54.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:54.384 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:54.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:54.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:54.518 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:54.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:54.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:54.620 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:54.664 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:54.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:54.765 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:54.794 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:54.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:54.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:54.836 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:29:54.869 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:29:54.869 INFO:teuthology.orchestra.run.smithi045.stdout: "success": "" 2024-03-11T22:29:54.869 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:29:54.886 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:29:54.896 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:54.938 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:29:54.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:54.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:54.981 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:29:55.026 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:29:55.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[39715]: 11.15 scrub ok 2024-03-11T22:29:55.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[39715]: 3.12 scrub starts 2024-03-11T22:29:55.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[39715]: 3.12 scrub ok 2024-03-11T22:29:55.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[39715]: osdmap e511: 8 total, 8 up, 8 in 2024-03-11T22:29:55.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[39715]: 3.5 scrub starts 2024-03-11T22:29:55.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[39715]: 3.5 scrub ok 2024-03-11T22:29:55.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[39715]: 3.0 scrub starts 2024-03-11T22:29:55.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[39715]: 3.0 scrub ok 2024-03-11T22:29:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[30178]: 11.15 scrub ok 2024-03-11T22:29:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[30178]: 3.12 scrub starts 2024-03-11T22:29:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[30178]: 3.12 scrub ok 2024-03-11T22:29:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[30178]: osdmap e511: 8 total, 8 up, 8 in 2024-03-11T22:29:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[30178]: 3.5 scrub starts 2024-03-11T22:29:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[30178]: 3.5 scrub ok 2024-03-11T22:29:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[30178]: 3.0 scrub starts 2024-03-11T22:29:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:54 smithi045 ceph-mon[30178]: 3.0 scrub ok 2024-03-11T22:29:55.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:55.066 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:29:55.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:54 smithi144 ceph-mon[33984]: 11.15 scrub ok 2024-03-11T22:29:55.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:54 smithi144 ceph-mon[33984]: 3.12 scrub starts 2024-03-11T22:29:55.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:54 smithi144 ceph-mon[33984]: 3.12 scrub ok 2024-03-11T22:29:55.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:54 smithi144 ceph-mon[33984]: osdmap e511: 8 total, 8 up, 8 in 2024-03-11T22:29:55.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:54 smithi144 ceph-mon[33984]: 3.5 scrub starts 2024-03-11T22:29:55.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:54 smithi144 ceph-mon[33984]: 3.5 scrub ok 2024-03-11T22:29:55.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:54 smithi144 ceph-mon[33984]: 3.0 scrub starts 2024-03-11T22:29:55.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:54 smithi144 ceph-mon[33984]: 3.0 scrub ok 2024-03-11T22:29:55.101 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:55.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:55.106 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:29:55.147 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:29:55.189 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:29:55.194 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:29:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:29:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:29:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:29:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:29:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:29:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:29:55.195 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:29:55.206 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:55.241 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:55.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:55.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:55.401 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:55.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:55.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:55.503 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:55.538 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:55.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:55.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:55.640 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:55.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:55.675 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:55.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:55.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:55.815 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:55.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:55.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:55.916 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:55.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:55.957 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:55.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:56.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[39715]: 5.9 scrub starts 2024-03-11T22:29:56.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[39715]: 5.9 scrub ok 2024-03-11T22:29:56.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[39715]: pgmap v1048: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 960 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1328/46647 objects misplaced (2.847%) 2024-03-11T22:29:56.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[39715]: 3.9 scrub starts 2024-03-11T22:29:56.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[39715]: 3.9 scrub ok 2024-03-11T22:29:56.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1428427659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:29:56.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[39715]: 5.1d scrub starts 2024-03-11T22:29:56.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[39715]: 5.1d scrub ok 2024-03-11T22:29:56.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[30178]: 5.9 scrub starts 2024-03-11T22:29:56.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[30178]: 5.9 scrub ok 2024-03-11T22:29:56.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[30178]: pgmap v1048: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 960 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1328/46647 objects misplaced (2.847%) 2024-03-11T22:29:56.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[30178]: 3.9 scrub starts 2024-03-11T22:29:56.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[30178]: 3.9 scrub ok 2024-03-11T22:29:56.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1428427659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:29:56.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[30178]: 5.1d scrub starts 2024-03-11T22:29:56.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:55 smithi045 ceph-mon[30178]: 5.1d scrub ok 2024-03-11T22:29:56.058 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:56.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:55 smithi144 ceph-mon[33984]: 5.9 scrub starts 2024-03-11T22:29:56.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:55 smithi144 ceph-mon[33984]: 5.9 scrub ok 2024-03-11T22:29:56.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:55 smithi144 ceph-mon[33984]: pgmap v1048: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 960 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1328/46647 objects misplaced (2.847%) 2024-03-11T22:29:56.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:55 smithi144 ceph-mon[33984]: 3.9 scrub starts 2024-03-11T22:29:56.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:55 smithi144 ceph-mon[33984]: 3.9 scrub ok 2024-03-11T22:29:56.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:55 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1428427659' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:29:56.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:55 smithi144 ceph-mon[33984]: 5.1d scrub starts 2024-03-11T22:29:56.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:55 smithi144 ceph-mon[33984]: 5.1d scrub ok 2024-03-11T22:29:56.103 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:56.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:56.205 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:56.243 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:56.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:56.344 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:29:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:56.384 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:56.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:29:56.485 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:56.521 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:56.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:56.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:56.667 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:56.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:56.769 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:56.802 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:56.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:56.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:56.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:56.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:56.944 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:56.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:57.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:57.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:57.084 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:57.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:57.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:29:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:29:57.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:56 smithi144 ceph-mon[33984]: 1.5 scrub starts 2024-03-11T22:29:57.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:56 smithi144 ceph-mon[33984]: 1.5 scrub ok 2024-03-11T22:29:57.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:56 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:29:57.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:56 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:29:57.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:56 smithi144 ceph-mon[33984]: osdmap e512: 8 total, 8 up, 8 in 2024-03-11T22:29:57.185 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:57.219 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:57.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:57.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:56 smithi045 ceph-mon[30178]: 1.5 scrub starts 2024-03-11T22:29:57.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:56 smithi045 ceph-mon[30178]: 1.5 scrub ok 2024-03-11T22:29:57.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:56 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:29:57.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:56 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:29:57.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:56 smithi045 ceph-mon[30178]: osdmap e512: 8 total, 8 up, 8 in 2024-03-11T22:29:57.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:56 smithi045 ceph-mon[39715]: 1.5 scrub starts 2024-03-11T22:29:57.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:56 smithi045 ceph-mon[39715]: 1.5 scrub ok 2024-03-11T22:29:57.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:56 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:29:57.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:56 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:29:57.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:56 smithi045 ceph-mon[39715]: osdmap e512: 8 total, 8 up, 8 in 2024-03-11T22:29:57.321 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:57.357 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:57.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:57.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:57.459 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:57.496 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:57.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:57.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:57.599 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:57.639 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:57.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:57.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:57.741 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:57.774 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:57.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:57.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:57.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:57.915 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:57.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:58.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:57 smithi045 ceph-mon[39715]: pgmap v1049: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 763 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1304/37206 objects misplaced (3.505%) 2024-03-11T22:29:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:57 smithi045 ceph-mon[39715]: 5.8 scrub starts 2024-03-11T22:29:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:57 smithi045 ceph-mon[39715]: 5.8 scrub ok 2024-03-11T22:29:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:57 smithi045 ceph-mon[39715]: 5.e scrub starts 2024-03-11T22:29:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:57 smithi045 ceph-mon[39715]: 5.e scrub ok 2024-03-11T22:29:58.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:57 smithi045 ceph-mon[30178]: pgmap v1049: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 763 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1304/37206 objects misplaced (3.505%) 2024-03-11T22:29:58.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:57 smithi045 ceph-mon[30178]: 5.8 scrub starts 2024-03-11T22:29:58.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:57 smithi045 ceph-mon[30178]: 5.8 scrub ok 2024-03-11T22:29:58.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:58.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:57 smithi045 ceph-mon[30178]: 5.e scrub starts 2024-03-11T22:29:58.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:57 smithi045 ceph-mon[30178]: 5.e scrub ok 2024-03-11T22:29:58.058 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:58.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:58.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:58.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:57 smithi144 ceph-mon[33984]: pgmap v1049: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 763 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1304/37206 objects misplaced (3.505%) 2024-03-11T22:29:58.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:57 smithi144 ceph-mon[33984]: 5.8 scrub starts 2024-03-11T22:29:58.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:57 smithi144 ceph-mon[33984]: 5.8 scrub ok 2024-03-11T22:29:58.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:29:58.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:57 smithi144 ceph-mon[33984]: 5.e scrub starts 2024-03-11T22:29:58.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:57 smithi144 ceph-mon[33984]: 5.e scrub ok 2024-03-11T22:29:58.160 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:58.202 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:58.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:58.304 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:58.340 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:58.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:58.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:58.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:58.478 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:58.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:58.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:58.617 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:58.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:58.718 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:58.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:58.757 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:58.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:58.858 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:58.902 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:58.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:58.937 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:58 smithi045 ceph-mon[39715]: 4.18 scrub starts 2024-03-11T22:29:58.937 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:58 smithi045 ceph-mon[39715]: 4.18 scrub ok 2024-03-11T22:29:58.938 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:58 smithi045 ceph-mon[30178]: 4.18 scrub starts 2024-03-11T22:29:58.938 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:58 smithi045 ceph-mon[30178]: 4.18 scrub ok 2024-03-11T22:29:59.004 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:59.035 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:59.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:59.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:29:59.049 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:29:59.050 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:29:59.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:58 smithi144 ceph-mon[33984]: 4.18 scrub starts 2024-03-11T22:29:59.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:58 smithi144 ceph-mon[33984]: 4.18 scrub ok 2024-03-11T22:29:59.137 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:29:59.170 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:59.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:29:59.272 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:29:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:59.316 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:59.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:29:59.417 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:29:59.454 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:59.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:59.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:29:59.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:29:59.593 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:59.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:59.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:29:59.695 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:29:59.731 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:29:59.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:59.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:29:59.815 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T22:29:59.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:29:59.861 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:59.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:29:59.963 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:29:59.992 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:29:59.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:29:59.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:00.094 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:00.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:29:59 smithi144 ceph-mon[33984]: pgmap v1051: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 572 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 781/28041 objects misplaced (2.785%) 2024-03-11T22:30:00.131 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:00.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:00.139 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:29:59 smithi045 ceph-mon[39715]: pgmap v1051: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 572 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 781/28041 objects misplaced (2.785%) 2024-03-11T22:30:00.141 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:29:59 smithi045 ceph-mon[30178]: pgmap v1051: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 572 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 781/28041 objects misplaced (2.785%) 2024-03-11T22:30:00.233 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:00.272 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:00.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:00.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:00.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:00.415 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:00.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:00.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:00.516 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:00.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:00.558 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:00.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:00.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:00.703 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:00.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:00.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:00.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:00.836 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:30:00.847 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:00.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:00.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:00.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:00.987 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:00.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:01.028 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T22:30:01.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T22:30:01.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[30178]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-11T22:30:01.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[30178]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-11T22:30:01.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[30178]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-11T22:30:01.068 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[30178]: pg 11.0 is active+remapped+backfill_toofull, acting [3,7,6] 2024-03-11T22:30:01.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[30178]: pg 11.10 is active+remapped+backfill_toofull, acting [3,6,7] 2024-03-11T22:30:01.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[30178]: pg 11.13 is active+remapped+backfill_toofull, acting [5,6,7] 2024-03-11T22:30:01.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1448586833' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:30:01.069 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:30:01.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[39715]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-11T22:30:01.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[39715]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-11T22:30:01.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[39715]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-11T22:30:01.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[39715]: pg 11.0 is active+remapped+backfill_toofull, acting [3,7,6] 2024-03-11T22:30:01.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[39715]: pg 11.10 is active+remapped+backfill_toofull, acting [3,6,7] 2024-03-11T22:30:01.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[39715]: pg 11.13 is active+remapped+backfill_toofull, acting [5,6,7] 2024-03-11T22:30:01.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1448586833' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:30:01.069 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:00 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:30:01.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:01.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:00 smithi144 ceph-mon[33984]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-11T22:30:01.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:00 smithi144 ceph-mon[33984]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-11T22:30:01.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:00 smithi144 ceph-mon[33984]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-11T22:30:01.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:00 smithi144 ceph-mon[33984]: pg 11.0 is active+remapped+backfill_toofull, acting [3,7,6] 2024-03-11T22:30:01.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:00 smithi144 ceph-mon[33984]: pg 11.10 is active+remapped+backfill_toofull, acting [3,6,7] 2024-03-11T22:30:01.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:00 smithi144 ceph-mon[33984]: pg 11.13 is active+remapped+backfill_toofull, acting [5,6,7] 2024-03-11T22:30:01.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:00 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1448586833' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:30:01.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:00 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:30:01.138 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:01.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:01.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:01.239 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:01.269 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:01.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:01.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:01.371 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:01.407 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:01.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:01.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:01.511 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:01.544 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:01.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:01.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:01.698 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:01.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:01.800 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:01.844 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:01.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:01.945 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:01.981 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:01.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:02.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[39715]: pgmap v1052: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 572 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 781/28041 objects misplaced (2.785%) 2024-03-11T22:30:02.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:30:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:30:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[39715]: osdmap e513: 8 total, 8 up, 8 in 2024-03-11T22:30:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:30:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[39715]: osdmap e514: 8 total, 8 up, 8 in 2024-03-11T22:30:02.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[30178]: pgmap v1052: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 572 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 781/28041 objects misplaced (2.785%) 2024-03-11T22:30:02.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:30:02.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:30:02.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[30178]: osdmap e513: 8 total, 8 up, 8 in 2024-03-11T22:30:02.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:30:02.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:02.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:01 smithi045 ceph-mon[30178]: osdmap e514: 8 total, 8 up, 8 in 2024-03-11T22:30:02.084 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:02.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:01 smithi144 ceph-mon[33984]: pgmap v1052: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 572 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 781/28041 objects misplaced (2.785%) 2024-03-11T22:30:02.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:01 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T22:30:02.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:01 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:30:02.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:01 smithi144 ceph-mon[33984]: osdmap e513: 8 total, 8 up, 8 in 2024-03-11T22:30:02.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:30:02.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:02.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:01 smithi144 ceph-mon[33984]: osdmap e514: 8 total, 8 up, 8 in 2024-03-11T22:30:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:02.123 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:02.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:02.224 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:02.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:02.257 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:02.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:02.358 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:02.399 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:02.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:02.501 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:02.533 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:02.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:02.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:02.663 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:02.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:02.764 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:02.794 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:02.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:02.828 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:30:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:30:02.895 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:02.940 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:02.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:02.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:03.042 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:03.076 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:03.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:03.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:03.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:02 smithi144 ceph-mon[33984]: 4.16 scrub starts 2024-03-11T22:30:03.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:02 smithi144 ceph-mon[33984]: 4.16 scrub ok 2024-03-11T22:30:03.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:02 smithi144 ceph-mon[33984]: 3.7 scrub starts 2024-03-11T22:30:03.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:02 smithi144 ceph-mon[33984]: 3.7 scrub ok 2024-03-11T22:30:03.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:02 smithi144 ceph-mon[33984]: 6.12 scrub starts 2024-03-11T22:30:03.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:02 smithi144 ceph-mon[33984]: 6.12 scrub ok 2024-03-11T22:30:03.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:02 smithi144 ceph-mon[33984]: 4.2 scrub starts 2024-03-11T22:30:03.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[30178]: 4.16 scrub starts 2024-03-11T22:30:03.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[30178]: 4.16 scrub ok 2024-03-11T22:30:03.103 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[30178]: 3.7 scrub starts 2024-03-11T22:30:03.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[30178]: 3.7 scrub ok 2024-03-11T22:30:03.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[30178]: 6.12 scrub starts 2024-03-11T22:30:03.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[30178]: 6.12 scrub ok 2024-03-11T22:30:03.104 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[30178]: 4.2 scrub starts 2024-03-11T22:30:03.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[39715]: 4.16 scrub starts 2024-03-11T22:30:03.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[39715]: 4.16 scrub ok 2024-03-11T22:30:03.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[39715]: 3.7 scrub starts 2024-03-11T22:30:03.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[39715]: 3.7 scrub ok 2024-03-11T22:30:03.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[39715]: 6.12 scrub starts 2024-03-11T22:30:03.104 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[39715]: 6.12 scrub ok 2024-03-11T22:30:03.105 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:02 smithi045 ceph-mon[39715]: 4.2 scrub starts 2024-03-11T22:30:03.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T22:30:03.178 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:03.207 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:03.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:03.309 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:03.355 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:03.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:03.457 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:03.490 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:03.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:03.592 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:03.623 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:03.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:03.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:03.725 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:03.764 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:03.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:03.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:03.866 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:03.921 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:03.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:03.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:03.962 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:04.017 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:04.023 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:04.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:04.079 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:04.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:04.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:03 smithi144 ceph-mon[33984]: pgmap v1055: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 201 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.50k op/s; 242/10230 objects misplaced (2.366%) 2024-03-11T22:30:04.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:03 smithi144 ceph-mon[33984]: 4.2 scrub ok 2024-03-11T22:30:04.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:03 smithi144 ceph-mon[33984]: 4.1b scrub starts 2024-03-11T22:30:04.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:03 smithi144 ceph-mon[33984]: 4.1b scrub ok 2024-03-11T22:30:04.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:03 smithi144 ceph-mon[33984]: 5.16 scrub starts 2024-03-11T22:30:04.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:03 smithi045 ceph-mon[39715]: pgmap v1055: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 201 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.50k op/s; 242/10230 objects misplaced (2.366%) 2024-03-11T22:30:04.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:03 smithi045 ceph-mon[39715]: 4.2 scrub ok 2024-03-11T22:30:04.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:03 smithi045 ceph-mon[39715]: 4.1b scrub starts 2024-03-11T22:30:04.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:03 smithi045 ceph-mon[39715]: 4.1b scrub ok 2024-03-11T22:30:04.126 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:03 smithi045 ceph-mon[39715]: 5.16 scrub starts 2024-03-11T22:30:04.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:03 smithi045 ceph-mon[30178]: pgmap v1055: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 201 MiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.50k op/s; 242/10230 objects misplaced (2.366%) 2024-03-11T22:30:04.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:03 smithi045 ceph-mon[30178]: 4.2 scrub ok 2024-03-11T22:30:04.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:03 smithi045 ceph-mon[30178]: 4.1b scrub starts 2024-03-11T22:30:04.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:03 smithi045 ceph-mon[30178]: 4.1b scrub ok 2024-03-11T22:30:04.127 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:03 smithi045 ceph-mon[30178]: 5.16 scrub starts 2024-03-11T22:30:04.158 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Removed 97328 objects 2024-03-11T22:30:04.158 INFO:tasks.radosbench.radosbench.0.smithi045.stdout:Clean up completed and total clean up time :96.9666 2024-03-11T22:30:04.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:04.207 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:04.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:04.258 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-11T22:30:04.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-11T22:30:04.308 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:04.334 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:04.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:04.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:04.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:04.461 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:04.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:04.562 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:04.589 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:04.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:04.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:04.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:04.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:04.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:04.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:04.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:04.841 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:04.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:04.942 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:04.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:04.967 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:04.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:05.068 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:05.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:05.097 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:05.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:05.099 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:04 smithi144 ceph-mon[33984]: 5.16 scrub ok 2024-03-11T22:30:05.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:04 smithi144 ceph-mon[33984]: 5.11 scrub starts 2024-03-11T22:30:05.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:04 smithi144 ceph-mon[33984]: 5.11 scrub ok 2024-03-11T22:30:05.198 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:05.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:05.225 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:05.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:05.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:04 smithi045 ceph-mon[39715]: 5.16 scrub ok 2024-03-11T22:30:05.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:04 smithi045 ceph-mon[39715]: 5.11 scrub starts 2024-03-11T22:30:05.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:04 smithi045 ceph-mon[39715]: 5.11 scrub ok 2024-03-11T22:30:05.259 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:04 smithi045 ceph-mon[30178]: 5.16 scrub ok 2024-03-11T22:30:05.259 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:04 smithi045 ceph-mon[30178]: 5.11 scrub starts 2024-03-11T22:30:05.259 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:04 smithi045 ceph-mon[30178]: 5.11 scrub ok 2024-03-11T22:30:05.326 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:05.351 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:05.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:05.453 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:05.479 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:05.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:05.575 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:05.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:05.609 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:05.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:05.711 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:05.736 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:05.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:05.838 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:05.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:05.863 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:05.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:05.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T22:30:05.964 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:05.988 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:05.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:06.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[39715]: 4.a scrub starts 2024-03-11T22:30:06.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[39715]: 4.a scrub ok 2024-03-11T22:30:06.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[39715]: 3.1f scrub starts 2024-03-11T22:30:06.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[39715]: 3.1f scrub ok 2024-03-11T22:30:06.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[39715]: pgmap v1056: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 178 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.19k op/s; 242/9093 objects misplaced (2.661%) 2024-03-11T22:30:06.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[39715]: 4.d scrub starts 2024-03-11T22:30:06.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[39715]: 4.d scrub ok 2024-03-11T22:30:06.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[39715]: 4.c scrub starts 2024-03-11T22:30:06.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[39715]: 4.c scrub ok 2024-03-11T22:30:06.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[30178]: 4.a scrub starts 2024-03-11T22:30:06.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[30178]: 4.a scrub ok 2024-03-11T22:30:06.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[30178]: 3.1f scrub starts 2024-03-11T22:30:06.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[30178]: 3.1f scrub ok 2024-03-11T22:30:06.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[30178]: pgmap v1056: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 178 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.19k op/s; 242/9093 objects misplaced (2.661%) 2024-03-11T22:30:06.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[30178]: 4.d scrub starts 2024-03-11T22:30:06.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[30178]: 4.d scrub ok 2024-03-11T22:30:06.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[30178]: 4.c scrub starts 2024-03-11T22:30:06.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:05 smithi045 ceph-mon[30178]: 4.c scrub ok 2024-03-11T22:30:06.090 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:06.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:05 smithi144 ceph-mon[33984]: 4.a scrub starts 2024-03-11T22:30:06.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:05 smithi144 ceph-mon[33984]: 4.a scrub ok 2024-03-11T22:30:06.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:05 smithi144 ceph-mon[33984]: 3.1f scrub starts 2024-03-11T22:30:06.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:05 smithi144 ceph-mon[33984]: 3.1f scrub ok 2024-03-11T22:30:06.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:05 smithi144 ceph-mon[33984]: pgmap v1056: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 178 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.19k op/s; 242/9093 objects misplaced (2.661%) 2024-03-11T22:30:06.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:05 smithi144 ceph-mon[33984]: 4.d scrub starts 2024-03-11T22:30:06.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:05 smithi144 ceph-mon[33984]: 4.d scrub ok 2024-03-11T22:30:06.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:05 smithi144 ceph-mon[33984]: 4.c scrub starts 2024-03-11T22:30:06.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:05 smithi144 ceph-mon[33984]: 4.c scrub ok 2024-03-11T22:30:06.114 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:06.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:06.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:06.216 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:06.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:06.246 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:06.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:06.347 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:06.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:06.373 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:06.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:06.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:06.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:06.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:06.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:06.600 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:06.625 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:06.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:06.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:06.727 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:06.750 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:06.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:06.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:06.852 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:06.875 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:06.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:06.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:06.977 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:07.005 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:07.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:07.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:07.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:30:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:30:07.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:06 smithi144 ceph-mon[33984]: 4.5 deep-scrub starts 2024-03-11T22:30:07.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:06 smithi144 ceph-mon[33984]: 4.5 deep-scrub ok 2024-03-11T22:30:07.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:06 smithi144 ceph-mon[33984]: 5.5 deep-scrub starts 2024-03-11T22:30:07.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:06 smithi144 ceph-mon[33984]: 5.5 deep-scrub ok 2024-03-11T22:30:07.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:06 smithi144 ceph-mon[33984]: 4.7 scrub starts 2024-03-11T22:30:07.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:06 smithi144 ceph-mon[33984]: 4.7 scrub ok 2024-03-11T22:30:07.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:06 smithi144 ceph-mon[33984]: 6.5 scrub starts 2024-03-11T22:30:07.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:06 smithi144 ceph-mon[33984]: osdmap e515: 8 total, 8 up, 8 in 2024-03-11T22:30:07.107 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:07.131 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:07.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:07.233 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:07.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:07.261 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:07.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:07.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[39715]: 4.5 deep-scrub starts 2024-03-11T22:30:07.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[39715]: 4.5 deep-scrub ok 2024-03-11T22:30:07.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[39715]: 5.5 deep-scrub starts 2024-03-11T22:30:07.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[39715]: 5.5 deep-scrub ok 2024-03-11T22:30:07.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[39715]: 4.7 scrub starts 2024-03-11T22:30:07.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[39715]: 4.7 scrub ok 2024-03-11T22:30:07.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[39715]: 6.5 scrub starts 2024-03-11T22:30:07.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[39715]: osdmap e515: 8 total, 8 up, 8 in 2024-03-11T22:30:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[30178]: 4.5 deep-scrub starts 2024-03-11T22:30:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[30178]: 4.5 deep-scrub ok 2024-03-11T22:30:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[30178]: 5.5 deep-scrub starts 2024-03-11T22:30:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[30178]: 5.5 deep-scrub ok 2024-03-11T22:30:07.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[30178]: 4.7 scrub starts 2024-03-11T22:30:07.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[30178]: 4.7 scrub ok 2024-03-11T22:30:07.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[30178]: 6.5 scrub starts 2024-03-11T22:30:07.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:06 smithi045 ceph-mon[30178]: osdmap e515: 8 total, 8 up, 8 in 2024-03-11T22:30:07.362 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:07.387 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:07.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:07.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:07.489 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:07.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:07.515 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:07.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:07.616 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:07.642 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:07.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:07.744 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:07.768 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:07.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:07.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:07.895 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:07.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:07.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:07.997 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:08.022 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:08.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:08.064 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:08.092 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[30178]: 6.5 scrub ok 2024-03-11T22:30:08.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[30178]: 1.0 deep-scrub starts 2024-03-11T22:30:08.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[30178]: 1.0 deep-scrub ok 2024-03-11T22:30:08.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[30178]: pgmap v1057: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 114 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 918 op/s; 238/6048 objects misplaced (3.935%) 2024-03-11T22:30:08.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[30178]: 6.1d deep-scrub starts 2024-03-11T22:30:08.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[30178]: 6.1d deep-scrub ok 2024-03-11T22:30:08.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:08.093 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[30178]: osdmap e516: 8 total, 8 up, 8 in 2024-03-11T22:30:08.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[39715]: 6.5 scrub ok 2024-03-11T22:30:08.093 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[39715]: 1.0 deep-scrub starts 2024-03-11T22:30:08.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[39715]: 1.0 deep-scrub ok 2024-03-11T22:30:08.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[39715]: pgmap v1057: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 114 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 918 op/s; 238/6048 objects misplaced (3.935%) 2024-03-11T22:30:08.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[39715]: 6.1d deep-scrub starts 2024-03-11T22:30:08.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[39715]: 6.1d deep-scrub ok 2024-03-11T22:30:08.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:08.094 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:07 smithi045 ceph-mon[39715]: osdmap e516: 8 total, 8 up, 8 in 2024-03-11T22:30:08.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:07 smithi144 ceph-mon[33984]: 6.5 scrub ok 2024-03-11T22:30:08.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:07 smithi144 ceph-mon[33984]: 1.0 deep-scrub starts 2024-03-11T22:30:08.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:07 smithi144 ceph-mon[33984]: 1.0 deep-scrub ok 2024-03-11T22:30:08.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:07 smithi144 ceph-mon[33984]: pgmap v1057: 168 pgs: 3 active+remapped+backfill_toofull, 165 active+clean; 114 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 918 op/s; 238/6048 objects misplaced (3.935%) 2024-03-11T22:30:08.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:07 smithi144 ceph-mon[33984]: 6.1d deep-scrub starts 2024-03-11T22:30:08.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:07 smithi144 ceph-mon[33984]: 6.1d deep-scrub ok 2024-03-11T22:30:08.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:08.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:07 smithi144 ceph-mon[33984]: osdmap e516: 8 total, 8 up, 8 in 2024-03-11T22:30:08.124 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:08.147 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:08.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:08.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:08.249 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:08.278 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:08.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:08.379 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:08.406 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:08.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:08.508 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:08.537 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:08.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:08.638 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:08.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:08.666 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:08.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:08.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:08.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:08.796 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:08.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:08.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:08.923 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:08.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:08.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:09.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:09.050 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:09.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:09.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:09.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:09.176 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:09.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:09.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:09.278 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:09.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:08 smithi045 ceph-mon[30178]: 5.6 scrub starts 2024-03-11T22:30:09.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:08 smithi045 ceph-mon[30178]: 5.6 scrub ok 2024-03-11T22:30:09.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:08 smithi045 ceph-mon[30178]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:30:09.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:08 smithi045 ceph-mon[39715]: 5.6 scrub starts 2024-03-11T22:30:09.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:08 smithi045 ceph-mon[39715]: 5.6 scrub ok 2024-03-11T22:30:09.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:08 smithi045 ceph-mon[39715]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:30:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:09.306 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:09.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:09.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:08 smithi144 ceph-mon[33984]: 5.6 scrub starts 2024-03-11T22:30:09.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:08 smithi144 ceph-mon[33984]: 5.6 scrub ok 2024-03-11T22:30:09.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:08 smithi144 ceph-mon[33984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T22:30:09.407 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:09.434 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:09.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:09.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:09.561 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:09.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:09.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:09.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:09.690 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:09.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:09.791 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:09.819 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:09.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:09.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:09.922 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:09.947 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:09.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:09.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:10.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[30178]: 5.15 scrub starts 2024-03-11T22:30:10.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[30178]: 5.15 scrub ok 2024-03-11T22:30:10.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[30178]: 3.1 scrub starts 2024-03-11T22:30:10.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[30178]: 3.1 scrub ok 2024-03-11T22:30:10.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[30178]: pgmap v1060: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 870 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:10.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[30178]: 3.1a deep-scrub starts 2024-03-11T22:30:10.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[30178]: 3.1a deep-scrub ok 2024-03-11T22:30:10.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[39715]: 5.15 scrub starts 2024-03-11T22:30:10.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[39715]: 5.15 scrub ok 2024-03-11T22:30:10.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[39715]: 3.1 scrub starts 2024-03-11T22:30:10.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[39715]: 3.1 scrub ok 2024-03-11T22:30:10.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[39715]: pgmap v1060: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 870 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:10.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[39715]: 3.1a deep-scrub starts 2024-03-11T22:30:10.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:09 smithi045 ceph-mon[39715]: 3.1a deep-scrub ok 2024-03-11T22:30:10.050 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:10.077 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:10.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:10.179 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:10.207 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:10.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:10.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:10.309 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:10.336 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:10.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:10.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:09 smithi144 ceph-mon[33984]: 5.15 scrub starts 2024-03-11T22:30:10.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:09 smithi144 ceph-mon[33984]: 5.15 scrub ok 2024-03-11T22:30:10.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:09 smithi144 ceph-mon[33984]: 3.1 scrub starts 2024-03-11T22:30:10.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:09 smithi144 ceph-mon[33984]: 3.1 scrub ok 2024-03-11T22:30:10.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:09 smithi144 ceph-mon[33984]: pgmap v1060: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 870 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:10.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:09 smithi144 ceph-mon[33984]: 3.1a deep-scrub starts 2024-03-11T22:30:10.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:09 smithi144 ceph-mon[33984]: 3.1a deep-scrub ok 2024-03-11T22:30:10.438 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:10.465 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:10.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:10.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:10.592 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:10.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:10.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:10.694 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:10.721 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:10.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:10.822 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:10.847 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:10.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:10.949 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:10.975 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:10.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:11.076 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:11.100 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:11.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:11.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:11.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:10 smithi045 ceph-mon[39715]: 3.b deep-scrub starts 2024-03-11T22:30:11.149 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:10 smithi045 ceph-mon[39715]: 3.b deep-scrub ok 2024-03-11T22:30:11.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:10 smithi045 ceph-mon[39715]: 6.f scrub starts 2024-03-11T22:30:11.150 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:10 smithi045 ceph-mon[39715]: 6.f scrub ok 2024-03-11T22:30:11.150 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:10 smithi045 ceph-mon[30178]: 3.b deep-scrub starts 2024-03-11T22:30:11.150 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:10 smithi045 ceph-mon[30178]: 3.b deep-scrub ok 2024-03-11T22:30:11.150 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:10 smithi045 ceph-mon[30178]: 6.f scrub starts 2024-03-11T22:30:11.150 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:10 smithi045 ceph-mon[30178]: 6.f scrub ok 2024-03-11T22:30:11.203 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:11.228 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:11.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:11.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:11.330 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:11.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:10 smithi144 ceph-mon[33984]: 3.b deep-scrub starts 2024-03-11T22:30:11.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:10 smithi144 ceph-mon[33984]: 3.b deep-scrub ok 2024-03-11T22:30:11.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:10 smithi144 ceph-mon[33984]: 6.f scrub starts 2024-03-11T22:30:11.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:10 smithi144 ceph-mon[33984]: 6.f scrub ok 2024-03-11T22:30:11.356 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:11.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:11.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:11.457 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:11.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:11.482 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:11.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:11.583 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:11.610 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:11.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:11.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:11.712 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:11.740 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:11.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:11.841 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:11.870 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:11.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:11.971 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:11.997 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:11.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:12.099 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:12.122 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:12.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:12.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[39715]: 6.1c scrub starts 2024-03-11T22:30:12.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[39715]: 6.1c scrub ok 2024-03-11T22:30:12.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[39715]: 4.e deep-scrub starts 2024-03-11T22:30:12.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[39715]: 4.e deep-scrub ok 2024-03-11T22:30:12.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[39715]: pgmap v1061: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 401 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:12.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[39715]: 3.e scrub starts 2024-03-11T22:30:12.154 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[39715]: 3.e scrub ok 2024-03-11T22:30:12.155 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[30178]: 6.1c scrub starts 2024-03-11T22:30:12.155 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[30178]: 6.1c scrub ok 2024-03-11T22:30:12.155 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[30178]: 4.e deep-scrub starts 2024-03-11T22:30:12.155 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[30178]: 4.e deep-scrub ok 2024-03-11T22:30:12.155 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[30178]: pgmap v1061: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 401 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:12.155 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[30178]: 3.e scrub starts 2024-03-11T22:30:12.155 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:11 smithi045 ceph-mon[30178]: 3.e scrub ok 2024-03-11T22:30:12.225 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:12.253 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:12.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:12.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:12.271 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:30:12.271 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:30:12.271 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:30:12.271 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:30:12.291 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:30:12.312 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:30:12.332 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:30:12.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:11 smithi144 ceph-mon[33984]: 6.1c scrub starts 2024-03-11T22:30:12.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:11 smithi144 ceph-mon[33984]: 6.1c scrub ok 2024-03-11T22:30:12.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:11 smithi144 ceph-mon[33984]: 4.e deep-scrub starts 2024-03-11T22:30:12.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:11 smithi144 ceph-mon[33984]: 4.e deep-scrub ok 2024-03-11T22:30:12.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:11 smithi144 ceph-mon[33984]: pgmap v1061: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 401 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:12.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:11 smithi144 ceph-mon[33984]: 3.e scrub starts 2024-03-11T22:30:12.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:11 smithi144 ceph-mon[33984]: 3.e scrub ok 2024-03-11T22:30:12.353 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:30:12.354 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:30:12.354 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:30:12.354 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:30:12.354 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:30:12.354 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:30:12.354 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:30:12.355 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:30:12.355 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:30:12.355 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:12.379 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:12.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:12.481 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:12.508 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:12.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:12.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:12.610 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:12.636 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:12.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:12.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:12.738 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:12.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:12.764 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:12.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:12.786 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:30:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:30:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:30:12.866 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:12.894 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:12.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:12.909 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T22:30:12.995 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:13.023 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:13.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:13.125 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:13.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:13.153 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:13.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:13.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:13.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:13.279 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:13.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:13.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:12 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2301248273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:30:13.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:12 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2914516927' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:30:13.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:12 smithi045 ceph-mon[39715]: 5.b scrub starts 2024-03-11T22:30:13.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:12 smithi045 ceph-mon[39715]: 5.b scrub ok 2024-03-11T22:30:13.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:12 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2301248273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:30:13.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:12 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2914516927' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:30:13.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:12 smithi045 ceph-mon[30178]: 5.b scrub starts 2024-03-11T22:30:13.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:12 smithi045 ceph-mon[30178]: 5.b scrub ok 2024-03-11T22:30:13.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:12 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2301248273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:30:13.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:12 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2914516927' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:30:13.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:12 smithi144 ceph-mon[33984]: 5.b scrub starts 2024-03-11T22:30:13.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:12 smithi144 ceph-mon[33984]: 5.b scrub ok 2024-03-11T22:30:13.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:13.395 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:30:13.395 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-03-11T22:30:13.395 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-03-11T22:30:13.395 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:30:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:13.405 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:13.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:13.506 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:13.530 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:13.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:13.613 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:13.632 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:13.660 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:13.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:13.762 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:13.789 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:13.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:13.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:13.891 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:13.918 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:13.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:13.994 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:13 smithi045 ceph-mon[30178]: pgmap v1062: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 354 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:13.994 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:13 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2914516927' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:30:13.994 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:13 smithi045 ceph-mon[30178]: 3.15 scrub starts 2024-03-11T22:30:13.994 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:13 smithi045 ceph-mon[30178]: 3.15 scrub ok 2024-03-11T22:30:13.994 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:13 smithi045 ceph-mon[30178]: osdmap e517: 8 total, 8 up, 8 in 2024-03-11T22:30:13.995 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:13 smithi045 ceph-mon[39715]: pgmap v1062: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 354 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:13.995 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:13 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2914516927' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:30:13.995 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:13 smithi045 ceph-mon[39715]: 3.15 scrub starts 2024-03-11T22:30:13.995 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:13 smithi045 ceph-mon[39715]: 3.15 scrub ok 2024-03-11T22:30:13.995 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:13 smithi045 ceph-mon[39715]: osdmap e517: 8 total, 8 up, 8 in 2024-03-11T22:30:14.020 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:14.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:14.045 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:14.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:14.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:14.171 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:14.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:14.272 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:14.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:14.300 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:14.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:14.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:13 smithi144 ceph-mon[33984]: pgmap v1062: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 354 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:14.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:13 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2914516927' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:30:14.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:13 smithi144 ceph-mon[33984]: 3.15 scrub starts 2024-03-11T22:30:14.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:13 smithi144 ceph-mon[33984]: 3.15 scrub ok 2024-03-11T22:30:14.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:13 smithi144 ceph-mon[33984]: osdmap e517: 8 total, 8 up, 8 in 2024-03-11T22:30:14.401 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:14.428 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:14.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:14.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:14.557 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:14.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:14.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:14.683 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:14.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:14.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:14.809 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:14.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:14.909 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:14.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:14.938 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:14.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:15.039 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:15.063 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:15.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:15.165 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:15.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:15.189 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:15.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:15.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:14 smithi045 ceph-mon[39715]: 3.18 scrub starts 2024-03-11T22:30:15.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:14 smithi045 ceph-mon[39715]: 3.18 scrub ok 2024-03-11T22:30:15.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:14 smithi045 ceph-mon[30178]: 3.18 scrub starts 2024-03-11T22:30:15.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:14 smithi045 ceph-mon[30178]: 3.18 scrub ok 2024-03-11T22:30:15.290 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:15.313 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:15.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:15.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:15.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:14 smithi144 ceph-mon[33984]: 3.18 scrub starts 2024-03-11T22:30:15.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:14 smithi144 ceph-mon[33984]: 3.18 scrub ok 2024-03-11T22:30:15.415 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:15.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T22:30:15.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:15.440 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:15.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:15.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:15.565 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:15.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:15.667 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:15.694 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:15.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:15.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:15.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:15.820 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:15.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:15.856 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T22:30:15.921 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:15.949 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:15.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:15.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:16.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:16.075 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:16.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:16.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:16.177 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:16.206 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:16.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:16.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:16.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:15 smithi045 ceph-mon[30178]: pgmap v1064: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 143 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:16.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-11T22:30:16.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-11T22:30:16.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:15 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-11T22:30:16.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:15 smithi045 ceph-mon[30178]: 2.1 scrub starts 2024-03-11T22:30:16.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:15 smithi045 ceph-mon[30178]: 2.1 scrub ok 2024-03-11T22:30:16.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:15 smithi045 ceph-mon[39715]: pgmap v1064: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 143 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:16.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-11T22:30:16.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-11T22:30:16.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:15 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-11T22:30:16.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:15 smithi045 ceph-mon[39715]: 2.1 scrub starts 2024-03-11T22:30:16.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:15 smithi045 ceph-mon[39715]: 2.1 scrub ok 2024-03-11T22:30:16.308 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:16.335 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:16.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:16.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:15 smithi144 ceph-mon[33984]: pgmap v1064: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 143 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:16.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-11T22:30:16.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-11T22:30:16.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:15 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-11T22:30:16.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:15 smithi144 ceph-mon[33984]: 2.1 scrub starts 2024-03-11T22:30:16.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:15 smithi144 ceph-mon[33984]: 2.1 scrub ok 2024-03-11T22:30:16.437 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:16.441 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T22:30:16.446 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:16.465 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:16.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:16.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:16.567 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:16.592 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:16.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:16.694 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:16.718 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:16.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:16.820 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:16.843 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:16.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:16.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:16.945 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:16.970 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:16.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:16.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:16.990 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[39715]: 3.1c scrub starts 2024-03-11T22:30:16.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[39715]: 3.1c scrub ok 2024-03-11T22:30:16.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[39715]: 5.18 scrub starts 2024-03-11T22:30:16.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[39715]: 5.18 scrub ok 2024-03-11T22:30:16.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[39715]: 5.13 scrub starts 2024-03-11T22:30:16.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[39715]: 5.13 scrub ok 2024-03-11T22:30:16.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-11T22:30:16.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-03-11T22:30:16.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-11T22:30:16.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[39715]: osdmap e518: 8 total, 8 up, 8 in 2024-03-11T22:30:16.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:16.991 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:30:16.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[30178]: 3.1c scrub starts 2024-03-11T22:30:16.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[30178]: 3.1c scrub ok 2024-03-11T22:30:16.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[30178]: 5.18 scrub starts 2024-03-11T22:30:16.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[30178]: 5.18 scrub ok 2024-03-11T22:30:16.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[30178]: 5.13 scrub starts 2024-03-11T22:30:16.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[30178]: 5.13 scrub ok 2024-03-11T22:30:16.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-11T22:30:16.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-03-11T22:30:16.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-11T22:30:16.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[30178]: osdmap e518: 8 total, 8 up, 8 in 2024-03-11T22:30:16.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:16.992 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:30:17.072 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:17.096 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:17.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:17.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:17.221 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:17.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:17.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:17.323 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:17.347 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:17.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:17.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:30:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:30:17.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:16 smithi144 ceph-mon[33984]: 3.1c scrub starts 2024-03-11T22:30:17.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:16 smithi144 ceph-mon[33984]: 3.1c scrub ok 2024-03-11T22:30:17.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:16 smithi144 ceph-mon[33984]: 5.18 scrub starts 2024-03-11T22:30:17.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:16 smithi144 ceph-mon[33984]: 5.18 scrub ok 2024-03-11T22:30:17.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:16 smithi144 ceph-mon[33984]: 5.13 scrub starts 2024-03-11T22:30:17.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:16 smithi144 ceph-mon[33984]: 5.13 scrub ok 2024-03-11T22:30:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-11T22:30:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-03-11T22:30:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-11T22:30:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:16 smithi144 ceph-mon[33984]: osdmap e518: 8 total, 8 up, 8 in 2024-03-11T22:30:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:30:17.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:17.471 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:17.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:17.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:17.573 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:17.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:17.597 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:17.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:17.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:17.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:17.723 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:17.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:17.824 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:17.848 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:17.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:17.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:17.950 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:17.975 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:17.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:18.077 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:18.101 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:18.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:18.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:18.203 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:18.215 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[39715]: 5.1b scrub starts 2024-03-11T22:30:18.216 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[39715]: 5.1b scrub ok 2024-03-11T22:30:18.216 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[39715]: pgmap v1066: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:18.216 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[39715]: 5.17 scrub starts 2024-03-11T22:30:18.216 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[39715]: 5.17 scrub ok 2024-03-11T22:30:18.216 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[39715]: osdmap e519: 8 total, 8 up, 8 in 2024-03-11T22:30:18.216 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:18.216 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[39715]: 6.1e deep-scrub starts 2024-03-11T22:30:18.216 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[39715]: 6.1e deep-scrub ok 2024-03-11T22:30:18.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[30178]: 5.1b scrub starts 2024-03-11T22:30:18.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[30178]: 5.1b scrub ok 2024-03-11T22:30:18.216 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[30178]: pgmap v1066: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:18.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[30178]: 5.17 scrub starts 2024-03-11T22:30:18.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[30178]: 5.17 scrub ok 2024-03-11T22:30:18.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[30178]: osdmap e519: 8 total, 8 up, 8 in 2024-03-11T22:30:18.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:18.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[30178]: 6.1e deep-scrub starts 2024-03-11T22:30:18.217 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:17 smithi045 ceph-mon[30178]: 6.1e deep-scrub ok 2024-03-11T22:30:18.228 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:18.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:18.330 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:18.350 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:30:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a[67536]: logger=cleanup t=2024-03-11T22:30:18.095819901Z level=info msg="Completed cleanup jobs" duration=32.100872ms 2024-03-11T22:30:18.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:17 smithi144 ceph-mon[33984]: 5.1b scrub starts 2024-03-11T22:30:18.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:17 smithi144 ceph-mon[33984]: 5.1b scrub ok 2024-03-11T22:30:18.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:17 smithi144 ceph-mon[33984]: pgmap v1066: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 32/642 objects misplaced (4.984%) 2024-03-11T22:30:18.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:17 smithi144 ceph-mon[33984]: 5.17 scrub starts 2024-03-11T22:30:18.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:17 smithi144 ceph-mon[33984]: 5.17 scrub ok 2024-03-11T22:30:18.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:17 smithi144 ceph-mon[33984]: osdmap e519: 8 total, 8 up, 8 in 2024-03-11T22:30:18.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:18.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:17 smithi144 ceph-mon[33984]: 6.1e deep-scrub starts 2024-03-11T22:30:18.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:17 smithi144 ceph-mon[33984]: 6.1e deep-scrub ok 2024-03-11T22:30:18.354 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:18.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:18.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:18.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:18.481 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:18.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:18.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:18.608 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:18.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:18.709 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:18.734 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:18.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:18.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:18.836 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:18.861 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:18.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:18.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:18.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:18.987 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:18.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:19.089 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:19.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:19.114 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:19.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:19.215 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:19.239 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:19.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:19.263 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:30:19.264 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:30:19.264 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[39715]: osdmap e520: 8 total, 8 up, 8 in 2024-03-11T22:30:19.264 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[39715]: 6.17 scrub starts 2024-03-11T22:30:19.264 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[39715]: 6.17 scrub ok 2024-03-11T22:30:19.264 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:19.264 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:30:19.264 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:30:19.264 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:30:19.264 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:30:19.265 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[30178]: osdmap e520: 8 total, 8 up, 8 in 2024-03-11T22:30:19.265 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[30178]: 6.17 scrub starts 2024-03-11T22:30:19.265 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[30178]: 6.17 scrub ok 2024-03-11T22:30:19.265 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:19.265 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:30:19.265 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:18 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:30:19.341 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:19.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:18 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:30:19.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:18 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:30:19.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:18 smithi144 ceph-mon[33984]: osdmap e520: 8 total, 8 up, 8 in 2024-03-11T22:30:19.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:18 smithi144 ceph-mon[33984]: 6.17 scrub starts 2024-03-11T22:30:19.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:18 smithi144 ceph-mon[33984]: 6.17 scrub ok 2024-03-11T22:30:19.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:18 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:30:19.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:18 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:30:19.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:18 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:30:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:19.365 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:19.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:19.466 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:19.489 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:19.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:19.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:19.591 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:19.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:19.616 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:19.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:19.718 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:19.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:19.743 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:19.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:19.844 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:19.871 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:19.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:19.962 INFO:teuthology.orchestra.run.smithi045.stderr:pool 'unique_pool_4' removed 2024-03-11T22:30:19.972 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:19.979 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:19.979 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:19.997 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:19.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:20.099 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:20.124 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:20.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:20.225 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:20.250 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:20.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:20.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:19 smithi045 ceph-mon[39715]: pgmap v1069: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 909 B/s rd, 0 op/s; 32/642 objects misplaced (4.984%); 0 B/s, 0 objects/s recovering 2024-03-11T22:30:20.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:19 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3494993596' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:30:20.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:19 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:30:20.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:19 smithi045 ceph-mon[39715]: 5.d scrub starts 2024-03-11T22:30:20.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:19 smithi045 ceph-mon[39715]: 5.d scrub ok 2024-03-11T22:30:20.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:19 smithi045 ceph-mon[30178]: pgmap v1069: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 909 B/s rd, 0 op/s; 32/642 objects misplaced (4.984%); 0 B/s, 0 objects/s recovering 2024-03-11T22:30:20.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:19 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3494993596' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:30:20.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:19 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:30:20.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:19 smithi045 ceph-mon[30178]: 5.d scrub starts 2024-03-11T22:30:20.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:19 smithi045 ceph-mon[30178]: 5.d scrub ok 2024-03-11T22:30:20.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:19 smithi144 ceph-mon[33984]: pgmap v1069: 168 pgs: 1 peering, 2 active+remapped+backfill_toofull, 165 active+clean; 1.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 909 B/s rd, 0 op/s; 32/642 objects misplaced (4.984%); 0 B/s, 0 objects/s recovering 2024-03-11T22:30:20.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:19 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3494993596' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:30:20.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:19 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T22:30:20.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:19 smithi144 ceph-mon[33984]: 5.d scrub starts 2024-03-11T22:30:20.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:19 smithi144 ceph-mon[33984]: 5.d scrub ok 2024-03-11T22:30:20.352 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:20.377 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:20.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:20.479 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:20.503 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:20.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:20.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:20.605 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:20.629 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:20.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:20.730 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:20.755 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:20.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:20.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:20.882 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:20.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:20.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:20.954 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T22:30:20.985 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:21.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:21.010 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:21.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:21.111 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:21.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:21.135 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:21.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:21.236 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:21.261 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:21.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:21.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:21.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[39715]: 4.12 scrub starts 2024-03-11T22:30:21.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[39715]: 4.12 scrub ok 2024-03-11T22:30:21.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:30:21.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[39715]: osdmap e521: 8 total, 8 up, 8 in 2024-03-11T22:30:21.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:30:21.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:30:21.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[39715]: 6.9 deep-scrub starts 2024-03-11T22:30:21.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/730803280' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:30:21.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:30:21.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[30178]: 4.12 scrub starts 2024-03-11T22:30:21.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[30178]: 4.12 scrub ok 2024-03-11T22:30:21.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:30:21.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[30178]: osdmap e521: 8 total, 8 up, 8 in 2024-03-11T22:30:21.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:30:21.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:30:21.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[30178]: 6.9 deep-scrub starts 2024-03-11T22:30:21.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/730803280' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:30:21.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:20 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:30:21.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:20 smithi144 ceph-mon[33984]: 4.12 scrub starts 2024-03-11T22:30:21.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:20 smithi144 ceph-mon[33984]: 4.12 scrub ok 2024-03-11T22:30:21.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:20 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T22:30:21.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:20 smithi144 ceph-mon[33984]: osdmap e521: 8 total, 8 up, 8 in 2024-03-11T22:30:21.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:20 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:30:21.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:20 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:30:21.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:20 smithi144 ceph-mon[33984]: 6.9 deep-scrub starts 2024-03-11T22:30:21.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:20 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/730803280' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:30:21.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:20 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:30:21.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:21.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:21.388 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:21.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:21.489 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5 2024-03-11T22:30:21.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:21.514 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:21.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.5' 2024-03-11T22:30:21.615 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:21.639 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:21.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:21.740 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:21.765 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:21.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:21.824 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T22:30:21.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:21.892 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:21.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:21.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:21.960 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is set 2024-03-11T22:30:21.994 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6 2024-03-11T22:30:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:22.020 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:22.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.6' 2024-03-11T22:30:22.121 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:22.146 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:22.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:22.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:22.272 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:22.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:22.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: 3.16 scrub starts 2024-03-11T22:30:22.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: 3.16 scrub ok 2024-03-11T22:30:22.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: 6.9 deep-scrub ok 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: pgmap v1071: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 887 B/s rd, 0 op/s 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: 3.19 scrub starts 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: 3.19 scrub ok 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: osdmap e522: 8 total, 8 up, 8 in 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3439648181' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: 6.16 scrub starts 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: 6.16 scrub ok 2024-03-11T22:30:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: 4.1c scrub starts 2024-03-11T22:30:22.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[39715]: 4.1c scrub ok 2024-03-11T22:30:22.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: 3.16 scrub starts 2024-03-11T22:30:22.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: 3.16 scrub ok 2024-03-11T22:30:22.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: 6.9 deep-scrub ok 2024-03-11T22:30:22.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: pgmap v1071: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 887 B/s rd, 0 op/s 2024-03-11T22:30:22.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: 3.19 scrub starts 2024-03-11T22:30:22.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: 3.19 scrub ok 2024-03-11T22:30:22.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:30:22.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:30:22.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-11T22:30:22.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:30:22.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:30:22.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: osdmap e522: 8 total, 8 up, 8 in 2024-03-11T22:30:22.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3439648181' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:30:22.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:30:22.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: 6.16 scrub starts 2024-03-11T22:30:22.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: 6.16 scrub ok 2024-03-11T22:30:22.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: 4.1c scrub starts 2024-03-11T22:30:22.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:21 smithi045 ceph-mon[30178]: 4.1c scrub ok 2024-03-11T22:30:22.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: 3.16 scrub starts 2024-03-11T22:30:22.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: 3.16 scrub ok 2024-03-11T22:30:22.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: 6.9 deep-scrub ok 2024-03-11T22:30:22.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: pgmap v1071: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 887 B/s rd, 0 op/s 2024-03-11T22:30:22.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: 3.19 scrub starts 2024-03-11T22:30:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: 3.19 scrub ok 2024-03-11T22:30:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:30:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:30:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-11T22:30:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:30:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:30:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: osdmap e522: 8 total, 8 up, 8 in 2024-03-11T22:30:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3439648181' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:30:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T22:30:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: 6.16 scrub starts 2024-03-11T22:30:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: 6.16 scrub ok 2024-03-11T22:30:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: 4.1c scrub starts 2024-03-11T22:30:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:21 smithi144 ceph-mon[33984]: 4.1c scrub ok 2024-03-11T22:30:22.373 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:22.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:22.399 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:22.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:22.500 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:22.524 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:22.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:22.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:22.626 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:22.650 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:22.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:22.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:22.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:22.775 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:22.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:22.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:22.900 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:22.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:22.947 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:30:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:30:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:30:23.001 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:23.026 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:23.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:23.127 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7 2024-03-11T22:30:23.155 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:23.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:23.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.7' 2024-03-11T22:30:23.199 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:22 smithi045 ceph-mon[39715]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:30:23.199 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:22 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:30:23.200 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:22 smithi045 ceph-mon[39715]: osdmap e523: 8 total, 8 up, 8 in 2024-03-11T22:30:23.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:22 smithi045 ceph-mon[30178]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:30:23.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:22 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:30:23.200 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:22 smithi045 ceph-mon[30178]: osdmap e523: 8 total, 8 up, 8 in 2024-03-11T22:30:23.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:23.282 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:23.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:23.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:22 smithi144 ceph-mon[33984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T22:30:23.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:22 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T22:30:23.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:22 smithi144 ceph-mon[33984]: osdmap e523: 8 total, 8 up, 8 in 2024-03-11T22:30:23.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:23.408 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:23.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:23.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2 2024-03-11T22:30:23.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:23.533 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:23.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.2' 2024-03-11T22:30:23.634 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3 2024-03-11T22:30:23.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:23.659 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:23.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.3' 2024-03-11T22:30:23.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:23.783 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:23.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:23.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0 2024-03-11T22:30:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:23.910 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:23.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.0' 2024-03-11T22:30:24.011 DEBUG:teuthology.orchestra.run.smithi045:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1 2024-03-11T22:30:24.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:24.035 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: docker: command not found 2024-03-11T22:30:24.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi045 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.1' 2024-03-11T22:30:24.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:23 smithi045 ceph-mon[30178]: pgmap v1074: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 890 B/s rd, 0 op/s 2024-03-11T22:30:24.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:23 smithi045 ceph-mon[30178]: 5.14 scrub starts 2024-03-11T22:30:24.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:23 smithi045 ceph-mon[39715]: pgmap v1074: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 890 B/s rd, 0 op/s 2024-03-11T22:30:24.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:23 smithi045 ceph-mon[39715]: 5.14 scrub starts 2024-03-11T22:30:24.136 DEBUG:teuthology.orchestra.run.smithi144:> sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4 2024-03-11T22:30:24.164 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: docker: command not found 2024-03-11T22:30:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:30:24.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi144 with status 1: 'sudo docker kill -s 1 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd.4' 2024-03-11T22:30:24.197 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-11T22:30:24.209 INFO:tasks.thrashosds:joining thrashosds 2024-03-11T22:30:24.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:23 smithi144 ceph-mon[33984]: pgmap v1074: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 890 B/s rd, 0 op/s 2024-03-11T22:30:24.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:23 smithi144 ceph-mon[33984]: 5.14 scrub starts 2024-03-11T22:30:25.048 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0: {} 2024-03-11T22:30:25.066 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1: {} 2024-03-11T22:30:25.086 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2: {} 2024-03-11T22:30:25.106 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3: {} 2024-03-11T22:30:25.126 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4: {} 2024-03-11T22:30:25.148 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5: {} 2024-03-11T22:30:25.168 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6: {} 2024-03-11T22:30:25.188 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7: {} 2024-03-11T22:30:25.189 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:24 smithi045 ceph-mon[39715]: 5.12 scrub starts 2024-03-11T22:30:25.189 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:24 smithi045 ceph-mon[30178]: 5.12 scrub starts 2024-03-11T22:30:25.189 INFO:teuthology.orchestra.run.smithi045.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T22:30:25.190 INFO:teuthology.orchestra.run.smithi045.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T22:30:25.190 INFO:teuthology.orchestra.run.smithi045.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T22:30:25.190 INFO:teuthology.orchestra.run.smithi045.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T22:30:25.190 INFO:teuthology.orchestra.run.smithi045.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T22:30:25.190 INFO:teuthology.orchestra.run.smithi045.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T22:30:25.190 INFO:teuthology.orchestra.run.smithi045.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T22:30:25.190 INFO:teuthology.orchestra.run.smithi045.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T22:30:25.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:24 smithi144 ceph-mon[33984]: 5.12 scrub starts 2024-03-11T22:30:25.747 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T22:30:26.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:25 smithi045 ceph-mon[39715]: pgmap v1075: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T22:30:26.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:25 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/976092547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:30:26.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:25 smithi045 ceph-mon[30178]: pgmap v1075: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T22:30:26.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:25 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/976092547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:30:26.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:25 smithi144 ceph-mon[33984]: pgmap v1075: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T22:30:26.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:25 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/976092547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T22:30:26.382 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:26.854 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:27.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:30:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:30:27.723 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:30:27.723 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:30:28.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:27 smithi045 ceph-mon[30178]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 776 B/s rd, 0 op/s; 70 B/s, 2 objects/s recovering 2024-03-11T22:30:28.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:27 smithi045 ceph-mon[30178]: osdmap e524: 8 total, 8 up, 8 in 2024-03-11T22:30:28.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:28.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:27 smithi045 ceph-mon[39715]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 776 B/s rd, 0 op/s; 70 B/s, 2 objects/s recovering 2024-03-11T22:30:28.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:27 smithi045 ceph-mon[39715]: osdmap e524: 8 total, 8 up, 8 in 2024-03-11T22:30:28.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:28.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:27 smithi144 ceph-mon[33984]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 776 B/s rd, 0 op/s; 70 B/s, 2 objects/s recovering 2024-03-11T22:30:28.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:27 smithi144 ceph-mon[33984]: osdmap e524: 8 total, 8 up, 8 in 2024-03-11T22:30:28.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:28.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T22:30:29.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:29 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:30:29.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:29 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:30:29.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:29 smithi045 ceph-mon[30178]: pgmap v1078: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-03-11T22:30:29.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:29 smithi045 ceph-mon[30178]: osdmap e525: 8 total, 8 up, 8 in 2024-03-11T22:30:29.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:29 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:30:29.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:29 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:30:29.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:29 smithi045 ceph-mon[39715]: pgmap v1078: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-03-11T22:30:29.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:29 smithi045 ceph-mon[39715]: osdmap e525: 8 total, 8 up, 8 in 2024-03-11T22:30:30.085 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:30.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:29 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:30:30.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:29 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:30:30.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:29 smithi144 ceph-mon[33984]: pgmap v1078: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 14 B/s, 0 objects/s recovering 2024-03-11T22:30:30.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:29 smithi144 ceph-mon[33984]: osdmap e525: 8 total, 8 up, 8 in 2024-03-11T22:30:31.646 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T22:30:31.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[39715]: pgmap v1080: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:30:31.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:30:31.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:31.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3916506871' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:30:31.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:30:31.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:30:31.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[39715]: osdmap e526: 8 total, 8 up, 8 in 2024-03-11T22:30:31.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[30178]: pgmap v1080: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:30:31.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:30:31.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:31.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3916506871' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:30:31.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:30:31.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:30:31.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:31 smithi045 ceph-mon[30178]: osdmap e526: 8 total, 8 up, 8 in 2024-03-11T22:30:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:31 smithi144 ceph-mon[33984]: pgmap v1080: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:30:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:30:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:31 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3916506871' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:30:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:31 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:30:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:31 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:30:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:31 smithi144 ceph-mon[33984]: osdmap e526: 8 total, 8 up, 8 in 2024-03-11T22:30:32.660 INFO:teuthology.orchestra.run.smithi045.stderr:noscrub is unset 2024-03-11T22:30:33.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:32 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3078095328' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:30:33.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:32 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3078095328' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:30:33.037 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:30:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:30:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:30:33.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:32 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3078095328' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T22:30:33.454 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T22:30:33.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:33 smithi045 ceph-mon[30178]: pgmap v1082: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:30:33.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:33 smithi045 ceph-mon[30178]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-11T22:30:33.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:33 smithi045 ceph-mon[30178]: Cluster is now healthy 2024-03-11T22:30:33.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:33 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3078095328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:30:33.682 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:33 smithi045 ceph-mon[30178]: osdmap e527: 8 total, 8 up, 8 in 2024-03-11T22:30:33.682 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:33 smithi045 ceph-mon[39715]: pgmap v1082: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:30:33.682 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:33 smithi045 ceph-mon[39715]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-11T22:30:33.682 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:33 smithi045 ceph-mon[39715]: Cluster is now healthy 2024-03-11T22:30:33.683 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:33 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3078095328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:30:33.683 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:33 smithi045 ceph-mon[39715]: osdmap e527: 8 total, 8 up, 8 in 2024-03-11T22:30:34.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:33 smithi144 ceph-mon[33984]: pgmap v1082: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:30:34.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:33 smithi144 ceph-mon[33984]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-11T22:30:34.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:33 smithi144 ceph-mon[33984]: Cluster is now healthy 2024-03-11T22:30:34.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:33 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3078095328' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T22:30:34.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:33 smithi144 ceph-mon[33984]: osdmap e527: 8 total, 8 up, 8 in 2024-03-11T22:30:34.117 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-03-11T22:30:34.117 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:30:34.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T22:30:35.263 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T22:30:35.336 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:35.376 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:35.603 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:30:35.759 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:35 smithi045 ceph-mon[39715]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:35.760 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:35 smithi045 ceph-mon[39715]: 5.14 deep-scrub starts 2024-03-11T22:30:35.760 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:35 smithi045 ceph-mon[39715]: 5.14 deep-scrub ok 2024-03-11T22:30:35.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:35 smithi045 ceph-mon[30178]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:35.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:35 smithi045 ceph-mon[30178]: 5.14 deep-scrub starts 2024-03-11T22:30:35.760 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:35 smithi045 ceph-mon[30178]: 5.14 deep-scrub ok 2024-03-11T22:30:35.977 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:35 smithi144 ceph-mon[33984]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:35.977 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:35 smithi144 ceph-mon[33984]: 5.14 deep-scrub starts 2024-03-11T22:30:35.977 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:35 smithi144 ceph-mon[33984]: 5.14 deep-scrub ok 2024-03-11T22:30:37.343 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:30:36 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:30:37.721 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:37 smithi045 ceph-mon[39715]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-03-11T22:30:37.721 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:37 smithi045 ceph-mon[39715]: osdmap e528: 8 total, 8 up, 8 in 2024-03-11T22:30:37.721 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:37 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:37.721 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:37 smithi045 ceph-mon[30178]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-03-11T22:30:37.722 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:37 smithi045 ceph-mon[30178]: osdmap e528: 8 total, 8 up, 8 in 2024-03-11T22:30:37.722 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:37 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:38.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:37 smithi144 ceph-mon[33984]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-03-11T22:30:38.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:37 smithi144 ceph-mon[33984]: osdmap e528: 8 total, 8 up, 8 in 2024-03-11T22:30:38.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:37 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:38.198 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:30:38.199 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:30:38.652 INFO:teuthology.orchestra.run.smithi045.stderr:nodeep-scrub is unset 2024-03-11T22:30:38.653 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:38 smithi045 ceph-mon[30178]: 5.3 scrub starts 2024-03-11T22:30:38.653 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:38 smithi045 ceph-mon[30178]: 5.3 scrub ok 2024-03-11T22:30:38.653 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:38 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1052402224' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:30:38.653 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:38 smithi045 ceph-mon[39715]: 5.3 scrub starts 2024-03-11T22:30:38.653 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:38 smithi045 ceph-mon[39715]: 5.3 scrub ok 2024-03-11T22:30:38.653 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:38 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1052402224' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:30:38.755 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:38 smithi144 ceph-mon[33984]: 5.3 scrub starts 2024-03-11T22:30:38.755 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:38 smithi144 ceph-mon[33984]: 5.3 scrub ok 2024-03-11T22:30:38.755 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:38 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1052402224' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T22:30:38.783 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T22:30:39.034 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:30:39.655 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T22:30:39.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:39 smithi144 ceph-mon[33984]: pgmap v1087: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:30:39.851 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:39 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1052402224' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:30:39.851 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:39 smithi144 ceph-mon[33984]: osdmap e529: 8 total, 8 up, 8 in 2024-03-11T22:30:39.851 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:39 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3469142130' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:30:39.851 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:39 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:30:39.978 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:39 smithi045 ceph-mon[39715]: pgmap v1087: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:30:39.978 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:39 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1052402224' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:30:39.978 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:39 smithi045 ceph-mon[39715]: osdmap e529: 8 total, 8 up, 8 in 2024-03-11T22:30:39.978 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:39 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3469142130' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:30:39.978 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:39 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:30:39.978 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:39 smithi045 ceph-mon[30178]: pgmap v1087: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:30:39.979 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:39 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1052402224' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T22:30:39.979 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:39 smithi045 ceph-mon[30178]: osdmap e529: 8 total, 8 up, 8 in 2024-03-11T22:30:39.979 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:39 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3469142130' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:30:39.979 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:39 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:30:40.306 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T22:30:40.553 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:40.731 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:30:40.731 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:30:40.731 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:40 smithi144 ceph-mon[33984]: 6.7 scrub starts 2024-03-11T22:30:40.731 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:40 smithi144 ceph-mon[33984]: 6.7 scrub ok 2024-03-11T22:30:40.731 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:40 smithi144 ceph-mon[33984]: 1.7 deep-scrub starts 2024-03-11T22:30:40.731 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:40 smithi144 ceph-mon[33984]: 1.7 deep-scrub ok 2024-03-11T22:30:40.731 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:40 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:30:40.731 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:40 smithi144 ceph-mon[33984]: osdmap e530: 8 total, 8 up, 8 in 2024-03-11T22:30:40.732 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:40 smithi144 ceph-mon[33984]: 3.1b scrub starts 2024-03-11T22:30:40.732 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:40 smithi144 ceph-mon[33984]: 3.1b scrub ok 2024-03-11T22:30:40.732 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:40 smithi144 ceph-mon[33984]: 5.12 scrub starts 2024-03-11T22:30:40.732 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:40 smithi144 ceph-mon[33984]: 5.12 scrub ok 2024-03-11T22:30:40.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[30178]: 6.7 scrub starts 2024-03-11T22:30:40.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[30178]: 6.7 scrub ok 2024-03-11T22:30:40.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[30178]: 1.7 deep-scrub starts 2024-03-11T22:30:40.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[30178]: 1.7 deep-scrub ok 2024-03-11T22:30:40.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:30:40.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[30178]: osdmap e530: 8 total, 8 up, 8 in 2024-03-11T22:30:40.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[30178]: 3.1b scrub starts 2024-03-11T22:30:40.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[30178]: 3.1b scrub ok 2024-03-11T22:30:40.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[30178]: 5.12 scrub starts 2024-03-11T22:30:40.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[30178]: 5.12 scrub ok 2024-03-11T22:30:41.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[39715]: 6.7 scrub starts 2024-03-11T22:30:41.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[39715]: 6.7 scrub ok 2024-03-11T22:30:41.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[39715]: 1.7 deep-scrub starts 2024-03-11T22:30:41.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[39715]: 1.7 deep-scrub ok 2024-03-11T22:30:41.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:30:41.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[39715]: osdmap e530: 8 total, 8 up, 8 in 2024-03-11T22:30:41.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[39715]: 3.1b scrub starts 2024-03-11T22:30:41.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[39715]: 3.1b scrub ok 2024-03-11T22:30:41.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[39715]: 5.12 scrub starts 2024-03-11T22:30:41.000 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:40 smithi045 ceph-mon[39715]: 5.12 scrub ok 2024-03-11T22:30:41.424 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T22:30:41.660 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: 5.f scrub starts 2024-03-11T22:30:41.671 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:30:41.924 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: 5.f scrub ok 2024-03-11T22:30:41.924 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:30:41.924 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:30:41.924 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: pgmap v1090: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:41.924 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: 4.13 scrub starts 2024-03-11T22:30:41.924 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: 4.13 scrub ok 2024-03-11T22:30:41.925 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: osdmap e531: 8 total, 8 up, 8 in 2024-03-11T22:30:41.925 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: 3.2 scrub starts 2024-03-11T22:30:41.925 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: 3.2 scrub ok 2024-03-11T22:30:41.925 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: 4.1e scrub starts 2024-03-11T22:30:41.925 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: 4.1e scrub ok 2024-03-11T22:30:41.925 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: 5.19 scrub starts 2024-03-11T22:30:41.925 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: 5.19 scrub ok 2024-03-11T22:30:41.925 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:41 smithi144 ceph-mon[33984]: osdmap e532: 8 total, 8 up, 8 in 2024-03-11T22:30:42.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: 5.f scrub starts 2024-03-11T22:30:42.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: 5.f scrub ok 2024-03-11T22:30:42.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:30:42.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:30:42.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: pgmap v1090: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:42.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: 4.13 scrub starts 2024-03-11T22:30:42.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: 4.13 scrub ok 2024-03-11T22:30:42.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: osdmap e531: 8 total, 8 up, 8 in 2024-03-11T22:30:42.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: 3.2 scrub starts 2024-03-11T22:30:42.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: 3.2 scrub ok 2024-03-11T22:30:42.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: 4.1e scrub starts 2024-03-11T22:30:42.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: 4.1e scrub ok 2024-03-11T22:30:42.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: 5.19 scrub starts 2024-03-11T22:30:42.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: 5.19 scrub ok 2024-03-11T22:30:42.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[39715]: osdmap e532: 8 total, 8 up, 8 in 2024-03-11T22:30:42.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: 5.f scrub starts 2024-03-11T22:30:42.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: 5.f scrub ok 2024-03-11T22:30:42.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:30:42.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:30:42.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: pgmap v1090: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:42.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: 4.13 scrub starts 2024-03-11T22:30:42.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: 4.13 scrub ok 2024-03-11T22:30:42.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: osdmap e531: 8 total, 8 up, 8 in 2024-03-11T22:30:42.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: 3.2 scrub starts 2024-03-11T22:30:42.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: 3.2 scrub ok 2024-03-11T22:30:42.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: 4.1e scrub starts 2024-03-11T22:30:42.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: 4.1e scrub ok 2024-03-11T22:30:42.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: 5.19 scrub starts 2024-03-11T22:30:42.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: 5.19 scrub ok 2024-03-11T22:30:42.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:41 smithi045 ceph-mon[30178]: osdmap e532: 8 total, 8 up, 8 in 2024-03-11T22:30:42.665 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:30:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:30:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:30:42.786 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:42 smithi144 ceph-mon[33984]: 5.7 scrub starts 2024-03-11T22:30:42.786 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:42 smithi144 ceph-mon[33984]: 5.7 scrub ok 2024-03-11T22:30:42.786 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:42 smithi144 ceph-mon[33984]: 3.6 scrub starts 2024-03-11T22:30:42.786 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:42 smithi144 ceph-mon[33984]: 3.6 scrub ok 2024-03-11T22:30:43.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:42 smithi045 ceph-mon[30178]: 5.7 scrub starts 2024-03-11T22:30:43.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:42 smithi045 ceph-mon[30178]: 5.7 scrub ok 2024-03-11T22:30:43.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:42 smithi045 ceph-mon[30178]: 3.6 scrub starts 2024-03-11T22:30:43.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:42 smithi045 ceph-mon[30178]: 3.6 scrub ok 2024-03-11T22:30:43.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:42 smithi045 ceph-mon[39715]: 5.7 scrub starts 2024-03-11T22:30:43.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:42 smithi045 ceph-mon[39715]: 5.7 scrub ok 2024-03-11T22:30:43.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:42 smithi045 ceph-mon[39715]: 3.6 scrub starts 2024-03-11T22:30:43.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:42 smithi045 ceph-mon[39715]: 3.6 scrub ok 2024-03-11T22:30:43.367 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:30:43.367 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:30:43.676 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T22:30:43.987 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:43 smithi144 ceph-mon[33984]: pgmap v1093: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:30:43.987 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:43 smithi144 ceph-mon[33984]: 5.1f scrub starts 2024-03-11T22:30:43.987 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:43 smithi144 ceph-mon[33984]: 5.1f scrub ok 2024-03-11T22:30:43.988 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:43 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4132265336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:30:43.988 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:43 smithi144 ceph-mon[33984]: 5.1a scrub starts 2024-03-11T22:30:43.988 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:43 smithi144 ceph-mon[33984]: 5.1a scrub ok 2024-03-11T22:30:44.014 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T22:30:44.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:43 smithi045 ceph-mon[39715]: pgmap v1093: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:30:44.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:43 smithi045 ceph-mon[39715]: 5.1f scrub starts 2024-03-11T22:30:44.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:43 smithi045 ceph-mon[39715]: 5.1f scrub ok 2024-03-11T22:30:44.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:43 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4132265336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:30:44.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:43 smithi045 ceph-mon[39715]: 5.1a scrub starts 2024-03-11T22:30:44.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:43 smithi045 ceph-mon[39715]: 5.1a scrub ok 2024-03-11T22:30:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:43 smithi045 ceph-mon[30178]: pgmap v1093: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:30:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:43 smithi045 ceph-mon[30178]: 5.1f scrub starts 2024-03-11T22:30:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:43 smithi045 ceph-mon[30178]: 5.1f scrub ok 2024-03-11T22:30:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:43 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4132265336' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:30:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:43 smithi045 ceph-mon[30178]: 5.1a scrub starts 2024-03-11T22:30:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:43 smithi045 ceph-mon[30178]: 5.1a scrub ok 2024-03-11T22:30:44.267 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:30:44.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T22:30:44.765 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:44.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:44 smithi045 ceph-mon[39715]: 1.a scrub starts 2024-03-11T22:30:44.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:44 smithi045 ceph-mon[39715]: 1.a scrub ok 2024-03-11T22:30:44.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:44 smithi045 ceph-mon[39715]: 4.1f scrub starts 2024-03-11T22:30:44.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:44 smithi045 ceph-mon[39715]: 4.1f scrub ok 2024-03-11T22:30:44.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:44 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/4132265336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:30:44.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:44 smithi045 ceph-mon[39715]: osdmap e533: 8 total, 8 up, 8 in 2024-03-11T22:30:44.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:44 smithi045 ceph-mon[30178]: 1.a scrub starts 2024-03-11T22:30:44.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:44 smithi045 ceph-mon[30178]: 1.a scrub ok 2024-03-11T22:30:44.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:44 smithi045 ceph-mon[30178]: 4.1f scrub starts 2024-03-11T22:30:44.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:44 smithi045 ceph-mon[30178]: 4.1f scrub ok 2024-03-11T22:30:44.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:44 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/4132265336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:30:44.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:44 smithi045 ceph-mon[30178]: osdmap e533: 8 total, 8 up, 8 in 2024-03-11T22:30:45.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:44 smithi144 ceph-mon[33984]: 1.a scrub starts 2024-03-11T22:30:45.094 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:44 smithi144 ceph-mon[33984]: 1.a scrub ok 2024-03-11T22:30:45.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:44 smithi144 ceph-mon[33984]: 4.1f scrub starts 2024-03-11T22:30:45.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:44 smithi144 ceph-mon[33984]: 4.1f scrub ok 2024-03-11T22:30:45.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:44 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/4132265336' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:30:45.095 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:44 smithi144 ceph-mon[33984]: osdmap e533: 8 total, 8 up, 8 in 2024-03-11T22:30:45.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:45 smithi045 ceph-mon[30178]: pgmap v1095: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:30:45.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:45 smithi045 ceph-mon[30178]: 6.0 scrub starts 2024-03-11T22:30:45.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:45 smithi045 ceph-mon[30178]: 6.0 scrub ok 2024-03-11T22:30:45.911 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:45 smithi045 ceph-mon[39715]: pgmap v1095: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:30:45.911 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:45 smithi045 ceph-mon[39715]: 6.0 scrub starts 2024-03-11T22:30:45.911 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:45 smithi045 ceph-mon[39715]: 6.0 scrub ok 2024-03-11T22:30:45.924 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:30:45.924 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:30:45.924 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:45 smithi144 ceph-mon[33984]: pgmap v1095: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:30:45.924 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:45 smithi144 ceph-mon[33984]: 6.0 scrub starts 2024-03-11T22:30:45.925 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:45 smithi144 ceph-mon[33984]: 6.0 scrub ok 2024-03-11T22:30:46.534 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T22:30:46.776 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:30:46.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:46 smithi144 ceph-mon[33984]: 3.8 scrub starts 2024-03-11T22:30:46.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:46 smithi144 ceph-mon[33984]: 3.8 scrub ok 2024-03-11T22:30:46.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:30:46.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:46.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:46 smithi144 ceph-mon[33984]: osdmap e534: 8 total, 8 up, 8 in 2024-03-11T22:30:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:46 smithi045 ceph-mon[30178]: 3.8 scrub starts 2024-03-11T22:30:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:46 smithi045 ceph-mon[30178]: 3.8 scrub ok 2024-03-11T22:30:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:30:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:46 smithi045 ceph-mon[30178]: osdmap e534: 8 total, 8 up, 8 in 2024-03-11T22:30:47.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:46 smithi045 ceph-mon[39715]: 3.8 scrub starts 2024-03-11T22:30:47.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:46 smithi045 ceph-mon[39715]: 3.8 scrub ok 2024-03-11T22:30:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:30:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:30:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:46 smithi045 ceph-mon[39715]: osdmap e534: 8 total, 8 up, 8 in 2024-03-11T22:30:47.164 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:30:46 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:30:47.694 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T22:30:48.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[30178]: pgmap v1096: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:48.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[30178]: 4.f scrub starts 2024-03-11T22:30:48.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[30178]: 4.f scrub ok 2024-03-11T22:30:48.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[30178]: 3.11 scrub starts 2024-03-11T22:30:48.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[30178]: 3.11 scrub ok 2024-03-11T22:30:48.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[30178]: 5.1c scrub starts 2024-03-11T22:30:48.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[30178]: 5.1c scrub ok 2024-03-11T22:30:48.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/119548892' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:30:48.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:30:48.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:48.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[39715]: pgmap v1096: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:48.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[39715]: 4.f scrub starts 2024-03-11T22:30:48.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[39715]: 4.f scrub ok 2024-03-11T22:30:48.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[39715]: 3.11 scrub starts 2024-03-11T22:30:48.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[39715]: 3.11 scrub ok 2024-03-11T22:30:48.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[39715]: 5.1c scrub starts 2024-03-11T22:30:48.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[39715]: 5.1c scrub ok 2024-03-11T22:30:48.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/119548892' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:30:48.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:30:48.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:47 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:48.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:47 smithi144 ceph-mon[33984]: pgmap v1096: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:48.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:47 smithi144 ceph-mon[33984]: 4.f scrub starts 2024-03-11T22:30:48.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:47 smithi144 ceph-mon[33984]: 4.f scrub ok 2024-03-11T22:30:48.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:47 smithi144 ceph-mon[33984]: 3.11 scrub starts 2024-03-11T22:30:48.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:47 smithi144 ceph-mon[33984]: 3.11 scrub ok 2024-03-11T22:30:48.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:47 smithi144 ceph-mon[33984]: 5.1c scrub starts 2024-03-11T22:30:48.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:47 smithi144 ceph-mon[33984]: 5.1c scrub ok 2024-03-11T22:30:48.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:47 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/119548892' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:30:48.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:47 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:30:48.055 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:47 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:48.441 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:30:48.441 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:30:48.569 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T22:30:48.845 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:48.967 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[39715]: 6.8 deep-scrub starts 2024-03-11T22:30:48.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[39715]: 6.8 deep-scrub ok 2024-03-11T22:30:48.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[39715]: 3.4 scrub starts 2024-03-11T22:30:48.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[39715]: 3.4 scrub ok 2024-03-11T22:30:48.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:30:48.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[39715]: osdmap e535: 8 total, 8 up, 8 in 2024-03-11T22:30:48.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[39715]: 6.10 scrub starts 2024-03-11T22:30:48.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[39715]: 6.10 scrub ok 2024-03-11T22:30:48.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[30178]: 6.8 deep-scrub starts 2024-03-11T22:30:48.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[30178]: 6.8 deep-scrub ok 2024-03-11T22:30:48.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[30178]: 3.4 scrub starts 2024-03-11T22:30:48.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[30178]: 3.4 scrub ok 2024-03-11T22:30:48.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:30:48.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[30178]: osdmap e535: 8 total, 8 up, 8 in 2024-03-11T22:30:48.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[30178]: 6.10 scrub starts 2024-03-11T22:30:48.969 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:48 smithi045 ceph-mon[30178]: 6.10 scrub ok 2024-03-11T22:30:49.062 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:48 smithi144 ceph-mon[33984]: 6.8 deep-scrub starts 2024-03-11T22:30:49.062 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:48 smithi144 ceph-mon[33984]: 6.8 deep-scrub ok 2024-03-11T22:30:49.062 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:48 smithi144 ceph-mon[33984]: 3.4 scrub starts 2024-03-11T22:30:49.062 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:48 smithi144 ceph-mon[33984]: 3.4 scrub ok 2024-03-11T22:30:49.063 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:48 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:30:49.063 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:48 smithi144 ceph-mon[33984]: osdmap e535: 8 total, 8 up, 8 in 2024-03-11T22:30:49.063 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:48 smithi144 ceph-mon[33984]: 6.10 scrub starts 2024-03-11T22:30:49.063 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:48 smithi144 ceph-mon[33984]: 6.10 scrub ok 2024-03-11T22:30:49.090 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T22:30:49.329 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:30:49.729 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:49 smithi144 ceph-mon[33984]: 4.1 scrub starts 2024-03-11T22:30:49.729 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:49 smithi144 ceph-mon[33984]: 4.1 scrub ok 2024-03-11T22:30:49.729 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:49 smithi144 ceph-mon[33984]: pgmap v1099: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:49.729 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:49 smithi144 ceph-mon[33984]: 4.9 scrub starts 2024-03-11T22:30:49.729 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:49 smithi144 ceph-mon[33984]: 4.9 scrub ok 2024-03-11T22:30:50.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:49 smithi045 ceph-mon[39715]: 4.1 scrub starts 2024-03-11T22:30:50.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:49 smithi045 ceph-mon[39715]: 4.1 scrub ok 2024-03-11T22:30:50.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:49 smithi045 ceph-mon[39715]: pgmap v1099: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:50.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:49 smithi045 ceph-mon[39715]: 4.9 scrub starts 2024-03-11T22:30:50.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:49 smithi045 ceph-mon[39715]: 4.9 scrub ok 2024-03-11T22:30:50.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:49 smithi045 ceph-mon[30178]: 4.1 scrub starts 2024-03-11T22:30:50.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:49 smithi045 ceph-mon[30178]: 4.1 scrub ok 2024-03-11T22:30:50.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:49 smithi045 ceph-mon[30178]: pgmap v1099: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:50.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:49 smithi045 ceph-mon[30178]: 4.9 scrub starts 2024-03-11T22:30:50.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:49 smithi045 ceph-mon[30178]: 4.9 scrub ok 2024-03-11T22:30:50.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:50 smithi045 ceph-mon[39715]: 6.14 scrub starts 2024-03-11T22:30:50.786 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:50 smithi045 ceph-mon[39715]: 6.14 scrub ok 2024-03-11T22:30:50.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:50 smithi045 ceph-mon[30178]: 6.14 scrub starts 2024-03-11T22:30:50.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:50 smithi045 ceph-mon[30178]: 6.14 scrub ok 2024-03-11T22:30:50.833 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:50 smithi144 ceph-mon[33984]: 6.14 scrub starts 2024-03-11T22:30:50.833 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:50 smithi144 ceph-mon[33984]: 6.14 scrub ok 2024-03-11T22:30:51.100 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:30:51.101 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:30:51.655 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T22:30:51.949 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:51 smithi144 ceph-mon[33984]: 3.c scrub starts 2024-03-11T22:30:51.949 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:51 smithi144 ceph-mon[33984]: 3.c scrub ok 2024-03-11T22:30:51.949 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:51 smithi144 ceph-mon[33984]: pgmap v1100: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 745 B/s rd, 0 op/s 2024-03-11T22:30:51.950 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:51 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2113629616' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:30:51.950 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:51 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:30:51.950 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:51 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:30:51.950 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:51 smithi144 ceph-mon[33984]: osdmap e536: 8 total, 8 up, 8 in 2024-03-11T22:30:51.975 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T22:30:51.977 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[39715]: 3.c scrub starts 2024-03-11T22:30:51.977 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[39715]: 3.c scrub ok 2024-03-11T22:30:51.977 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[39715]: pgmap v1100: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 745 B/s rd, 0 op/s 2024-03-11T22:30:51.977 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2113629616' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:30:51.977 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:30:51.977 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:30:51.977 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[39715]: osdmap e536: 8 total, 8 up, 8 in 2024-03-11T22:30:51.978 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[30178]: 3.c scrub starts 2024-03-11T22:30:51.978 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[30178]: 3.c scrub ok 2024-03-11T22:30:51.978 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[30178]: pgmap v1100: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 745 B/s rd, 0 op/s 2024-03-11T22:30:51.978 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2113629616' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:30:51.978 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:30:51.978 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:30:51.978 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:51 smithi045 ceph-mon[30178]: osdmap e536: 8 total, 8 up, 8 in 2024-03-11T22:30:52.217 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:30:52.411 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T22:30:52.665 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:52.701 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:30:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:30:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:30:52.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:52 smithi045 ceph-mon[39715]: 6.19 scrub starts 2024-03-11T22:30:52.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:52 smithi045 ceph-mon[39715]: 6.19 scrub ok 2024-03-11T22:30:52.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:52 smithi045 ceph-mon[39715]: 6.4 scrub starts 2024-03-11T22:30:52.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:52 smithi045 ceph-mon[39715]: 6.4 scrub ok 2024-03-11T22:30:52.961 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:52 smithi045 ceph-mon[30178]: 6.19 scrub starts 2024-03-11T22:30:52.961 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:52 smithi045 ceph-mon[30178]: 6.19 scrub ok 2024-03-11T22:30:52.961 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:52 smithi045 ceph-mon[30178]: 6.4 scrub starts 2024-03-11T22:30:52.961 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:52 smithi045 ceph-mon[30178]: 6.4 scrub ok 2024-03-11T22:30:53.089 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:52 smithi144 ceph-mon[33984]: 6.19 scrub starts 2024-03-11T22:30:53.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:52 smithi144 ceph-mon[33984]: 6.19 scrub ok 2024-03-11T22:30:53.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:52 smithi144 ceph-mon[33984]: 6.4 scrub starts 2024-03-11T22:30:53.090 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:52 smithi144 ceph-mon[33984]: 6.4 scrub ok 2024-03-11T22:30:53.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[39715]: pgmap v1102: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:30:53.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[39715]: 1.6 scrub starts 2024-03-11T22:30:53.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[39715]: 1.6 scrub ok 2024-03-11T22:30:53.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[39715]: 6.1 scrub starts 2024-03-11T22:30:53.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[39715]: 6.1 scrub ok 2024-03-11T22:30:53.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[39715]: 6.13 scrub starts 2024-03-11T22:30:53.811 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[39715]: 6.13 scrub ok 2024-03-11T22:30:53.811 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[30178]: pgmap v1102: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:30:53.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[30178]: 1.6 scrub starts 2024-03-11T22:30:53.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[30178]: 1.6 scrub ok 2024-03-11T22:30:53.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[30178]: 6.1 scrub starts 2024-03-11T22:30:53.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[30178]: 6.1 scrub ok 2024-03-11T22:30:53.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[30178]: 6.13 scrub starts 2024-03-11T22:30:53.812 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:53 smithi045 ceph-mon[30178]: 6.13 scrub ok 2024-03-11T22:30:53.937 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:30:53.937 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:30:54.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:53 smithi144 ceph-mon[33984]: pgmap v1102: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:30:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:53 smithi144 ceph-mon[33984]: 1.6 scrub starts 2024-03-11T22:30:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:53 smithi144 ceph-mon[33984]: 1.6 scrub ok 2024-03-11T22:30:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:53 smithi144 ceph-mon[33984]: 6.1 scrub starts 2024-03-11T22:30:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:53 smithi144 ceph-mon[33984]: 6.1 scrub ok 2024-03-11T22:30:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:53 smithi144 ceph-mon[33984]: 6.13 scrub starts 2024-03-11T22:30:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:53 smithi144 ceph-mon[33984]: 6.13 scrub ok 2024-03-11T22:30:54.617 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T22:30:54.713 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T22:30:54.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:54 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:30:54.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:54 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:30:54.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:54 smithi144 ceph-mon[33984]: 3.13 scrub starts 2024-03-11T22:30:54.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:54 smithi144 ceph-mon[33984]: 3.13 scrub ok 2024-03-11T22:30:54.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:54 smithi144 ceph-mon[33984]: osdmap e537: 8 total, 8 up, 8 in 2024-03-11T22:30:54.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:54 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/438379362' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:30:54.858 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:30:55.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:54 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:30:55.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:54 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:30:55.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:54 smithi045 ceph-mon[39715]: 3.13 scrub starts 2024-03-11T22:30:55.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:54 smithi045 ceph-mon[39715]: 3.13 scrub ok 2024-03-11T22:30:55.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:54 smithi045 ceph-mon[39715]: osdmap e537: 8 total, 8 up, 8 in 2024-03-11T22:30:55.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:54 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/438379362' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:30:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:54 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:30:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:54 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:30:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:54 smithi045 ceph-mon[30178]: 3.13 scrub starts 2024-03-11T22:30:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:54 smithi045 ceph-mon[30178]: 3.13 scrub ok 2024-03-11T22:30:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:54 smithi045 ceph-mon[30178]: osdmap e537: 8 total, 8 up, 8 in 2024-03-11T22:30:55.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:54 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/438379362' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:30:55.551 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T22:30:55.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:30:55.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:55 smithi045 ceph-mon[30178]: pgmap v1104: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 747 B/s rd, 0 op/s 2024-03-11T22:30:55.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:55 smithi045 ceph-mon[30178]: 2.0 scrub starts 2024-03-11T22:30:55.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:55 smithi045 ceph-mon[30178]: 2.0 scrub ok 2024-03-11T22:30:55.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:55 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/438379362' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:30:55.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:55 smithi045 ceph-mon[30178]: osdmap e538: 8 total, 8 up, 8 in 2024-03-11T22:30:55.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:55 smithi045 ceph-mon[39715]: pgmap v1104: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 747 B/s rd, 0 op/s 2024-03-11T22:30:55.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:55 smithi045 ceph-mon[39715]: 2.0 scrub starts 2024-03-11T22:30:55.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:55 smithi045 ceph-mon[39715]: 2.0 scrub ok 2024-03-11T22:30:55.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:55 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/438379362' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:30:55.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:55 smithi045 ceph-mon[39715]: osdmap e538: 8 total, 8 up, 8 in 2024-03-11T22:30:55.807 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:55.971 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:55 smithi144 ceph-mon[33984]: pgmap v1104: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 747 B/s rd, 0 op/s 2024-03-11T22:30:55.972 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:55 smithi144 ceph-mon[33984]: 2.0 scrub starts 2024-03-11T22:30:55.972 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:55 smithi144 ceph-mon[33984]: 2.0 scrub ok 2024-03-11T22:30:55.972 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:55 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/438379362' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:30:55.972 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:55 smithi144 ceph-mon[33984]: osdmap e538: 8 total, 8 up, 8 in 2024-03-11T22:30:56.527 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:30:56.528 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:30:56.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:56 smithi144 ceph-mon[33984]: 5.a scrub starts 2024-03-11T22:30:56.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:56 smithi144 ceph-mon[33984]: 5.a scrub ok 2024-03-11T22:30:56.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:56 smithi144 ceph-mon[33984]: 6.e scrub starts 2024-03-11T22:30:56.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:56 smithi144 ceph-mon[33984]: 6.e scrub ok 2024-03-11T22:30:56.851 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:56 smithi144 ceph-mon[33984]: 4.1a scrub starts 2024-03-11T22:30:56.851 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:56 smithi144 ceph-mon[33984]: 4.1a scrub ok 2024-03-11T22:30:57.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:56 smithi045 ceph-mon[39715]: 5.a scrub starts 2024-03-11T22:30:57.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:56 smithi045 ceph-mon[39715]: 5.a scrub ok 2024-03-11T22:30:57.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:56 smithi045 ceph-mon[39715]: 6.e scrub starts 2024-03-11T22:30:57.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:56 smithi045 ceph-mon[39715]: 6.e scrub ok 2024-03-11T22:30:57.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:56 smithi045 ceph-mon[39715]: 4.1a scrub starts 2024-03-11T22:30:57.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:56 smithi045 ceph-mon[39715]: 4.1a scrub ok 2024-03-11T22:30:57.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:56 smithi045 ceph-mon[30178]: 5.a scrub starts 2024-03-11T22:30:57.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:56 smithi045 ceph-mon[30178]: 5.a scrub ok 2024-03-11T22:30:57.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:56 smithi045 ceph-mon[30178]: 6.e scrub starts 2024-03-11T22:30:57.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:56 smithi045 ceph-mon[30178]: 6.e scrub ok 2024-03-11T22:30:57.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:56 smithi045 ceph-mon[30178]: 4.1a scrub starts 2024-03-11T22:30:57.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:56 smithi045 ceph-mon[30178]: 4.1a scrub ok 2024-03-11T22:30:57.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:30:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:30:57.174 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T22:30:57.416 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:30:57.833 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:57 smithi144 ceph-mon[33984]: pgmap v1106: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:57.833 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:57 smithi144 ceph-mon[33984]: 4.1d scrub starts 2024-03-11T22:30:57.833 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:57 smithi144 ceph-mon[33984]: 4.1d scrub ok 2024-03-11T22:30:57.833 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:57 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:57.852 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:30:57.854 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:30:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:57 smithi045 ceph-mon[39715]: pgmap v1106: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:57 smithi045 ceph-mon[39715]: 4.1d scrub starts 2024-03-11T22:30:58.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:57 smithi045 ceph-mon[39715]: 4.1d scrub ok 2024-03-11T22:30:58.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:57 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:58.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:57 smithi045 ceph-mon[30178]: pgmap v1106: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:30:58.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:57 smithi045 ceph-mon[30178]: 4.1d scrub starts 2024-03-11T22:30:58.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:57 smithi045 ceph-mon[30178]: 4.1d scrub ok 2024-03-11T22:30:58.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:57 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:30:58.838 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":1106,"stamp":"2024-03-11T22:30:56.533622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1841,"num_read_kb":1593,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":353,"num_bytes_recovered":336286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1137,"ondisk_log_size":1137,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6177980,"kb_used_data":18520,"kb_used_omap":12,"kb_used_meta":6159347,"kb_avail":743783236,"statfs":{"total":767960285184,"available":761634033664,"internally_reserved":0,"allocated":18964480,"data_stored":12457298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":6307171891},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001345"},"pg_stats":[{"pgid":"4.c","version":"479'35","reported_seq":829,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646935+0000","last_change":"2024-03-11T22:30:04.763741+0000","last_active":"2024-03-11T22:30:48.646935+0000","last_peered":"2024-03-11T22:30:48.646935+0000","last_clean":"2024-03-11T22:30:48.646935+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:30:48.646935+0000","last_undegraded":"2024-03-11T22:30:48.646935+0000","last_fullsized":"2024-03-11T22:30:48.646935+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'35","last_scrub_stamp":"2024-03-11T22:30:04.763606+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:30:04.763606+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:24.323735+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1001,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646037+0000","last_change":"2024-03-11T22:30:09.860422+0000","last_active":"2024-03-11T22:30:48.646037+0000","last_peered":"2024-03-11T22:30:48.646037+0000","last_clean":"2024-03-11T22:30:48.646037+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:30:48.646037+0000","last_undegraded":"2024-03-11T22:30:48.646037+0000","last_fullsized":"2024-03-11T22:30:48.646037+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:09.860304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:09.860304+0000","last_clean_scrub_stamp":"2024-03-11T22:30:09.860304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:31.943754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":1318,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705102+0000","last_change":"2024-03-11T22:30:19.637446+0000","last_active":"2024-03-11T22:30:54.705102+0000","last_peered":"2024-03-11T22:30:54.705102+0000","last_clean":"2024-03-11T22:30:54.705102+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:30:54.705102+0000","last_undegraded":"2024-03-11T22:30:54.705102+0000","last_fullsized":"2024-03-11T22:30:54.705102+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:30:19.637348+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:26:11.939962+0000","last_clean_scrub_stamp":"2024-03-11T22:30:19.637348+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:40.836875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":110,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.668118+0000","last_change":"2024-03-11T22:29:04.728008+0000","last_active":"2024-03-11T22:30:51.668118+0000","last_peered":"2024-03-11T22:30:51.668118+0000","last_clean":"2024-03-11T22:30:51.668118+0000","last_became_active":"2024-03-11T22:29:04.727120+0000","last_became_peered":"2024-03-11T22:29:04.727120+0000","last_unstale":"2024-03-11T22:30:51.668118+0000","last_undegraded":"2024-03-11T22:30:51.668118+0000","last_fullsized":"2024-03-11T22:30:51.668118+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:49.601908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:49.601908+0000","last_clean_scrub_stamp":"2024-03-11T22:26:49.601908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":1346,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.645445+0000","last_change":"2024-03-11T22:26:43.904135+0000","last_active":"2024-03-11T22:30:48.645445+0000","last_peered":"2024-03-11T22:30:48.645445+0000","last_clean":"2024-03-11T22:30:48.645445+0000","last_became_active":"2024-03-11T22:20:32.293356+0000","last_became_peered":"2024-03-11T22:20:32.293356+0000","last_unstale":"2024-03-11T22:30:48.645445+0000","last_undegraded":"2024-03-11T22:30:48.645445+0000","last_fullsized":"2024-03-11T22:30:48.645445+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:43.904050+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:43.904050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"479'53","reported_seq":944,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704828+0000","last_change":"2024-03-11T22:30:04.672014+0000","last_active":"2024-03-11T22:30:54.704828+0000","last_peered":"2024-03-11T22:30:54.704828+0000","last_clean":"2024-03-11T22:30:54.704828+0000","last_became_active":"2024-03-11T22:29:04.695558+0000","last_became_peered":"2024-03-11T22:29:04.695558+0000","last_unstale":"2024-03-11T22:30:54.704828+0000","last_undegraded":"2024-03-11T22:30:54.704828+0000","last_fullsized":"2024-03-11T22:30:54.704828+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'53","last_scrub_stamp":"2024-03-11T22:30:04.671906+0000","last_deep_scrub":"353'39","last_deep_scrub_stamp":"2024-03-11T22:23:05.461729+0000","last_clean_scrub_stamp":"2024-03-11T22:30:04.671906+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:28.141862+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1043,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672488+0000","last_change":"2024-03-11T22:29:04.719200+0000","last_active":"2024-03-11T22:30:55.672488+0000","last_peered":"2024-03-11T22:30:55.672488+0000","last_clean":"2024-03-11T22:30:55.672488+0000","last_became_active":"2024-03-11T22:29:04.718290+0000","last_became_peered":"2024-03-11T22:29:04.718290+0000","last_unstale":"2024-03-11T22:30:55.672488+0000","last_undegraded":"2024-03-11T22:30:55.672488+0000","last_fullsized":"2024-03-11T22:30:55.672488+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:28:21.605401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:28:21.605401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":110,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667655+0000","last_change":"2024-03-11T22:29:04.706086+0000","last_active":"2024-03-11T22:30:51.667655+0000","last_peered":"2024-03-11T22:30:51.667655+0000","last_clean":"2024-03-11T22:30:51.667655+0000","last_became_active":"2024-03-11T22:29:04.705661+0000","last_became_peered":"2024-03-11T22:29:04.705661+0000","last_unstale":"2024-03-11T22:30:51.667655+0000","last_undegraded":"2024-03-11T22:30:51.667655+0000","last_fullsized":"2024-03-11T22:30:51.667655+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:50.571821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.571821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1028,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704725+0000","last_change":"2024-03-11T22:30:10.839723+0000","last_active":"2024-03-11T22:30:54.704725+0000","last_peered":"2024-03-11T22:30:54.704725+0000","last_clean":"2024-03-11T22:30:54.704725+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:30:54.704725+0000","last_undegraded":"2024-03-11T22:30:54.704725+0000","last_fullsized":"2024-03-11T22:30:54.704725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:10.839608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:10.839608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:30.826822+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":1329,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401170+0000","last_change":"2024-03-11T22:26:03.097798+0000","last_active":"2024-03-11T22:30:55.401170+0000","last_peered":"2024-03-11T22:30:55.401170+0000","last_clean":"2024-03-11T22:30:55.401170+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:30:55.401170+0000","last_undegraded":"2024-03-11T22:30:55.401170+0000","last_fullsized":"2024-03-11T22:30:55.401170+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:03.097636+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:03.097636+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"516'32","reported_seq":910,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672880+0000","last_change":"2024-03-11T22:30:16.937051+0000","last_active":"2024-03-11T22:30:55.672880+0000","last_peered":"2024-03-11T22:30:55.672880+0000","last_clean":"2024-03-11T22:30:55.672880+0000","last_became_active":"2024-03-11T22:30:16.930541+0000","last_became_peered":"2024-03-11T22:30:16.930541+0000","last_unstale":"2024-03-11T22:30:55.672880+0000","last_undegraded":"2024-03-11T22:30:55.672880+0000","last_fullsized":"2024-03-11T22:30:55.672880+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'32","last_scrub_stamp":"2024-03-11T22:30:10.404123+0000","last_deep_scrub":"516'32","last_deep_scrub_stamp":"2024-03-11T22:30:10.404123+0000","last_clean_scrub_stamp":"2024-03-11T22:30:10.404123+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:34.707986+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":800,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705982+0000","last_change":"2024-03-11T22:29:54.744324+0000","last_active":"2024-03-11T22:30:54.705982+0000","last_peered":"2024-03-11T22:30:54.705982+0000","last_clean":"2024-03-11T22:30:54.705982+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:30:54.705982+0000","last_undegraded":"2024-03-11T22:30:54.705982+0000","last_fullsized":"2024-03-11T22:30:54.705982+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:54.744093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:29:54.744093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:30:58.841387+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1007,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401878+0000","last_change":"2024-03-11T22:30:40.303604+0000","last_active":"2024-03-11T22:30:55.401878+0000","last_peered":"2024-03-11T22:30:55.401878+0000","last_clean":"2024-03-11T22:30:55.401878+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:30:55.401878+0000","last_undegraded":"2024-03-11T22:30:55.401878+0000","last_fullsized":"2024-03-11T22:30:55.401878+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:40.303527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.303527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:46.392625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":981,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646585+0000","last_change":"2024-03-11T22:26:48.845265+0000","last_active":"2024-03-11T22:30:48.646585+0000","last_peered":"2024-03-11T22:30:48.646585+0000","last_clean":"2024-03-11T22:30:48.646585+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:30:48.646585+0000","last_undegraded":"2024-03-11T22:30:48.646585+0000","last_fullsized":"2024-03-11T22:30:48.646585+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:48.845177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:48.845177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"480'46","reported_seq":134,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.668259+0000","last_change":"2024-03-11T22:30:46.625195+0000","last_active":"2024-03-11T22:30:51.668259+0000","last_peered":"2024-03-11T22:30:51.668259+0000","last_clean":"2024-03-11T22:30:51.668259+0000","last_became_active":"2024-03-11T22:29:04.726632+0000","last_became_peered":"2024-03-11T22:29:04.726632+0000","last_unstale":"2024-03-11T22:30:51.668259+0000","last_undegraded":"2024-03-11T22:30:51.668259+0000","last_fullsized":"2024-03-11T22:30:51.668259+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'46","last_scrub_stamp":"2024-03-11T22:30:46.625090+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:30:46.625090+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:06.299995+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1026,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.645760+0000","last_change":"2024-03-11T22:30:45.799130+0000","last_active":"2024-03-11T22:30:48.645760+0000","last_peered":"2024-03-11T22:30:48.645760+0000","last_clean":"2024-03-11T22:30:48.645760+0000","last_became_active":"2024-03-11T22:29:04.701072+0000","last_became_peered":"2024-03-11T22:29:04.701072+0000","last_unstale":"2024-03-11T22:30:48.645760+0000","last_undegraded":"2024-03-11T22:30:48.645760+0000","last_fullsized":"2024-03-11T22:30:48.645760+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:45.799037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:45.799037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:49.472786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":993,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401782+0000","last_change":"2024-03-11T22:29:57.068709+0000","last_active":"2024-03-11T22:30:55.401782+0000","last_peered":"2024-03-11T22:30:55.401782+0000","last_clean":"2024-03-11T22:30:55.401782+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:30:55.401782+0000","last_undegraded":"2024-03-11T22:30:55.401782+0000","last_fullsized":"2024-03-11T22:30:55.401782+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:29:57.068625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:29:57.068625+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:01.261357+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1007,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401537+0000","last_change":"2024-03-11T22:29:04.787987+0000","last_active":"2024-03-11T22:30:55.401537+0000","last_peered":"2024-03-11T22:30:55.401537+0000","last_clean":"2024-03-11T22:30:55.401537+0000","last_became_active":"2024-03-11T22:29:04.787650+0000","last_became_peered":"2024-03-11T22:29:04.787650+0000","last_unstale":"2024-03-11T22:30:55.401537+0000","last_undegraded":"2024-03-11T22:30:55.401537+0000","last_fullsized":"2024-03-11T22:30:55.401537+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:36.095984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:26:36.095984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":1338,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401204+0000","last_change":"2024-03-11T22:30:43.217270+0000","last_active":"2024-03-11T22:30:55.401204+0000","last_peered":"2024-03-11T22:30:55.401204+0000","last_clean":"2024-03-11T22:30:55.401204+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:30:55.401204+0000","last_undegraded":"2024-03-11T22:30:55.401204+0000","last_fullsized":"2024-03-11T22:30:55.401204+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:30:43.217187+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:30:43.217187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:01.077102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":778,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.645986+0000","last_change":"2024-03-11T22:26:09.040467+0000","last_active":"2024-03-11T22:30:48.645986+0000","last_peered":"2024-03-11T22:30:48.645986+0000","last_clean":"2024-03-11T22:30:48.645986+0000","last_became_active":"2024-03-11T22:20:29.438896+0000","last_became_peered":"2024-03-11T22:20:29.438896+0000","last_unstale":"2024-03-11T22:30:48.645986+0000","last_undegraded":"2024-03-11T22:30:48.645986+0000","last_fullsized":"2024-03-11T22:30:48.645986+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:09.040331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:09.040331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"479'42","reported_seq":1110,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.645821+0000","last_change":"2024-03-11T22:26:05.020921+0000","last_active":"2024-03-11T22:30:48.645821+0000","last_peered":"2024-03-11T22:30:48.645821+0000","last_clean":"2024-03-11T22:30:48.645821+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:30:48.645821+0000","last_undegraded":"2024-03-11T22:30:48.645821+0000","last_fullsized":"2024-03-11T22:30:48.645821+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'30","last_scrub_stamp":"2024-03-11T22:26:05.020726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:05.020726+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1014,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.647062+0000","last_change":"2024-03-11T22:30:01.607109+0000","last_active":"2024-03-11T22:30:48.647062+0000","last_peered":"2024-03-11T22:30:48.647062+0000","last_clean":"2024-03-11T22:30:48.647062+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:30:48.647062+0000","last_undegraded":"2024-03-11T22:30:48.647062+0000","last_fullsized":"2024-03-11T22:30:48.647062+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:01.606988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:01.606988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:25.665371+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":760,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646039+0000","last_change":"2024-03-11T22:25:51.005350+0000","last_active":"2024-03-11T22:30:48.646039+0000","last_peered":"2024-03-11T22:30:48.646039+0000","last_clean":"2024-03-11T22:30:48.646039+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:30:48.646039+0000","last_undegraded":"2024-03-11T22:30:48.646039+0000","last_fullsized":"2024-03-11T22:30:48.646039+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:51.005219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:51.005219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":1309,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705189+0000","last_change":"2024-03-11T22:29:55.711127+0000","last_active":"2024-03-11T22:30:54.705189+0000","last_peered":"2024-03-11T22:30:54.705189+0000","last_clean":"2024-03-11T22:30:54.705189+0000","last_became_active":"2024-03-11T22:26:49.698651+0000","last_became_peered":"2024-03-11T22:26:49.698651+0000","last_unstale":"2024-03-11T22:30:54.705189+0000","last_undegraded":"2024-03-11T22:30:54.705189+0000","last_fullsized":"2024-03-11T22:30:54.705189+0000","mapping_epoch":175,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:29:55.710911+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:25:05.773963+0000","last_clean_scrub_stamp":"2024-03-11T22:29:55.710911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:07.856356+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"488'10","reported_seq":1284,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.645906+0000","last_change":"2024-03-11T22:29:04.839059+0000","last_active":"2024-03-11T22:30:48.645906+0000","last_peered":"2024-03-11T22:30:48.645906+0000","last_clean":"2024-03-11T22:30:48.645906+0000","last_became_active":"2024-03-11T22:29:04.721065+0000","last_became_peered":"2024-03-11T22:29:04.721065+0000","last_unstale":"2024-03-11T22:30:48.645906+0000","last_undegraded":"2024-03-11T22:30:48.645906+0000","last_fullsized":"2024-03-11T22:30:48.645906+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:26:13.061151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:13.061151+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"480'42","reported_seq":856,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401107+0000","last_change":"2024-03-11T22:30:48.353744+0000","last_active":"2024-03-11T22:30:55.401107+0000","last_peered":"2024-03-11T22:30:55.401107+0000","last_clean":"2024-03-11T22:30:55.401107+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:30:55.401107+0000","last_undegraded":"2024-03-11T22:30:55.401107+0000","last_fullsized":"2024-03-11T22:30:55.401107+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'42","last_scrub_stamp":"2024-03-11T22:30:48.353647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:48.353647+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:14.430459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":1032,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.644401+0000","last_change":"2024-03-11T22:30:42.177735+0000","last_active":"2024-03-11T22:30:51.644401+0000","last_peered":"2024-03-11T22:30:51.644401+0000","last_clean":"2024-03-11T22:30:51.644401+0000","last_became_active":"2024-03-11T22:29:04.718160+0000","last_became_peered":"2024-03-11T22:29:04.718160+0000","last_unstale":"2024-03-11T22:30:51.644401+0000","last_undegraded":"2024-03-11T22:30:51.644401+0000","last_fullsized":"2024-03-11T22:30:51.644401+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:30:42.177651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:42.177651+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:52.743855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":961,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672538+0000","last_change":"2024-03-11T22:28:34.728469+0000","last_active":"2024-03-11T22:30:55.672538+0000","last_peered":"2024-03-11T22:30:55.672538+0000","last_clean":"2024-03-11T22:30:55.672538+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:30:55.672538+0000","last_undegraded":"2024-03-11T22:30:55.672538+0000","last_fullsized":"2024-03-11T22:30:55.672538+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:28:34.728337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:28:34.728337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":1360,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646500+0000","last_change":"2024-03-11T22:26:53.558080+0000","last_active":"2024-03-11T22:30:48.646500+0000","last_peered":"2024-03-11T22:30:48.646500+0000","last_clean":"2024-03-11T22:30:48.646500+0000","last_became_active":"2024-03-11T22:26:53.557811+0000","last_became_peered":"2024-03-11T22:26:53.557811+0000","last_unstale":"2024-03-11T22:30:48.646500+0000","last_undegraded":"2024-03-11T22:30:48.646500+0000","last_fullsized":"2024-03-11T22:30:48.646500+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:45.837466+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:45.837466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"378'9","reported_seq":1280,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.645194+0000","last_change":"2024-03-11T22:30:38.192095+0000","last_active":"2024-03-11T22:30:51.645194+0000","last_peered":"2024-03-11T22:30:51.645194+0000","last_clean":"2024-03-11T22:30:51.645194+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:30:51.645194+0000","last_undegraded":"2024-03-11T22:30:51.645194+0000","last_fullsized":"2024-03-11T22:30:51.645194+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9","last_scrub_stamp":"2024-03-11T22:30:38.191965+0000","last_deep_scrub":"378'9","last_deep_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_clean_scrub_stamp":"2024-03-11T22:30:38.191965+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:58.989567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'30","reported_seq":130,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667392+0000","last_change":"2024-03-11T22:30:02.607278+0000","last_active":"2024-03-11T22:30:51.667392+0000","last_peered":"2024-03-11T22:30:51.667392+0000","last_clean":"2024-03-11T22:30:51.667392+0000","last_became_active":"2024-03-11T22:29:04.727100+0000","last_became_peered":"2024-03-11T22:29:04.727100+0000","last_unstale":"2024-03-11T22:30:51.667392+0000","last_undegraded":"2024-03-11T22:30:51.667392+0000","last_fullsized":"2024-03-11T22:30:51.667392+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-03-11T22:30:02.607142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:02.607142+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:31.923645+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1016,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401706+0000","last_change":"2024-03-11T22:29:54.153237+0000","last_active":"2024-03-11T22:30:55.401706+0000","last_peered":"2024-03-11T22:30:55.401706+0000","last_clean":"2024-03-11T22:30:55.401706+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:30:55.401706+0000","last_undegraded":"2024-03-11T22:30:55.401706+0000","last_fullsized":"2024-03-11T22:30:55.401706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:54.153119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:29:54.153119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:01.613626+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1000,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.643337+0000","last_change":"2024-03-11T22:30:45.264761+0000","last_active":"2024-03-11T22:30:51.643337+0000","last_peered":"2024-03-11T22:30:51.643337+0000","last_clean":"2024-03-11T22:30:51.643337+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:30:51.643337+0000","last_undegraded":"2024-03-11T22:30:51.643337+0000","last_fullsized":"2024-03-11T22:30:51.643337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:45.264658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:45.264658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:14.204199+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":1288,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673132+0000","last_change":"2024-03-11T22:30:39.559483+0000","last_active":"2024-03-11T22:30:55.673132+0000","last_peered":"2024-03-11T22:30:55.673132+0000","last_clean":"2024-03-11T22:30:55.673132+0000","last_became_active":"2024-03-11T22:26:41.661375+0000","last_became_peered":"2024-03-11T22:26:41.661375+0000","last_unstale":"2024-03-11T22:30:55.673132+0000","last_undegraded":"2024-03-11T22:30:55.673132+0000","last_fullsized":"2024-03-11T22:30:55.673132+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":423,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:30:39.559403+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:30:39.559403+0000","last_clean_scrub_stamp":"2024-03-11T22:30:39.559403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:59.861593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1023,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672410+0000","last_change":"2024-03-11T22:28:37.585104+0000","last_active":"2024-03-11T22:30:55.672410+0000","last_peered":"2024-03-11T22:30:55.672410+0000","last_clean":"2024-03-11T22:30:55.672410+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:30:55.672410+0000","last_undegraded":"2024-03-11T22:30:55.672410+0000","last_fullsized":"2024-03-11T22:30:55.672410+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:28:37.584991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:28:37.584991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"479'47","reported_seq":1073,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.644770+0000","last_change":"2024-03-11T22:26:47.889297+0000","last_active":"2024-03-11T22:30:51.644770+0000","last_peered":"2024-03-11T22:30:51.644770+0000","last_clean":"2024-03-11T22:30:51.644770+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:30:51.644770+0000","last_undegraded":"2024-03-11T22:30:51.644770+0000","last_fullsized":"2024-03-11T22:30:51.644770+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'37","last_scrub_stamp":"2024-03-11T22:26:47.889156+0000","last_deep_scrub":"429'37","last_deep_scrub_stamp":"2024-03-11T22:26:47.889156+0000","last_clean_scrub_stamp":"2024-03-11T22:26:47.889156+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":118,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667812+0000","last_change":"2024-03-11T22:30:47.657180+0000","last_active":"2024-03-11T22:30:51.667812+0000","last_peered":"2024-03-11T22:30:51.667812+0000","last_clean":"2024-03-11T22:30:51.667812+0000","last_became_active":"2024-03-11T22:29:04.722813+0000","last_became_peered":"2024-03-11T22:29:04.722813+0000","last_unstale":"2024-03-11T22:30:51.667812+0000","last_undegraded":"2024-03-11T22:30:51.667812+0000","last_fullsized":"2024-03-11T22:30:51.667812+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:47.657052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:04.706082+0000","last_clean_scrub_stamp":"2024-03-11T22:30:47.657052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:03.794039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":110,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667036+0000","last_change":"2024-03-11T22:29:04.729789+0000","last_active":"2024-03-11T22:30:51.667036+0000","last_peered":"2024-03-11T22:30:51.667036+0000","last_clean":"2024-03-11T22:30:51.667036+0000","last_became_active":"2024-03-11T22:29:04.729455+0000","last_became_peered":"2024-03-11T22:29:04.729455+0000","last_unstale":"2024-03-11T22:30:51.667036+0000","last_undegraded":"2024-03-11T22:30:51.667036+0000","last_fullsized":"2024-03-11T22:30:51.667036+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.604198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.604198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":1263,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673111+0000","last_change":"2024-03-11T22:30:52.611262+0000","last_active":"2024-03-11T22:30:55.673111+0000","last_peered":"2024-03-11T22:30:55.673111+0000","last_clean":"2024-03-11T22:30:55.673111+0000","last_became_active":"2024-03-11T22:26:45.688609+0000","last_became_peered":"2024-03-11T22:26:45.688609+0000","last_unstale":"2024-03-11T22:30:55.673111+0000","last_undegraded":"2024-03-11T22:30:55.673111+0000","last_fullsized":"2024-03-11T22:30:55.673111+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:30:52.611147+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:30:52.611147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:05.456604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1020,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401905+0000","last_change":"2024-03-11T22:30:41.301144+0000","last_active":"2024-03-11T22:30:55.401905+0000","last_peered":"2024-03-11T22:30:55.401905+0000","last_clean":"2024-03-11T22:30:55.401905+0000","last_became_active":"2024-03-11T22:29:04.718259+0000","last_became_peered":"2024-03-11T22:29:04.718259+0000","last_unstale":"2024-03-11T22:30:55.401905+0000","last_undegraded":"2024-03-11T22:30:55.401905+0000","last_fullsized":"2024-03-11T22:30:55.401905+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:41.301047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:41.301047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:43.242569+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"479'36","reported_seq":1101,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.644308+0000","last_change":"2024-03-11T22:29:04.818952+0000","last_active":"2024-03-11T22:30:51.644308+0000","last_peered":"2024-03-11T22:30:51.644308+0000","last_clean":"2024-03-11T22:30:51.644308+0000","last_became_active":"2024-03-11T22:29:04.720793+0000","last_became_peered":"2024-03-11T22:29:04.720793+0000","last_unstale":"2024-03-11T22:30:51.644308+0000","last_undegraded":"2024-03-11T22:30:51.644308+0000","last_fullsized":"2024-03-11T22:30:51.644308+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'24","last_scrub_stamp":"2024-03-11T22:26:35.660739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:35.660739+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":221,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667058+0000","last_change":"2024-03-11T22:29:04.716256+0000","last_active":"2024-03-11T22:30:51.667058+0000","last_peered":"2024-03-11T22:30:51.667058+0000","last_clean":"2024-03-11T22:30:51.667058+0000","last_became_active":"2024-03-11T22:29:04.702609+0000","last_became_peered":"2024-03-11T22:29:04.702609+0000","last_unstale":"2024-03-11T22:30:51.667058+0000","last_undegraded":"2024-03-11T22:30:51.667058+0000","last_fullsized":"2024-03-11T22:30:51.667058+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:26:35.669961+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:26:35.669961+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":297,"num_read_kb":292,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1011,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.643627+0000","last_change":"2024-03-11T22:30:08.418564+0000","last_active":"2024-03-11T22:30:51.643627+0000","last_peered":"2024-03-11T22:30:51.643627+0000","last_clean":"2024-03-11T22:30:51.643627+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:30:51.643627+0000","last_undegraded":"2024-03-11T22:30:51.643627+0000","last_fullsized":"2024-03-11T22:30:51.643627+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:08.418488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:08.418488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:35.702001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":118,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667711+0000","last_change":"2024-03-11T22:30:51.665235+0000","last_active":"2024-03-11T22:30:51.667711+0000","last_peered":"2024-03-11T22:30:51.667711+0000","last_clean":"2024-03-11T22:30:51.667711+0000","last_became_active":"2024-03-11T22:29:04.696141+0000","last_became_peered":"2024-03-11T22:29:04.696141+0000","last_unstale":"2024-03-11T22:30:51.667711+0000","last_undegraded":"2024-03-11T22:30:51.667711+0000","last_fullsized":"2024-03-11T22:30:51.667711+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:51.665137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:22.750560+0000","last_clean_scrub_stamp":"2024-03-11T22:30:51.665137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:54.684247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":567,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667453+0000","last_change":"2024-03-11T22:29:49.468313+0000","last_active":"2024-03-11T22:30:51.667453+0000","last_peered":"2024-03-11T22:30:51.667453+0000","last_clean":"2024-03-11T22:30:51.667453+0000","last_became_active":"2024-03-11T22:29:37.643184+0000","last_became_peered":"2024-03-11T22:29:37.643184+0000","last_unstale":"2024-03-11T22:30:51.667453+0000","last_undegraded":"2024-03-11T22:30:51.667453+0000","last_fullsized":"2024-03-11T22:30:51.667453+0000","mapping_epoch":502,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:29:49.467957+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:28:24.623016+0000","last_clean_scrub_stamp":"2024-03-11T22:29:49.467957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.316172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":983,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705621+0000","last_change":"2024-03-11T22:26:42.960839+0000","last_active":"2024-03-11T22:30:54.705621+0000","last_peered":"2024-03-11T22:30:54.705621+0000","last_clean":"2024-03-11T22:30:54.705621+0000","last_became_active":"2024-03-11T22:20:29.420497+0000","last_became_peered":"2024-03-11T22:20:29.420497+0000","last_unstale":"2024-03-11T22:30:54.705621+0000","last_undegraded":"2024-03-11T22:30:54.705621+0000","last_fullsized":"2024-03-11T22:30:54.705621+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:42.960739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:26:42.960739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"514'48","reported_seq":1362,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672890+0000","last_change":"2024-03-11T22:30:05.380085+0000","last_active":"2024-03-11T22:30:55.672890+0000","last_peered":"2024-03-11T22:30:55.672890+0000","last_clean":"2024-03-11T22:30:55.672890+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:30:55.672890+0000","last_undegraded":"2024-03-11T22:30:55.672890+0000","last_fullsized":"2024-03-11T22:30:55.672890+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"514'48","last_scrub_stamp":"2024-03-11T22:30:05.379958+0000","last_deep_scrub":"514'48","last_deep_scrub_stamp":"2024-03-11T22:30:05.379958+0000","last_clean_scrub_stamp":"2024-03-11T22:30:05.379958+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:12.891919+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1002,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.645819+0000","last_change":"2024-03-11T22:30:40.798528+0000","last_active":"2024-03-11T22:30:48.645819+0000","last_peered":"2024-03-11T22:30:48.645819+0000","last_clean":"2024-03-11T22:30:48.645819+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:30:48.645819+0000","last_undegraded":"2024-03-11T22:30:48.645819+0000","last_fullsized":"2024-03-11T22:30:48.645819+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:40.798443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.798443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:54.912063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":1359,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.643443+0000","last_change":"2024-03-11T22:30:06.479012+0000","last_active":"2024-03-11T22:30:51.643443+0000","last_peered":"2024-03-11T22:30:51.643443+0000","last_clean":"2024-03-11T22:30:51.643443+0000","last_became_active":"2024-03-11T22:26:37.632412+0000","last_became_peered":"2024-03-11T22:26:37.632412+0000","last_unstale":"2024-03-11T22:30:51.643443+0000","last_undegraded":"2024-03-11T22:30:51.643443+0000","last_fullsized":"2024-03-11T22:30:51.643443+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":19,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:30:06.478921+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:30:06.478921+0000","last_clean_scrub_stamp":"2024-03-11T22:30:06.478921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:10.660047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1004,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.400948+0000","last_change":"2024-03-11T22:30:39.454150+0000","last_active":"2024-03-11T22:30:55.400948+0000","last_peered":"2024-03-11T22:30:55.400948+0000","last_clean":"2024-03-11T22:30:55.400948+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:30:55.400948+0000","last_undegraded":"2024-03-11T22:30:55.400948+0000","last_fullsized":"2024-03-11T22:30:55.400948+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:39.454015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:39.454015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:49.055198+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1033,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704591+0000","last_change":"2024-03-11T22:30:08.659249+0000","last_active":"2024-03-11T22:30:54.704591+0000","last_peered":"2024-03-11T22:30:54.704591+0000","last_clean":"2024-03-11T22:30:54.704591+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:30:54.704591+0000","last_undegraded":"2024-03-11T22:30:54.704591+0000","last_fullsized":"2024-03-11T22:30:54.704591+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:08.659154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:08.659154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:16.522418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'21","reported_seq":130,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.668251+0000","last_change":"2024-03-11T22:30:05.683743+0000","last_active":"2024-03-11T22:30:51.668251+0000","last_peered":"2024-03-11T22:30:51.668251+0000","last_clean":"2024-03-11T22:30:51.668251+0000","last_became_active":"2024-03-11T22:29:04.705980+0000","last_became_peered":"2024-03-11T22:29:04.705980+0000","last_unstale":"2024-03-11T22:30:51.668251+0000","last_undegraded":"2024-03-11T22:30:51.668251+0000","last_fullsized":"2024-03-11T22:30:51.668251+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'21","last_scrub_stamp":"2024-03-11T22:30:05.683641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:05.683641+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:09.121346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1058,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706077+0000","last_change":"2024-03-11T22:30:15.663376+0000","last_active":"2024-03-11T22:30:54.706077+0000","last_peered":"2024-03-11T22:30:54.706077+0000","last_clean":"2024-03-11T22:30:54.706077+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:30:54.706077+0000","last_undegraded":"2024-03-11T22:30:54.706077+0000","last_fullsized":"2024-03-11T22:30:54.706077+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:15.663298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:30:15.663298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:39.777684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":118,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667052+0000","last_change":"2024-03-11T22:29:54.581275+0000","last_active":"2024-03-11T22:30:51.667052+0000","last_peered":"2024-03-11T22:30:51.667052+0000","last_clean":"2024-03-11T22:30:51.667052+0000","last_became_active":"2024-03-11T22:29:04.713225+0000","last_became_peered":"2024-03-11T22:29:04.713225+0000","last_unstale":"2024-03-11T22:30:51.667052+0000","last_undegraded":"2024-03-11T22:30:51.667052+0000","last_fullsized":"2024-03-11T22:30:51.667052+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:54.581156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:29:54.581156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.689997+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":961,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672907+0000","last_change":"2024-03-11T22:30:06.412273+0000","last_active":"2024-03-11T22:30:55.672907+0000","last_peered":"2024-03-11T22:30:55.672907+0000","last_clean":"2024-03-11T22:30:55.672907+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:30:55.672907+0000","last_undegraded":"2024-03-11T22:30:55.672907+0000","last_fullsized":"2024-03-11T22:30:55.672907+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:06.412160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:30:06.412160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:19.024605+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":1366,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.643560+0000","last_change":"2024-03-11T22:26:32.769183+0000","last_active":"2024-03-11T22:30:51.643560+0000","last_peered":"2024-03-11T22:30:51.643560+0000","last_clean":"2024-03-11T22:30:51.643560+0000","last_became_active":"2024-03-11T22:26:29.590669+0000","last_became_peered":"2024-03-11T22:26:29.590669+0000","last_unstale":"2024-03-11T22:30:51.643560+0000","last_undegraded":"2024-03-11T22:30:51.643560+0000","last_fullsized":"2024-03-11T22:30:51.643560+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:32.769096+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:32.769096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1015,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.644419+0000","last_change":"2024-03-11T22:30:05.507577+0000","last_active":"2024-03-11T22:30:51.644419+0000","last_peered":"2024-03-11T22:30:51.644419+0000","last_clean":"2024-03-11T22:30:51.644419+0000","last_became_active":"2024-03-11T22:29:04.707688+0000","last_became_peered":"2024-03-11T22:29:04.707688+0000","last_unstale":"2024-03-11T22:30:51.644419+0000","last_undegraded":"2024-03-11T22:30:51.644419+0000","last_fullsized":"2024-03-11T22:30:51.644419+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:05.507462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:05.507462+0000","last_clean_scrub_stamp":"2024-03-11T22:30:05.507462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:31.012916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"520'55","reported_seq":79,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401377+0000","last_change":"2024-03-11T22:30:16.943159+0000","last_active":"2024-03-11T22:30:55.401377+0000","last_peered":"2024-03-11T22:30:55.401377+0000","last_clean":"2024-03-11T22:30:55.401377+0000","last_became_active":"2024-03-11T22:30:16.930211+0000","last_became_peered":"2024-03-11T22:30:16.930211+0000","last_unstale":"2024-03-11T22:30:55.401377+0000","last_undegraded":"2024-03-11T22:30:55.401377+0000","last_fullsized":"2024-03-11T22:30:55.401377+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'54","last_scrub_stamp":"2024-03-11T22:29:52.527782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:29:52.527782+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:20.718332+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":770,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.644657+0000","last_change":"2024-03-11T22:26:44.809895+0000","last_active":"2024-03-11T22:30:51.644657+0000","last_peered":"2024-03-11T22:30:51.644657+0000","last_clean":"2024-03-11T22:30:51.644657+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:30:51.644657+0000","last_undegraded":"2024-03-11T22:30:51.644657+0000","last_fullsized":"2024-03-11T22:30:51.644657+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.809803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.809803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":1048,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401277+0000","last_change":"2024-03-11T22:29:04.812619+0000","last_active":"2024-03-11T22:30:55.401277+0000","last_peered":"2024-03-11T22:30:55.401277+0000","last_clean":"2024-03-11T22:30:55.401277+0000","last_became_active":"2024-03-11T22:29:04.718809+0000","last_became_peered":"2024-03-11T22:29:04.718809+0000","last_unstale":"2024-03-11T22:30:55.401277+0000","last_undegraded":"2024-03-11T22:30:55.401277+0000","last_fullsized":"2024-03-11T22:30:55.401277+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:26:31.988426+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:26:31.988426+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":994,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.647211+0000","last_change":"2024-03-11T22:25:58.161115+0000","last_active":"2024-03-11T22:30:48.647211+0000","last_peered":"2024-03-11T22:30:48.647211+0000","last_clean":"2024-03-11T22:30:48.647211+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:30:48.647211+0000","last_undegraded":"2024-03-11T22:30:48.647211+0000","last_fullsized":"2024-03-11T22:30:48.647211+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:58.160972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:25:58.160972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":1345,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401081+0000","last_change":"2024-03-11T22:26:34.036467+0000","last_active":"2024-03-11T22:30:55.401081+0000","last_peered":"2024-03-11T22:30:55.401081+0000","last_clean":"2024-03-11T22:30:55.401081+0000","last_became_active":"2024-03-11T22:26:33.607145+0000","last_became_peered":"2024-03-11T22:26:33.607145+0000","last_unstale":"2024-03-11T22:30:55.401081+0000","last_undegraded":"2024-03-11T22:30:55.401081+0000","last_fullsized":"2024-03-11T22:30:55.401081+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:34.036387+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:34.036387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"479'21","reported_seq":1080,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.645076+0000","last_change":"2024-03-11T22:29:04.732685+0000","last_active":"2024-03-11T22:30:51.645076+0000","last_peered":"2024-03-11T22:30:51.645076+0000","last_clean":"2024-03-11T22:30:51.645076+0000","last_became_active":"2024-03-11T22:29:04.707867+0000","last_became_peered":"2024-03-11T22:29:04.707867+0000","last_unstale":"2024-03-11T22:30:51.645076+0000","last_undegraded":"2024-03-11T22:30:51.645076+0000","last_fullsized":"2024-03-11T22:30:51.645076+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:26:12.661479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:12.661479+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401587+0000","last_change":"2024-03-11T22:30:50.377246+0000","last_active":"2024-03-11T22:30:55.401587+0000","last_peered":"2024-03-11T22:30:55.401587+0000","last_clean":"2024-03-11T22:30:55.401587+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:30:55.401587+0000","last_undegraded":"2024-03-11T22:30:55.401587+0000","last_fullsized":"2024-03-11T22:30:55.401587+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:50.377151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:50.377151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:15.151224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1041,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.393858+0000","last_change":"2024-03-11T22:30:55.393883+0000","last_active":"2024-03-11T22:30:55.393858+0000","last_peered":"2024-03-11T22:30:55.393858+0000","last_clean":"2024-03-11T22:30:55.393858+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:30:55.393858+0000","last_undegraded":"2024-03-11T22:30:55.393858+0000","last_fullsized":"2024-03-11T22:30:55.393858+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:55.393786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:30:55.393786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:10.301036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1011,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.643891+0000","last_change":"2024-03-11T22:30:20.400482+0000","last_active":"2024-03-11T22:30:51.643891+0000","last_peered":"2024-03-11T22:30:51.643891+0000","last_clean":"2024-03-11T22:30:51.643891+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:30:51.643891+0000","last_undegraded":"2024-03-11T22:30:51.643891+0000","last_fullsized":"2024-03-11T22:30:51.643891+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:20.400349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:20.400349+0000","last_clean_scrub_stamp":"2024-03-11T22:30:20.400349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:39.045173+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"480'51","reported_seq":1210,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672311+0000","last_change":"2024-03-11T22:30:04.401901+0000","last_active":"2024-03-11T22:30:55.672311+0000","last_peered":"2024-03-11T22:30:55.672311+0000","last_clean":"2024-03-11T22:30:55.672311+0000","last_became_active":"2024-03-11T22:29:04.718797+0000","last_became_peered":"2024-03-11T22:29:04.718797+0000","last_unstale":"2024-03-11T22:30:55.672311+0000","last_undegraded":"2024-03-11T22:30:55.672311+0000","last_fullsized":"2024-03-11T22:30:55.672311+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'51","last_scrub_stamp":"2024-03-11T22:30:04.401771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:04.401771+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:32.893247+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":959,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401790+0000","last_change":"2024-03-11T22:26:04.099295+0000","last_active":"2024-03-11T22:30:55.401790+0000","last_peered":"2024-03-11T22:30:55.401790+0000","last_clean":"2024-03-11T22:30:55.401790+0000","last_became_active":"2024-03-11T22:20:29.413664+0000","last_became_peered":"2024-03-11T22:20:29.413664+0000","last_unstale":"2024-03-11T22:30:55.401790+0000","last_undegraded":"2024-03-11T22:30:55.401790+0000","last_fullsized":"2024-03-11T22:30:55.401790+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:04.099165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:04.099165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1032,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705447+0000","last_change":"2024-03-11T22:30:12.675279+0000","last_active":"2024-03-11T22:30:54.705447+0000","last_peered":"2024-03-11T22:30:54.705447+0000","last_clean":"2024-03-11T22:30:54.705447+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:30:54.705447+0000","last_undegraded":"2024-03-11T22:30:54.705447+0000","last_fullsized":"2024-03-11T22:30:54.705447+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:12.675202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:30:12.675202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:40.937625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":991,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704426+0000","last_change":"2024-03-11T22:30:47.568602+0000","last_active":"2024-03-11T22:30:54.704426+0000","last_peered":"2024-03-11T22:30:54.704426+0000","last_clean":"2024-03-11T22:30:54.704426+0000","last_became_active":"2024-03-11T22:20:29.415565+0000","last_became_peered":"2024-03-11T22:20:29.415565+0000","last_unstale":"2024-03-11T22:30:54.704426+0000","last_undegraded":"2024-03-11T22:30:54.704426+0000","last_fullsized":"2024-03-11T22:30:54.704426+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:47.568500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:47.568500+0000","last_clean_scrub_stamp":"2024-03-11T22:30:47.568500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:49.067116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1045,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704211+0000","last_change":"2024-03-11T22:29:56.665159+0000","last_active":"2024-03-11T22:30:54.704211+0000","last_peered":"2024-03-11T22:30:54.704211+0000","last_clean":"2024-03-11T22:30:54.704211+0000","last_became_active":"2024-03-11T22:29:04.718513+0000","last_became_peered":"2024-03-11T22:29:04.718513+0000","last_unstale":"2024-03-11T22:30:54.704211+0000","last_undegraded":"2024-03-11T22:30:54.704211+0000","last_fullsized":"2024-03-11T22:30:54.704211+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:56.665023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:29:56.665023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:13.354927+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"478'24","reported_seq":130,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.666909+0000","last_change":"2024-03-11T22:30:48.649205+0000","last_active":"2024-03-11T22:30:51.666909+0000","last_peered":"2024-03-11T22:30:51.666909+0000","last_clean":"2024-03-11T22:30:51.666909+0000","last_became_active":"2024-03-11T22:29:04.694343+0000","last_became_peered":"2024-03-11T22:29:04.694343+0000","last_unstale":"2024-03-11T22:30:51.666909+0000","last_undegraded":"2024-03-11T22:30:51.666909+0000","last_fullsized":"2024-03-11T22:30:51.666909+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'24","last_scrub_stamp":"2024-03-11T22:30:48.649070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:48.649070+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:03.230178+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":118,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.668121+0000","last_change":"2024-03-11T22:30:10.740917+0000","last_active":"2024-03-11T22:30:51.668121+0000","last_peered":"2024-03-11T22:30:51.668121+0000","last_clean":"2024-03-11T22:30:51.668121+0000","last_became_active":"2024-03-11T22:29:04.727524+0000","last_became_peered":"2024-03-11T22:29:04.727524+0000","last_unstale":"2024-03-11T22:30:51.668121+0000","last_undegraded":"2024-03-11T22:30:51.668121+0000","last_fullsized":"2024-03-11T22:30:51.668121+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:10.740798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:10.740798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:36.857785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":1023,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646268+0000","last_change":"2024-03-11T22:29:04.742292+0000","last_active":"2024-03-11T22:30:48.646268+0000","last_peered":"2024-03-11T22:30:48.646268+0000","last_clean":"2024-03-11T22:30:48.646268+0000","last_became_active":"2024-03-11T22:29:04.698531+0000","last_became_peered":"2024-03-11T22:29:04.698531+0000","last_unstale":"2024-03-11T22:30:48.646268+0000","last_undegraded":"2024-03-11T22:30:48.646268+0000","last_fullsized":"2024-03-11T22:30:48.646268+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:26:35.021902+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:26:35.021902+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"310'11","reported_seq":1252,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672191+0000","last_change":"2024-03-11T22:29:54.360194+0000","last_active":"2024-03-11T22:30:55.672191+0000","last_peered":"2024-03-11T22:30:55.672191+0000","last_clean":"2024-03-11T22:30:55.672191+0000","last_became_active":"2024-03-11T22:20:29.418355+0000","last_became_peered":"2024-03-11T22:20:29.418355+0000","last_unstale":"2024-03-11T22:30:55.672191+0000","last_undegraded":"2024-03-11T22:30:55.672191+0000","last_fullsized":"2024-03-11T22:30:55.672191+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:29:54.360080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:29:54.360080+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:07.077073+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"479'46","reported_seq":1124,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.400978+0000","last_change":"2024-03-11T22:26:45.212688+0000","last_active":"2024-03-11T22:30:55.400978+0000","last_peered":"2024-03-11T22:30:55.400978+0000","last_clean":"2024-03-11T22:30:55.400978+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:30:55.400978+0000","last_undegraded":"2024-03-11T22:30:55.400978+0000","last_fullsized":"2024-03-11T22:30:55.400978+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:26:45.212591+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:26:45.212591+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"432'4","reported_seq":805,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.645062+0000","last_change":"2024-03-11T22:26:50.847434+0000","last_active":"2024-03-11T22:30:51.645062+0000","last_peered":"2024-03-11T22:30:51.645062+0000","last_clean":"2024-03-11T22:30:51.645062+0000","last_became_active":"2024-03-11T22:20:29.415226+0000","last_became_peered":"2024-03-11T22:20:29.415226+0000","last_unstale":"2024-03-11T22:30:51.645062+0000","last_undegraded":"2024-03-11T22:30:51.645062+0000","last_fullsized":"2024-03-11T22:30:51.645062+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'4","last_scrub_stamp":"2024-03-11T22:26:50.847349+0000","last_deep_scrub":"432'4","last_deep_scrub_stamp":"2024-03-11T22:26:50.847349+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.847349+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":996,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401409+0000","last_change":"2024-03-11T22:26:49.194654+0000","last_active":"2024-03-11T22:30:55.401409+0000","last_peered":"2024-03-11T22:30:55.401409+0000","last_clean":"2024-03-11T22:30:55.401409+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:30:55.401409+0000","last_undegraded":"2024-03-11T22:30:55.401409+0000","last_fullsized":"2024-03-11T22:30:55.401409+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:49.194569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:49.194569+0000","last_clean_scrub_stamp":"2024-03-11T22:26:49.194569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"479'18","reported_seq":1086,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.647244+0000","last_change":"2024-03-11T22:30:16.934222+0000","last_active":"2024-03-11T22:30:48.647244+0000","last_peered":"2024-03-11T22:30:48.647244+0000","last_clean":"2024-03-11T22:30:48.647244+0000","last_became_active":"2024-03-11T22:30:16.929794+0000","last_became_peered":"2024-03-11T22:30:16.929794+0000","last_unstale":"2024-03-11T22:30:48.647244+0000","last_undegraded":"2024-03-11T22:30:48.647244+0000","last_fullsized":"2024-03-11T22:30:48.647244+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'12","last_scrub_stamp":"2024-03-11T22:25:52.052338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:52.052338+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1022,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.671990+0000","last_change":"2024-03-11T22:28:33.573448+0000","last_active":"2024-03-11T22:30:55.671990+0000","last_peered":"2024-03-11T22:30:55.671990+0000","last_clean":"2024-03-11T22:30:55.671990+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:30:55.671990+0000","last_undegraded":"2024-03-11T22:30:55.671990+0000","last_fullsized":"2024-03-11T22:30:55.671990+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:28:33.573346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:28:33.573346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1019,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401611+0000","last_change":"2024-03-11T22:30:03.168689+0000","last_active":"2024-03-11T22:30:55.401611+0000","last_peered":"2024-03-11T22:30:55.401611+0000","last_clean":"2024-03-11T22:30:55.401611+0000","last_became_active":"2024-03-11T22:29:04.731305+0000","last_became_peered":"2024-03-11T22:29:04.731305+0000","last_unstale":"2024-03-11T22:30:55.401611+0000","last_undegraded":"2024-03-11T22:30:55.401611+0000","last_fullsized":"2024-03-11T22:30:55.401611+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:03.168564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:30:03.168564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:32.143726+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":952,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672984+0000","last_change":"2024-03-11T22:29:52.451412+0000","last_active":"2024-03-11T22:30:55.672984+0000","last_peered":"2024-03-11T22:30:55.672984+0000","last_clean":"2024-03-11T22:30:55.672984+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:30:55.672984+0000","last_undegraded":"2024-03-11T22:30:55.672984+0000","last_fullsized":"2024-03-11T22:30:55.672984+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:52.451186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:29:52.451186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:30:59.862313+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"478'25","reported_seq":1076,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.644603+0000","last_change":"2024-03-11T22:30:01.530506+0000","last_active":"2024-03-11T22:30:51.644603+0000","last_peered":"2024-03-11T22:30:51.644603+0000","last_clean":"2024-03-11T22:30:51.644603+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:30:51.644603+0000","last_undegraded":"2024-03-11T22:30:51.644603+0000","last_fullsized":"2024-03-11T22:30:51.644603+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:30:01.530350+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_clean_scrub_stamp":"2024-03-11T22:30:01.530350+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:27.843442+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":792,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672072+0000","last_change":"2024-03-11T22:30:46.646058+0000","last_active":"2024-03-11T22:30:55.672072+0000","last_peered":"2024-03-11T22:30:55.672072+0000","last_clean":"2024-03-11T22:30:55.672072+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:30:55.672072+0000","last_undegraded":"2024-03-11T22:30:55.672072+0000","last_fullsized":"2024-03-11T22:30:55.672072+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:46.645943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:46.645943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:15.950611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1017,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.647170+0000","last_change":"2024-03-11T22:30:16.883019+0000","last_active":"2024-03-11T22:30:48.647170+0000","last_peered":"2024-03-11T22:30:48.647170+0000","last_clean":"2024-03-11T22:30:48.647170+0000","last_became_active":"2024-03-11T22:29:04.700408+0000","last_became_peered":"2024-03-11T22:29:04.700408+0000","last_unstale":"2024-03-11T22:30:48.647170+0000","last_undegraded":"2024-03-11T22:30:48.647170+0000","last_fullsized":"2024-03-11T22:30:48.647170+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:16.882883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:16.882883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:40.426358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1036,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705535+0000","last_change":"2024-03-11T22:30:49.480356+0000","last_active":"2024-03-11T22:30:54.705535+0000","last_peered":"2024-03-11T22:30:54.705535+0000","last_clean":"2024-03-11T22:30:54.705535+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:30:54.705535+0000","last_undegraded":"2024-03-11T22:30:54.705535+0000","last_fullsized":"2024-03-11T22:30:54.705535+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:49.480247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:30:49.480247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:59.538719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"479'25","reported_seq":1049,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401791+0000","last_change":"2024-03-11T22:26:12.149477+0000","last_active":"2024-03-11T22:30:55.401791+0000","last_peered":"2024-03-11T22:30:55.401791+0000","last_clean":"2024-03-11T22:30:55.401791+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:30:55.401791+0000","last_undegraded":"2024-03-11T22:30:55.401791+0000","last_fullsized":"2024-03-11T22:30:55.401791+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:26:12.149360+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:26:12.149360+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1022,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.643439+0000","last_change":"2024-03-11T22:29:53.461268+0000","last_active":"2024-03-11T22:30:51.643439+0000","last_peered":"2024-03-11T22:30:51.643439+0000","last_clean":"2024-03-11T22:30:51.643439+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:30:51.643439+0000","last_undegraded":"2024-03-11T22:30:51.643439+0000","last_fullsized":"2024-03-11T22:30:51.643439+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:53.461031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:50.030272+0000","last_clean_scrub_stamp":"2024-03-11T22:29:53.461031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:14.083004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"527'12","reported_seq":1308,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646758+0000","last_change":"2024-03-11T22:30:34.882548+0000","last_active":"2024-03-11T22:30:48.646758+0000","last_peered":"2024-03-11T22:30:48.646758+0000","last_clean":"2024-03-11T22:30:48.646758+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:30:48.646758+0000","last_undegraded":"2024-03-11T22:30:48.646758+0000","last_fullsized":"2024-03-11T22:30:48.646758+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'12","last_scrub_stamp":"2024-03-11T22:30:34.882462+0000","last_deep_scrub":"527'12","last_deep_scrub_stamp":"2024-03-11T22:30:34.882462+0000","last_clean_scrub_stamp":"2024-03-11T22:30:34.882462+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:45.890672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":798,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704582+0000","last_change":"2024-03-11T22:30:18.817369+0000","last_active":"2024-03-11T22:30:54.704582+0000","last_peered":"2024-03-11T22:30:54.704582+0000","last_clean":"2024-03-11T22:30:54.704582+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:30:54.704582+0000","last_undegraded":"2024-03-11T22:30:54.704582+0000","last_fullsized":"2024-03-11T22:30:54.704582+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:18.817224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:48.490757+0000","last_clean_scrub_stamp":"2024-03-11T22:30:18.817224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:31.548740+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"479'30","reported_seq":1088,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646695+0000","last_change":"2024-03-11T22:29:04.830471+0000","last_active":"2024-03-11T22:30:48.646695+0000","last_peered":"2024-03-11T22:30:48.646695+0000","last_clean":"2024-03-11T22:30:48.646695+0000","last_became_active":"2024-03-11T22:29:04.720501+0000","last_became_peered":"2024-03-11T22:29:04.720501+0000","last_unstale":"2024-03-11T22:30:48.646695+0000","last_undegraded":"2024-03-11T22:30:48.646695+0000","last_fullsized":"2024-03-11T22:30:48.646695+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:25:56.196035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:56.196035+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":822,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705361+0000","last_change":"2024-03-11T22:30:53.455852+0000","last_active":"2024-03-11T22:30:54.705361+0000","last_peered":"2024-03-11T22:30:54.705361+0000","last_clean":"2024-03-11T22:30:54.705361+0000","last_became_active":"2024-03-11T22:20:29.415019+0000","last_became_peered":"2024-03-11T22:20:29.415019+0000","last_unstale":"2024-03-11T22:30:54.705361+0000","last_undegraded":"2024-03-11T22:30:54.705361+0000","last_fullsized":"2024-03-11T22:30:54.705361+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:53.455748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:51.034868+0000","last_clean_scrub_stamp":"2024-03-11T22:30:53.455748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:55.363500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"488'10","reported_seq":1305,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401688+0000","last_change":"2024-03-11T22:30:08.228206+0000","last_active":"2024-03-11T22:30:55.401688+0000","last_peered":"2024-03-11T22:30:55.401688+0000","last_clean":"2024-03-11T22:30:55.401688+0000","last_became_active":"2024-03-11T22:29:04.787739+0000","last_became_peered":"2024-03-11T22:29:04.787739+0000","last_unstale":"2024-03-11T22:30:55.401688+0000","last_undegraded":"2024-03-11T22:30:55.401688+0000","last_fullsized":"2024-03-11T22:30:55.401688+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'10","last_scrub_stamp":"2024-03-11T22:30:08.228075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:08.228075+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:10.466725+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1002,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.644210+0000","last_change":"2024-03-11T22:30:21.303542+0000","last_active":"2024-03-11T22:30:51.644210+0000","last_peered":"2024-03-11T22:30:51.644210+0000","last_clean":"2024-03-11T22:30:51.644210+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:30:51.644210+0000","last_undegraded":"2024-03-11T22:30:51.644210+0000","last_fullsized":"2024-03-11T22:30:51.644210+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:21.303440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:21.303440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:36.442252+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"479'27","reported_seq":886,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672384+0000","last_change":"2024-03-11T22:30:40.594125+0000","last_active":"2024-03-11T22:30:55.672384+0000","last_peered":"2024-03-11T22:30:55.672384+0000","last_clean":"2024-03-11T22:30:55.672384+0000","last_became_active":"2024-03-11T22:29:04.720174+0000","last_became_peered":"2024-03-11T22:29:04.720174+0000","last_unstale":"2024-03-11T22:30:55.672384+0000","last_undegraded":"2024-03-11T22:30:55.672384+0000","last_fullsized":"2024-03-11T22:30:55.672384+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-03-11T22:30:40.594016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.594016+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:55.638323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":809,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672786+0000","last_change":"2024-03-11T22:26:38.840815+0000","last_active":"2024-03-11T22:30:55.672786+0000","last_peered":"2024-03-11T22:30:55.672786+0000","last_clean":"2024-03-11T22:30:55.672786+0000","last_became_active":"2024-03-11T22:20:29.420867+0000","last_became_peered":"2024-03-11T22:20:29.420867+0000","last_unstale":"2024-03-11T22:30:55.672786+0000","last_undegraded":"2024-03-11T22:30:55.672786+0000","last_fullsized":"2024-03-11T22:30:55.672786+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:26:38.840685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:38.840685+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":121,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667512+0000","last_change":"2024-03-11T22:30:40.510945+0000","last_active":"2024-03-11T22:30:51.667512+0000","last_peered":"2024-03-11T22:30:51.667512+0000","last_clean":"2024-03-11T22:30:51.667512+0000","last_became_active":"2024-03-11T22:29:04.723400+0000","last_became_peered":"2024-03-11T22:29:04.723400+0000","last_unstale":"2024-03-11T22:30:51.667512+0000","last_undegraded":"2024-03-11T22:30:51.667512+0000","last_fullsized":"2024-03-11T22:30:51.667512+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:40.510823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.510823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:09.057333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":981,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.645705+0000","last_change":"2024-03-11T22:29:50.712811+0000","last_active":"2024-03-11T22:30:48.645705+0000","last_peered":"2024-03-11T22:30:48.645705+0000","last_clean":"2024-03-11T22:30:48.645705+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:30:48.645705+0000","last_undegraded":"2024-03-11T22:30:48.645705+0000","last_fullsized":"2024-03-11T22:30:48.645705+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:50.712416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:29:50.712416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:10.140704+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"478'25","reported_seq":128,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667243+0000","last_change":"2024-03-11T22:30:19.687640+0000","last_active":"2024-03-11T22:30:51.667243+0000","last_peered":"2024-03-11T22:30:51.667243+0000","last_clean":"2024-03-11T22:30:51.667243+0000","last_became_active":"2024-03-11T22:29:04.730485+0000","last_became_peered":"2024-03-11T22:29:04.730485+0000","last_unstale":"2024-03-11T22:30:51.667243+0000","last_undegraded":"2024-03-11T22:30:51.667243+0000","last_fullsized":"2024-03-11T22:30:51.667243+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:30:19.687537+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:30:19.687537+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:19.724923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":944,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646647+0000","last_change":"2024-03-11T22:30:12.898819+0000","last_active":"2024-03-11T22:30:48.646647+0000","last_peered":"2024-03-11T22:30:48.646647+0000","last_clean":"2024-03-11T22:30:48.646647+0000","last_became_active":"2024-03-11T22:20:29.422747+0000","last_became_peered":"2024-03-11T22:20:29.422747+0000","last_unstale":"2024-03-11T22:30:48.646647+0000","last_undegraded":"2024-03-11T22:30:48.646647+0000","last_fullsized":"2024-03-11T22:30:48.646647+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:12.898649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:12.898649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.975372+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1005,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.645994+0000","last_change":"2024-03-11T22:30:15.836456+0000","last_active":"2024-03-11T22:30:48.645994+0000","last_peered":"2024-03-11T22:30:48.645994+0000","last_clean":"2024-03-11T22:30:48.645994+0000","last_became_active":"2024-03-11T22:29:04.700352+0000","last_became_peered":"2024-03-11T22:29:04.700352+0000","last_unstale":"2024-03-11T22:30:48.645994+0000","last_undegraded":"2024-03-11T22:30:48.645994+0000","last_fullsized":"2024-03-11T22:30:48.645994+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:15.836335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:30:15.836335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:21.846380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1010,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.643211+0000","last_change":"2024-03-11T22:30:48.240248+0000","last_active":"2024-03-11T22:30:51.643211+0000","last_peered":"2024-03-11T22:30:51.643211+0000","last_clean":"2024-03-11T22:30:51.643211+0000","last_became_active":"2024-03-11T22:29:04.707420+0000","last_became_peered":"2024-03-11T22:29:04.707420+0000","last_unstale":"2024-03-11T22:30:51.643211+0000","last_undegraded":"2024-03-11T22:30:51.643211+0000","last_fullsized":"2024-03-11T22:30:51.643211+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:48.240128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:09.057970+0000","last_clean_scrub_stamp":"2024-03-11T22:30:48.240128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:12.685122+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"480'28","reported_seq":1069,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646024+0000","last_change":"2024-03-11T22:29:51.655446+0000","last_active":"2024-03-11T22:30:48.646024+0000","last_peered":"2024-03-11T22:30:48.646024+0000","last_clean":"2024-03-11T22:30:48.646024+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:30:48.646024+0000","last_undegraded":"2024-03-11T22:30:48.646024+0000","last_fullsized":"2024-03-11T22:30:48.646024+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'28","last_scrub_stamp":"2024-03-11T22:29:51.655136+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:29:51.655136+0000","objects_scrubbed":5,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:30:52.761466+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1039,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401613+0000","last_change":"2024-03-11T22:30:20.359726+0000","last_active":"2024-03-11T22:30:55.401613+0000","last_peered":"2024-03-11T22:30:55.401613+0000","last_clean":"2024-03-11T22:30:55.401613+0000","last_became_active":"2024-03-11T22:29:04.722573+0000","last_became_peered":"2024-03-11T22:29:04.722573+0000","last_unstale":"2024-03-11T22:30:55.401613+0000","last_undegraded":"2024-03-11T22:30:55.401613+0000","last_fullsized":"2024-03-11T22:30:55.401613+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:20.359623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:20.359623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:46.022838+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":783,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.671964+0000","last_change":"2024-03-11T22:26:44.807269+0000","last_active":"2024-03-11T22:30:55.671964+0000","last_peered":"2024-03-11T22:30:55.671964+0000","last_clean":"2024-03-11T22:30:55.671964+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:30:55.671964+0000","last_undegraded":"2024-03-11T22:30:55.671964+0000","last_fullsized":"2024-03-11T22:30:55.671964+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.807187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.807187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":981,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646473+0000","last_change":"2024-03-11T22:24:59.829935+0000","last_active":"2024-03-11T22:30:48.646473+0000","last_peered":"2024-03-11T22:30:48.646473+0000","last_clean":"2024-03-11T22:30:48.646473+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:30:48.646473+0000","last_undegraded":"2024-03-11T22:30:48.646473+0000","last_fullsized":"2024-03-11T22:30:48.646473+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:24:59.829811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:24:59.829811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"478'12","reported_seq":1051,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.645710+0000","last_change":"2024-03-11T22:29:04.843107+0000","last_active":"2024-03-11T22:30:48.645710+0000","last_peered":"2024-03-11T22:30:48.645710+0000","last_clean":"2024-03-11T22:30:48.645710+0000","last_became_active":"2024-03-11T22:29:04.734207+0000","last_became_peered":"2024-03-11T22:29:04.734207+0000","last_unstale":"2024-03-11T22:30:48.645710+0000","last_undegraded":"2024-03-11T22:30:48.645710+0000","last_fullsized":"2024-03-11T22:30:48.645710+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'8","last_scrub_stamp":"2024-03-11T22:26:50.833149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.833149+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1002,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.645227+0000","last_change":"2024-03-11T22:26:05.787798+0000","last_active":"2024-03-11T22:30:51.645227+0000","last_peered":"2024-03-11T22:30:51.645227+0000","last_clean":"2024-03-11T22:30:51.645227+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:30:51.645227+0000","last_undegraded":"2024-03-11T22:30:51.645227+0000","last_fullsized":"2024-03-11T22:30:51.645227+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:05.787543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:05.787543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":600,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672075+0000","last_change":"2024-03-11T22:30:03.392102+0000","last_active":"2024-03-11T22:30:55.672075+0000","last_peered":"2024-03-11T22:30:55.672075+0000","last_clean":"2024-03-11T22:30:55.672075+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:30:55.672075+0000","last_undegraded":"2024-03-11T22:30:55.672075+0000","last_fullsized":"2024-03-11T22:30:55.672075+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:03.391983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:03.391983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:28.249700+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":970,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705070+0000","last_change":"2024-03-11T22:30:01.668808+0000","last_active":"2024-03-11T22:30:54.705070+0000","last_peered":"2024-03-11T22:30:54.705070+0000","last_clean":"2024-03-11T22:30:54.705070+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:30:54.705070+0000","last_undegraded":"2024-03-11T22:30:54.705070+0000","last_fullsized":"2024-03-11T22:30:54.705070+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:01.668594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:01.668594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:07.076683+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"480'27","reported_seq":1106,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673172+0000","last_change":"2024-03-11T22:30:43.611613+0000","last_active":"2024-03-11T22:30:55.673172+0000","last_peered":"2024-03-11T22:30:55.673172+0000","last_clean":"2024-03-11T22:30:55.673172+0000","last_became_active":"2024-03-11T22:29:04.720148+0000","last_became_peered":"2024-03-11T22:29:04.720148+0000","last_unstale":"2024-03-11T22:30:55.673172+0000","last_undegraded":"2024-03-11T22:30:55.673172+0000","last_fullsized":"2024-03-11T22:30:55.673172+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'27","last_scrub_stamp":"2024-03-11T22:30:43.611513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:43.611513+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:00.303736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1013,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.645653+0000","last_change":"2024-03-11T22:30:13.866705+0000","last_active":"2024-03-11T22:30:48.645653+0000","last_peered":"2024-03-11T22:30:48.645653+0000","last_clean":"2024-03-11T22:30:48.645653+0000","last_became_active":"2024-03-11T22:29:04.733742+0000","last_became_peered":"2024-03-11T22:29:04.733742+0000","last_unstale":"2024-03-11T22:30:48.645653+0000","last_undegraded":"2024-03-11T22:30:48.645653+0000","last_fullsized":"2024-03-11T22:30:48.645653+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:13.866615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:13.866615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:42.154858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1004,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.644780+0000","last_change":"2024-03-11T22:26:43.777178+0000","last_active":"2024-03-11T22:30:51.644780+0000","last_peered":"2024-03-11T22:30:51.644780+0000","last_clean":"2024-03-11T22:30:51.644780+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:30:51.644780+0000","last_undegraded":"2024-03-11T22:30:51.644780+0000","last_fullsized":"2024-03-11T22:30:51.644780+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:43.777100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:43.777100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":118,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667926+0000","last_change":"2024-03-11T22:30:06.824568+0000","last_active":"2024-03-11T22:30:51.667926+0000","last_peered":"2024-03-11T22:30:51.667926+0000","last_clean":"2024-03-11T22:30:51.667926+0000","last_became_active":"2024-03-11T22:29:04.786119+0000","last_became_peered":"2024-03-11T22:29:04.786119+0000","last_unstale":"2024-03-11T22:30:51.667926+0000","last_undegraded":"2024-03-11T22:30:51.667926+0000","last_fullsized":"2024-03-11T22:30:51.667926+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:06.824468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:06.824468+0000","last_clean_scrub_stamp":"2024-03-11T22:30:06.824468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:29.075598+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"479'30","reported_seq":1089,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.644499+0000","last_change":"2024-03-11T22:30:41.197575+0000","last_active":"2024-03-11T22:30:51.644499+0000","last_peered":"2024-03-11T22:30:51.644499+0000","last_clean":"2024-03-11T22:30:51.644499+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:30:51.644499+0000","last_undegraded":"2024-03-11T22:30:51.644499+0000","last_fullsized":"2024-03-11T22:30:51.644499+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-03-11T22:30:41.197457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:41.197457+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:01.110242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":122,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.666900+0000","last_change":"2024-03-11T22:30:20.763816+0000","last_active":"2024-03-11T22:30:51.666900+0000","last_peered":"2024-03-11T22:30:51.666900+0000","last_clean":"2024-03-11T22:30:51.666900+0000","last_became_active":"2024-03-11T22:29:04.693762+0000","last_became_peered":"2024-03-11T22:29:04.693762+0000","last_unstale":"2024-03-11T22:30:51.666900+0000","last_undegraded":"2024-03-11T22:30:51.666900+0000","last_fullsized":"2024-03-11T22:30:51.666900+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:30:20.763628+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:30:20.763628+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:38.282863+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'11","reported_seq":1333,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673080+0000","last_change":"2024-03-11T22:30:42.614513+0000","last_active":"2024-03-11T22:30:55.673080+0000","last_peered":"2024-03-11T22:30:55.673080+0000","last_clean":"2024-03-11T22:30:55.673080+0000","last_became_active":"2024-03-11T22:20:29.415781+0000","last_became_peered":"2024-03-11T22:20:29.415781+0000","last_unstale":"2024-03-11T22:30:55.673080+0000","last_undegraded":"2024-03-11T22:30:55.673080+0000","last_fullsized":"2024-03-11T22:30:55.673080+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-11T22:30:42.614396+0000","last_deep_scrub":"315'11","last_deep_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_clean_scrub_stamp":"2024-03-11T22:30:42.614396+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:56.150179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":956,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401482+0000","last_change":"2024-03-11T22:30:10.198091+0000","last_active":"2024-03-11T22:30:55.401482+0000","last_peered":"2024-03-11T22:30:55.401482+0000","last_clean":"2024-03-11T22:30:55.401482+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:30:55.401482+0000","last_undegraded":"2024-03-11T22:30:55.401482+0000","last_fullsized":"2024-03-11T22:30:55.401482+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:30:10.197974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:10.197974+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.414872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"480'24","reported_seq":1065,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646879+0000","last_change":"2024-03-11T22:29:04.833923+0000","last_active":"2024-03-11T22:30:48.646879+0000","last_peered":"2024-03-11T22:30:48.646879+0000","last_clean":"2024-03-11T22:30:48.646879+0000","last_became_active":"2024-03-11T22:29:04.720341+0000","last_became_peered":"2024-03-11T22:29:04.720341+0000","last_unstale":"2024-03-11T22:30:48.646879+0000","last_undegraded":"2024-03-11T22:30:48.646879+0000","last_fullsized":"2024-03-11T22:30:48.646879+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'18","last_scrub_stamp":"2024-03-11T22:25:01.629321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:01.629321+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":118,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667054+0000","last_change":"2024-03-11T22:30:08.741018+0000","last_active":"2024-03-11T22:30:51.667054+0000","last_peered":"2024-03-11T22:30:51.667054+0000","last_clean":"2024-03-11T22:30:51.667054+0000","last_became_active":"2024-03-11T22:29:04.725678+0000","last_became_peered":"2024-03-11T22:29:04.725678+0000","last_unstale":"2024-03-11T22:30:51.667054+0000","last_undegraded":"2024-03-11T22:30:51.667054+0000","last_fullsized":"2024-03-11T22:30:51.667054+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:08.740926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:08.740926+0000","last_clean_scrub_stamp":"2024-03-11T22:30:08.740926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:21.051131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":759,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646959+0000","last_change":"2024-03-11T22:30:46.780258+0000","last_active":"2024-03-11T22:30:48.646959+0000","last_peered":"2024-03-11T22:30:48.646959+0000","last_clean":"2024-03-11T22:30:48.646959+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:30:48.646959+0000","last_undegraded":"2024-03-11T22:30:48.646959+0000","last_fullsized":"2024-03-11T22:30:48.646959+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:46.780169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:46.780169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:02.985331+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":982,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646948+0000","last_change":"2024-03-11T22:26:37.062133+0000","last_active":"2024-03-11T22:30:48.646948+0000","last_peered":"2024-03-11T22:30:48.646948+0000","last_clean":"2024-03-11T22:30:48.646948+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:30:48.646948+0000","last_undegraded":"2024-03-11T22:30:48.646948+0000","last_fullsized":"2024-03-11T22:30:48.646948+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:37.062022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:37.062022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"479'39","reported_seq":1148,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705296+0000","last_change":"2024-03-11T22:30:21.635123+0000","last_active":"2024-03-11T22:30:54.705296+0000","last_peered":"2024-03-11T22:30:54.705296+0000","last_clean":"2024-03-11T22:30:54.705296+0000","last_became_active":"2024-03-11T22:29:04.721324+0000","last_became_peered":"2024-03-11T22:29:04.721324+0000","last_unstale":"2024-03-11T22:30:54.705296+0000","last_undegraded":"2024-03-11T22:30:54.705296+0000","last_fullsized":"2024-03-11T22:30:54.705296+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'39","last_scrub_stamp":"2024-03-11T22:30:21.634960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:21.634960+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:39.506214+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1021,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.644904+0000","last_change":"2024-03-11T22:30:40.231188+0000","last_active":"2024-03-11T22:30:51.644904+0000","last_peered":"2024-03-11T22:30:51.644904+0000","last_clean":"2024-03-11T22:30:51.644904+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:30:51.644904+0000","last_undegraded":"2024-03-11T22:30:51.644904+0000","last_fullsized":"2024-03-11T22:30:51.644904+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:40.231100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.231100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:49.098484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":118,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667815+0000","last_change":"2024-03-11T22:29:55.548541+0000","last_active":"2024-03-11T22:30:51.667815+0000","last_peered":"2024-03-11T22:30:51.667815+0000","last_clean":"2024-03-11T22:30:51.667815+0000","last_became_active":"2024-03-11T22:29:04.724856+0000","last_became_peered":"2024-03-11T22:29:04.724856+0000","last_unstale":"2024-03-11T22:30:51.667815+0000","last_undegraded":"2024-03-11T22:30:51.667815+0000","last_fullsized":"2024-03-11T22:30:51.667815+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:55.546853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:29:55.546853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:25.517836+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":790,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673097+0000","last_change":"2024-03-11T22:30:17.307828+0000","last_active":"2024-03-11T22:30:55.673097+0000","last_peered":"2024-03-11T22:30:55.673097+0000","last_clean":"2024-03-11T22:30:55.673097+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:30:55.673097+0000","last_undegraded":"2024-03-11T22:30:55.673097+0000","last_fullsized":"2024-03-11T22:30:55.673097+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:17.307729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:17.307729+0000","last_clean_scrub_stamp":"2024-03-11T22:30:17.307729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:24.381369+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"478'11","reported_seq":863,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.645776+0000","last_change":"2024-03-11T22:30:02.742323+0000","last_active":"2024-03-11T22:30:48.645776+0000","last_peered":"2024-03-11T22:30:48.645776+0000","last_clean":"2024-03-11T22:30:48.645776+0000","last_became_active":"2024-03-11T22:29:04.698810+0000","last_became_peered":"2024-03-11T22:29:04.698810+0000","last_unstale":"2024-03-11T22:30:48.645776+0000","last_undegraded":"2024-03-11T22:30:48.645776+0000","last_fullsized":"2024-03-11T22:30:48.645776+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'11","last_scrub_stamp":"2024-03-11T22:30:02.742197+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:30:02.742197+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:26.951844+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1026,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401582+0000","last_change":"2024-03-11T22:30:15.199268+0000","last_active":"2024-03-11T22:30:55.401582+0000","last_peered":"2024-03-11T22:30:55.401582+0000","last_clean":"2024-03-11T22:30:55.401582+0000","last_became_active":"2024-03-11T22:29:04.721192+0000","last_became_peered":"2024-03-11T22:29:04.721192+0000","last_unstale":"2024-03-11T22:30:55.401582+0000","last_undegraded":"2024-03-11T22:30:55.401582+0000","last_fullsized":"2024-03-11T22:30:55.401582+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:15.199165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:30:15.199165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:43.996210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":118,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667512+0000","last_change":"2024-03-11T22:30:43.546407+0000","last_active":"2024-03-11T22:30:51.667512+0000","last_peered":"2024-03-11T22:30:51.667512+0000","last_clean":"2024-03-11T22:30:51.667512+0000","last_became_active":"2024-03-11T22:29:04.726418+0000","last_became_peered":"2024-03-11T22:29:04.726418+0000","last_unstale":"2024-03-11T22:30:51.667512+0000","last_undegraded":"2024-03-11T22:30:51.667512+0000","last_fullsized":"2024-03-11T22:30:51.667512+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:43.546317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:09.575187+0000","last_clean_scrub_stamp":"2024-03-11T22:30:43.546317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:10.181400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401239+0000","last_change":"2024-03-11T22:30:51.393304+0000","last_active":"2024-03-11T22:30:55.401239+0000","last_peered":"2024-03-11T22:30:55.401239+0000","last_clean":"2024-03-11T22:30:55.401239+0000","last_became_active":"2024-03-11T22:29:04.721807+0000","last_became_peered":"2024-03-11T22:29:04.721807+0000","last_unstale":"2024-03-11T22:30:55.401239+0000","last_undegraded":"2024-03-11T22:30:55.401239+0000","last_fullsized":"2024-03-11T22:30:55.401239+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:51.393206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:51.393206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:52.157560+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"479'21","reported_seq":812,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.645966+0000","last_change":"2024-03-11T22:25:03.685762+0000","last_active":"2024-03-11T22:30:48.645966+0000","last_peered":"2024-03-11T22:30:48.645966+0000","last_clean":"2024-03-11T22:30:48.645966+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:30:48.645966+0000","last_undegraded":"2024-03-11T22:30:48.645966+0000","last_fullsized":"2024-03-11T22:30:48.645966+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:25:03.685629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:03.685629+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1014,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401563+0000","last_change":"2024-03-11T22:29:53.126868+0000","last_active":"2024-03-11T22:30:55.401563+0000","last_peered":"2024-03-11T22:30:55.401563+0000","last_clean":"2024-03-11T22:30:55.401563+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:30:55.401563+0000","last_undegraded":"2024-03-11T22:30:55.401563+0000","last_fullsized":"2024-03-11T22:30:55.401563+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:53.126769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:29:53.126769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:12.911808+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1018,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401263+0000","last_change":"2024-03-11T22:30:16.197266+0000","last_active":"2024-03-11T22:30:55.401263+0000","last_peered":"2024-03-11T22:30:55.401263+0000","last_clean":"2024-03-11T22:30:55.401263+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:30:55.401263+0000","last_undegraded":"2024-03-11T22:30:55.401263+0000","last_fullsized":"2024-03-11T22:30:55.401263+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:16.197172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:30:16.197172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.268824+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1002,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.643949+0000","last_change":"2024-03-11T22:29:04.708231+0000","last_active":"2024-03-11T22:30:51.643949+0000","last_peered":"2024-03-11T22:30:51.643949+0000","last_clean":"2024-03-11T22:30:51.643949+0000","last_became_active":"2024-03-11T22:29:04.707444+0000","last_became_peered":"2024-03-11T22:29:04.707444+0000","last_unstale":"2024-03-11T22:30:51.643949+0000","last_undegraded":"2024-03-11T22:30:51.643949+0000","last_fullsized":"2024-03-11T22:30:51.643949+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:37.738562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:37.738562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"480'39","reported_seq":1050,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646092+0000","last_change":"2024-03-11T22:26:38.945645+0000","last_active":"2024-03-11T22:30:48.646092+0000","last_peered":"2024-03-11T22:30:48.646092+0000","last_clean":"2024-03-11T22:30:48.646092+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:30:48.646092+0000","last_undegraded":"2024-03-11T22:30:48.646092+0000","last_fullsized":"2024-03-11T22:30:48.646092+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:26:38.945505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:38.945505+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":1009,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.647383+0000","last_change":"2024-03-11T22:26:39.902683+0000","last_active":"2024-03-11T22:30:48.647383+0000","last_peered":"2024-03-11T22:30:48.647383+0000","last_clean":"2024-03-11T22:30:48.647383+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:30:48.647383+0000","last_undegraded":"2024-03-11T22:30:48.647383+0000","last_fullsized":"2024-03-11T22:30:48.647383+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:26:39.902588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:39.902588+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":803,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672111+0000","last_change":"2024-03-11T22:30:15.364972+0000","last_active":"2024-03-11T22:30:55.672111+0000","last_peered":"2024-03-11T22:30:55.672111+0000","last_clean":"2024-03-11T22:30:55.672111+0000","last_became_active":"2024-03-11T22:29:04.719011+0000","last_became_peered":"2024-03-11T22:29:04.719011+0000","last_unstale":"2024-03-11T22:30:55.672111+0000","last_undegraded":"2024-03-11T22:30:55.672111+0000","last_fullsized":"2024-03-11T22:30:55.672111+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:15.364881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:15.364881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.215173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":992,"reported_epoch":535,"state":"active+clean","last_fresh":"2024-03-11T22:30:48.646158+0000","last_change":"2024-03-11T22:26:34.021098+0000","last_active":"2024-03-11T22:30:48.646158+0000","last_peered":"2024-03-11T22:30:48.646158+0000","last_clean":"2024-03-11T22:30:48.646158+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:30:48.646158+0000","last_undegraded":"2024-03-11T22:30:48.646158+0000","last_fullsized":"2024-03-11T22:30:48.646158+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:34.020992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:34.020992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"478'25","reported_seq":869,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672223+0000","last_change":"2024-03-11T22:29:57.388072+0000","last_active":"2024-03-11T22:30:55.672223+0000","last_peered":"2024-03-11T22:30:55.672223+0000","last_clean":"2024-03-11T22:30:55.672223+0000","last_became_active":"2024-03-11T22:29:04.718521+0000","last_became_peered":"2024-03-11T22:29:04.718521+0000","last_unstale":"2024-03-11T22:30:55.672223+0000","last_undegraded":"2024-03-11T22:30:55.672223+0000","last_fullsized":"2024-03-11T22:30:55.672223+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:29:57.387926+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:29:57.387926+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:05.421282+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1011,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.644897+0000","last_change":"2024-03-11T22:30:04.486340+0000","last_active":"2024-03-11T22:30:51.644897+0000","last_peered":"2024-03-11T22:30:51.644897+0000","last_clean":"2024-03-11T22:30:51.644897+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:30:51.644897+0000","last_undegraded":"2024-03-11T22:30:51.644897+0000","last_fullsized":"2024-03-11T22:30:51.644897+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:04.486254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:04.486254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:17.389356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":118,"reported_epoch":536,"state":"active+clean","last_fresh":"2024-03-11T22:30:51.667042+0000","last_change":"2024-03-11T22:30:41.486573+0000","last_active":"2024-03-11T22:30:51.667042+0000","last_peered":"2024-03-11T22:30:51.667042+0000","last_clean":"2024-03-11T22:30:51.667042+0000","last_became_active":"2024-03-11T22:29:04.701716+0000","last_became_peered":"2024-03-11T22:29:04.701716+0000","last_unstale":"2024-03-11T22:30:51.667042+0000","last_undegraded":"2024-03-11T22:30:51.667042+0000","last_fullsized":"2024-03-11T22:30:51.667042+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:41.486494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:41.486494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:43.179038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":785,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401118+0000","last_change":"2024-03-11T22:29:04.718466+0000","last_active":"2024-03-11T22:30:55.401118+0000","last_peered":"2024-03-11T22:30:55.401118+0000","last_clean":"2024-03-11T22:30:55.401118+0000","last_became_active":"2024-03-11T22:29:04.718122+0000","last_became_peered":"2024-03-11T22:29:04.718122+0000","last_unstale":"2024-03-11T22:30:55.401118+0000","last_undegraded":"2024-03-11T22:30:55.401118+0000","last_fullsized":"2024-03-11T22:30:55.401118+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:10.104642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:10.104642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":297,"num_read_kb":292,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":6574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1035,"ondisk_log_size":1035,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005329,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":841604,"kb_used_data":2424,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92903548,"statfs":{"total":95995035648,"available":95133233152,"internally_reserved":0,"allocated":2482176,"data_stored":1666660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":859306443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332374,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":691312,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93053840,"statfs":{"total":95995035648,"available":95287132160,"internally_reserved":0,"allocated":2134016,"data_stored":1338790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":705755590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575151,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":751040,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92994112,"statfs":{"total":95995035648,"available":95225970688,"internally_reserved":0,"allocated":2150400,"data_stored":1323310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738677,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957876,"kb_used_data":2088,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92787276,"statfs":{"total":95995035648,"available":95014170624,"internally_reserved":0,"allocated":2138112,"data_stored":1330522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411627,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":647164,"kb_used_data":2352,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93097988,"statfs":{"total":95995035648,"available":95332339712,"internally_reserved":0,"allocated":2408448,"data_stored":1609052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149988,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":777980,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92967172,"statfs":{"total":95995035648,"available":95198384128,"internally_reserved":0,"allocated":2740224,"data_stored":1914126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":793901511},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921057,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":714972,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93030180,"statfs":{"total":95995035648,"available":95262904320,"internally_reserved":0,"allocated":2170880,"data_stored":1346269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659418,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":796032,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92949120,"statfs":{"total":95995035648,"available":95179898880,"internally_reserved":0,"allocated":2740224,"data_stored":1928569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":812382662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:30:58.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:30:58.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:58 smithi144 ceph-mon[33984]: 1.8 deep-scrub starts 2024-03-11T22:30:58.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:58 smithi144 ceph-mon[33984]: 1.8 deep-scrub ok 2024-03-11T22:30:58.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:58 smithi144 ceph-mon[33984]: from='client.15975 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:30:58.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:58 smithi144 ceph-mon[33984]: 4.14 scrub starts 2024-03-11T22:30:58.846 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:58 smithi144 ceph-mon[33984]: 4.14 scrub ok 2024-03-11T22:30:59.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:58 smithi045 ceph-mon[30178]: 1.8 deep-scrub starts 2024-03-11T22:30:59.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:58 smithi045 ceph-mon[30178]: 1.8 deep-scrub ok 2024-03-11T22:30:59.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:58 smithi045 ceph-mon[30178]: from='client.15975 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:30:59.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:58 smithi045 ceph-mon[30178]: 4.14 scrub starts 2024-03-11T22:30:59.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:58 smithi045 ceph-mon[30178]: 4.14 scrub ok 2024-03-11T22:30:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:58 smithi045 ceph-mon[39715]: 1.8 deep-scrub starts 2024-03-11T22:30:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:58 smithi045 ceph-mon[39715]: 1.8 deep-scrub ok 2024-03-11T22:30:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:58 smithi045 ceph-mon[39715]: from='client.15975 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:30:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:58 smithi045 ceph-mon[39715]: 4.14 scrub starts 2024-03-11T22:30:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:58 smithi045 ceph-mon[39715]: 4.14 scrub ok 2024-03-11T22:30:59.077 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:30:59.077 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:30:59.099 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:30:59.664 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T22:30:59.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:59 smithi045 ceph-mon[30178]: pgmap v1107: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:30:59.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:59 smithi045 ceph-mon[30178]: 3.14 scrub starts 2024-03-11T22:30:59.786 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:30:59 smithi045 ceph-mon[30178]: 3.14 scrub ok 2024-03-11T22:30:59.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:59 smithi045 ceph-mon[39715]: pgmap v1107: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:30:59.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:59 smithi045 ceph-mon[39715]: 3.14 scrub starts 2024-03-11T22:30:59.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:30:59 smithi045 ceph-mon[39715]: 3.14 scrub ok 2024-03-11T22:30:59.905 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:31:00.028 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:59 smithi144 ceph-mon[33984]: pgmap v1107: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:31:00.029 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:59 smithi144 ceph-mon[33984]: 3.14 scrub starts 2024-03-11T22:31:00.029 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:30:59 smithi144 ceph-mon[33984]: 3.14 scrub ok 2024-03-11T22:31:01.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:00 smithi144 ceph-mon[33984]: 5.4 scrub starts 2024-03-11T22:31:01.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:00 smithi144 ceph-mon[33984]: 5.4 scrub ok 2024-03-11T22:31:01.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:00 smithi144 ceph-mon[33984]: 5.c scrub starts 2024-03-11T22:31:01.018 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:00 smithi144 ceph-mon[33984]: 5.c scrub ok 2024-03-11T22:31:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:00 smithi045 ceph-mon[39715]: 5.4 scrub starts 2024-03-11T22:31:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:00 smithi045 ceph-mon[39715]: 5.4 scrub ok 2024-03-11T22:31:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:00 smithi045 ceph-mon[39715]: 5.c scrub starts 2024-03-11T22:31:01.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:00 smithi045 ceph-mon[39715]: 5.c scrub ok 2024-03-11T22:31:01.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:00 smithi045 ceph-mon[30178]: 5.4 scrub starts 2024-03-11T22:31:01.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:00 smithi045 ceph-mon[30178]: 5.4 scrub ok 2024-03-11T22:31:01.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:00 smithi045 ceph-mon[30178]: 5.c scrub starts 2024-03-11T22:31:01.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:00 smithi045 ceph-mon[30178]: 5.c scrub ok 2024-03-11T22:31:01.270 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:31:01.273 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:31:01.548 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:31:01.548 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:31:01.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:01 smithi144 ceph-mon[33984]: pgmap v1108: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:31:01.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:01.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:01 smithi045 ceph-mon[39715]: pgmap v1108: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:31:01.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:01.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:01 smithi045 ceph-mon[30178]: pgmap v1108: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:31:01.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:01.972 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":1108,"stamp":"2024-03-11T22:31:00.534756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1846,"num_read_kb":1598,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":353,"num_bytes_recovered":336286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1137,"ondisk_log_size":1137,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6178080,"kb_used_data":18628,"kb_used_omap":12,"kb_used_meta":6159347,"kb_avail":743783136,"statfs":{"total":767960285184,"available":761633931264,"internally_reserved":0,"allocated":19075072,"data_stored":12527156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":6307171891},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002000"},"pg_stats":[{"pgid":"4.c","version":"479'35","reported_seq":835,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392097+0000","last_change":"2024-03-11T22:30:04.763741+0000","last_active":"2024-03-11T22:30:55.392097+0000","last_peered":"2024-03-11T22:30:55.392097+0000","last_clean":"2024-03-11T22:30:55.392097+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:30:55.392097+0000","last_undegraded":"2024-03-11T22:30:55.392097+0000","last_fullsized":"2024-03-11T22:30:55.392097+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'35","last_scrub_stamp":"2024-03-11T22:30:04.763606+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:30:04.763606+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:24.323735+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1007,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391437+0000","last_change":"2024-03-11T22:30:09.860422+0000","last_active":"2024-03-11T22:30:55.391437+0000","last_peered":"2024-03-11T22:30:55.391437+0000","last_clean":"2024-03-11T22:30:55.391437+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:30:55.391437+0000","last_undegraded":"2024-03-11T22:30:55.391437+0000","last_fullsized":"2024-03-11T22:30:55.391437+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:09.860304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:09.860304+0000","last_clean_scrub_stamp":"2024-03-11T22:30:09.860304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:31.943754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":1318,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705102+0000","last_change":"2024-03-11T22:30:19.637446+0000","last_active":"2024-03-11T22:30:54.705102+0000","last_peered":"2024-03-11T22:30:54.705102+0000","last_clean":"2024-03-11T22:30:54.705102+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:30:54.705102+0000","last_undegraded":"2024-03-11T22:30:54.705102+0000","last_fullsized":"2024-03-11T22:30:54.705102+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:30:19.637348+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:26:11.939962+0000","last_clean_scrub_stamp":"2024-03-11T22:30:19.637348+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:40.836875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690185+0000","last_change":"2024-03-11T22:30:55.690226+0000","last_active":"2024-03-11T22:30:55.690185+0000","last_peered":"2024-03-11T22:30:55.690185+0000","last_clean":"2024-03-11T22:30:55.690185+0000","last_became_active":"2024-03-11T22:29:04.727120+0000","last_became_peered":"2024-03-11T22:29:04.727120+0000","last_unstale":"2024-03-11T22:30:55.690185+0000","last_undegraded":"2024-03-11T22:30:55.690185+0000","last_fullsized":"2024-03-11T22:30:55.690185+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:55.689796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:49.601908+0000","last_clean_scrub_stamp":"2024-03-11T22:30:55.689796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:12.015039+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":1353,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391270+0000","last_change":"2024-03-11T22:26:43.904135+0000","last_active":"2024-03-11T22:30:55.391270+0000","last_peered":"2024-03-11T22:30:55.391270+0000","last_clean":"2024-03-11T22:30:55.391270+0000","last_became_active":"2024-03-11T22:20:32.293356+0000","last_became_peered":"2024-03-11T22:20:32.293356+0000","last_unstale":"2024-03-11T22:30:55.391270+0000","last_undegraded":"2024-03-11T22:30:55.391270+0000","last_fullsized":"2024-03-11T22:30:55.391270+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:43.904050+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:43.904050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"479'53","reported_seq":944,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704828+0000","last_change":"2024-03-11T22:30:04.672014+0000","last_active":"2024-03-11T22:30:54.704828+0000","last_peered":"2024-03-11T22:30:54.704828+0000","last_clean":"2024-03-11T22:30:54.704828+0000","last_became_active":"2024-03-11T22:29:04.695558+0000","last_became_peered":"2024-03-11T22:29:04.695558+0000","last_unstale":"2024-03-11T22:30:54.704828+0000","last_undegraded":"2024-03-11T22:30:54.704828+0000","last_fullsized":"2024-03-11T22:30:54.704828+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'53","last_scrub_stamp":"2024-03-11T22:30:04.671906+0000","last_deep_scrub":"353'39","last_deep_scrub_stamp":"2024-03-11T22:23:05.461729+0000","last_clean_scrub_stamp":"2024-03-11T22:30:04.671906+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:28.141862+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1043,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672488+0000","last_change":"2024-03-11T22:29:04.719200+0000","last_active":"2024-03-11T22:30:55.672488+0000","last_peered":"2024-03-11T22:30:55.672488+0000","last_clean":"2024-03-11T22:30:55.672488+0000","last_became_active":"2024-03-11T22:29:04.718290+0000","last_became_peered":"2024-03-11T22:29:04.718290+0000","last_unstale":"2024-03-11T22:30:55.672488+0000","last_undegraded":"2024-03-11T22:30:55.672488+0000","last_fullsized":"2024-03-11T22:30:55.672488+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:28:21.605401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:28:21.605401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":114,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705903+0000","last_change":"2024-03-11T22:29:04.706086+0000","last_active":"2024-03-11T22:30:54.705903+0000","last_peered":"2024-03-11T22:30:54.705903+0000","last_clean":"2024-03-11T22:30:54.705903+0000","last_became_active":"2024-03-11T22:29:04.705661+0000","last_became_peered":"2024-03-11T22:29:04.705661+0000","last_unstale":"2024-03-11T22:30:54.705903+0000","last_undegraded":"2024-03-11T22:30:54.705903+0000","last_fullsized":"2024-03-11T22:30:54.705903+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:50.571821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.571821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1028,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704725+0000","last_change":"2024-03-11T22:30:10.839723+0000","last_active":"2024-03-11T22:30:54.704725+0000","last_peered":"2024-03-11T22:30:54.704725+0000","last_clean":"2024-03-11T22:30:54.704725+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:30:54.704725+0000","last_undegraded":"2024-03-11T22:30:54.704725+0000","last_fullsized":"2024-03-11T22:30:54.704725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:10.839608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:10.839608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:30.826822+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":1329,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401170+0000","last_change":"2024-03-11T22:26:03.097798+0000","last_active":"2024-03-11T22:30:55.401170+0000","last_peered":"2024-03-11T22:30:55.401170+0000","last_clean":"2024-03-11T22:30:55.401170+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:30:55.401170+0000","last_undegraded":"2024-03-11T22:30:55.401170+0000","last_fullsized":"2024-03-11T22:30:55.401170+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:03.097636+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:03.097636+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"516'32","reported_seq":910,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672880+0000","last_change":"2024-03-11T22:30:16.937051+0000","last_active":"2024-03-11T22:30:55.672880+0000","last_peered":"2024-03-11T22:30:55.672880+0000","last_clean":"2024-03-11T22:30:55.672880+0000","last_became_active":"2024-03-11T22:30:16.930541+0000","last_became_peered":"2024-03-11T22:30:16.930541+0000","last_unstale":"2024-03-11T22:30:55.672880+0000","last_undegraded":"2024-03-11T22:30:55.672880+0000","last_fullsized":"2024-03-11T22:30:55.672880+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'32","last_scrub_stamp":"2024-03-11T22:30:10.404123+0000","last_deep_scrub":"516'32","last_deep_scrub_stamp":"2024-03-11T22:30:10.404123+0000","last_clean_scrub_stamp":"2024-03-11T22:30:10.404123+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:34.707986+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":800,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705982+0000","last_change":"2024-03-11T22:29:54.744324+0000","last_active":"2024-03-11T22:30:54.705982+0000","last_peered":"2024-03-11T22:30:54.705982+0000","last_clean":"2024-03-11T22:30:54.705982+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:30:54.705982+0000","last_undegraded":"2024-03-11T22:30:54.705982+0000","last_fullsized":"2024-03-11T22:30:54.705982+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:54.744093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:29:54.744093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:30:58.841387+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1007,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401878+0000","last_change":"2024-03-11T22:30:40.303604+0000","last_active":"2024-03-11T22:30:55.401878+0000","last_peered":"2024-03-11T22:30:55.401878+0000","last_clean":"2024-03-11T22:30:55.401878+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:30:55.401878+0000","last_undegraded":"2024-03-11T22:30:55.401878+0000","last_fullsized":"2024-03-11T22:30:55.401878+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:40.303527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.303527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:46.392625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":987,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391638+0000","last_change":"2024-03-11T22:26:48.845265+0000","last_active":"2024-03-11T22:30:55.391638+0000","last_peered":"2024-03-11T22:30:55.391638+0000","last_clean":"2024-03-11T22:30:55.391638+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:30:55.391638+0000","last_undegraded":"2024-03-11T22:30:55.391638+0000","last_fullsized":"2024-03-11T22:30:55.391638+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:48.845177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:48.845177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"480'46","reported_seq":138,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706706+0000","last_change":"2024-03-11T22:30:46.625195+0000","last_active":"2024-03-11T22:30:54.706706+0000","last_peered":"2024-03-11T22:30:54.706706+0000","last_clean":"2024-03-11T22:30:54.706706+0000","last_became_active":"2024-03-11T22:29:04.726632+0000","last_became_peered":"2024-03-11T22:29:04.726632+0000","last_unstale":"2024-03-11T22:30:54.706706+0000","last_undegraded":"2024-03-11T22:30:54.706706+0000","last_fullsized":"2024-03-11T22:30:54.706706+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'46","last_scrub_stamp":"2024-03-11T22:30:46.625090+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:30:46.625090+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:06.299995+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1032,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391355+0000","last_change":"2024-03-11T22:30:45.799130+0000","last_active":"2024-03-11T22:30:55.391355+0000","last_peered":"2024-03-11T22:30:55.391355+0000","last_clean":"2024-03-11T22:30:55.391355+0000","last_became_active":"2024-03-11T22:29:04.701072+0000","last_became_peered":"2024-03-11T22:29:04.701072+0000","last_unstale":"2024-03-11T22:30:55.391355+0000","last_undegraded":"2024-03-11T22:30:55.391355+0000","last_fullsized":"2024-03-11T22:30:55.391355+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:45.799037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:45.799037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:49.472786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":993,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401782+0000","last_change":"2024-03-11T22:29:57.068709+0000","last_active":"2024-03-11T22:30:55.401782+0000","last_peered":"2024-03-11T22:30:55.401782+0000","last_clean":"2024-03-11T22:30:55.401782+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:30:55.401782+0000","last_undegraded":"2024-03-11T22:30:55.401782+0000","last_fullsized":"2024-03-11T22:30:55.401782+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:29:57.068625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:29:57.068625+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:01.261357+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1007,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401537+0000","last_change":"2024-03-11T22:29:04.787987+0000","last_active":"2024-03-11T22:30:55.401537+0000","last_peered":"2024-03-11T22:30:55.401537+0000","last_clean":"2024-03-11T22:30:55.401537+0000","last_became_active":"2024-03-11T22:29:04.787650+0000","last_became_peered":"2024-03-11T22:29:04.787650+0000","last_unstale":"2024-03-11T22:30:55.401537+0000","last_undegraded":"2024-03-11T22:30:55.401537+0000","last_fullsized":"2024-03-11T22:30:55.401537+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:36.095984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:26:36.095984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":1338,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401204+0000","last_change":"2024-03-11T22:30:43.217270+0000","last_active":"2024-03-11T22:30:55.401204+0000","last_peered":"2024-03-11T22:30:55.401204+0000","last_clean":"2024-03-11T22:30:55.401204+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:30:55.401204+0000","last_undegraded":"2024-03-11T22:30:55.401204+0000","last_fullsized":"2024-03-11T22:30:55.401204+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:30:43.217187+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:30:43.217187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:01.077102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":784,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391616+0000","last_change":"2024-03-11T22:26:09.040467+0000","last_active":"2024-03-11T22:30:55.391616+0000","last_peered":"2024-03-11T22:30:55.391616+0000","last_clean":"2024-03-11T22:30:55.391616+0000","last_became_active":"2024-03-11T22:20:29.438896+0000","last_became_peered":"2024-03-11T22:20:29.438896+0000","last_unstale":"2024-03-11T22:30:55.391616+0000","last_undegraded":"2024-03-11T22:30:55.391616+0000","last_fullsized":"2024-03-11T22:30:55.391616+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:09.040331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:09.040331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"479'42","reported_seq":1116,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391383+0000","last_change":"2024-03-11T22:26:05.020921+0000","last_active":"2024-03-11T22:30:55.391383+0000","last_peered":"2024-03-11T22:30:55.391383+0000","last_clean":"2024-03-11T22:30:55.391383+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:30:55.391383+0000","last_undegraded":"2024-03-11T22:30:55.391383+0000","last_fullsized":"2024-03-11T22:30:55.391383+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'30","last_scrub_stamp":"2024-03-11T22:26:05.020726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:05.020726+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1020,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392410+0000","last_change":"2024-03-11T22:30:01.607109+0000","last_active":"2024-03-11T22:30:55.392410+0000","last_peered":"2024-03-11T22:30:55.392410+0000","last_clean":"2024-03-11T22:30:55.392410+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:30:55.392410+0000","last_undegraded":"2024-03-11T22:30:55.392410+0000","last_fullsized":"2024-03-11T22:30:55.392410+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:01.606988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:01.606988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:25.665371+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":766,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391628+0000","last_change":"2024-03-11T22:25:51.005350+0000","last_active":"2024-03-11T22:30:55.391628+0000","last_peered":"2024-03-11T22:30:55.391628+0000","last_clean":"2024-03-11T22:30:55.391628+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:30:55.391628+0000","last_undegraded":"2024-03-11T22:30:55.391628+0000","last_fullsized":"2024-03-11T22:30:55.391628+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:51.005219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:51.005219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":1309,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705189+0000","last_change":"2024-03-11T22:29:55.711127+0000","last_active":"2024-03-11T22:30:54.705189+0000","last_peered":"2024-03-11T22:30:54.705189+0000","last_clean":"2024-03-11T22:30:54.705189+0000","last_became_active":"2024-03-11T22:26:49.698651+0000","last_became_peered":"2024-03-11T22:26:49.698651+0000","last_unstale":"2024-03-11T22:30:54.705189+0000","last_undegraded":"2024-03-11T22:30:54.705189+0000","last_fullsized":"2024-03-11T22:30:54.705189+0000","mapping_epoch":175,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:29:55.710911+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:25:05.773963+0000","last_clean_scrub_stamp":"2024-03-11T22:29:55.710911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:07.856356+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"488'10","reported_seq":1291,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391481+0000","last_change":"2024-03-11T22:29:04.839059+0000","last_active":"2024-03-11T22:30:55.391481+0000","last_peered":"2024-03-11T22:30:55.391481+0000","last_clean":"2024-03-11T22:30:55.391481+0000","last_became_active":"2024-03-11T22:29:04.721065+0000","last_became_peered":"2024-03-11T22:29:04.721065+0000","last_unstale":"2024-03-11T22:30:55.391481+0000","last_undegraded":"2024-03-11T22:30:55.391481+0000","last_fullsized":"2024-03-11T22:30:55.391481+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:26:13.061151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:13.061151+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"480'42","reported_seq":856,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401107+0000","last_change":"2024-03-11T22:30:48.353744+0000","last_active":"2024-03-11T22:30:55.401107+0000","last_peered":"2024-03-11T22:30:55.401107+0000","last_clean":"2024-03-11T22:30:55.401107+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:30:55.401107+0000","last_undegraded":"2024-03-11T22:30:55.401107+0000","last_fullsized":"2024-03-11T22:30:55.401107+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'42","last_scrub_stamp":"2024-03-11T22:30:48.353647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:48.353647+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:14.430459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":1036,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690460+0000","last_change":"2024-03-11T22:30:42.177735+0000","last_active":"2024-03-11T22:30:55.690460+0000","last_peered":"2024-03-11T22:30:55.690460+0000","last_clean":"2024-03-11T22:30:55.690460+0000","last_became_active":"2024-03-11T22:29:04.718160+0000","last_became_peered":"2024-03-11T22:29:04.718160+0000","last_unstale":"2024-03-11T22:30:55.690460+0000","last_undegraded":"2024-03-11T22:30:55.690460+0000","last_fullsized":"2024-03-11T22:30:55.690460+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:30:42.177651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:42.177651+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:52.743855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":961,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672538+0000","last_change":"2024-03-11T22:28:34.728469+0000","last_active":"2024-03-11T22:30:55.672538+0000","last_peered":"2024-03-11T22:30:55.672538+0000","last_clean":"2024-03-11T22:30:55.672538+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:30:55.672538+0000","last_undegraded":"2024-03-11T22:30:55.672538+0000","last_fullsized":"2024-03-11T22:30:55.672538+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:28:34.728337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:28:34.728337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":1366,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391508+0000","last_change":"2024-03-11T22:26:53.558080+0000","last_active":"2024-03-11T22:30:55.391508+0000","last_peered":"2024-03-11T22:30:55.391508+0000","last_clean":"2024-03-11T22:30:55.391508+0000","last_became_active":"2024-03-11T22:26:53.557811+0000","last_became_peered":"2024-03-11T22:26:53.557811+0000","last_unstale":"2024-03-11T22:30:55.391508+0000","last_undegraded":"2024-03-11T22:30:55.391508+0000","last_fullsized":"2024-03-11T22:30:55.391508+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:45.837466+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:45.837466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"378'9","reported_seq":1285,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.691199+0000","last_change":"2024-03-11T22:30:38.192095+0000","last_active":"2024-03-11T22:30:55.691199+0000","last_peered":"2024-03-11T22:30:55.691199+0000","last_clean":"2024-03-11T22:30:55.691199+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:30:55.691199+0000","last_undegraded":"2024-03-11T22:30:55.691199+0000","last_fullsized":"2024-03-11T22:30:55.691199+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9","last_scrub_stamp":"2024-03-11T22:30:38.191965+0000","last_deep_scrub":"378'9","last_deep_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_clean_scrub_stamp":"2024-03-11T22:30:38.191965+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:58.989567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'30","reported_seq":134,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705358+0000","last_change":"2024-03-11T22:30:02.607278+0000","last_active":"2024-03-11T22:30:54.705358+0000","last_peered":"2024-03-11T22:30:54.705358+0000","last_clean":"2024-03-11T22:30:54.705358+0000","last_became_active":"2024-03-11T22:29:04.727100+0000","last_became_peered":"2024-03-11T22:29:04.727100+0000","last_unstale":"2024-03-11T22:30:54.705358+0000","last_undegraded":"2024-03-11T22:30:54.705358+0000","last_fullsized":"2024-03-11T22:30:54.705358+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-03-11T22:30:02.607142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:02.607142+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:31.923645+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1016,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401706+0000","last_change":"2024-03-11T22:29:54.153237+0000","last_active":"2024-03-11T22:30:55.401706+0000","last_peered":"2024-03-11T22:30:55.401706+0000","last_clean":"2024-03-11T22:30:55.401706+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:30:55.401706+0000","last_undegraded":"2024-03-11T22:30:55.401706+0000","last_fullsized":"2024-03-11T22:30:55.401706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:54.153119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:29:54.153119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:01.613626+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1004,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.689702+0000","last_change":"2024-03-11T22:30:45.264761+0000","last_active":"2024-03-11T22:30:55.689702+0000","last_peered":"2024-03-11T22:30:55.689702+0000","last_clean":"2024-03-11T22:30:55.689702+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:30:55.689702+0000","last_undegraded":"2024-03-11T22:30:55.689702+0000","last_fullsized":"2024-03-11T22:30:55.689702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:45.264658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:45.264658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:14.204199+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":1288,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673132+0000","last_change":"2024-03-11T22:30:39.559483+0000","last_active":"2024-03-11T22:30:55.673132+0000","last_peered":"2024-03-11T22:30:55.673132+0000","last_clean":"2024-03-11T22:30:55.673132+0000","last_became_active":"2024-03-11T22:26:41.661375+0000","last_became_peered":"2024-03-11T22:26:41.661375+0000","last_unstale":"2024-03-11T22:30:55.673132+0000","last_undegraded":"2024-03-11T22:30:55.673132+0000","last_fullsized":"2024-03-11T22:30:55.673132+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":423,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:30:39.559403+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:30:39.559403+0000","last_clean_scrub_stamp":"2024-03-11T22:30:39.559403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:59.861593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1023,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672410+0000","last_change":"2024-03-11T22:28:37.585104+0000","last_active":"2024-03-11T22:30:55.672410+0000","last_peered":"2024-03-11T22:30:55.672410+0000","last_clean":"2024-03-11T22:30:55.672410+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:30:55.672410+0000","last_undegraded":"2024-03-11T22:30:55.672410+0000","last_fullsized":"2024-03-11T22:30:55.672410+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:28:37.584991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:28:37.584991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"479'47","reported_seq":1077,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690929+0000","last_change":"2024-03-11T22:26:47.889297+0000","last_active":"2024-03-11T22:30:55.690929+0000","last_peered":"2024-03-11T22:30:55.690929+0000","last_clean":"2024-03-11T22:30:55.690929+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:30:55.690929+0000","last_undegraded":"2024-03-11T22:30:55.690929+0000","last_fullsized":"2024-03-11T22:30:55.690929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'37","last_scrub_stamp":"2024-03-11T22:26:47.889156+0000","last_deep_scrub":"429'37","last_deep_scrub_stamp":"2024-03-11T22:26:47.889156+0000","last_clean_scrub_stamp":"2024-03-11T22:26:47.889156+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706131+0000","last_change":"2024-03-11T22:30:47.657180+0000","last_active":"2024-03-11T22:30:54.706131+0000","last_peered":"2024-03-11T22:30:54.706131+0000","last_clean":"2024-03-11T22:30:54.706131+0000","last_became_active":"2024-03-11T22:29:04.722813+0000","last_became_peered":"2024-03-11T22:29:04.722813+0000","last_unstale":"2024-03-11T22:30:54.706131+0000","last_undegraded":"2024-03-11T22:30:54.706131+0000","last_fullsized":"2024-03-11T22:30:54.706131+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:47.657052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:04.706082+0000","last_clean_scrub_stamp":"2024-03-11T22:30:47.657052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:03.794039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704665+0000","last_change":"2024-03-11T22:30:52.727011+0000","last_active":"2024-03-11T22:30:54.704665+0000","last_peered":"2024-03-11T22:30:54.704665+0000","last_clean":"2024-03-11T22:30:54.704665+0000","last_became_active":"2024-03-11T22:29:04.729455+0000","last_became_peered":"2024-03-11T22:29:04.729455+0000","last_unstale":"2024-03-11T22:30:54.704665+0000","last_undegraded":"2024-03-11T22:30:54.704665+0000","last_fullsized":"2024-03-11T22:30:54.704665+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:52.726908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:52.726908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:07.437314+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":1263,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673111+0000","last_change":"2024-03-11T22:30:52.611262+0000","last_active":"2024-03-11T22:30:55.673111+0000","last_peered":"2024-03-11T22:30:55.673111+0000","last_clean":"2024-03-11T22:30:55.673111+0000","last_became_active":"2024-03-11T22:26:45.688609+0000","last_became_peered":"2024-03-11T22:26:45.688609+0000","last_unstale":"2024-03-11T22:30:55.673111+0000","last_undegraded":"2024-03-11T22:30:55.673111+0000","last_fullsized":"2024-03-11T22:30:55.673111+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:30:52.611147+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:30:52.611147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:05.456604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1020,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401905+0000","last_change":"2024-03-11T22:30:41.301144+0000","last_active":"2024-03-11T22:30:55.401905+0000","last_peered":"2024-03-11T22:30:55.401905+0000","last_clean":"2024-03-11T22:30:55.401905+0000","last_became_active":"2024-03-11T22:29:04.718259+0000","last_became_peered":"2024-03-11T22:29:04.718259+0000","last_unstale":"2024-03-11T22:30:55.401905+0000","last_undegraded":"2024-03-11T22:30:55.401905+0000","last_fullsized":"2024-03-11T22:30:55.401905+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:41.301047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:41.301047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:43.242569+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"479'36","reported_seq":1105,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690327+0000","last_change":"2024-03-11T22:29:04.818952+0000","last_active":"2024-03-11T22:30:55.690327+0000","last_peered":"2024-03-11T22:30:55.690327+0000","last_clean":"2024-03-11T22:30:55.690327+0000","last_became_active":"2024-03-11T22:29:04.720793+0000","last_became_peered":"2024-03-11T22:29:04.720793+0000","last_unstale":"2024-03-11T22:30:55.690327+0000","last_undegraded":"2024-03-11T22:30:55.690327+0000","last_fullsized":"2024-03-11T22:30:55.690327+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'24","last_scrub_stamp":"2024-03-11T22:26:35.660739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:35.660739+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":238,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.002162+0000","last_change":"2024-03-11T22:30:54.684229+0000","last_active":"2024-03-11T22:30:55.002162+0000","last_peered":"2024-03-11T22:30:55.002162+0000","last_clean":"2024-03-11T22:30:55.002162+0000","last_became_active":"2024-03-11T22:29:04.702609+0000","last_became_peered":"2024-03-11T22:29:04.702609+0000","last_unstale":"2024-03-11T22:30:55.002162+0000","last_undegraded":"2024-03-11T22:30:55.002162+0000","last_fullsized":"2024-03-11T22:30:55.002162+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:30:54.684108+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:30:54.684108+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:07.958873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":302,"num_read_kb":297,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690826+0000","last_change":"2024-03-11T22:30:08.418564+0000","last_active":"2024-03-11T22:30:55.690826+0000","last_peered":"2024-03-11T22:30:55.690826+0000","last_clean":"2024-03-11T22:30:55.690826+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:30:55.690826+0000","last_undegraded":"2024-03-11T22:30:55.690826+0000","last_fullsized":"2024-03-11T22:30:55.690826+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:08.418488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:08.418488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:35.702001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705994+0000","last_change":"2024-03-11T22:30:51.665235+0000","last_active":"2024-03-11T22:30:54.705994+0000","last_peered":"2024-03-11T22:30:54.705994+0000","last_clean":"2024-03-11T22:30:54.705994+0000","last_became_active":"2024-03-11T22:29:04.696141+0000","last_became_peered":"2024-03-11T22:29:04.696141+0000","last_unstale":"2024-03-11T22:30:54.705994+0000","last_undegraded":"2024-03-11T22:30:54.705994+0000","last_fullsized":"2024-03-11T22:30:54.705994+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:51.665137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:22.750560+0000","last_clean_scrub_stamp":"2024-03-11T22:30:51.665137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:54.684247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":571,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706033+0000","last_change":"2024-03-11T22:29:49.468313+0000","last_active":"2024-03-11T22:30:54.706033+0000","last_peered":"2024-03-11T22:30:54.706033+0000","last_clean":"2024-03-11T22:30:54.706033+0000","last_became_active":"2024-03-11T22:29:37.643184+0000","last_became_peered":"2024-03-11T22:29:37.643184+0000","last_unstale":"2024-03-11T22:30:54.706033+0000","last_undegraded":"2024-03-11T22:30:54.706033+0000","last_fullsized":"2024-03-11T22:30:54.706033+0000","mapping_epoch":502,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:29:49.467957+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:28:24.623016+0000","last_clean_scrub_stamp":"2024-03-11T22:29:49.467957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.316172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":983,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705621+0000","last_change":"2024-03-11T22:26:42.960839+0000","last_active":"2024-03-11T22:30:54.705621+0000","last_peered":"2024-03-11T22:30:54.705621+0000","last_clean":"2024-03-11T22:30:54.705621+0000","last_became_active":"2024-03-11T22:20:29.420497+0000","last_became_peered":"2024-03-11T22:20:29.420497+0000","last_unstale":"2024-03-11T22:30:54.705621+0000","last_undegraded":"2024-03-11T22:30:54.705621+0000","last_fullsized":"2024-03-11T22:30:54.705621+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:42.960739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:26:42.960739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"514'48","reported_seq":1362,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672890+0000","last_change":"2024-03-11T22:30:05.380085+0000","last_active":"2024-03-11T22:30:55.672890+0000","last_peered":"2024-03-11T22:30:55.672890+0000","last_clean":"2024-03-11T22:30:55.672890+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:30:55.672890+0000","last_undegraded":"2024-03-11T22:30:55.672890+0000","last_fullsized":"2024-03-11T22:30:55.672890+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"514'48","last_scrub_stamp":"2024-03-11T22:30:05.379958+0000","last_deep_scrub":"514'48","last_deep_scrub_stamp":"2024-03-11T22:30:05.379958+0000","last_clean_scrub_stamp":"2024-03-11T22:30:05.379958+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:12.891919+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1008,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391301+0000","last_change":"2024-03-11T22:30:40.798528+0000","last_active":"2024-03-11T22:30:55.391301+0000","last_peered":"2024-03-11T22:30:55.391301+0000","last_clean":"2024-03-11T22:30:55.391301+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:30:55.391301+0000","last_undegraded":"2024-03-11T22:30:55.391301+0000","last_fullsized":"2024-03-11T22:30:55.391301+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:40.798443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.798443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:54.912063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":1363,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.689975+0000","last_change":"2024-03-11T22:30:06.479012+0000","last_active":"2024-03-11T22:30:55.689975+0000","last_peered":"2024-03-11T22:30:55.689975+0000","last_clean":"2024-03-11T22:30:55.689975+0000","last_became_active":"2024-03-11T22:26:37.632412+0000","last_became_peered":"2024-03-11T22:26:37.632412+0000","last_unstale":"2024-03-11T22:30:55.689975+0000","last_undegraded":"2024-03-11T22:30:55.689975+0000","last_fullsized":"2024-03-11T22:30:55.689975+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":19,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:30:06.478921+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:30:06.478921+0000","last_clean_scrub_stamp":"2024-03-11T22:30:06.478921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:10.660047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1004,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.400948+0000","last_change":"2024-03-11T22:30:39.454150+0000","last_active":"2024-03-11T22:30:55.400948+0000","last_peered":"2024-03-11T22:30:55.400948+0000","last_clean":"2024-03-11T22:30:55.400948+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:30:55.400948+0000","last_undegraded":"2024-03-11T22:30:55.400948+0000","last_fullsized":"2024-03-11T22:30:55.400948+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:39.454015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:39.454015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:49.055198+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1033,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704591+0000","last_change":"2024-03-11T22:30:08.659249+0000","last_active":"2024-03-11T22:30:54.704591+0000","last_peered":"2024-03-11T22:30:54.704591+0000","last_clean":"2024-03-11T22:30:54.704591+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:30:54.704591+0000","last_undegraded":"2024-03-11T22:30:54.704591+0000","last_fullsized":"2024-03-11T22:30:54.704591+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:08.659154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:08.659154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:16.522418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'21","reported_seq":134,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706596+0000","last_change":"2024-03-11T22:30:05.683743+0000","last_active":"2024-03-11T22:30:54.706596+0000","last_peered":"2024-03-11T22:30:54.706596+0000","last_clean":"2024-03-11T22:30:54.706596+0000","last_became_active":"2024-03-11T22:29:04.705980+0000","last_became_peered":"2024-03-11T22:29:04.705980+0000","last_unstale":"2024-03-11T22:30:54.706596+0000","last_undegraded":"2024-03-11T22:30:54.706596+0000","last_fullsized":"2024-03-11T22:30:54.706596+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'21","last_scrub_stamp":"2024-03-11T22:30:05.683641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:05.683641+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:09.121346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1058,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706077+0000","last_change":"2024-03-11T22:30:15.663376+0000","last_active":"2024-03-11T22:30:54.706077+0000","last_peered":"2024-03-11T22:30:54.706077+0000","last_clean":"2024-03-11T22:30:54.706077+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:30:54.706077+0000","last_undegraded":"2024-03-11T22:30:54.706077+0000","last_fullsized":"2024-03-11T22:30:54.706077+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:15.663298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:30:15.663298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:39.777684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705251+0000","last_change":"2024-03-11T22:29:54.581275+0000","last_active":"2024-03-11T22:30:54.705251+0000","last_peered":"2024-03-11T22:30:54.705251+0000","last_clean":"2024-03-11T22:30:54.705251+0000","last_became_active":"2024-03-11T22:29:04.713225+0000","last_became_peered":"2024-03-11T22:29:04.713225+0000","last_unstale":"2024-03-11T22:30:54.705251+0000","last_undegraded":"2024-03-11T22:30:54.705251+0000","last_fullsized":"2024-03-11T22:30:54.705251+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:54.581156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:29:54.581156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.689997+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":961,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672907+0000","last_change":"2024-03-11T22:30:06.412273+0000","last_active":"2024-03-11T22:30:55.672907+0000","last_peered":"2024-03-11T22:30:55.672907+0000","last_clean":"2024-03-11T22:30:55.672907+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:30:55.672907+0000","last_undegraded":"2024-03-11T22:30:55.672907+0000","last_fullsized":"2024-03-11T22:30:55.672907+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:06.412160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:30:06.412160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:19.024605+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":1370,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690025+0000","last_change":"2024-03-11T22:26:32.769183+0000","last_active":"2024-03-11T22:30:55.690025+0000","last_peered":"2024-03-11T22:30:55.690025+0000","last_clean":"2024-03-11T22:30:55.690025+0000","last_became_active":"2024-03-11T22:26:29.590669+0000","last_became_peered":"2024-03-11T22:26:29.590669+0000","last_unstale":"2024-03-11T22:30:55.690025+0000","last_undegraded":"2024-03-11T22:30:55.690025+0000","last_fullsized":"2024-03-11T22:30:55.690025+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:32.769096+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:32.769096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1019,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690385+0000","last_change":"2024-03-11T22:30:05.507577+0000","last_active":"2024-03-11T22:30:55.690385+0000","last_peered":"2024-03-11T22:30:55.690385+0000","last_clean":"2024-03-11T22:30:55.690385+0000","last_became_active":"2024-03-11T22:29:04.707688+0000","last_became_peered":"2024-03-11T22:29:04.707688+0000","last_unstale":"2024-03-11T22:30:55.690385+0000","last_undegraded":"2024-03-11T22:30:55.690385+0000","last_fullsized":"2024-03-11T22:30:55.690385+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:05.507462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:05.507462+0000","last_clean_scrub_stamp":"2024-03-11T22:30:05.507462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:31.012916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"520'55","reported_seq":79,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401377+0000","last_change":"2024-03-11T22:30:16.943159+0000","last_active":"2024-03-11T22:30:55.401377+0000","last_peered":"2024-03-11T22:30:55.401377+0000","last_clean":"2024-03-11T22:30:55.401377+0000","last_became_active":"2024-03-11T22:30:16.930211+0000","last_became_peered":"2024-03-11T22:30:16.930211+0000","last_unstale":"2024-03-11T22:30:55.401377+0000","last_undegraded":"2024-03-11T22:30:55.401377+0000","last_fullsized":"2024-03-11T22:30:55.401377+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'54","last_scrub_stamp":"2024-03-11T22:29:52.527782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:29:52.527782+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:20.718332+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":774,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690420+0000","last_change":"2024-03-11T22:26:44.809895+0000","last_active":"2024-03-11T22:30:55.690420+0000","last_peered":"2024-03-11T22:30:55.690420+0000","last_clean":"2024-03-11T22:30:55.690420+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:30:55.690420+0000","last_undegraded":"2024-03-11T22:30:55.690420+0000","last_fullsized":"2024-03-11T22:30:55.690420+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.809803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.809803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":1048,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401277+0000","last_change":"2024-03-11T22:29:04.812619+0000","last_active":"2024-03-11T22:30:55.401277+0000","last_peered":"2024-03-11T22:30:55.401277+0000","last_clean":"2024-03-11T22:30:55.401277+0000","last_became_active":"2024-03-11T22:29:04.718809+0000","last_became_peered":"2024-03-11T22:29:04.718809+0000","last_unstale":"2024-03-11T22:30:55.401277+0000","last_undegraded":"2024-03-11T22:30:55.401277+0000","last_fullsized":"2024-03-11T22:30:55.401277+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:26:31.988426+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:26:31.988426+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1000,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392410+0000","last_change":"2024-03-11T22:25:58.161115+0000","last_active":"2024-03-11T22:30:55.392410+0000","last_peered":"2024-03-11T22:30:55.392410+0000","last_clean":"2024-03-11T22:30:55.392410+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:30:55.392410+0000","last_undegraded":"2024-03-11T22:30:55.392410+0000","last_fullsized":"2024-03-11T22:30:55.392410+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:58.160972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:25:58.160972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":1345,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401081+0000","last_change":"2024-03-11T22:26:34.036467+0000","last_active":"2024-03-11T22:30:55.401081+0000","last_peered":"2024-03-11T22:30:55.401081+0000","last_clean":"2024-03-11T22:30:55.401081+0000","last_became_active":"2024-03-11T22:26:33.607145+0000","last_became_peered":"2024-03-11T22:26:33.607145+0000","last_unstale":"2024-03-11T22:30:55.401081+0000","last_undegraded":"2024-03-11T22:30:55.401081+0000","last_fullsized":"2024-03-11T22:30:55.401081+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:34.036387+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:34.036387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"479'21","reported_seq":1084,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.691092+0000","last_change":"2024-03-11T22:29:04.732685+0000","last_active":"2024-03-11T22:30:55.691092+0000","last_peered":"2024-03-11T22:30:55.691092+0000","last_clean":"2024-03-11T22:30:55.691092+0000","last_became_active":"2024-03-11T22:29:04.707867+0000","last_became_peered":"2024-03-11T22:29:04.707867+0000","last_unstale":"2024-03-11T22:30:55.691092+0000","last_undegraded":"2024-03-11T22:30:55.691092+0000","last_fullsized":"2024-03-11T22:30:55.691092+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:26:12.661479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:12.661479+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401587+0000","last_change":"2024-03-11T22:30:50.377246+0000","last_active":"2024-03-11T22:30:55.401587+0000","last_peered":"2024-03-11T22:30:55.401587+0000","last_clean":"2024-03-11T22:30:55.401587+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:30:55.401587+0000","last_undegraded":"2024-03-11T22:30:55.401587+0000","last_fullsized":"2024-03-11T22:30:55.401587+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:50.377151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:50.377151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:15.151224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1041,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.393858+0000","last_change":"2024-03-11T22:30:55.393883+0000","last_active":"2024-03-11T22:30:55.393858+0000","last_peered":"2024-03-11T22:30:55.393858+0000","last_clean":"2024-03-11T22:30:55.393858+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:30:55.393858+0000","last_undegraded":"2024-03-11T22:30:55.393858+0000","last_fullsized":"2024-03-11T22:30:55.393858+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:55.393786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:30:55.393786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:10.301036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690637+0000","last_change":"2024-03-11T22:30:20.400482+0000","last_active":"2024-03-11T22:30:55.690637+0000","last_peered":"2024-03-11T22:30:55.690637+0000","last_clean":"2024-03-11T22:30:55.690637+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:30:55.690637+0000","last_undegraded":"2024-03-11T22:30:55.690637+0000","last_fullsized":"2024-03-11T22:30:55.690637+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:20.400349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:20.400349+0000","last_clean_scrub_stamp":"2024-03-11T22:30:20.400349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:39.045173+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"480'51","reported_seq":1210,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672311+0000","last_change":"2024-03-11T22:30:04.401901+0000","last_active":"2024-03-11T22:30:55.672311+0000","last_peered":"2024-03-11T22:30:55.672311+0000","last_clean":"2024-03-11T22:30:55.672311+0000","last_became_active":"2024-03-11T22:29:04.718797+0000","last_became_peered":"2024-03-11T22:29:04.718797+0000","last_unstale":"2024-03-11T22:30:55.672311+0000","last_undegraded":"2024-03-11T22:30:55.672311+0000","last_fullsized":"2024-03-11T22:30:55.672311+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'51","last_scrub_stamp":"2024-03-11T22:30:04.401771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:04.401771+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:32.893247+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":959,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401790+0000","last_change":"2024-03-11T22:26:04.099295+0000","last_active":"2024-03-11T22:30:55.401790+0000","last_peered":"2024-03-11T22:30:55.401790+0000","last_clean":"2024-03-11T22:30:55.401790+0000","last_became_active":"2024-03-11T22:20:29.413664+0000","last_became_peered":"2024-03-11T22:20:29.413664+0000","last_unstale":"2024-03-11T22:30:55.401790+0000","last_undegraded":"2024-03-11T22:30:55.401790+0000","last_fullsized":"2024-03-11T22:30:55.401790+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:04.099165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:04.099165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1032,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705447+0000","last_change":"2024-03-11T22:30:12.675279+0000","last_active":"2024-03-11T22:30:54.705447+0000","last_peered":"2024-03-11T22:30:54.705447+0000","last_clean":"2024-03-11T22:30:54.705447+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:30:54.705447+0000","last_undegraded":"2024-03-11T22:30:54.705447+0000","last_fullsized":"2024-03-11T22:30:54.705447+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:12.675202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:30:12.675202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:40.937625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":991,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704426+0000","last_change":"2024-03-11T22:30:47.568602+0000","last_active":"2024-03-11T22:30:54.704426+0000","last_peered":"2024-03-11T22:30:54.704426+0000","last_clean":"2024-03-11T22:30:54.704426+0000","last_became_active":"2024-03-11T22:20:29.415565+0000","last_became_peered":"2024-03-11T22:20:29.415565+0000","last_unstale":"2024-03-11T22:30:54.704426+0000","last_undegraded":"2024-03-11T22:30:54.704426+0000","last_fullsized":"2024-03-11T22:30:54.704426+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:47.568500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:47.568500+0000","last_clean_scrub_stamp":"2024-03-11T22:30:47.568500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:49.067116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1045,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704211+0000","last_change":"2024-03-11T22:29:56.665159+0000","last_active":"2024-03-11T22:30:54.704211+0000","last_peered":"2024-03-11T22:30:54.704211+0000","last_clean":"2024-03-11T22:30:54.704211+0000","last_became_active":"2024-03-11T22:29:04.718513+0000","last_became_peered":"2024-03-11T22:29:04.718513+0000","last_unstale":"2024-03-11T22:30:54.704211+0000","last_undegraded":"2024-03-11T22:30:54.704211+0000","last_fullsized":"2024-03-11T22:30:54.704211+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:56.665023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:29:56.665023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:13.354927+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"478'24","reported_seq":134,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704572+0000","last_change":"2024-03-11T22:30:48.649205+0000","last_active":"2024-03-11T22:30:54.704572+0000","last_peered":"2024-03-11T22:30:54.704572+0000","last_clean":"2024-03-11T22:30:54.704572+0000","last_became_active":"2024-03-11T22:29:04.694343+0000","last_became_peered":"2024-03-11T22:29:04.694343+0000","last_unstale":"2024-03-11T22:30:54.704572+0000","last_undegraded":"2024-03-11T22:30:54.704572+0000","last_fullsized":"2024-03-11T22:30:54.704572+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'24","last_scrub_stamp":"2024-03-11T22:30:48.649070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:48.649070+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:03.230178+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706389+0000","last_change":"2024-03-11T22:30:10.740917+0000","last_active":"2024-03-11T22:30:54.706389+0000","last_peered":"2024-03-11T22:30:54.706389+0000","last_clean":"2024-03-11T22:30:54.706389+0000","last_became_active":"2024-03-11T22:29:04.727524+0000","last_became_peered":"2024-03-11T22:29:04.727524+0000","last_unstale":"2024-03-11T22:30:54.706389+0000","last_undegraded":"2024-03-11T22:30:54.706389+0000","last_fullsized":"2024-03-11T22:30:54.706389+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:10.740798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:10.740798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:36.857785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":1029,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391787+0000","last_change":"2024-03-11T22:29:04.742292+0000","last_active":"2024-03-11T22:30:55.391787+0000","last_peered":"2024-03-11T22:30:55.391787+0000","last_clean":"2024-03-11T22:30:55.391787+0000","last_became_active":"2024-03-11T22:29:04.698531+0000","last_became_peered":"2024-03-11T22:29:04.698531+0000","last_unstale":"2024-03-11T22:30:55.391787+0000","last_undegraded":"2024-03-11T22:30:55.391787+0000","last_fullsized":"2024-03-11T22:30:55.391787+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:26:35.021902+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:26:35.021902+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"310'11","reported_seq":1252,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672191+0000","last_change":"2024-03-11T22:29:54.360194+0000","last_active":"2024-03-11T22:30:55.672191+0000","last_peered":"2024-03-11T22:30:55.672191+0000","last_clean":"2024-03-11T22:30:55.672191+0000","last_became_active":"2024-03-11T22:20:29.418355+0000","last_became_peered":"2024-03-11T22:20:29.418355+0000","last_unstale":"2024-03-11T22:30:55.672191+0000","last_undegraded":"2024-03-11T22:30:55.672191+0000","last_fullsized":"2024-03-11T22:30:55.672191+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:29:54.360080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:29:54.360080+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:07.077073+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"479'46","reported_seq":1124,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.400978+0000","last_change":"2024-03-11T22:26:45.212688+0000","last_active":"2024-03-11T22:30:55.400978+0000","last_peered":"2024-03-11T22:30:55.400978+0000","last_clean":"2024-03-11T22:30:55.400978+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:30:55.400978+0000","last_undegraded":"2024-03-11T22:30:55.400978+0000","last_fullsized":"2024-03-11T22:30:55.400978+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:26:45.212591+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:26:45.212591+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"432'4","reported_seq":809,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690550+0000","last_change":"2024-03-11T22:26:50.847434+0000","last_active":"2024-03-11T22:30:55.690550+0000","last_peered":"2024-03-11T22:30:55.690550+0000","last_clean":"2024-03-11T22:30:55.690550+0000","last_became_active":"2024-03-11T22:20:29.415226+0000","last_became_peered":"2024-03-11T22:20:29.415226+0000","last_unstale":"2024-03-11T22:30:55.690550+0000","last_undegraded":"2024-03-11T22:30:55.690550+0000","last_fullsized":"2024-03-11T22:30:55.690550+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'4","last_scrub_stamp":"2024-03-11T22:26:50.847349+0000","last_deep_scrub":"432'4","last_deep_scrub_stamp":"2024-03-11T22:26:50.847349+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.847349+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":996,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401409+0000","last_change":"2024-03-11T22:26:49.194654+0000","last_active":"2024-03-11T22:30:55.401409+0000","last_peered":"2024-03-11T22:30:55.401409+0000","last_clean":"2024-03-11T22:30:55.401409+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:30:55.401409+0000","last_undegraded":"2024-03-11T22:30:55.401409+0000","last_fullsized":"2024-03-11T22:30:55.401409+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:49.194569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:49.194569+0000","last_clean_scrub_stamp":"2024-03-11T22:26:49.194569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"479'18","reported_seq":1092,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392556+0000","last_change":"2024-03-11T22:30:16.934222+0000","last_active":"2024-03-11T22:30:55.392556+0000","last_peered":"2024-03-11T22:30:55.392556+0000","last_clean":"2024-03-11T22:30:55.392556+0000","last_became_active":"2024-03-11T22:30:16.929794+0000","last_became_peered":"2024-03-11T22:30:16.929794+0000","last_unstale":"2024-03-11T22:30:55.392556+0000","last_undegraded":"2024-03-11T22:30:55.392556+0000","last_fullsized":"2024-03-11T22:30:55.392556+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'12","last_scrub_stamp":"2024-03-11T22:25:52.052338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:52.052338+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1022,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.671990+0000","last_change":"2024-03-11T22:28:33.573448+0000","last_active":"2024-03-11T22:30:55.671990+0000","last_peered":"2024-03-11T22:30:55.671990+0000","last_clean":"2024-03-11T22:30:55.671990+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:30:55.671990+0000","last_undegraded":"2024-03-11T22:30:55.671990+0000","last_fullsized":"2024-03-11T22:30:55.671990+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:28:33.573346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:28:33.573346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1019,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401611+0000","last_change":"2024-03-11T22:30:03.168689+0000","last_active":"2024-03-11T22:30:55.401611+0000","last_peered":"2024-03-11T22:30:55.401611+0000","last_clean":"2024-03-11T22:30:55.401611+0000","last_became_active":"2024-03-11T22:29:04.731305+0000","last_became_peered":"2024-03-11T22:29:04.731305+0000","last_unstale":"2024-03-11T22:30:55.401611+0000","last_undegraded":"2024-03-11T22:30:55.401611+0000","last_fullsized":"2024-03-11T22:30:55.401611+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:03.168564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:30:03.168564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:32.143726+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":952,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672984+0000","last_change":"2024-03-11T22:29:52.451412+0000","last_active":"2024-03-11T22:30:55.672984+0000","last_peered":"2024-03-11T22:30:55.672984+0000","last_clean":"2024-03-11T22:30:55.672984+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:30:55.672984+0000","last_undegraded":"2024-03-11T22:30:55.672984+0000","last_fullsized":"2024-03-11T22:30:55.672984+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:52.451186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:29:52.451186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:30:59.862313+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"478'25","reported_seq":1080,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690854+0000","last_change":"2024-03-11T22:30:01.530506+0000","last_active":"2024-03-11T22:30:55.690854+0000","last_peered":"2024-03-11T22:30:55.690854+0000","last_clean":"2024-03-11T22:30:55.690854+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:30:55.690854+0000","last_undegraded":"2024-03-11T22:30:55.690854+0000","last_fullsized":"2024-03-11T22:30:55.690854+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:30:01.530350+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_clean_scrub_stamp":"2024-03-11T22:30:01.530350+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:27.843442+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":792,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672072+0000","last_change":"2024-03-11T22:30:46.646058+0000","last_active":"2024-03-11T22:30:55.672072+0000","last_peered":"2024-03-11T22:30:55.672072+0000","last_clean":"2024-03-11T22:30:55.672072+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:30:55.672072+0000","last_undegraded":"2024-03-11T22:30:55.672072+0000","last_fullsized":"2024-03-11T22:30:55.672072+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:46.645943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:46.645943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:15.950611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1023,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392507+0000","last_change":"2024-03-11T22:30:16.883019+0000","last_active":"2024-03-11T22:30:55.392507+0000","last_peered":"2024-03-11T22:30:55.392507+0000","last_clean":"2024-03-11T22:30:55.392507+0000","last_became_active":"2024-03-11T22:29:04.700408+0000","last_became_peered":"2024-03-11T22:29:04.700408+0000","last_unstale":"2024-03-11T22:30:55.392507+0000","last_undegraded":"2024-03-11T22:30:55.392507+0000","last_fullsized":"2024-03-11T22:30:55.392507+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:16.882883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:16.882883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:40.426358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1036,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705535+0000","last_change":"2024-03-11T22:30:49.480356+0000","last_active":"2024-03-11T22:30:54.705535+0000","last_peered":"2024-03-11T22:30:54.705535+0000","last_clean":"2024-03-11T22:30:54.705535+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:30:54.705535+0000","last_undegraded":"2024-03-11T22:30:54.705535+0000","last_fullsized":"2024-03-11T22:30:54.705535+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:49.480247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:30:49.480247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:59.538719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"479'25","reported_seq":1049,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401791+0000","last_change":"2024-03-11T22:26:12.149477+0000","last_active":"2024-03-11T22:30:55.401791+0000","last_peered":"2024-03-11T22:30:55.401791+0000","last_clean":"2024-03-11T22:30:55.401791+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:30:55.401791+0000","last_undegraded":"2024-03-11T22:30:55.401791+0000","last_fullsized":"2024-03-11T22:30:55.401791+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:26:12.149360+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:26:12.149360+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1026,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.689912+0000","last_change":"2024-03-11T22:29:53.461268+0000","last_active":"2024-03-11T22:30:55.689912+0000","last_peered":"2024-03-11T22:30:55.689912+0000","last_clean":"2024-03-11T22:30:55.689912+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:30:55.689912+0000","last_undegraded":"2024-03-11T22:30:55.689912+0000","last_fullsized":"2024-03-11T22:30:55.689912+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:53.461031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:50.030272+0000","last_clean_scrub_stamp":"2024-03-11T22:29:53.461031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:14.083004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"527'12","reported_seq":1315,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391889+0000","last_change":"2024-03-11T22:30:34.882548+0000","last_active":"2024-03-11T22:30:55.391889+0000","last_peered":"2024-03-11T22:30:55.391889+0000","last_clean":"2024-03-11T22:30:55.391889+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:30:55.391889+0000","last_undegraded":"2024-03-11T22:30:55.391889+0000","last_fullsized":"2024-03-11T22:30:55.391889+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'12","last_scrub_stamp":"2024-03-11T22:30:34.882462+0000","last_deep_scrub":"527'12","last_deep_scrub_stamp":"2024-03-11T22:30:34.882462+0000","last_clean_scrub_stamp":"2024-03-11T22:30:34.882462+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:45.890672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":798,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704582+0000","last_change":"2024-03-11T22:30:18.817369+0000","last_active":"2024-03-11T22:30:54.704582+0000","last_peered":"2024-03-11T22:30:54.704582+0000","last_clean":"2024-03-11T22:30:54.704582+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:30:54.704582+0000","last_undegraded":"2024-03-11T22:30:54.704582+0000","last_fullsized":"2024-03-11T22:30:54.704582+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:18.817224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:48.490757+0000","last_clean_scrub_stamp":"2024-03-11T22:30:18.817224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:31.548740+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"479'30","reported_seq":1094,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391859+0000","last_change":"2024-03-11T22:29:04.830471+0000","last_active":"2024-03-11T22:30:55.391859+0000","last_peered":"2024-03-11T22:30:55.391859+0000","last_clean":"2024-03-11T22:30:55.391859+0000","last_became_active":"2024-03-11T22:29:04.720501+0000","last_became_peered":"2024-03-11T22:29:04.720501+0000","last_unstale":"2024-03-11T22:30:55.391859+0000","last_undegraded":"2024-03-11T22:30:55.391859+0000","last_fullsized":"2024-03-11T22:30:55.391859+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'20","last_scrub_stamp":"2024-03-11T22:25:56.196035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:25:56.196035+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":822,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705361+0000","last_change":"2024-03-11T22:30:53.455852+0000","last_active":"2024-03-11T22:30:54.705361+0000","last_peered":"2024-03-11T22:30:54.705361+0000","last_clean":"2024-03-11T22:30:54.705361+0000","last_became_active":"2024-03-11T22:20:29.415019+0000","last_became_peered":"2024-03-11T22:20:29.415019+0000","last_unstale":"2024-03-11T22:30:54.705361+0000","last_undegraded":"2024-03-11T22:30:54.705361+0000","last_fullsized":"2024-03-11T22:30:54.705361+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:53.455748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:51.034868+0000","last_clean_scrub_stamp":"2024-03-11T22:30:53.455748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:55.363500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"488'10","reported_seq":1305,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401688+0000","last_change":"2024-03-11T22:30:08.228206+0000","last_active":"2024-03-11T22:30:55.401688+0000","last_peered":"2024-03-11T22:30:55.401688+0000","last_clean":"2024-03-11T22:30:55.401688+0000","last_became_active":"2024-03-11T22:29:04.787739+0000","last_became_peered":"2024-03-11T22:29:04.787739+0000","last_unstale":"2024-03-11T22:30:55.401688+0000","last_undegraded":"2024-03-11T22:30:55.401688+0000","last_fullsized":"2024-03-11T22:30:55.401688+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'10","last_scrub_stamp":"2024-03-11T22:30:08.228075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:08.228075+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:10.466725+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1006,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690302+0000","last_change":"2024-03-11T22:30:21.303542+0000","last_active":"2024-03-11T22:30:55.690302+0000","last_peered":"2024-03-11T22:30:55.690302+0000","last_clean":"2024-03-11T22:30:55.690302+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:30:55.690302+0000","last_undegraded":"2024-03-11T22:30:55.690302+0000","last_fullsized":"2024-03-11T22:30:55.690302+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:21.303440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:21.303440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:36.442252+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"479'27","reported_seq":886,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672384+0000","last_change":"2024-03-11T22:30:40.594125+0000","last_active":"2024-03-11T22:30:55.672384+0000","last_peered":"2024-03-11T22:30:55.672384+0000","last_clean":"2024-03-11T22:30:55.672384+0000","last_became_active":"2024-03-11T22:29:04.720174+0000","last_became_peered":"2024-03-11T22:29:04.720174+0000","last_unstale":"2024-03-11T22:30:55.672384+0000","last_undegraded":"2024-03-11T22:30:55.672384+0000","last_fullsized":"2024-03-11T22:30:55.672384+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-03-11T22:30:40.594016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.594016+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:55.638323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":809,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672786+0000","last_change":"2024-03-11T22:26:38.840815+0000","last_active":"2024-03-11T22:30:55.672786+0000","last_peered":"2024-03-11T22:30:55.672786+0000","last_clean":"2024-03-11T22:30:55.672786+0000","last_became_active":"2024-03-11T22:20:29.420867+0000","last_became_peered":"2024-03-11T22:20:29.420867+0000","last_unstale":"2024-03-11T22:30:55.672786+0000","last_undegraded":"2024-03-11T22:30:55.672786+0000","last_fullsized":"2024-03-11T22:30:55.672786+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:26:38.840685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:38.840685+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":125,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705457+0000","last_change":"2024-03-11T22:30:40.510945+0000","last_active":"2024-03-11T22:30:54.705457+0000","last_peered":"2024-03-11T22:30:54.705457+0000","last_clean":"2024-03-11T22:30:54.705457+0000","last_became_active":"2024-03-11T22:29:04.723400+0000","last_became_peered":"2024-03-11T22:29:04.723400+0000","last_unstale":"2024-03-11T22:30:54.705457+0000","last_undegraded":"2024-03-11T22:30:54.705457+0000","last_fullsized":"2024-03-11T22:30:54.705457+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:40.510823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.510823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:09.057333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":987,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391290+0000","last_change":"2024-03-11T22:29:50.712811+0000","last_active":"2024-03-11T22:30:55.391290+0000","last_peered":"2024-03-11T22:30:55.391290+0000","last_clean":"2024-03-11T22:30:55.391290+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:30:55.391290+0000","last_undegraded":"2024-03-11T22:30:55.391290+0000","last_fullsized":"2024-03-11T22:30:55.391290+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:50.712416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:29:50.712416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:10.140704+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"478'25","reported_seq":132,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705098+0000","last_change":"2024-03-11T22:30:19.687640+0000","last_active":"2024-03-11T22:30:54.705098+0000","last_peered":"2024-03-11T22:30:54.705098+0000","last_clean":"2024-03-11T22:30:54.705098+0000","last_became_active":"2024-03-11T22:29:04.730485+0000","last_became_peered":"2024-03-11T22:29:04.730485+0000","last_unstale":"2024-03-11T22:30:54.705098+0000","last_undegraded":"2024-03-11T22:30:54.705098+0000","last_fullsized":"2024-03-11T22:30:54.705098+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:30:19.687537+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:30:19.687537+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:19.724923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":950,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392008+0000","last_change":"2024-03-11T22:30:12.898819+0000","last_active":"2024-03-11T22:30:55.392008+0000","last_peered":"2024-03-11T22:30:55.392008+0000","last_clean":"2024-03-11T22:30:55.392008+0000","last_became_active":"2024-03-11T22:20:29.422747+0000","last_became_peered":"2024-03-11T22:20:29.422747+0000","last_unstale":"2024-03-11T22:30:55.392008+0000","last_undegraded":"2024-03-11T22:30:55.392008+0000","last_fullsized":"2024-03-11T22:30:55.392008+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:12.898649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:12.898649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.975372+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1011,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391622+0000","last_change":"2024-03-11T22:30:15.836456+0000","last_active":"2024-03-11T22:30:55.391622+0000","last_peered":"2024-03-11T22:30:55.391622+0000","last_clean":"2024-03-11T22:30:55.391622+0000","last_became_active":"2024-03-11T22:29:04.700352+0000","last_became_peered":"2024-03-11T22:29:04.700352+0000","last_unstale":"2024-03-11T22:30:55.391622+0000","last_undegraded":"2024-03-11T22:30:55.391622+0000","last_fullsized":"2024-03-11T22:30:55.391622+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:15.836335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:30:15.836335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:21.846380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1014,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.689829+0000","last_change":"2024-03-11T22:30:48.240248+0000","last_active":"2024-03-11T22:30:55.689829+0000","last_peered":"2024-03-11T22:30:55.689829+0000","last_clean":"2024-03-11T22:30:55.689829+0000","last_became_active":"2024-03-11T22:29:04.707420+0000","last_became_peered":"2024-03-11T22:29:04.707420+0000","last_unstale":"2024-03-11T22:30:55.689829+0000","last_undegraded":"2024-03-11T22:30:55.689829+0000","last_fullsized":"2024-03-11T22:30:55.689829+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:48.240128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:09.057970+0000","last_clean_scrub_stamp":"2024-03-11T22:30:48.240128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:12.685122+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"480'28","reported_seq":1075,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391729+0000","last_change":"2024-03-11T22:29:51.655446+0000","last_active":"2024-03-11T22:30:55.391729+0000","last_peered":"2024-03-11T22:30:55.391729+0000","last_clean":"2024-03-11T22:30:55.391729+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:30:55.391729+0000","last_undegraded":"2024-03-11T22:30:55.391729+0000","last_fullsized":"2024-03-11T22:30:55.391729+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'28","last_scrub_stamp":"2024-03-11T22:29:51.655136+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:29:51.655136+0000","objects_scrubbed":5,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1039,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401613+0000","last_change":"2024-03-11T22:30:20.359726+0000","last_active":"2024-03-11T22:30:55.401613+0000","last_peered":"2024-03-11T22:30:55.401613+0000","last_clean":"2024-03-11T22:30:55.401613+0000","last_became_active":"2024-03-11T22:29:04.722573+0000","last_became_peered":"2024-03-11T22:29:04.722573+0000","last_unstale":"2024-03-11T22:30:55.401613+0000","last_undegraded":"2024-03-11T22:30:55.401613+0000","last_fullsized":"2024-03-11T22:30:55.401613+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:20.359623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:20.359623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:46.022838+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":783,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.671964+0000","last_change":"2024-03-11T22:26:44.807269+0000","last_active":"2024-03-11T22:30:55.671964+0000","last_peered":"2024-03-11T22:30:55.671964+0000","last_clean":"2024-03-11T22:30:55.671964+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:30:55.671964+0000","last_undegraded":"2024-03-11T22:30:55.671964+0000","last_fullsized":"2024-03-11T22:30:55.671964+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.807187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.807187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":995,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391915+0000","last_change":"2024-03-11T22:30:53.091266+0000","last_active":"2024-03-11T22:30:55.391915+0000","last_peered":"2024-03-11T22:30:55.391915+0000","last_clean":"2024-03-11T22:30:55.391915+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:30:55.391915+0000","last_undegraded":"2024-03-11T22:30:55.391915+0000","last_fullsized":"2024-03-11T22:30:55.391915+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:53.091163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:53.091163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:16.620952+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"478'12","reported_seq":1057,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391233+0000","last_change":"2024-03-11T22:29:04.843107+0000","last_active":"2024-03-11T22:30:55.391233+0000","last_peered":"2024-03-11T22:30:55.391233+0000","last_clean":"2024-03-11T22:30:55.391233+0000","last_became_active":"2024-03-11T22:29:04.734207+0000","last_became_peered":"2024-03-11T22:29:04.734207+0000","last_unstale":"2024-03-11T22:30:55.391233+0000","last_undegraded":"2024-03-11T22:30:55.391233+0000","last_fullsized":"2024-03-11T22:30:55.391233+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'8","last_scrub_stamp":"2024-03-11T22:26:50.833149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.833149+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1006,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690611+0000","last_change":"2024-03-11T22:26:05.787798+0000","last_active":"2024-03-11T22:30:55.690611+0000","last_peered":"2024-03-11T22:30:55.690611+0000","last_clean":"2024-03-11T22:30:55.690611+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:30:55.690611+0000","last_undegraded":"2024-03-11T22:30:55.690611+0000","last_fullsized":"2024-03-11T22:30:55.690611+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:05.787543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:05.787543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":600,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672075+0000","last_change":"2024-03-11T22:30:03.392102+0000","last_active":"2024-03-11T22:30:55.672075+0000","last_peered":"2024-03-11T22:30:55.672075+0000","last_clean":"2024-03-11T22:30:55.672075+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:30:55.672075+0000","last_undegraded":"2024-03-11T22:30:55.672075+0000","last_fullsized":"2024-03-11T22:30:55.672075+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:03.391983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:03.391983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:28.249700+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":970,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705070+0000","last_change":"2024-03-11T22:30:01.668808+0000","last_active":"2024-03-11T22:30:54.705070+0000","last_peered":"2024-03-11T22:30:54.705070+0000","last_clean":"2024-03-11T22:30:54.705070+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:30:54.705070+0000","last_undegraded":"2024-03-11T22:30:54.705070+0000","last_fullsized":"2024-03-11T22:30:54.705070+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:01.668594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:01.668594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:07.076683+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"480'27","reported_seq":1106,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673172+0000","last_change":"2024-03-11T22:30:43.611613+0000","last_active":"2024-03-11T22:30:55.673172+0000","last_peered":"2024-03-11T22:30:55.673172+0000","last_clean":"2024-03-11T22:30:55.673172+0000","last_became_active":"2024-03-11T22:29:04.720148+0000","last_became_peered":"2024-03-11T22:29:04.720148+0000","last_unstale":"2024-03-11T22:30:55.673172+0000","last_undegraded":"2024-03-11T22:30:55.673172+0000","last_fullsized":"2024-03-11T22:30:55.673172+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'27","last_scrub_stamp":"2024-03-11T22:30:43.611513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:43.611513+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:00.303736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1019,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391232+0000","last_change":"2024-03-11T22:30:13.866705+0000","last_active":"2024-03-11T22:30:55.391232+0000","last_peered":"2024-03-11T22:30:55.391232+0000","last_clean":"2024-03-11T22:30:55.391232+0000","last_became_active":"2024-03-11T22:29:04.733742+0000","last_became_peered":"2024-03-11T22:29:04.733742+0000","last_unstale":"2024-03-11T22:30:55.391232+0000","last_undegraded":"2024-03-11T22:30:55.391232+0000","last_fullsized":"2024-03-11T22:30:55.391232+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:13.866615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:13.866615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:42.154858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1008,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690912+0000","last_change":"2024-03-11T22:26:43.777178+0000","last_active":"2024-03-11T22:30:55.690912+0000","last_peered":"2024-03-11T22:30:55.690912+0000","last_clean":"2024-03-11T22:30:55.690912+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:30:55.690912+0000","last_undegraded":"2024-03-11T22:30:55.690912+0000","last_fullsized":"2024-03-11T22:30:55.690912+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:43.777100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:43.777100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706579+0000","last_change":"2024-03-11T22:30:06.824568+0000","last_active":"2024-03-11T22:30:54.706579+0000","last_peered":"2024-03-11T22:30:54.706579+0000","last_clean":"2024-03-11T22:30:54.706579+0000","last_became_active":"2024-03-11T22:29:04.786119+0000","last_became_peered":"2024-03-11T22:29:04.786119+0000","last_unstale":"2024-03-11T22:30:54.706579+0000","last_undegraded":"2024-03-11T22:30:54.706579+0000","last_fullsized":"2024-03-11T22:30:54.706579+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:06.824468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:06.824468+0000","last_clean_scrub_stamp":"2024-03-11T22:30:06.824468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:29.075598+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"479'30","reported_seq":1093,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690716+0000","last_change":"2024-03-11T22:30:41.197575+0000","last_active":"2024-03-11T22:30:55.690716+0000","last_peered":"2024-03-11T22:30:55.690716+0000","last_clean":"2024-03-11T22:30:55.690716+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:30:55.690716+0000","last_undegraded":"2024-03-11T22:30:55.690716+0000","last_fullsized":"2024-03-11T22:30:55.690716+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-03-11T22:30:41.197457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:41.197457+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:01.110242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":126,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704469+0000","last_change":"2024-03-11T22:30:20.763816+0000","last_active":"2024-03-11T22:30:54.704469+0000","last_peered":"2024-03-11T22:30:54.704469+0000","last_clean":"2024-03-11T22:30:54.704469+0000","last_became_active":"2024-03-11T22:29:04.693762+0000","last_became_peered":"2024-03-11T22:29:04.693762+0000","last_unstale":"2024-03-11T22:30:54.704469+0000","last_undegraded":"2024-03-11T22:30:54.704469+0000","last_fullsized":"2024-03-11T22:30:54.704469+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:30:20.763628+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:30:20.763628+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:38.282863+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'11","reported_seq":1333,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673080+0000","last_change":"2024-03-11T22:30:42.614513+0000","last_active":"2024-03-11T22:30:55.673080+0000","last_peered":"2024-03-11T22:30:55.673080+0000","last_clean":"2024-03-11T22:30:55.673080+0000","last_became_active":"2024-03-11T22:20:29.415781+0000","last_became_peered":"2024-03-11T22:20:29.415781+0000","last_unstale":"2024-03-11T22:30:55.673080+0000","last_undegraded":"2024-03-11T22:30:55.673080+0000","last_fullsized":"2024-03-11T22:30:55.673080+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-11T22:30:42.614396+0000","last_deep_scrub":"315'11","last_deep_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_clean_scrub_stamp":"2024-03-11T22:30:42.614396+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:56.150179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":956,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401482+0000","last_change":"2024-03-11T22:30:10.198091+0000","last_active":"2024-03-11T22:30:55.401482+0000","last_peered":"2024-03-11T22:30:55.401482+0000","last_clean":"2024-03-11T22:30:55.401482+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:30:55.401482+0000","last_undegraded":"2024-03-11T22:30:55.401482+0000","last_fullsized":"2024-03-11T22:30:55.401482+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:30:10.197974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:10.197974+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.414872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"480'24","reported_seq":1079,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:56.848659+0000","last_change":"2024-03-11T22:30:56.848699+0000","last_active":"2024-03-11T22:30:56.848659+0000","last_peered":"2024-03-11T22:30:56.848659+0000","last_clean":"2024-03-11T22:30:56.848659+0000","last_became_active":"2024-03-11T22:29:04.720341+0000","last_became_peered":"2024-03-11T22:29:04.720341+0000","last_unstale":"2024-03-11T22:30:56.848659+0000","last_undegraded":"2024-03-11T22:30:56.848659+0000","last_fullsized":"2024-03-11T22:30:56.848659+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'24","last_scrub_stamp":"2024-03-11T22:30:56.848590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:56.848590+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:58.594672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704997+0000","last_change":"2024-03-11T22:30:08.741018+0000","last_active":"2024-03-11T22:30:54.704997+0000","last_peered":"2024-03-11T22:30:54.704997+0000","last_clean":"2024-03-11T22:30:54.704997+0000","last_became_active":"2024-03-11T22:29:04.725678+0000","last_became_peered":"2024-03-11T22:29:04.725678+0000","last_unstale":"2024-03-11T22:30:54.704997+0000","last_undegraded":"2024-03-11T22:30:54.704997+0000","last_fullsized":"2024-03-11T22:30:54.704997+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:08.740926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:08.740926+0000","last_clean_scrub_stamp":"2024-03-11T22:30:08.740926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:21.051131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":765,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392090+0000","last_change":"2024-03-11T22:30:46.780258+0000","last_active":"2024-03-11T22:30:55.392090+0000","last_peered":"2024-03-11T22:30:55.392090+0000","last_clean":"2024-03-11T22:30:55.392090+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:30:55.392090+0000","last_undegraded":"2024-03-11T22:30:55.392090+0000","last_fullsized":"2024-03-11T22:30:55.392090+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:46.780169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:46.780169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:02.985331+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":988,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392290+0000","last_change":"2024-03-11T22:26:37.062133+0000","last_active":"2024-03-11T22:30:55.392290+0000","last_peered":"2024-03-11T22:30:55.392290+0000","last_clean":"2024-03-11T22:30:55.392290+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:30:55.392290+0000","last_undegraded":"2024-03-11T22:30:55.392290+0000","last_fullsized":"2024-03-11T22:30:55.392290+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:37.062022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:37.062022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"479'39","reported_seq":1148,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705296+0000","last_change":"2024-03-11T22:30:21.635123+0000","last_active":"2024-03-11T22:30:54.705296+0000","last_peered":"2024-03-11T22:30:54.705296+0000","last_clean":"2024-03-11T22:30:54.705296+0000","last_became_active":"2024-03-11T22:29:04.721324+0000","last_became_peered":"2024-03-11T22:29:04.721324+0000","last_unstale":"2024-03-11T22:30:54.705296+0000","last_undegraded":"2024-03-11T22:30:54.705296+0000","last_fullsized":"2024-03-11T22:30:54.705296+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'39","last_scrub_stamp":"2024-03-11T22:30:21.634960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:21.634960+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:39.506214+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1025,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.691304+0000","last_change":"2024-03-11T22:30:40.231188+0000","last_active":"2024-03-11T22:30:55.691304+0000","last_peered":"2024-03-11T22:30:55.691304+0000","last_clean":"2024-03-11T22:30:55.691304+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:30:55.691304+0000","last_undegraded":"2024-03-11T22:30:55.691304+0000","last_fullsized":"2024-03-11T22:30:55.691304+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:40.231100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.231100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:49.098484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706564+0000","last_change":"2024-03-11T22:29:55.548541+0000","last_active":"2024-03-11T22:30:54.706564+0000","last_peered":"2024-03-11T22:30:54.706564+0000","last_clean":"2024-03-11T22:30:54.706564+0000","last_became_active":"2024-03-11T22:29:04.724856+0000","last_became_peered":"2024-03-11T22:29:04.724856+0000","last_unstale":"2024-03-11T22:30:54.706564+0000","last_undegraded":"2024-03-11T22:30:54.706564+0000","last_fullsized":"2024-03-11T22:30:54.706564+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:55.546853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:29:55.546853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:25.517836+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":790,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673097+0000","last_change":"2024-03-11T22:30:17.307828+0000","last_active":"2024-03-11T22:30:55.673097+0000","last_peered":"2024-03-11T22:30:55.673097+0000","last_clean":"2024-03-11T22:30:55.673097+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:30:55.673097+0000","last_undegraded":"2024-03-11T22:30:55.673097+0000","last_fullsized":"2024-03-11T22:30:55.673097+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:17.307729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:17.307729+0000","last_clean_scrub_stamp":"2024-03-11T22:30:17.307729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:24.381369+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"478'11","reported_seq":869,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391398+0000","last_change":"2024-03-11T22:30:02.742323+0000","last_active":"2024-03-11T22:30:55.391398+0000","last_peered":"2024-03-11T22:30:55.391398+0000","last_clean":"2024-03-11T22:30:55.391398+0000","last_became_active":"2024-03-11T22:29:04.698810+0000","last_became_peered":"2024-03-11T22:29:04.698810+0000","last_unstale":"2024-03-11T22:30:55.391398+0000","last_undegraded":"2024-03-11T22:30:55.391398+0000","last_fullsized":"2024-03-11T22:30:55.391398+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'11","last_scrub_stamp":"2024-03-11T22:30:02.742197+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:30:02.742197+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:26.951844+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1026,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401582+0000","last_change":"2024-03-11T22:30:15.199268+0000","last_active":"2024-03-11T22:30:55.401582+0000","last_peered":"2024-03-11T22:30:55.401582+0000","last_clean":"2024-03-11T22:30:55.401582+0000","last_became_active":"2024-03-11T22:29:04.721192+0000","last_became_peered":"2024-03-11T22:29:04.721192+0000","last_unstale":"2024-03-11T22:30:55.401582+0000","last_undegraded":"2024-03-11T22:30:55.401582+0000","last_fullsized":"2024-03-11T22:30:55.401582+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:15.199165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:30:15.199165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:43.996210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705264+0000","last_change":"2024-03-11T22:30:43.546407+0000","last_active":"2024-03-11T22:30:54.705264+0000","last_peered":"2024-03-11T22:30:54.705264+0000","last_clean":"2024-03-11T22:30:54.705264+0000","last_became_active":"2024-03-11T22:29:04.726418+0000","last_became_peered":"2024-03-11T22:29:04.726418+0000","last_unstale":"2024-03-11T22:30:54.705264+0000","last_undegraded":"2024-03-11T22:30:54.705264+0000","last_fullsized":"2024-03-11T22:30:54.705264+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:43.546317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:09.575187+0000","last_clean_scrub_stamp":"2024-03-11T22:30:43.546317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:10.181400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401239+0000","last_change":"2024-03-11T22:30:51.393304+0000","last_active":"2024-03-11T22:30:55.401239+0000","last_peered":"2024-03-11T22:30:55.401239+0000","last_clean":"2024-03-11T22:30:55.401239+0000","last_became_active":"2024-03-11T22:29:04.721807+0000","last_became_peered":"2024-03-11T22:29:04.721807+0000","last_unstale":"2024-03-11T22:30:55.401239+0000","last_undegraded":"2024-03-11T22:30:55.401239+0000","last_fullsized":"2024-03-11T22:30:55.401239+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:51.393206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:51.393206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:52.157560+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"479'21","reported_seq":826,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.873642+0000","last_change":"2024-03-11T22:30:55.873677+0000","last_active":"2024-03-11T22:30:55.873642+0000","last_peered":"2024-03-11T22:30:55.873642+0000","last_clean":"2024-03-11T22:30:55.873642+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:30:55.873642+0000","last_undegraded":"2024-03-11T22:30:55.873642+0000","last_fullsized":"2024-03-11T22:30:55.873642+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'21","last_scrub_stamp":"2024-03-11T22:30:55.873576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:55.873576+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:18.938926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1014,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401563+0000","last_change":"2024-03-11T22:29:53.126868+0000","last_active":"2024-03-11T22:30:55.401563+0000","last_peered":"2024-03-11T22:30:55.401563+0000","last_clean":"2024-03-11T22:30:55.401563+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:30:55.401563+0000","last_undegraded":"2024-03-11T22:30:55.401563+0000","last_fullsized":"2024-03-11T22:30:55.401563+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:53.126769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:29:53.126769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:12.911808+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1018,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401263+0000","last_change":"2024-03-11T22:30:16.197266+0000","last_active":"2024-03-11T22:30:55.401263+0000","last_peered":"2024-03-11T22:30:55.401263+0000","last_clean":"2024-03-11T22:30:55.401263+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:30:55.401263+0000","last_undegraded":"2024-03-11T22:30:55.401263+0000","last_fullsized":"2024-03-11T22:30:55.401263+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:16.197172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:30:16.197172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.268824+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1006,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690161+0000","last_change":"2024-03-11T22:29:04.708231+0000","last_active":"2024-03-11T22:30:55.690161+0000","last_peered":"2024-03-11T22:30:55.690161+0000","last_clean":"2024-03-11T22:30:55.690161+0000","last_became_active":"2024-03-11T22:29:04.707444+0000","last_became_peered":"2024-03-11T22:29:04.707444+0000","last_unstale":"2024-03-11T22:30:55.690161+0000","last_undegraded":"2024-03-11T22:30:55.690161+0000","last_fullsized":"2024-03-11T22:30:55.690161+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:37.738562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:37.738562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"480'39","reported_seq":1056,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391884+0000","last_change":"2024-03-11T22:26:38.945645+0000","last_active":"2024-03-11T22:30:55.391884+0000","last_peered":"2024-03-11T22:30:55.391884+0000","last_clean":"2024-03-11T22:30:55.391884+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:30:55.391884+0000","last_undegraded":"2024-03-11T22:30:55.391884+0000","last_fullsized":"2024-03-11T22:30:55.391884+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:26:38.945505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:38.945505+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392257+0000","last_change":"2024-03-11T22:26:39.902683+0000","last_active":"2024-03-11T22:30:55.392257+0000","last_peered":"2024-03-11T22:30:55.392257+0000","last_clean":"2024-03-11T22:30:55.392257+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:30:55.392257+0000","last_undegraded":"2024-03-11T22:30:55.392257+0000","last_fullsized":"2024-03-11T22:30:55.392257+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:26:39.902588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:39.902588+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":803,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672111+0000","last_change":"2024-03-11T22:30:15.364972+0000","last_active":"2024-03-11T22:30:55.672111+0000","last_peered":"2024-03-11T22:30:55.672111+0000","last_clean":"2024-03-11T22:30:55.672111+0000","last_became_active":"2024-03-11T22:29:04.719011+0000","last_became_peered":"2024-03-11T22:29:04.719011+0000","last_unstale":"2024-03-11T22:30:55.672111+0000","last_undegraded":"2024-03-11T22:30:55.672111+0000","last_fullsized":"2024-03-11T22:30:55.672111+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:15.364881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:15.364881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.215173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":998,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391581+0000","last_change":"2024-03-11T22:26:34.021098+0000","last_active":"2024-03-11T22:30:55.391581+0000","last_peered":"2024-03-11T22:30:55.391581+0000","last_clean":"2024-03-11T22:30:55.391581+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:30:55.391581+0000","last_undegraded":"2024-03-11T22:30:55.391581+0000","last_fullsized":"2024-03-11T22:30:55.391581+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:34.020992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:34.020992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"478'25","reported_seq":869,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672223+0000","last_change":"2024-03-11T22:29:57.388072+0000","last_active":"2024-03-11T22:30:55.672223+0000","last_peered":"2024-03-11T22:30:55.672223+0000","last_clean":"2024-03-11T22:30:55.672223+0000","last_became_active":"2024-03-11T22:29:04.718521+0000","last_became_peered":"2024-03-11T22:29:04.718521+0000","last_unstale":"2024-03-11T22:30:55.672223+0000","last_undegraded":"2024-03-11T22:30:55.672223+0000","last_fullsized":"2024-03-11T22:30:55.672223+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:29:57.387926+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:29:57.387926+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:05.421282+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690302+0000","last_change":"2024-03-11T22:30:04.486340+0000","last_active":"2024-03-11T22:30:55.690302+0000","last_peered":"2024-03-11T22:30:55.690302+0000","last_clean":"2024-03-11T22:30:55.690302+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:30:55.690302+0000","last_undegraded":"2024-03-11T22:30:55.690302+0000","last_fullsized":"2024-03-11T22:30:55.690302+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:04.486254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:04.486254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:17.389356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704753+0000","last_change":"2024-03-11T22:30:41.486573+0000","last_active":"2024-03-11T22:30:54.704753+0000","last_peered":"2024-03-11T22:30:54.704753+0000","last_clean":"2024-03-11T22:30:54.704753+0000","last_became_active":"2024-03-11T22:29:04.701716+0000","last_became_peered":"2024-03-11T22:29:04.701716+0000","last_unstale":"2024-03-11T22:30:54.704753+0000","last_undegraded":"2024-03-11T22:30:54.704753+0000","last_fullsized":"2024-03-11T22:30:54.704753+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:41.486494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:41.486494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:43.179038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":785,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401118+0000","last_change":"2024-03-11T22:29:04.718466+0000","last_active":"2024-03-11T22:30:55.401118+0000","last_peered":"2024-03-11T22:30:55.401118+0000","last_clean":"2024-03-11T22:30:55.401118+0000","last_became_active":"2024-03-11T22:29:04.718122+0000","last_became_peered":"2024-03-11T22:29:04.718122+0000","last_unstale":"2024-03-11T22:30:55.401118+0000","last_undegraded":"2024-03-11T22:30:55.401118+0000","last_fullsized":"2024-03-11T22:30:55.401118+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:10.104642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:10.104642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":302,"num_read_kb":297,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":6574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1035,"ondisk_log_size":1035,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005330,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":841612,"kb_used_data":2436,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92903540,"statfs":{"total":95995035648,"available":95133224960,"internally_reserved":0,"allocated":2494464,"data_stored":1674408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":859306443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332374,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":691312,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93053840,"statfs":{"total":95995035648,"available":95287132160,"internally_reserved":0,"allocated":2134016,"data_stored":1338790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":705755590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575152,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":751080,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92994072,"statfs":{"total":95995035648,"available":95225929728,"internally_reserved":0,"allocated":2187264,"data_stored":1346596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738678,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957900,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92787252,"statfs":{"total":95995035648,"available":95014146048,"internally_reserved":0,"allocated":2162688,"data_stored":1346060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411627,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":647164,"kb_used_data":2352,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93097988,"statfs":{"total":95995035648,"available":95332339712,"internally_reserved":0,"allocated":2408448,"data_stored":1609052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149989,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":778020,"kb_used_data":2712,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92967132,"statfs":{"total":95995035648,"available":95198343168,"internally_reserved":0,"allocated":2777088,"data_stored":1937412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":793901511},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921058,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":714960,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93030192,"statfs":{"total":95995035648,"available":95262916608,"internally_reserved":0,"allocated":2170880,"data_stored":1346269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659418,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":796032,"kb_used_data":2676,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92949120,"statfs":{"total":95995035648,"available":95179898880,"internally_reserved":0,"allocated":2740224,"data_stored":1928569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":812382662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:31:01.978 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T22:31:01.978 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T22:31:01.978 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-11T22:31:01.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T22:31:02.203 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T22:31:02.223 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:02.445 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:31:02.624 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:31:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:31:02.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:02 smithi144 ceph-mon[33984]: from='client.25811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:31:02.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:02 smithi144 ceph-mon[33984]: 3.9 scrub starts 2024-03-11T22:31:02.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:02 smithi144 ceph-mon[33984]: 3.9 scrub ok 2024-03-11T22:31:02.851 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:02 smithi144 ceph-mon[33984]: 5.10 scrub starts 2024-03-11T22:31:02.851 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:02 smithi144 ceph-mon[33984]: 5.10 scrub ok 2024-03-11T22:31:02.851 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:02 smithi144 ceph-mon[33984]: 4.17 scrub starts 2024-03-11T22:31:02.851 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:02 smithi144 ceph-mon[33984]: 4.17 scrub ok 2024-03-11T22:31:03.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[30178]: from='client.25811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:31:03.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[30178]: 3.9 scrub starts 2024-03-11T22:31:03.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[30178]: 3.9 scrub ok 2024-03-11T22:31:03.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[30178]: 5.10 scrub starts 2024-03-11T22:31:03.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[30178]: 5.10 scrub ok 2024-03-11T22:31:03.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[30178]: 4.17 scrub starts 2024-03-11T22:31:03.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[30178]: 4.17 scrub ok 2024-03-11T22:31:03.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[39715]: from='client.25811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:31:03.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[39715]: 3.9 scrub starts 2024-03-11T22:31:03.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[39715]: 3.9 scrub ok 2024-03-11T22:31:03.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[39715]: 5.10 scrub starts 2024-03-11T22:31:03.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[39715]: 5.10 scrub ok 2024-03-11T22:31:03.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[39715]: 4.17 scrub starts 2024-03-11T22:31:03.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:02 smithi045 ceph-mon[39715]: 4.17 scrub ok 2024-03-11T22:31:03.845 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:03 smithi144 ceph-mon[33984]: pgmap v1109: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:31:04.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:03 smithi045 ceph-mon[30178]: pgmap v1109: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:31:04.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:03 smithi045 ceph-mon[39715]: pgmap v1109: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:31:04.125 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:31:04.125 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:31:04.162 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:31:04.162 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-03-11T22:31:04.162 INFO:teuthology.orchestra.run.smithi045.stdout: "epoch": 538, 2024-03-11T22:31:04.162 INFO:teuthology.orchestra.run.smithi045.stdout: "fsid": "f75aafa4-dff2-11ee-95c8-87774f69a715", 2024-03-11T22:31:04.162 INFO:teuthology.orchestra.run.smithi045.stdout: "created": "2024-03-11T22:02:51.075940+0000", 2024-03-11T22:31:04.162 INFO:teuthology.orchestra.run.smithi045.stdout: "modified": "2024-03-11T22:30:54.701461+0000", 2024-03-11T22:31:04.162 INFO:teuthology.orchestra.run.smithi045.stdout: "last_up_change": "2024-03-11T22:08:17.992246+0000", 2024-03-11T22:31:04.162 INFO:teuthology.orchestra.run.smithi045.stdout: "last_in_change": "2024-03-11T22:29:03.667666+0000", 2024-03-11T22:31:04.162 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T22:31:04.162 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_num": 5799936, 2024-03-11T22:31:04.162 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_set": [ 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: "pglog_hardlimit", 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: "purged_snapdirs", 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: "recovery_deletes", 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: "sortbitwise" 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_version": 18, 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_snapshot": "", 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_max": 11, 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: "max_osd": 8, 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: "require_min_compat_client": "luminous", 2024-03-11T22:31:04.163 INFO:teuthology.orchestra.run.smithi045.stdout: "min_compat_client": "luminous", 2024-03-11T22:31:04.164 INFO:teuthology.orchestra.run.smithi045.stdout: "require_osd_release": "squid", 2024-03-11T22:31:04.164 INFO:teuthology.orchestra.run.smithi045.stdout: "allow_crimson": false, 2024-03-11T22:31:04.164 INFO:teuthology.orchestra.run.smithi045.stdout: "pools": [ 2024-03-11T22:31:04.164 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.164 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 1, 2024-03-11T22:31:04.164 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": ".mgr", 2024-03-11T22:31:04.164 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:06:12.903386+0000", 2024-03-11T22:31:04.164 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:31:04.164 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:31:04.164 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:31:04.164 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:31:04.164 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:31:04.164 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 11, 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 11, 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 11, 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 11, 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 11, 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "1.b", 2024-03-11T22:31:04.165 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 438, 2024-03-11T22:31:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 437, 2024-03-11T22:31:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 437, 2024-03-11T22:31:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "93'130", 2024-03-11T22:31:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "93'130" 2024-03-11T22:31:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "438", 2024-03-11T22:31:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:31:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "438", 2024-03-11T22:31:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "436", 2024-03-11T22:31:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:31:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:31:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:31:04.167 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:31:04.168 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:31:04.168 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:31:04.168 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:31:04.168 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:31:04.168 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:31:04.168 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:31:04.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.168 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:31:04.168 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:31:04.168 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:31:04.168 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:31:04.168 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:31:04.168 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:31:04.169 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:31:04.169 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:31:04.169 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:31:04.169 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:31:04.169 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:31:04.169 INFO:teuthology.orchestra.run.smithi045.stdout: "options": { 2024-03-11T22:31:04.169 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_max": 32, 2024-03-11T22:31:04.169 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_min": 1 2024-03-11T22:31:04.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.169 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:31:04.169 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": {} 2024-03-11T22:31:04.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.169 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:31:04.170 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:31:04.170 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.6399999856948853, 2024-03-11T22:31:04.170 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.6399999856948853, 2024-03-11T22:31:04.170 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T22:31:04.170 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2.1800000667572021, 2024-03-11T22:31:04.170 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2.1800000667572021, 2024-03-11T22:31:04.170 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T22:31:04.170 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:31:04.170 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T22:31:04.170 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.170 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 2, 2024-03-11T22:31:04.171 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "datapool", 2024-03-11T22:31:04.171 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:31.014060+0000", 2024-03-11T22:31:04.171 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 8193, 2024-03-11T22:31:04.171 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T22:31:04.171 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:31:04.171 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:31:04.172 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:31:04.172 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:31:04.172 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:31:04.172 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:31:04.172 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:31:04.172 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:31:04.173 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:31:04.173 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:31:04.173 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 3, 2024-03-11T22:31:04.173 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 3, 2024-03-11T22:31:04.173 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 3, 2024-03-11T22:31:04.173 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 3, 2024-03-11T22:31:04.173 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 3, 2024-03-11T22:31:04.174 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:31:04.174 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:31:04.174 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:31:04.174 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:31:04.174 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:31:04.174 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:31:04.174 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:31:04.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.174 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "71", 2024-03-11T22:31:04.174 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:31:04.174 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:31:04.174 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:31:04.175 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:31:04.175 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:31:04.175 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 2, 2024-03-11T22:31:04.175 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 71, 2024-03-11T22:31:04.175 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:31:04.175 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:31:04.175 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:31:04.175 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:31:04.175 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:31:04.175 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:31:04.175 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:31:04.175 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:31:04.176 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:31:04.176 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:31:04.176 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:31:04.176 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:31:04.176 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:31:04.176 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:31:04.176 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:31:04.176 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:31:04.176 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:31:04.176 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:31:04.176 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:31:04.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.176 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:31:04.177 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:31:04.177 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:31:04.177 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:31:04.177 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:31:04.177 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:31:04.177 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:31:04.177 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:31:04.177 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:31:04.177 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:31:04.177 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:31:04.177 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:31:04.177 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:31:04.177 INFO:teuthology.orchestra.run.smithi045.stdout: "rbd": {} 2024-03-11T22:31:04.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.178 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:31:04.178 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:31:04.178 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.9900000095367432, 2024-03-11T22:31:04.178 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.9900000095367432, 2024-03-11T22:31:04.178 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.1699999570846558, 2024-03-11T22:31:04.178 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-11T22:31:04.178 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-11T22:31:04.178 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T22:31:04.178 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:31:04.178 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T22:31:04.178 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 3, 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": ".rgw.root", 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:31.591167+0000", 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:31:04.179 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:31:04.180 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:31:04.180 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:31:04.180 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 32, 2024-03-11T22:31:04.180 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 32, 2024-03-11T22:31:04.180 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 32, 2024-03-11T22:31:04.180 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 32, 2024-03-11T22:31:04.180 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 32, 2024-03-11T22:31:04.180 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:31:04.180 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:31:04.180 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:31:04.180 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:31:04.180 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:31:04.180 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "69", 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:31:04.181 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:31:04.182 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:31:04.182 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:31:04.182 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:31:04.182 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:31:04.182 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:31:04.182 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:31:04.182 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:31:04.182 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:31:04.182 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:31:04.182 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:31:04.182 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:31:04.182 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:31:04.182 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:31:04.183 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw": {} 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.5, 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.5, 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2, 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2, 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 4, 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "default.rgw.log", 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:33.606516+0000", 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:31:04.185 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:31:04.186 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:31:04.186 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:31:04.186 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 32, 2024-03-11T22:31:04.186 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 32, 2024-03-11T22:31:04.186 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 32, 2024-03-11T22:31:04.186 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 32, 2024-03-11T22:31:04.186 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 32, 2024-03-11T22:31:04.186 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:31:04.186 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:31:04.186 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:31:04.186 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:31:04.187 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:31:04.187 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:31:04.187 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:31:04.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.187 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "71", 2024-03-11T22:31:04.187 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:31:04.187 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:31:04.187 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:31:04.187 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:31:04.187 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:31:04.187 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:31:04.188 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:31:04.189 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:31:04.190 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:31:04.190 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:31:04.190 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:31:04.190 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:31:04.190 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:31:04.190 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:31:04.190 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw": {} 2024-03-11T22:31:04.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.190 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:31:04.190 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:31:04.190 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.8799999952316284, 2024-03-11T22:31:04.190 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.8799999952316284, 2024-03-11T22:31:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T22:31:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2.5, 2024-03-11T22:31:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2.5, 2024-03-11T22:31:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T22:31:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:31:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T22:31:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 5, 2024-03-11T22:31:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "default.rgw.control", 2024-03-11T22:31:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:35.315522+0000", 2024-03-11T22:31:04.191 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 32, 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 32, 2024-03-11T22:31:04.192 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 32, 2024-03-11T22:31:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 32, 2024-03-11T22:31:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 32, 2024-03-11T22:31:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:31:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:31:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:31:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:31:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:31:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:31:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:31:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "73", 2024-03-11T22:31:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:31:04.193 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:31:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:31:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:31:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:31:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:31:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:31:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:31:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:31:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:31:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:31:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:31:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:31:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:31:04.194 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:31:04.195 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:31:04.196 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:31:04.196 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:31:04.196 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:31:04.196 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:31:04.196 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:31:04.196 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:31:04.196 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:31:04.196 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:31:04.196 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:31:04.196 INFO:teuthology.orchestra.run.smithi045.stdout: "options": {}, 2024-03-11T22:31:04.196 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:31:04.196 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw": {} 2024-03-11T22:31:04.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.197 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:31:04.197 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:31:04.197 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.1299999952316284, 2024-03-11T22:31:04.197 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.1299999952316284, 2024-03-11T22:31:04.197 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T22:31:04.197 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 1.5, 2024-03-11T22:31:04.197 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 1.5, 2024-03-11T22:31:04.197 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T22:31:04.198 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:31:04.198 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T22:31:04.198 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.198 INFO:teuthology.orchestra.run.smithi045.stdout: "pool": 6, 2024-03-11T22:31:04.200 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_name": "default.rgw.meta", 2024-03-11T22:31:04.200 INFO:teuthology.orchestra.run.smithi045.stdout: "create_time": "2024-03-11T22:08:37.383856+0000", 2024-03-11T22:31:04.200 INFO:teuthology.orchestra.run.smithi045.stdout: "flags": 1, 2024-03-11T22:31:04.201 INFO:teuthology.orchestra.run.smithi045.stdout: "flags_names": "hashpspool", 2024-03-11T22:31:04.201 INFO:teuthology.orchestra.run.smithi045.stdout: "type": 1, 2024-03-11T22:31:04.201 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 3, 2024-03-11T22:31:04.201 INFO:teuthology.orchestra.run.smithi045.stdout: "min_size": 2, 2024-03-11T22:31:04.201 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_rule": 0, 2024-03-11T22:31:04.202 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_count": 0, 2024-03-11T22:31:04.202 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_target": 0, 2024-03-11T22:31:04.202 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T22:31:04.202 INFO:teuthology.orchestra.run.smithi045.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T22:31:04.202 INFO:teuthology.orchestra.run.smithi045.stdout: "object_hash": 2, 2024-03-11T22:31:04.202 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_mode": "off", 2024-03-11T22:31:04.202 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num": 32, 2024-03-11T22:31:04.202 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num": 32, 2024-03-11T22:31:04.202 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_placement_num_target": 32, 2024-03-11T22:31:04.202 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_target": 32, 2024-03-11T22:31:04.202 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_num_pending": 32, 2024-03-11T22:31:04.202 INFO:teuthology.orchestra.run.smithi045.stdout: "last_pg_merge_meta": { 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: "source_pgid": "0.0", 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: "ready_epoch": 0, 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_started": 0, 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: "last_epoch_clean": 0, 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: "source_version": "0'0", 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: "target_version": "0'0" 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: "last_change": "75", 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend": "0", 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: "auid": 0, 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_mode": "selfmanaged", 2024-03-11T22:31:04.203 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_seq": 0, 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "snap_epoch": 0, 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "pool_snaps": [], 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps": "[]", 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_bytes": 0, 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "quota_max_objects": 0, 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "tiers": [], 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "tier_of": -1, 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "read_tier": -1, 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "write_tier": -1, 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_mode": "none", 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_bytes": 0, 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "target_max_objects": 0, 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T22:31:04.204 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T22:31:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T22:31:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_flush_age": 0, 2024-03-11T22:31:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: "cache_min_evict_age": 0, 2024-03-11T22:31:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profile": "", 2024-03-11T22:31:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_params": { 2024-03-11T22:31:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "none" 2024-03-11T22:31:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_period": 0, 2024-03-11T22:31:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_count": 0, 2024-03-11T22:31:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: "use_gmt_hitset": true, 2024-03-11T22:31:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: "min_read_recency_for_promote": 0, 2024-03-11T22:31:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: "min_write_recency_for_promote": 0, 2024-03-11T22:31:04.205 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T22:31:04.206 INFO:teuthology.orchestra.run.smithi045.stdout: "hit_set_search_last_n": 0, 2024-03-11T22:31:04.206 INFO:teuthology.orchestra.run.smithi045.stdout: "grade_table": [], 2024-03-11T22:31:04.206 INFO:teuthology.orchestra.run.smithi045.stdout: "stripe_width": 0, 2024-03-11T22:31:04.206 INFO:teuthology.orchestra.run.smithi045.stdout: "expected_num_objects": 0, 2024-03-11T22:31:04.206 INFO:teuthology.orchestra.run.smithi045.stdout: "fast_read": false, 2024-03-11T22:31:04.206 INFO:teuthology.orchestra.run.smithi045.stdout: "options": { 2024-03-11T22:31:04.206 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_autoscale_bias": 4 2024-03-11T22:31:04.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.206 INFO:teuthology.orchestra.run.smithi045.stdout: "application_metadata": { 2024-03-11T22:31:04.206 INFO:teuthology.orchestra.run.smithi045.stdout: "rgw": {} 2024-03-11T22:31:04.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.206 INFO:teuthology.orchestra.run.smithi045.stdout: "read_balance": { 2024-03-11T22:31:04.207 INFO:teuthology.orchestra.run.smithi045.stdout: "score_type": "Fair distribution", 2024-03-11T22:31:04.207 INFO:teuthology.orchestra.run.smithi045.stdout: "score_acting": 1.5, 2024-03-11T22:31:04.207 INFO:teuthology.orchestra.run.smithi045.stdout: "score_stable": 1.5, 2024-03-11T22:31:04.207 INFO:teuthology.orchestra.run.smithi045.stdout: "optimal_score": 1.3300000429153442, 2024-03-11T22:31:04.207 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_acting": 2, 2024-03-11T22:31:04.207 INFO:teuthology.orchestra.run.smithi045.stdout: "raw_score_stable": 2, 2024-03-11T22:31:04.207 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity_weighted": 0.75, 2024-03-11T22:31:04.207 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity": 1, 2024-03-11T22:31:04.207 INFO:teuthology.orchestra.run.smithi045.stdout: "average_primary_affinity_weighted": 0.75 2024-03-11T22:31:04.207 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.207 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.208 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:31:04.208 INFO:teuthology.orchestra.run.smithi045.stdout: "osds": [ 2024-03-11T22:31:04.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.208 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 0, 2024-03-11T22:31:04.208 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "55a4e633-f2a6-4d27-b10c-de1c0f87cf32", 2024-03-11T22:31:04.208 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:31:04.208 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:31:04.208 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:31:04.208 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:31:04.208 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:31:04.208 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:31:04.208 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 8, 2024-03-11T22:31:04.208 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 506, 2024-03-11T22:31:04.209 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:31:04.209 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:31:04.209 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:31:04.209 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.209 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.209 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6802", 2024-03-11T22:31:04.209 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:31:04.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.209 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6803", 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6804", 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.210 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6805", 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6808", 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.211 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6809", 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6806", 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.212 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6807", 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2216353820 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6803/2216353820", 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.45:6805/2216353820", 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.45:6809/2216353820", 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.45:6807/2216353820", 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 1, 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "60274eeb-591a-43fe-a579-f13d13631e14", 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 13, 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 515, 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:31:04.214 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:31:04.215 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.215 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.215 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6810", 2024-03-11T22:31:04.215 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:31:04.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.215 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.215 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6811", 2024-03-11T22:31:04.215 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:31:04.215 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.215 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.216 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:31:04.216 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.216 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.216 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6812", 2024-03-11T22:31:04.216 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:31:04.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.216 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.216 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6813", 2024-03-11T22:31:04.216 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:31:04.216 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.216 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.217 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:31:04.217 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.217 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.217 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6816", 2024-03-11T22:31:04.217 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:31:04.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.217 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.217 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6817", 2024-03-11T22:31:04.217 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:31:04.217 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6814", 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6815", 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 1241486091 2024-03-11T22:31:04.218 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6811/1241486091", 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.45:6813/1241486091", 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.45:6817/1241486091", 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.45:6815/1241486091", 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 2, 2024-03-11T22:31:04.219 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b", 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 18, 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 504, 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.220 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6818", 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6819", 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.221 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6820", 2024-03-11T22:31:04.222 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:31:04.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.222 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.222 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6821", 2024-03-11T22:31:04.222 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:31:04.222 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.222 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.222 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:31:04.222 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6824", 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6825", 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.224 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.224 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6822", 2024-03-11T22:31:04.224 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:31:04.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.224 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.224 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6823", 2024-03-11T22:31:04.224 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 4251983518 2024-03-11T22:31:04.224 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.224 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.224 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6819/4251983518", 2024-03-11T22:31:04.224 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.45:6821/4251983518", 2024-03-11T22:31:04.225 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.45:6825/4251983518", 2024-03-11T22:31:04.225 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.45:6823/4251983518", 2024-03-11T22:31:04.225 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:31:04.225 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:31:04.225 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:31:04.225 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.225 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 3, 2024-03-11T22:31:04.225 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "e51a65f3-96f6-478c-8d26-df1ace4a5058", 2024-03-11T22:31:04.225 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:31:04.225 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:31:04.225 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:31:04.226 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:31:04.226 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:31:04.226 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:31:04.226 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 26, 2024-03-11T22:31:04.226 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 518, 2024-03-11T22:31:04.226 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:31:04.226 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:31:04.226 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:31:04.226 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.226 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.226 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6826", 2024-03-11T22:31:04.226 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:31:04.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.227 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.227 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6827", 2024-03-11T22:31:04.227 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:31:04.227 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.227 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.227 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:31:04.227 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6828", 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6829", 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:31:04.228 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6832", 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6833", 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:31:04.229 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.230 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.230 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6830", 2024-03-11T22:31:04.230 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:31:04.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.230 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.230 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.45:6831", 2024-03-11T22:31:04.230 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3178755830 2024-03-11T22:31:04.230 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.230 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.231 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.45:6827/3178755830", 2024-03-11T22:31:04.231 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.45:6829/3178755830", 2024-03-11T22:31:04.231 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.45:6833/3178755830", 2024-03-11T22:31:04.231 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.45:6831/3178755830", 2024-03-11T22:31:04.231 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:31:04.231 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:31:04.231 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:31:04.231 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.231 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 4, 2024-03-11T22:31:04.231 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "32d0b71f-c696-4677-bdd3-b88a04c97604", 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 0, 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 33, 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 501, 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.232 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6800", 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6801", 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6802", 2024-03-11T22:31:04.233 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:31:04.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.234 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.234 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6803", 2024-03-11T22:31:04.234 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:31:04.234 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.234 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.234 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:31:04.234 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.235 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.235 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6806", 2024-03-11T22:31:04.235 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:31:04.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.235 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.235 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6807", 2024-03-11T22:31:04.236 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:31:04.236 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.236 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.236 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:31:04.236 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.237 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.237 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6804", 2024-03-11T22:31:04.237 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:31:04.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.237 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.237 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6805", 2024-03-11T22:31:04.237 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 912745159 2024-03-11T22:31:04.238 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.238 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.238 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.144:6801/912745159", 2024-03-11T22:31:04.238 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.144:6803/912745159", 2024-03-11T22:31:04.238 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.144:6807/912745159", 2024-03-11T22:31:04.238 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.144:6805/912745159", 2024-03-11T22:31:04.238 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:31:04.238 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:31:04.238 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 5, 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "f46b91af-d67d-404c-aad1-75348e35d1c4", 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 41, 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 518, 2024-03-11T22:31:04.239 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6808", 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6809", 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6810", 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6811", 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.241 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6814", 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6815", 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.242 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:31:04.243 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.243 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.243 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6812", 2024-03-11T22:31:04.243 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:31:04.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.243 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.243 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6813", 2024-03-11T22:31:04.243 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3162869174 2024-03-11T22:31:04.243 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.144:6809/3162869174", 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.144:6811/3162869174", 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.144:6815/3162869174", 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.144:6813/3162869174", 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 6, 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "970b128d-cd56-4dff-8345-3196b46c1257", 2024-03-11T22:31:04.244 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:31:04.245 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:31:04.245 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:31:04.245 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 1, 2024-03-11T22:31:04.245 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:31:04.245 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:31:04.245 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 51, 2024-03-11T22:31:04.245 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 518, 2024-03-11T22:31:04.245 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:31:04.245 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:31:04.245 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:31:04.245 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.245 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.246 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6816", 2024-03-11T22:31:04.246 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:31:04.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.246 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.246 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6817", 2024-03-11T22:31:04.246 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:31:04.246 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.246 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.246 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6818", 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6819", 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:31:04.247 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.248 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.248 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6822", 2024-03-11T22:31:04.248 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:31:04.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.248 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.248 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6823", 2024-03-11T22:31:04.248 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:31:04.248 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.248 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.248 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6820", 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6821", 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 3439446288 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.144:6817/3439446288", 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.144:6819/3439446288", 2024-03-11T22:31:04.249 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.144:6823/3439446288", 2024-03-11T22:31:04.250 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.144:6821/3439446288", 2024-03-11T22:31:04.250 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:31:04.250 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:31:04.250 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:31:04.250 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.250 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 7, 2024-03-11T22:31:04.250 INFO:teuthology.orchestra.run.smithi045.stdout: "uuid": "40100aaf-830d-4137-b037-5cf167cba308", 2024-03-11T22:31:04.250 INFO:teuthology.orchestra.run.smithi045.stdout: "up": 1, 2024-03-11T22:31:04.250 INFO:teuthology.orchestra.run.smithi045.stdout: "in": 1, 2024-03-11T22:31:04.250 INFO:teuthology.orchestra.run.smithi045.stdout: "weight": 1, 2024-03-11T22:31:04.250 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_affinity": 0, 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_begin": 0, 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: "last_clean_end": 0, 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: "up_from": 61, 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: "up_thru": 73, 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: "down_at": 0, 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: "lost_at": 0, 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addrs": { 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6824", 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6825", 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addrs": { 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6826", 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.252 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6827", 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addrs": { 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6830", 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.253 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6831", 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addrs": { 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: "addrvec": [ 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v2", 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6828", 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.254 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "172.21.15.144:6829", 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 2371064429 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: "public_addr": "172.21.15.144:6825/2371064429", 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_addr": "172.21.15.144:6827/2371064429", 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_back_addr": "172.21.15.144:6831/2371064429", 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: "heartbeat_front_addr": "172.21.15.144:6829/2371064429", 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: "state": [ 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: "exists", 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: "up" 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.255 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:31:04.256 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_xinfo": [ 2024-03-11T22:31:04.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.256 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 0, 2024-03-11T22:31:04.256 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:31:04.256 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:31:04.256 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:31:04.256 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:31:04.256 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:31:04.256 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:30:20.123858+0000", 2024-03-11T22:31:04.256 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:31:04.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.256 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 1, 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:29:56.332395+0000", 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 2, 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:31:04.257 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:31:04.258 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:31:04.258 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:30:27.982816+0000", 2024-03-11T22:31:04.258 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:31:04.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.258 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 3, 2024-03-11T22:31:04.258 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:31:04.258 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:31:04.258 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:31:04.258 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:31:04.258 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:31:04.258 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:30:53.327967+0000", 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 4, 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:29:52.989185+0000", 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 5, 2024-03-11T22:31:04.259 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:31:04.260 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:31:04.260 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:31:04.260 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:31:04.260 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:31:04.260 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:30:20.928119+0000", 2024-03-11T22:31:04.260 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:31:04.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.260 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 6, 2024-03-11T22:31:04.260 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:31:04.260 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:31:04.260 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:30:17.503923+0000", 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": 7, 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: "down_stamp": "0.000000", 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_probability": 0, 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: "laggy_interval": 0, 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: "features": 4540701547738038271, 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: "old_weight": 0, 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: "last_purged_snaps_scrub": "2024-03-11T22:30:40.448372+0000", 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: "dead_epoch": 0 2024-03-11T22:31:04.261 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_upmap": [], 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_upmap_items": [ 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "4.4", 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: "mappings": [ 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: "from": 1, 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: "to": 5 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.262 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "4.e", 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: "mappings": [ 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: "from": 1, 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: "to": 2 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: "pgid": "4.17", 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: "mappings": [ 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: "from": 1, 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: "to": 2 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.263 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_upmap_primaries": [], 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: "pg_temp": [], 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: "primary_temp": [], 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: "blocklist": { 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/1901139892": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/1110781721": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6800/2299156110": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/2331445880": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6801/4256719031": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6801/2299156110": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6800/3882489981": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:31:04.264 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/272906373": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/11898002": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/2777740355": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/1903379086": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/926341889": "2024-03-12T22:03:25.087713+0000", 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6800/4256719031": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/2513993410": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/2584267737": "2024-03-12T22:03:52.557006+0000", 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:0/3509006862": "2024-03-12T22:09:14.021723+0000", 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "172.21.15.45:6801/3882489981": "2024-03-12T22:09:14.021723+0000" 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "range_blocklist": {}, 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "erasure_code_profiles": { 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "default": { 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "crush-failure-domain": "osd", 2024-03-11T22:31:04.265 INFO:teuthology.orchestra.run.smithi045.stdout: "k": "2", 2024-03-11T22:31:04.266 INFO:teuthology.orchestra.run.smithi045.stdout: "m": "1", 2024-03-11T22:31:04.266 INFO:teuthology.orchestra.run.smithi045.stdout: "plugin": "jerasure", 2024-03-11T22:31:04.266 INFO:teuthology.orchestra.run.smithi045.stdout: "technique": "reed_sol_van" 2024-03-11T22:31:04.266 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-03-11T22:31:04.266 INFO:teuthology.orchestra.run.smithi045.stdout: "removed_snaps_queue": [], 2024-03-11T22:31:04.266 INFO:teuthology.orchestra.run.smithi045.stdout: "new_removed_snaps": [], 2024-03-11T22:31:04.266 INFO:teuthology.orchestra.run.smithi045.stdout: "new_purged_snaps": [], 2024-03-11T22:31:04.266 INFO:teuthology.orchestra.run.smithi045.stdout: "crush_node_flags": {}, 2024-03-11T22:31:04.266 INFO:teuthology.orchestra.run.smithi045.stdout: "device_class_flags": {}, 2024-03-11T22:31:04.266 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode": { 2024-03-11T22:31:04.267 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode_enabled": false, 2024-03-11T22:31:04.267 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_bucket_count": 0, 2024-03-11T22:31:04.267 INFO:teuthology.orchestra.run.smithi045.stdout: "degraded_stretch_mode": 0, 2024-03-11T22:31:04.267 INFO:teuthology.orchestra.run.smithi045.stdout: "recovering_stretch_mode": 0, 2024-03-11T22:31:04.267 INFO:teuthology.orchestra.run.smithi045.stdout: "stretch_mode_bucket": 0 2024-03-11T22:31:04.267 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-03-11T22:31:04.267 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-03-11T22:31:04.777 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T22:31:05.023 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:31:05.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:04 smithi045 ceph-mon[39715]: 3.17 deep-scrub starts 2024-03-11T22:31:05.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:04 smithi045 ceph-mon[39715]: 3.17 deep-scrub ok 2024-03-11T22:31:05.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:04 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1353198743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T22:31:05.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:04 smithi045 ceph-mon[30178]: 3.17 deep-scrub starts 2024-03-11T22:31:05.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:04 smithi045 ceph-mon[30178]: 3.17 deep-scrub ok 2024-03-11T22:31:05.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:04 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1353198743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T22:31:05.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:04 smithi144 ceph-mon[33984]: 3.17 deep-scrub starts 2024-03-11T22:31:05.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:04 smithi144 ceph-mon[33984]: 3.17 deep-scrub ok 2024-03-11T22:31:05.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:04 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1353198743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T22:31:05.261 INFO:tasks.thrashosds.thrasher:j is {'epoch': 538, 'fsid': 'f75aafa4-dff2-11ee-95c8-87774f69a715', 'created': '2024-03-11T22:02:51.075940+0000', 'modified': '2024-03-11T22:30:54.701461+0000', 'last_up_change': '2024-03-11T22:08:17.992246+0000', 'last_in_change': '2024-03-11T22:29:03.667666+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T22:06:12.903386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 438, 'last_epoch_started': 437, 'last_epoch_clean': 437, 'source_version': "93'130", 'target_version': "93'130"}, 'last_change': '438', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '438', 'last_force_op_resend_preluminous': '436', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6399999856948853, 'score_stable': 1.6399999856948853, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-11T22:08:31.014060+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T22:08:31.591167+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T22:08:33.606516+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T22:08:35.315522+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.1299999952316284, 'score_stable': 1.1299999952316284, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T22:08:37.383856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.75, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.75}}], 'osds': [{'osd': 0, 'uuid': '55a4e633-f2a6-4d27-b10c-de1c0f87cf32', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 506, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6802', 'nonce': 2216353820}, {'type': 'v1', 'addr': '172.21.15.45:6803', 'nonce': 2216353820}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6804', 'nonce': 2216353820}, {'type': 'v1', 'addr': '172.21.15.45:6805', 'nonce': 2216353820}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6808', 'nonce': 2216353820}, {'type': 'v1', 'addr': '172.21.15.45:6809', 'nonce': 2216353820}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6806', 'nonce': 2216353820}, {'type': 'v1', 'addr': '172.21.15.45:6807', 'nonce': 2216353820}]}, 'public_addr': '172.21.15.45:6803/2216353820', 'cluster_addr': '172.21.15.45:6805/2216353820', 'heartbeat_back_addr': '172.21.15.45:6809/2216353820', 'heartbeat_front_addr': '172.21.15.45:6807/2216353820', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '60274eeb-591a-43fe-a579-f13d13631e14', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 515, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6810', 'nonce': 1241486091}, {'type': 'v1', 'addr': '172.21.15.45:6811', 'nonce': 1241486091}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6812', 'nonce': 1241486091}, {'type': 'v1', 'addr': '172.21.15.45:6813', 'nonce': 1241486091}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6816', 'nonce': 1241486091}, {'type': 'v1', 'addr': '172.21.15.45:6817', 'nonce': 1241486091}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6814', 'nonce': 1241486091}, {'type': 'v1', 'addr': '172.21.15.45:6815', 'nonce': 1241486091}]}, 'public_addr': '172.21.15.45:6811/1241486091', 'cluster_addr': '172.21.15.45:6813/1241486091', 'heartbeat_back_addr': '172.21.15.45:6817/1241486091', 'heartbeat_front_addr': '172.21.15.45:6815/1241486091', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 504, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6818', 'nonce': 4251983518}, {'type': 'v1', 'addr': '172.21.15.45:6819', 'nonce': 4251983518}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6820', 'nonce': 4251983518}, {'type': 'v1', 'addr': '172.21.15.45:6821', 'nonce': 4251983518}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6824', 'nonce': 4251983518}, {'type': 'v1', 'addr': '172.21.15.45:6825', 'nonce': 4251983518}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6822', 'nonce': 4251983518}, {'type': 'v1', 'addr': '172.21.15.45:6823', 'nonce': 4251983518}]}, 'public_addr': '172.21.15.45:6819/4251983518', 'cluster_addr': '172.21.15.45:6821/4251983518', 'heartbeat_back_addr': '172.21.15.45:6825/4251983518', 'heartbeat_front_addr': '172.21.15.45:6823/4251983518', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'e51a65f3-96f6-478c-8d26-df1ace4a5058', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 518, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6826', 'nonce': 3178755830}, {'type': 'v1', 'addr': '172.21.15.45:6827', 'nonce': 3178755830}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6828', 'nonce': 3178755830}, {'type': 'v1', 'addr': '172.21.15.45:6829', 'nonce': 3178755830}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6832', 'nonce': 3178755830}, {'type': 'v1', 'addr': '172.21.15.45:6833', 'nonce': 3178755830}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6830', 'nonce': 3178755830}, {'type': 'v1', 'addr': '172.21.15.45:6831', 'nonce': 3178755830}]}, 'public_addr': '172.21.15.45:6827/3178755830', 'cluster_addr': '172.21.15.45:6829/3178755830', 'heartbeat_back_addr': '172.21.15.45:6833/3178755830', 'heartbeat_front_addr': '172.21.15.45:6831/3178755830', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '32d0b71f-c696-4677-bdd3-b88a04c97604', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 501, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6800', 'nonce': 912745159}, {'type': 'v1', 'addr': '172.21.15.144:6801', 'nonce': 912745159}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6802', 'nonce': 912745159}, {'type': 'v1', 'addr': '172.21.15.144:6803', 'nonce': 912745159}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6806', 'nonce': 912745159}, {'type': 'v1', 'addr': '172.21.15.144:6807', 'nonce': 912745159}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6804', 'nonce': 912745159}, {'type': 'v1', 'addr': '172.21.15.144:6805', 'nonce': 912745159}]}, 'public_addr': '172.21.15.144:6801/912745159', 'cluster_addr': '172.21.15.144:6803/912745159', 'heartbeat_back_addr': '172.21.15.144:6807/912745159', 'heartbeat_front_addr': '172.21.15.144:6805/912745159', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f46b91af-d67d-404c-aad1-75348e35d1c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 518, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6808', 'nonce': 3162869174}, {'type': 'v1', 'addr': '172.21.15.144:6809', 'nonce': 3162869174}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6810', 'nonce': 3162869174}, {'type': 'v1', 'addr': '172.21.15.144:6811', 'nonce': 3162869174}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6814', 'nonce': 3162869174}, {'type': 'v1', 'addr': '172.21.15.144:6815', 'nonce': 3162869174}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6812', 'nonce': 3162869174}, {'type': 'v1', 'addr': '172.21.15.144:6813', 'nonce': 3162869174}]}, 'public_addr': '172.21.15.144:6809/3162869174', 'cluster_addr': '172.21.15.144:6811/3162869174', 'heartbeat_back_addr': '172.21.15.144:6815/3162869174', 'heartbeat_front_addr': '172.21.15.144:6813/3162869174', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '970b128d-cd56-4dff-8345-3196b46c1257', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 518, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6816', 'nonce': 3439446288}, {'type': 'v1', 'addr': '172.21.15.144:6817', 'nonce': 3439446288}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6818', 'nonce': 3439446288}, {'type': 'v1', 'addr': '172.21.15.144:6819', 'nonce': 3439446288}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6822', 'nonce': 3439446288}, {'type': 'v1', 'addr': '172.21.15.144:6823', 'nonce': 3439446288}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6820', 'nonce': 3439446288}, {'type': 'v1', 'addr': '172.21.15.144:6821', 'nonce': 3439446288}]}, 'public_addr': '172.21.15.144:6817/3439446288', 'cluster_addr': '172.21.15.144:6819/3439446288', 'heartbeat_back_addr': '172.21.15.144:6823/3439446288', 'heartbeat_front_addr': '172.21.15.144:6821/3439446288', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '40100aaf-830d-4137-b037-5cf167cba308', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 61, 'up_thru': 73, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6824', 'nonce': 2371064429}, {'type': 'v1', 'addr': '172.21.15.144:6825', 'nonce': 2371064429}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6826', 'nonce': 2371064429}, {'type': 'v1', 'addr': '172.21.15.144:6827', 'nonce': 2371064429}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6830', 'nonce': 2371064429}, {'type': 'v1', 'addr': '172.21.15.144:6831', 'nonce': 2371064429}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6828', 'nonce': 2371064429}, {'type': 'v1', 'addr': '172.21.15.144:6829', 'nonce': 2371064429}]}, 'public_addr': '172.21.15.144:6825/2371064429', 'cluster_addr': '172.21.15.144:6827/2371064429', 'heartbeat_back_addr': '172.21.15.144:6831/2371064429', 'heartbeat_front_addr': '172.21.15.144:6829/2371064429', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:30:20.123858+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:29:56.332395+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:30:27.982816+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:30:53.327967+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:29:52.989185+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:30:20.928119+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:30:17.503923+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T22:30:40.448372+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.45:0/1901139892': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/1110781721': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:6800/2299156110': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:0/2331445880': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:6801/4256719031': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:6801/2299156110': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:6800/3882489981': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/272906373': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/11898002': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/2777740355': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:0/1903379086': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:0/926341889': '2024-03-12T22:03:25.087713+0000', '172.21.15.45:6800/4256719031': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:0/2513993410': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:0/2584267737': '2024-03-12T22:03:52.557006+0000', '172.21.15.45:0/3509006862': '2024-03-12T22:09:14.021723+0000', '172.21.15.45:6801/3882489981': '2024-03-12T22:09:14.021723+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T22:31:05.261 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:31:05.509 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:05.911 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:05 smithi045 ceph-mon[39715]: pgmap v1110: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:31:05.911 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:05 smithi045 ceph-mon[39715]: 5.1 scrub starts 2024-03-11T22:31:05.911 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:05 smithi045 ceph-mon[39715]: 5.1 scrub ok 2024-03-11T22:31:05.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:05 smithi045 ceph-mon[30178]: pgmap v1110: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:31:05.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:05 smithi045 ceph-mon[30178]: 5.1 scrub starts 2024-03-11T22:31:05.911 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:05 smithi045 ceph-mon[30178]: 5.1 scrub ok 2024-03-11T22:31:06.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:05 smithi144 ceph-mon[33984]: pgmap v1110: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:31:06.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:05 smithi144 ceph-mon[33984]: 5.1 scrub starts 2024-03-11T22:31:06.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:05 smithi144 ceph-mon[33984]: 5.1 scrub ok 2024-03-11T22:31:06.824 INFO:teuthology.orchestra.run.smithi144.stdout:ERROR: (22) Invalid argument 2024-03-11T22:31:06.824 INFO:teuthology.orchestra.run.smithi144.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T22:31:07.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:31:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:31:07.489 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:31:07.492 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:31:07.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:07 smithi144 ceph-mon[33984]: 4.15 scrub starts 2024-03-11T22:31:07.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:07 smithi144 ceph-mon[33984]: 4.15 scrub ok 2024-03-11T22:31:07.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:07 smithi144 ceph-mon[33984]: pgmap v1111: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-03-11T22:31:07.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:07 smithi144 ceph-mon[33984]: 6.3 scrub starts 2024-03-11T22:31:07.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:07 smithi144 ceph-mon[33984]: 6.3 scrub ok 2024-03-11T22:31:07.851 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:07 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:08.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:07 smithi045 ceph-mon[39715]: 4.15 scrub starts 2024-03-11T22:31:08.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:07 smithi045 ceph-mon[39715]: 4.15 scrub ok 2024-03-11T22:31:08.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:07 smithi045 ceph-mon[39715]: pgmap v1111: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-03-11T22:31:08.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:07 smithi045 ceph-mon[39715]: 6.3 scrub starts 2024-03-11T22:31:08.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:07 smithi045 ceph-mon[39715]: 6.3 scrub ok 2024-03-11T22:31:08.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:07 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:08.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:07 smithi045 ceph-mon[30178]: 4.15 scrub starts 2024-03-11T22:31:08.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:07 smithi045 ceph-mon[30178]: 4.15 scrub ok 2024-03-11T22:31:08.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:07 smithi045 ceph-mon[30178]: pgmap v1111: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-03-11T22:31:08.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:07 smithi045 ceph-mon[30178]: 6.3 scrub starts 2024-03-11T22:31:08.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:07 smithi045 ceph-mon[30178]: 6.3 scrub ok 2024-03-11T22:31:08.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:07 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:08.142 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":1111,"stamp":"2024-03-11T22:31:06.536365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1851,"num_read_kb":1603,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":353,"num_bytes_recovered":336286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1137,"ondisk_log_size":1137,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6182240,"kb_used_data":18652,"kb_used_omap":12,"kb_used_meta":6163507,"kb_avail":743778976,"statfs":{"total":767960285184,"available":761629671424,"internally_reserved":0,"allocated":19099648,"data_stored":12542652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":6311431731},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.833245"},"pg_stats":[{"pgid":"4.c","version":"479'35","reported_seq":835,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392097+0000","last_change":"2024-03-11T22:30:04.763741+0000","last_active":"2024-03-11T22:30:55.392097+0000","last_peered":"2024-03-11T22:30:55.392097+0000","last_clean":"2024-03-11T22:30:55.392097+0000","last_became_active":"2024-03-11T22:14:52.237576+0000","last_became_peered":"2024-03-11T22:14:52.237576+0000","last_unstale":"2024-03-11T22:30:55.392097+0000","last_undegraded":"2024-03-11T22:30:55.392097+0000","last_fullsized":"2024-03-11T22:30:55.392097+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'35","last_scrub_stamp":"2024-03-11T22:30:04.763606+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:30:04.763606+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:24.323735+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1008,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391482+0000","last_change":"2024-03-11T22:30:09.860422+0000","last_active":"2024-03-11T22:30:55.391482+0000","last_peered":"2024-03-11T22:30:55.391482+0000","last_clean":"2024-03-11T22:30:55.391482+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:30:55.391482+0000","last_undegraded":"2024-03-11T22:30:55.391482+0000","last_fullsized":"2024-03-11T22:30:55.391482+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:09.860304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:09.860304+0000","last_clean_scrub_stamp":"2024-03-11T22:30:09.860304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:31.943754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":1320,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:58.624475+0000","last_change":"2024-03-11T22:30:19.637446+0000","last_active":"2024-03-11T22:30:58.624475+0000","last_peered":"2024-03-11T22:30:58.624475+0000","last_clean":"2024-03-11T22:30:58.624475+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:30:58.624475+0000","last_undegraded":"2024-03-11T22:30:58.624475+0000","last_fullsized":"2024-03-11T22:30:58.624475+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:30:19.637348+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:26:11.939962+0000","last_clean_scrub_stamp":"2024-03-11T22:30:19.637348+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:40.836875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690185+0000","last_change":"2024-03-11T22:30:55.690226+0000","last_active":"2024-03-11T22:30:55.690185+0000","last_peered":"2024-03-11T22:30:55.690185+0000","last_clean":"2024-03-11T22:30:55.690185+0000","last_became_active":"2024-03-11T22:29:04.727120+0000","last_became_peered":"2024-03-11T22:29:04.727120+0000","last_unstale":"2024-03-11T22:30:55.690185+0000","last_undegraded":"2024-03-11T22:30:55.690185+0000","last_fullsized":"2024-03-11T22:30:55.690185+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:55.689796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:49.601908+0000","last_clean_scrub_stamp":"2024-03-11T22:30:55.689796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:12.015039+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":1353,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391270+0000","last_change":"2024-03-11T22:26:43.904135+0000","last_active":"2024-03-11T22:30:55.391270+0000","last_peered":"2024-03-11T22:30:55.391270+0000","last_clean":"2024-03-11T22:30:55.391270+0000","last_became_active":"2024-03-11T22:20:32.293356+0000","last_became_peered":"2024-03-11T22:20:32.293356+0000","last_unstale":"2024-03-11T22:30:55.391270+0000","last_undegraded":"2024-03-11T22:30:55.391270+0000","last_fullsized":"2024-03-11T22:30:55.391270+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:43.904050+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:43.904050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"479'53","reported_seq":944,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704828+0000","last_change":"2024-03-11T22:30:04.672014+0000","last_active":"2024-03-11T22:30:54.704828+0000","last_peered":"2024-03-11T22:30:54.704828+0000","last_clean":"2024-03-11T22:30:54.704828+0000","last_became_active":"2024-03-11T22:29:04.695558+0000","last_became_peered":"2024-03-11T22:29:04.695558+0000","last_unstale":"2024-03-11T22:30:54.704828+0000","last_undegraded":"2024-03-11T22:30:54.704828+0000","last_fullsized":"2024-03-11T22:30:54.704828+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'53","last_scrub_stamp":"2024-03-11T22:30:04.671906+0000","last_deep_scrub":"353'39","last_deep_scrub_stamp":"2024-03-11T22:23:05.461729+0000","last_clean_scrub_stamp":"2024-03-11T22:30:04.671906+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:28.141862+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1043,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672488+0000","last_change":"2024-03-11T22:29:04.719200+0000","last_active":"2024-03-11T22:30:55.672488+0000","last_peered":"2024-03-11T22:30:55.672488+0000","last_clean":"2024-03-11T22:30:55.672488+0000","last_became_active":"2024-03-11T22:29:04.718290+0000","last_became_peered":"2024-03-11T22:29:04.718290+0000","last_unstale":"2024-03-11T22:30:55.672488+0000","last_undegraded":"2024-03-11T22:30:55.672488+0000","last_fullsized":"2024-03-11T22:30:55.672488+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:28:21.605401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:15:42.136540+0000","last_clean_scrub_stamp":"2024-03-11T22:28:21.605401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:59.632668+0000","last_change":"2024-03-11T22:30:59.632689+0000","last_active":"2024-03-11T22:30:59.632668+0000","last_peered":"2024-03-11T22:30:59.632668+0000","last_clean":"2024-03-11T22:30:59.632668+0000","last_became_active":"2024-03-11T22:29:04.705661+0000","last_became_peered":"2024-03-11T22:29:04.705661+0000","last_unstale":"2024-03-11T22:30:59.632668+0000","last_undegraded":"2024-03-11T22:30:59.632668+0000","last_fullsized":"2024-03-11T22:30:59.632668+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:59.632610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:16.594756+0000","last_clean_scrub_stamp":"2024-03-11T22:30:59.632610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:11.652033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1028,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704725+0000","last_change":"2024-03-11T22:30:10.839723+0000","last_active":"2024-03-11T22:30:54.704725+0000","last_peered":"2024-03-11T22:30:54.704725+0000","last_clean":"2024-03-11T22:30:54.704725+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:30:54.704725+0000","last_undegraded":"2024-03-11T22:30:54.704725+0000","last_fullsized":"2024-03-11T22:30:54.704725+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:10.839608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:10.839608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:30.826822+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":1337,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:57.479332+0000","last_change":"2024-03-11T22:30:57.479385+0000","last_active":"2024-03-11T22:30:57.479332+0000","last_peered":"2024-03-11T22:30:57.479332+0000","last_clean":"2024-03-11T22:30:57.479332+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:30:57.479332+0000","last_undegraded":"2024-03-11T22:30:57.479332+0000","last_fullsized":"2024-03-11T22:30:57.479332+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:30:57.479260+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:30:57.479260+0000","last_clean_scrub_stamp":"2024-03-11T22:30:57.479260+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:17.612404+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"516'32","reported_seq":910,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672880+0000","last_change":"2024-03-11T22:30:16.937051+0000","last_active":"2024-03-11T22:30:55.672880+0000","last_peered":"2024-03-11T22:30:55.672880+0000","last_clean":"2024-03-11T22:30:55.672880+0000","last_became_active":"2024-03-11T22:30:16.930541+0000","last_became_peered":"2024-03-11T22:30:16.930541+0000","last_unstale":"2024-03-11T22:30:55.672880+0000","last_undegraded":"2024-03-11T22:30:55.672880+0000","last_fullsized":"2024-03-11T22:30:55.672880+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'32","last_scrub_stamp":"2024-03-11T22:30:10.404123+0000","last_deep_scrub":"516'32","last_deep_scrub_stamp":"2024-03-11T22:30:10.404123+0000","last_clean_scrub_stamp":"2024-03-11T22:30:10.404123+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:34.707986+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":808,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:31:01.269737+0000","last_change":"2024-03-11T22:31:01.269760+0000","last_active":"2024-03-11T22:31:01.269737+0000","last_peered":"2024-03-11T22:31:01.269737+0000","last_clean":"2024-03-11T22:31:01.269737+0000","last_became_active":"2024-03-11T22:14:52.241717+0000","last_became_peered":"2024-03-11T22:14:52.241717+0000","last_unstale":"2024-03-11T22:31:01.269737+0000","last_undegraded":"2024-03-11T22:31:01.269737+0000","last_fullsized":"2024-03-11T22:31:01.269737+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:01.269678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:31:01.269678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:17.923093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1007,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401878+0000","last_change":"2024-03-11T22:30:40.303604+0000","last_active":"2024-03-11T22:30:55.401878+0000","last_peered":"2024-03-11T22:30:55.401878+0000","last_clean":"2024-03-11T22:30:55.401878+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:30:55.401878+0000","last_undegraded":"2024-03-11T22:30:55.401878+0000","last_fullsized":"2024-03-11T22:30:55.401878+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:40.303527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.303527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:46.392625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":987,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391638+0000","last_change":"2024-03-11T22:26:48.845265+0000","last_active":"2024-03-11T22:30:55.391638+0000","last_peered":"2024-03-11T22:30:55.391638+0000","last_clean":"2024-03-11T22:30:55.391638+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:30:55.391638+0000","last_undegraded":"2024-03-11T22:30:55.391638+0000","last_fullsized":"2024-03-11T22:30:55.391638+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:48.845177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:48.845177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"480'46","reported_seq":138,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706706+0000","last_change":"2024-03-11T22:30:46.625195+0000","last_active":"2024-03-11T22:30:54.706706+0000","last_peered":"2024-03-11T22:30:54.706706+0000","last_clean":"2024-03-11T22:30:54.706706+0000","last_became_active":"2024-03-11T22:29:04.726632+0000","last_became_peered":"2024-03-11T22:29:04.726632+0000","last_unstale":"2024-03-11T22:30:54.706706+0000","last_undegraded":"2024-03-11T22:30:54.706706+0000","last_fullsized":"2024-03-11T22:30:54.706706+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'46","last_scrub_stamp":"2024-03-11T22:30:46.625090+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:30:46.625090+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:06.299995+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1032,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391355+0000","last_change":"2024-03-11T22:30:45.799130+0000","last_active":"2024-03-11T22:30:55.391355+0000","last_peered":"2024-03-11T22:30:55.391355+0000","last_clean":"2024-03-11T22:30:55.391355+0000","last_became_active":"2024-03-11T22:29:04.701072+0000","last_became_peered":"2024-03-11T22:29:04.701072+0000","last_unstale":"2024-03-11T22:30:55.391355+0000","last_undegraded":"2024-03-11T22:30:55.391355+0000","last_fullsized":"2024-03-11T22:30:55.391355+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:45.799037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:45.799037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:49.472786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"176'9","reported_seq":995,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:58.624402+0000","last_change":"2024-03-11T22:29:57.068709+0000","last_active":"2024-03-11T22:30:58.624402+0000","last_peered":"2024-03-11T22:30:58.624402+0000","last_clean":"2024-03-11T22:30:58.624402+0000","last_became_active":"2024-03-11T22:14:52.247600+0000","last_became_peered":"2024-03-11T22:14:52.247600+0000","last_unstale":"2024-03-11T22:30:58.624402+0000","last_undegraded":"2024-03-11T22:30:58.624402+0000","last_fullsized":"2024-03-11T22:30:58.624402+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'9","last_scrub_stamp":"2024-03-11T22:29:57.068625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:29:57.068625+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1007,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401537+0000","last_change":"2024-03-11T22:29:04.787987+0000","last_active":"2024-03-11T22:30:55.401537+0000","last_peered":"2024-03-11T22:30:55.401537+0000","last_clean":"2024-03-11T22:30:55.401537+0000","last_became_active":"2024-03-11T22:29:04.787650+0000","last_became_peered":"2024-03-11T22:29:04.787650+0000","last_unstale":"2024-03-11T22:30:55.401537+0000","last_undegraded":"2024-03-11T22:30:55.401537+0000","last_fullsized":"2024-03-11T22:30:55.401537+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:36.095984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:26:36.095984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":1338,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401204+0000","last_change":"2024-03-11T22:30:43.217270+0000","last_active":"2024-03-11T22:30:55.401204+0000","last_peered":"2024-03-11T22:30:55.401204+0000","last_clean":"2024-03-11T22:30:55.401204+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:30:55.401204+0000","last_undegraded":"2024-03-11T22:30:55.401204+0000","last_fullsized":"2024-03-11T22:30:55.401204+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:30:43.217187+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:30:43.217187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:01.077102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":784,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391616+0000","last_change":"2024-03-11T22:26:09.040467+0000","last_active":"2024-03-11T22:30:55.391616+0000","last_peered":"2024-03-11T22:30:55.391616+0000","last_clean":"2024-03-11T22:30:55.391616+0000","last_became_active":"2024-03-11T22:20:29.438896+0000","last_became_peered":"2024-03-11T22:20:29.438896+0000","last_unstale":"2024-03-11T22:30:55.391616+0000","last_undegraded":"2024-03-11T22:30:55.391616+0000","last_fullsized":"2024-03-11T22:30:55.391616+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:09.040331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:09.040331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"479'42","reported_seq":1116,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391383+0000","last_change":"2024-03-11T22:26:05.020921+0000","last_active":"2024-03-11T22:30:55.391383+0000","last_peered":"2024-03-11T22:30:55.391383+0000","last_clean":"2024-03-11T22:30:55.391383+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:30:55.391383+0000","last_undegraded":"2024-03-11T22:30:55.391383+0000","last_fullsized":"2024-03-11T22:30:55.391383+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'30","last_scrub_stamp":"2024-03-11T22:26:05.020726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:05.020726+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1020,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392410+0000","last_change":"2024-03-11T22:30:01.607109+0000","last_active":"2024-03-11T22:30:55.392410+0000","last_peered":"2024-03-11T22:30:55.392410+0000","last_clean":"2024-03-11T22:30:55.392410+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:30:55.392410+0000","last_undegraded":"2024-03-11T22:30:55.392410+0000","last_fullsized":"2024-03-11T22:30:55.392410+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:01.606988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:01.606988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:25.665371+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":766,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391628+0000","last_change":"2024-03-11T22:25:51.005350+0000","last_active":"2024-03-11T22:30:55.391628+0000","last_peered":"2024-03-11T22:30:55.391628+0000","last_clean":"2024-03-11T22:30:55.391628+0000","last_became_active":"2024-03-11T22:14:52.232107+0000","last_became_peered":"2024-03-11T22:14:52.232107+0000","last_unstale":"2024-03-11T22:30:55.391628+0000","last_undegraded":"2024-03-11T22:30:55.391628+0000","last_fullsized":"2024-03-11T22:30:55.391628+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:51.005219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:25:51.005219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":1309,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705189+0000","last_change":"2024-03-11T22:29:55.711127+0000","last_active":"2024-03-11T22:30:54.705189+0000","last_peered":"2024-03-11T22:30:54.705189+0000","last_clean":"2024-03-11T22:30:54.705189+0000","last_became_active":"2024-03-11T22:26:49.698651+0000","last_became_peered":"2024-03-11T22:26:49.698651+0000","last_unstale":"2024-03-11T22:30:54.705189+0000","last_undegraded":"2024-03-11T22:30:54.705189+0000","last_fullsized":"2024-03-11T22:30:54.705189+0000","mapping_epoch":175,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:29:55.710911+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:25:05.773963+0000","last_clean_scrub_stamp":"2024-03-11T22:29:55.710911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:07.856356+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"488'10","reported_seq":1292,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391556+0000","last_change":"2024-03-11T22:29:04.839059+0000","last_active":"2024-03-11T22:30:55.391556+0000","last_peered":"2024-03-11T22:30:55.391556+0000","last_clean":"2024-03-11T22:30:55.391556+0000","last_became_active":"2024-03-11T22:29:04.721065+0000","last_became_peered":"2024-03-11T22:29:04.721065+0000","last_unstale":"2024-03-11T22:30:55.391556+0000","last_undegraded":"2024-03-11T22:30:55.391556+0000","last_fullsized":"2024-03-11T22:30:55.391556+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-11T22:26:13.061151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:13.061151+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"480'42","reported_seq":856,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401107+0000","last_change":"2024-03-11T22:30:48.353744+0000","last_active":"2024-03-11T22:30:55.401107+0000","last_peered":"2024-03-11T22:30:55.401107+0000","last_clean":"2024-03-11T22:30:55.401107+0000","last_became_active":"2024-03-11T22:14:52.247862+0000","last_became_peered":"2024-03-11T22:14:52.247862+0000","last_unstale":"2024-03-11T22:30:55.401107+0000","last_undegraded":"2024-03-11T22:30:55.401107+0000","last_fullsized":"2024-03-11T22:30:55.401107+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'42","last_scrub_stamp":"2024-03-11T22:30:48.353647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:48.353647+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:14.430459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":1036,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690460+0000","last_change":"2024-03-11T22:30:42.177735+0000","last_active":"2024-03-11T22:30:55.690460+0000","last_peered":"2024-03-11T22:30:55.690460+0000","last_clean":"2024-03-11T22:30:55.690460+0000","last_became_active":"2024-03-11T22:29:04.718160+0000","last_became_peered":"2024-03-11T22:29:04.718160+0000","last_unstale":"2024-03-11T22:30:55.690460+0000","last_undegraded":"2024-03-11T22:30:55.690460+0000","last_fullsized":"2024-03-11T22:30:55.690460+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:30:42.177651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:42.177651+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:52.743855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":961,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672538+0000","last_change":"2024-03-11T22:28:34.728469+0000","last_active":"2024-03-11T22:30:55.672538+0000","last_peered":"2024-03-11T22:30:55.672538+0000","last_clean":"2024-03-11T22:30:55.672538+0000","last_became_active":"2024-03-11T22:10:35.409838+0000","last_became_peered":"2024-03-11T22:10:35.409838+0000","last_unstale":"2024-03-11T22:30:55.672538+0000","last_undegraded":"2024-03-11T22:30:55.672538+0000","last_fullsized":"2024-03-11T22:30:55.672538+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:28:34.728337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:28:34.728337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":1366,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391508+0000","last_change":"2024-03-11T22:26:53.558080+0000","last_active":"2024-03-11T22:30:55.391508+0000","last_peered":"2024-03-11T22:30:55.391508+0000","last_clean":"2024-03-11T22:30:55.391508+0000","last_became_active":"2024-03-11T22:26:53.557811+0000","last_became_peered":"2024-03-11T22:26:53.557811+0000","last_unstale":"2024-03-11T22:30:55.391508+0000","last_undegraded":"2024-03-11T22:30:55.391508+0000","last_fullsized":"2024-03-11T22:30:55.391508+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:45.837466+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:45.837466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"378'9","reported_seq":1286,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.691244+0000","last_change":"2024-03-11T22:30:38.192095+0000","last_active":"2024-03-11T22:30:55.691244+0000","last_peered":"2024-03-11T22:30:55.691244+0000","last_clean":"2024-03-11T22:30:55.691244+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:30:55.691244+0000","last_undegraded":"2024-03-11T22:30:55.691244+0000","last_fullsized":"2024-03-11T22:30:55.691244+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9","last_scrub_stamp":"2024-03-11T22:30:38.191965+0000","last_deep_scrub":"378'9","last_deep_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_clean_scrub_stamp":"2024-03-11T22:30:38.191965+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:58.989567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'30","reported_seq":134,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705358+0000","last_change":"2024-03-11T22:30:02.607278+0000","last_active":"2024-03-11T22:30:54.705358+0000","last_peered":"2024-03-11T22:30:54.705358+0000","last_clean":"2024-03-11T22:30:54.705358+0000","last_became_active":"2024-03-11T22:29:04.727100+0000","last_became_peered":"2024-03-11T22:29:04.727100+0000","last_unstale":"2024-03-11T22:30:54.705358+0000","last_undegraded":"2024-03-11T22:30:54.705358+0000","last_fullsized":"2024-03-11T22:30:54.705358+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-03-11T22:30:02.607142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:02.607142+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:31.923645+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1016,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401706+0000","last_change":"2024-03-11T22:29:54.153237+0000","last_active":"2024-03-11T22:30:55.401706+0000","last_peered":"2024-03-11T22:30:55.401706+0000","last_clean":"2024-03-11T22:30:55.401706+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:30:55.401706+0000","last_undegraded":"2024-03-11T22:30:55.401706+0000","last_fullsized":"2024-03-11T22:30:55.401706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:54.153119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:29:54.153119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:01.613626+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1004,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.689702+0000","last_change":"2024-03-11T22:30:45.264761+0000","last_active":"2024-03-11T22:30:55.689702+0000","last_peered":"2024-03-11T22:30:55.689702+0000","last_clean":"2024-03-11T22:30:55.689702+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:30:55.689702+0000","last_undegraded":"2024-03-11T22:30:55.689702+0000","last_fullsized":"2024-03-11T22:30:55.689702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:45.264658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:45.264658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:14.204199+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":1288,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673132+0000","last_change":"2024-03-11T22:30:39.559483+0000","last_active":"2024-03-11T22:30:55.673132+0000","last_peered":"2024-03-11T22:30:55.673132+0000","last_clean":"2024-03-11T22:30:55.673132+0000","last_became_active":"2024-03-11T22:26:41.661375+0000","last_became_peered":"2024-03-11T22:26:41.661375+0000","last_unstale":"2024-03-11T22:30:55.673132+0000","last_undegraded":"2024-03-11T22:30:55.673132+0000","last_fullsized":"2024-03-11T22:30:55.673132+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":423,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:30:39.559403+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:30:39.559403+0000","last_clean_scrub_stamp":"2024-03-11T22:30:39.559403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:59.861593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1023,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672410+0000","last_change":"2024-03-11T22:28:37.585104+0000","last_active":"2024-03-11T22:30:55.672410+0000","last_peered":"2024-03-11T22:30:55.672410+0000","last_clean":"2024-03-11T22:30:55.672410+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:30:55.672410+0000","last_undegraded":"2024-03-11T22:30:55.672410+0000","last_fullsized":"2024-03-11T22:30:55.672410+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:28:37.584991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:28:37.584991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"479'47","reported_seq":1077,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690929+0000","last_change":"2024-03-11T22:26:47.889297+0000","last_active":"2024-03-11T22:30:55.690929+0000","last_peered":"2024-03-11T22:30:55.690929+0000","last_clean":"2024-03-11T22:30:55.690929+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:30:55.690929+0000","last_undegraded":"2024-03-11T22:30:55.690929+0000","last_fullsized":"2024-03-11T22:30:55.690929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'37","last_scrub_stamp":"2024-03-11T22:26:47.889156+0000","last_deep_scrub":"429'37","last_deep_scrub_stamp":"2024-03-11T22:26:47.889156+0000","last_clean_scrub_stamp":"2024-03-11T22:26:47.889156+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706131+0000","last_change":"2024-03-11T22:30:47.657180+0000","last_active":"2024-03-11T22:30:54.706131+0000","last_peered":"2024-03-11T22:30:54.706131+0000","last_clean":"2024-03-11T22:30:54.706131+0000","last_became_active":"2024-03-11T22:29:04.722813+0000","last_became_peered":"2024-03-11T22:29:04.722813+0000","last_unstale":"2024-03-11T22:30:54.706131+0000","last_undegraded":"2024-03-11T22:30:54.706131+0000","last_fullsized":"2024-03-11T22:30:54.706131+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:47.657052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:04.706082+0000","last_clean_scrub_stamp":"2024-03-11T22:30:47.657052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:03.794039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704665+0000","last_change":"2024-03-11T22:30:52.727011+0000","last_active":"2024-03-11T22:30:54.704665+0000","last_peered":"2024-03-11T22:30:54.704665+0000","last_clean":"2024-03-11T22:30:54.704665+0000","last_became_active":"2024-03-11T22:29:04.729455+0000","last_became_peered":"2024-03-11T22:29:04.729455+0000","last_unstale":"2024-03-11T22:30:54.704665+0000","last_undegraded":"2024-03-11T22:30:54.704665+0000","last_fullsized":"2024-03-11T22:30:54.704665+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:52.726908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:52.726908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:07.437314+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":1263,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673111+0000","last_change":"2024-03-11T22:30:52.611262+0000","last_active":"2024-03-11T22:30:55.673111+0000","last_peered":"2024-03-11T22:30:55.673111+0000","last_clean":"2024-03-11T22:30:55.673111+0000","last_became_active":"2024-03-11T22:26:45.688609+0000","last_became_peered":"2024-03-11T22:26:45.688609+0000","last_unstale":"2024-03-11T22:30:55.673111+0000","last_undegraded":"2024-03-11T22:30:55.673111+0000","last_fullsized":"2024-03-11T22:30:55.673111+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:30:52.611147+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:30:52.611147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:05.456604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1020,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401905+0000","last_change":"2024-03-11T22:30:41.301144+0000","last_active":"2024-03-11T22:30:55.401905+0000","last_peered":"2024-03-11T22:30:55.401905+0000","last_clean":"2024-03-11T22:30:55.401905+0000","last_became_active":"2024-03-11T22:29:04.718259+0000","last_became_peered":"2024-03-11T22:29:04.718259+0000","last_unstale":"2024-03-11T22:30:55.401905+0000","last_undegraded":"2024-03-11T22:30:55.401905+0000","last_fullsized":"2024-03-11T22:30:55.401905+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:41.301047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:41.301047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:43.242569+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"479'36","reported_seq":1105,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690327+0000","last_change":"2024-03-11T22:29:04.818952+0000","last_active":"2024-03-11T22:30:55.690327+0000","last_peered":"2024-03-11T22:30:55.690327+0000","last_clean":"2024-03-11T22:30:55.690327+0000","last_became_active":"2024-03-11T22:29:04.720793+0000","last_became_peered":"2024-03-11T22:29:04.720793+0000","last_unstale":"2024-03-11T22:30:55.690327+0000","last_undegraded":"2024-03-11T22:30:55.690327+0000","last_fullsized":"2024-03-11T22:30:55.690327+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'24","last_scrub_stamp":"2024-03-11T22:26:35.660739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:35.660739+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":243,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:31:00.011384+0000","last_change":"2024-03-11T22:30:54.684229+0000","last_active":"2024-03-11T22:31:00.011384+0000","last_peered":"2024-03-11T22:31:00.011384+0000","last_clean":"2024-03-11T22:31:00.011384+0000","last_became_active":"2024-03-11T22:29:04.702609+0000","last_became_peered":"2024-03-11T22:29:04.702609+0000","last_unstale":"2024-03-11T22:31:00.011384+0000","last_undegraded":"2024-03-11T22:31:00.011384+0000","last_fullsized":"2024-03-11T22:31:00.011384+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:30:54.684108+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:30:54.684108+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:07.958873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":307,"num_read_kb":302,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690826+0000","last_change":"2024-03-11T22:30:08.418564+0000","last_active":"2024-03-11T22:30:55.690826+0000","last_peered":"2024-03-11T22:30:55.690826+0000","last_clean":"2024-03-11T22:30:55.690826+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:30:55.690826+0000","last_undegraded":"2024-03-11T22:30:55.690826+0000","last_fullsized":"2024-03-11T22:30:55.690826+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:08.418488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:08.418488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:35.702001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705994+0000","last_change":"2024-03-11T22:30:51.665235+0000","last_active":"2024-03-11T22:30:54.705994+0000","last_peered":"2024-03-11T22:30:54.705994+0000","last_clean":"2024-03-11T22:30:54.705994+0000","last_became_active":"2024-03-11T22:29:04.696141+0000","last_became_peered":"2024-03-11T22:29:04.696141+0000","last_unstale":"2024-03-11T22:30:54.705994+0000","last_undegraded":"2024-03-11T22:30:54.705994+0000","last_fullsized":"2024-03-11T22:30:54.705994+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:51.665137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:22:22.750560+0000","last_clean_scrub_stamp":"2024-03-11T22:30:51.665137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:54.684247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":571,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706033+0000","last_change":"2024-03-11T22:29:49.468313+0000","last_active":"2024-03-11T22:30:54.706033+0000","last_peered":"2024-03-11T22:30:54.706033+0000","last_clean":"2024-03-11T22:30:54.706033+0000","last_became_active":"2024-03-11T22:29:37.643184+0000","last_became_peered":"2024-03-11T22:29:37.643184+0000","last_unstale":"2024-03-11T22:30:54.706033+0000","last_undegraded":"2024-03-11T22:30:54.706033+0000","last_fullsized":"2024-03-11T22:30:54.706033+0000","mapping_epoch":502,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:29:49.467957+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:28:24.623016+0000","last_clean_scrub_stamp":"2024-03-11T22:29:49.467957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.316172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":991,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:59.233605+0000","last_change":"2024-03-11T22:30:59.233626+0000","last_active":"2024-03-11T22:30:59.233605+0000","last_peered":"2024-03-11T22:30:59.233605+0000","last_clean":"2024-03-11T22:30:59.233605+0000","last_became_active":"2024-03-11T22:20:29.420497+0000","last_became_peered":"2024-03-11T22:20:29.420497+0000","last_unstale":"2024-03-11T22:30:59.233605+0000","last_undegraded":"2024-03-11T22:30:59.233605+0000","last_fullsized":"2024-03-11T22:30:59.233605+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:59.233546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:30:59.233546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:06.529330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"514'48","reported_seq":1364,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:58.624410+0000","last_change":"2024-03-11T22:30:05.380085+0000","last_active":"2024-03-11T22:30:58.624410+0000","last_peered":"2024-03-11T22:30:58.624410+0000","last_clean":"2024-03-11T22:30:58.624410+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:30:58.624410+0000","last_undegraded":"2024-03-11T22:30:58.624410+0000","last_fullsized":"2024-03-11T22:30:58.624410+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"514'48","last_scrub_stamp":"2024-03-11T22:30:05.379958+0000","last_deep_scrub":"514'48","last_deep_scrub_stamp":"2024-03-11T22:30:05.379958+0000","last_clean_scrub_stamp":"2024-03-11T22:30:05.379958+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:12.891919+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1008,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391301+0000","last_change":"2024-03-11T22:30:40.798528+0000","last_active":"2024-03-11T22:30:55.391301+0000","last_peered":"2024-03-11T22:30:55.391301+0000","last_clean":"2024-03-11T22:30:55.391301+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:30:55.391301+0000","last_undegraded":"2024-03-11T22:30:55.391301+0000","last_fullsized":"2024-03-11T22:30:55.391301+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:40.798443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.798443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:54.912063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":1363,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.689975+0000","last_change":"2024-03-11T22:30:06.479012+0000","last_active":"2024-03-11T22:30:55.689975+0000","last_peered":"2024-03-11T22:30:55.689975+0000","last_clean":"2024-03-11T22:30:55.689975+0000","last_became_active":"2024-03-11T22:26:37.632412+0000","last_became_peered":"2024-03-11T22:26:37.632412+0000","last_unstale":"2024-03-11T22:30:55.689975+0000","last_undegraded":"2024-03-11T22:30:55.689975+0000","last_fullsized":"2024-03-11T22:30:55.689975+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":19,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:30:06.478921+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:30:06.478921+0000","last_clean_scrub_stamp":"2024-03-11T22:30:06.478921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:10.660047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1004,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.400948+0000","last_change":"2024-03-11T22:30:39.454150+0000","last_active":"2024-03-11T22:30:55.400948+0000","last_peered":"2024-03-11T22:30:55.400948+0000","last_clean":"2024-03-11T22:30:55.400948+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:30:55.400948+0000","last_undegraded":"2024-03-11T22:30:55.400948+0000","last_fullsized":"2024-03-11T22:30:55.400948+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:39.454015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:39.454015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:49.055198+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1033,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704591+0000","last_change":"2024-03-11T22:30:08.659249+0000","last_active":"2024-03-11T22:30:54.704591+0000","last_peered":"2024-03-11T22:30:54.704591+0000","last_clean":"2024-03-11T22:30:54.704591+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:30:54.704591+0000","last_undegraded":"2024-03-11T22:30:54.704591+0000","last_fullsized":"2024-03-11T22:30:54.704591+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:08.659154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:08.659154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:16.522418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"479'21","reported_seq":134,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706596+0000","last_change":"2024-03-11T22:30:05.683743+0000","last_active":"2024-03-11T22:30:54.706596+0000","last_peered":"2024-03-11T22:30:54.706596+0000","last_clean":"2024-03-11T22:30:54.706596+0000","last_became_active":"2024-03-11T22:29:04.705980+0000","last_became_peered":"2024-03-11T22:29:04.705980+0000","last_unstale":"2024-03-11T22:30:54.706596+0000","last_undegraded":"2024-03-11T22:30:54.706596+0000","last_fullsized":"2024-03-11T22:30:54.706596+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'21","last_scrub_stamp":"2024-03-11T22:30:05.683641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:05.683641+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:09.121346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1058,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706077+0000","last_change":"2024-03-11T22:30:15.663376+0000","last_active":"2024-03-11T22:30:54.706077+0000","last_peered":"2024-03-11T22:30:54.706077+0000","last_clean":"2024-03-11T22:30:54.706077+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:30:54.706077+0000","last_undegraded":"2024-03-11T22:30:54.706077+0000","last_fullsized":"2024-03-11T22:30:54.706077+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:15.663298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:30:15.663298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:39.777684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705251+0000","last_change":"2024-03-11T22:29:54.581275+0000","last_active":"2024-03-11T22:30:54.705251+0000","last_peered":"2024-03-11T22:30:54.705251+0000","last_clean":"2024-03-11T22:30:54.705251+0000","last_became_active":"2024-03-11T22:29:04.713225+0000","last_became_peered":"2024-03-11T22:29:04.713225+0000","last_unstale":"2024-03-11T22:30:54.705251+0000","last_undegraded":"2024-03-11T22:30:54.705251+0000","last_fullsized":"2024-03-11T22:30:54.705251+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:54.581156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:29:54.581156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.689997+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":961,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672907+0000","last_change":"2024-03-11T22:30:06.412273+0000","last_active":"2024-03-11T22:30:55.672907+0000","last_peered":"2024-03-11T22:30:55.672907+0000","last_clean":"2024-03-11T22:30:55.672907+0000","last_became_active":"2024-03-11T22:10:35.413011+0000","last_became_peered":"2024-03-11T22:10:35.413011+0000","last_unstale":"2024-03-11T22:30:55.672907+0000","last_undegraded":"2024-03-11T22:30:55.672907+0000","last_fullsized":"2024-03-11T22:30:55.672907+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:06.412160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:30:06.412160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:19.024605+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":1370,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690025+0000","last_change":"2024-03-11T22:26:32.769183+0000","last_active":"2024-03-11T22:30:55.690025+0000","last_peered":"2024-03-11T22:30:55.690025+0000","last_clean":"2024-03-11T22:30:55.690025+0000","last_became_active":"2024-03-11T22:26:29.590669+0000","last_became_peered":"2024-03-11T22:26:29.590669+0000","last_unstale":"2024-03-11T22:30:55.690025+0000","last_undegraded":"2024-03-11T22:30:55.690025+0000","last_fullsized":"2024-03-11T22:30:55.690025+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":411,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:32.769096+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:32.769096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1019,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690385+0000","last_change":"2024-03-11T22:30:05.507577+0000","last_active":"2024-03-11T22:30:55.690385+0000","last_peered":"2024-03-11T22:30:55.690385+0000","last_clean":"2024-03-11T22:30:55.690385+0000","last_became_active":"2024-03-11T22:29:04.707688+0000","last_became_peered":"2024-03-11T22:29:04.707688+0000","last_unstale":"2024-03-11T22:30:55.690385+0000","last_undegraded":"2024-03-11T22:30:55.690385+0000","last_fullsized":"2024-03-11T22:30:55.690385+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:05.507462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:05.507462+0000","last_clean_scrub_stamp":"2024-03-11T22:30:05.507462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:31.012916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"520'55","reported_seq":81,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:58.624569+0000","last_change":"2024-03-11T22:30:16.943159+0000","last_active":"2024-03-11T22:30:58.624569+0000","last_peered":"2024-03-11T22:30:58.624569+0000","last_clean":"2024-03-11T22:30:58.624569+0000","last_became_active":"2024-03-11T22:30:16.930211+0000","last_became_peered":"2024-03-11T22:30:16.930211+0000","last_unstale":"2024-03-11T22:30:58.624569+0000","last_undegraded":"2024-03-11T22:30:58.624569+0000","last_fullsized":"2024-03-11T22:30:58.624569+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'54","last_scrub_stamp":"2024-03-11T22:29:52.527782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:29:52.527782+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:20.718332+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":774,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690420+0000","last_change":"2024-03-11T22:26:44.809895+0000","last_active":"2024-03-11T22:30:55.690420+0000","last_peered":"2024-03-11T22:30:55.690420+0000","last_clean":"2024-03-11T22:30:55.690420+0000","last_became_active":"2024-03-11T22:14:52.241057+0000","last_became_peered":"2024-03-11T22:14:52.241057+0000","last_unstale":"2024-03-11T22:30:55.690420+0000","last_undegraded":"2024-03-11T22:30:55.690420+0000","last_fullsized":"2024-03-11T22:30:55.690420+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:44.809803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:44.809803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":1048,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401277+0000","last_change":"2024-03-11T22:29:04.812619+0000","last_active":"2024-03-11T22:30:55.401277+0000","last_peered":"2024-03-11T22:30:55.401277+0000","last_clean":"2024-03-11T22:30:55.401277+0000","last_became_active":"2024-03-11T22:29:04.718809+0000","last_became_peered":"2024-03-11T22:29:04.718809+0000","last_unstale":"2024-03-11T22:30:55.401277+0000","last_undegraded":"2024-03-11T22:30:55.401277+0000","last_fullsized":"2024-03-11T22:30:55.401277+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:26:31.988426+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:26:31.988426+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1000,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392410+0000","last_change":"2024-03-11T22:25:58.161115+0000","last_active":"2024-03-11T22:30:55.392410+0000","last_peered":"2024-03-11T22:30:55.392410+0000","last_clean":"2024-03-11T22:30:55.392410+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:30:55.392410+0000","last_undegraded":"2024-03-11T22:30:55.392410+0000","last_fullsized":"2024-03-11T22:30:55.392410+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:25:58.160972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:25:58.160972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":1345,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401081+0000","last_change":"2024-03-11T22:26:34.036467+0000","last_active":"2024-03-11T22:30:55.401081+0000","last_peered":"2024-03-11T22:30:55.401081+0000","last_clean":"2024-03-11T22:30:55.401081+0000","last_became_active":"2024-03-11T22:26:33.607145+0000","last_became_peered":"2024-03-11T22:26:33.607145+0000","last_unstale":"2024-03-11T22:30:55.401081+0000","last_undegraded":"2024-03-11T22:30:55.401081+0000","last_fullsized":"2024-03-11T22:30:55.401081+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:26:34.036387+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:26:34.036387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"479'21","reported_seq":1084,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.691092+0000","last_change":"2024-03-11T22:29:04.732685+0000","last_active":"2024-03-11T22:30:55.691092+0000","last_peered":"2024-03-11T22:30:55.691092+0000","last_clean":"2024-03-11T22:30:55.691092+0000","last_became_active":"2024-03-11T22:29:04.707867+0000","last_became_peered":"2024-03-11T22:29:04.707867+0000","last_unstale":"2024-03-11T22:30:55.691092+0000","last_undegraded":"2024-03-11T22:30:55.691092+0000","last_fullsized":"2024-03-11T22:30:55.691092+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'15","last_scrub_stamp":"2024-03-11T22:26:12.661479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:12.661479+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1016,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401623+0000","last_change":"2024-03-11T22:30:50.377246+0000","last_active":"2024-03-11T22:30:55.401623+0000","last_peered":"2024-03-11T22:30:55.401623+0000","last_clean":"2024-03-11T22:30:55.401623+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:30:55.401623+0000","last_undegraded":"2024-03-11T22:30:55.401623+0000","last_fullsized":"2024-03-11T22:30:55.401623+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:50.377151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:50.377151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:15.151224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1041,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.393858+0000","last_change":"2024-03-11T22:30:55.393883+0000","last_active":"2024-03-11T22:30:55.393858+0000","last_peered":"2024-03-11T22:30:55.393858+0000","last_clean":"2024-03-11T22:30:55.393858+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:30:55.393858+0000","last_undegraded":"2024-03-11T22:30:55.393858+0000","last_fullsized":"2024-03-11T22:30:55.393858+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:55.393786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:30:55.393786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:10.301036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690637+0000","last_change":"2024-03-11T22:30:20.400482+0000","last_active":"2024-03-11T22:30:55.690637+0000","last_peered":"2024-03-11T22:30:55.690637+0000","last_clean":"2024-03-11T22:30:55.690637+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:30:55.690637+0000","last_undegraded":"2024-03-11T22:30:55.690637+0000","last_fullsized":"2024-03-11T22:30:55.690637+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:20.400349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:20.400349+0000","last_clean_scrub_stamp":"2024-03-11T22:30:20.400349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:39.045173+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"480'51","reported_seq":1210,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672311+0000","last_change":"2024-03-11T22:30:04.401901+0000","last_active":"2024-03-11T22:30:55.672311+0000","last_peered":"2024-03-11T22:30:55.672311+0000","last_clean":"2024-03-11T22:30:55.672311+0000","last_became_active":"2024-03-11T22:29:04.718797+0000","last_became_peered":"2024-03-11T22:29:04.718797+0000","last_unstale":"2024-03-11T22:30:55.672311+0000","last_undegraded":"2024-03-11T22:30:55.672311+0000","last_fullsized":"2024-03-11T22:30:55.672311+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'51","last_scrub_stamp":"2024-03-11T22:30:04.401771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:04.401771+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:32.893247+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":959,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401790+0000","last_change":"2024-03-11T22:26:04.099295+0000","last_active":"2024-03-11T22:30:55.401790+0000","last_peered":"2024-03-11T22:30:55.401790+0000","last_clean":"2024-03-11T22:30:55.401790+0000","last_became_active":"2024-03-11T22:20:29.413664+0000","last_became_peered":"2024-03-11T22:20:29.413664+0000","last_unstale":"2024-03-11T22:30:55.401790+0000","last_undegraded":"2024-03-11T22:30:55.401790+0000","last_fullsized":"2024-03-11T22:30:55.401790+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:04.099165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:04.099165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1033,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705483+0000","last_change":"2024-03-11T22:30:12.675279+0000","last_active":"2024-03-11T22:30:54.705483+0000","last_peered":"2024-03-11T22:30:54.705483+0000","last_clean":"2024-03-11T22:30:54.705483+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:30:54.705483+0000","last_undegraded":"2024-03-11T22:30:54.705483+0000","last_fullsized":"2024-03-11T22:30:54.705483+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:12.675202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:30:12.675202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:40.937625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":991,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704426+0000","last_change":"2024-03-11T22:30:47.568602+0000","last_active":"2024-03-11T22:30:54.704426+0000","last_peered":"2024-03-11T22:30:54.704426+0000","last_clean":"2024-03-11T22:30:54.704426+0000","last_became_active":"2024-03-11T22:20:29.415565+0000","last_became_peered":"2024-03-11T22:20:29.415565+0000","last_unstale":"2024-03-11T22:30:54.704426+0000","last_undegraded":"2024-03-11T22:30:54.704426+0000","last_fullsized":"2024-03-11T22:30:54.704426+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:47.568500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:47.568500+0000","last_clean_scrub_stamp":"2024-03-11T22:30:47.568500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:49.067116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1045,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704211+0000","last_change":"2024-03-11T22:29:56.665159+0000","last_active":"2024-03-11T22:30:54.704211+0000","last_peered":"2024-03-11T22:30:54.704211+0000","last_clean":"2024-03-11T22:30:54.704211+0000","last_became_active":"2024-03-11T22:29:04.718513+0000","last_became_peered":"2024-03-11T22:29:04.718513+0000","last_unstale":"2024-03-11T22:30:54.704211+0000","last_undegraded":"2024-03-11T22:30:54.704211+0000","last_fullsized":"2024-03-11T22:30:54.704211+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:56.665023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:38.757164+0000","last_clean_scrub_stamp":"2024-03-11T22:29:56.665023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:13.354927+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"478'24","reported_seq":134,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704572+0000","last_change":"2024-03-11T22:30:48.649205+0000","last_active":"2024-03-11T22:30:54.704572+0000","last_peered":"2024-03-11T22:30:54.704572+0000","last_clean":"2024-03-11T22:30:54.704572+0000","last_became_active":"2024-03-11T22:29:04.694343+0000","last_became_peered":"2024-03-11T22:29:04.694343+0000","last_unstale":"2024-03-11T22:30:54.704572+0000","last_undegraded":"2024-03-11T22:30:54.704572+0000","last_fullsized":"2024-03-11T22:30:54.704572+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'24","last_scrub_stamp":"2024-03-11T22:30:48.649070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:48.649070+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:03.230178+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706389+0000","last_change":"2024-03-11T22:30:10.740917+0000","last_active":"2024-03-11T22:30:54.706389+0000","last_peered":"2024-03-11T22:30:54.706389+0000","last_clean":"2024-03-11T22:30:54.706389+0000","last_became_active":"2024-03-11T22:29:04.727524+0000","last_became_peered":"2024-03-11T22:29:04.727524+0000","last_unstale":"2024-03-11T22:30:54.706389+0000","last_undegraded":"2024-03-11T22:30:54.706389+0000","last_fullsized":"2024-03-11T22:30:54.706389+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:10.740798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:10.740798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:36.857785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":1029,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391787+0000","last_change":"2024-03-11T22:29:04.742292+0000","last_active":"2024-03-11T22:30:55.391787+0000","last_peered":"2024-03-11T22:30:55.391787+0000","last_clean":"2024-03-11T22:30:55.391787+0000","last_became_active":"2024-03-11T22:29:04.698531+0000","last_became_peered":"2024-03-11T22:29:04.698531+0000","last_unstale":"2024-03-11T22:30:55.391787+0000","last_undegraded":"2024-03-11T22:30:55.391787+0000","last_fullsized":"2024-03-11T22:30:55.391787+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:26:35.021902+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:26:35.021902+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"310'11","reported_seq":1254,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:58.624402+0000","last_change":"2024-03-11T22:29:54.360194+0000","last_active":"2024-03-11T22:30:58.624402+0000","last_peered":"2024-03-11T22:30:58.624402+0000","last_clean":"2024-03-11T22:30:58.624402+0000","last_became_active":"2024-03-11T22:20:29.418355+0000","last_became_peered":"2024-03-11T22:20:29.418355+0000","last_unstale":"2024-03-11T22:30:58.624402+0000","last_undegraded":"2024-03-11T22:30:58.624402+0000","last_fullsized":"2024-03-11T22:30:58.624402+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"310'11","last_scrub_stamp":"2024-03-11T22:29:54.360080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:29:54.360080+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:07.077073+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"479'46","reported_seq":1124,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.400978+0000","last_change":"2024-03-11T22:26:45.212688+0000","last_active":"2024-03-11T22:30:55.400978+0000","last_peered":"2024-03-11T22:30:55.400978+0000","last_clean":"2024-03-11T22:30:55.400978+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:30:55.400978+0000","last_undegraded":"2024-03-11T22:30:55.400978+0000","last_fullsized":"2024-03-11T22:30:55.400978+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'34","last_scrub_stamp":"2024-03-11T22:26:45.212591+0000","last_deep_scrub":"91'22","last_deep_scrub_stamp":"2024-03-11T22:09:36.274225+0000","last_clean_scrub_stamp":"2024-03-11T22:26:45.212591+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"432'4","reported_seq":809,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690550+0000","last_change":"2024-03-11T22:26:50.847434+0000","last_active":"2024-03-11T22:30:55.690550+0000","last_peered":"2024-03-11T22:30:55.690550+0000","last_clean":"2024-03-11T22:30:55.690550+0000","last_became_active":"2024-03-11T22:20:29.415226+0000","last_became_peered":"2024-03-11T22:20:29.415226+0000","last_unstale":"2024-03-11T22:30:55.690550+0000","last_undegraded":"2024-03-11T22:30:55.690550+0000","last_fullsized":"2024-03-11T22:30:55.690550+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'4","last_scrub_stamp":"2024-03-11T22:26:50.847349+0000","last_deep_scrub":"432'4","last_deep_scrub_stamp":"2024-03-11T22:26:50.847349+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.847349+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":996,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401409+0000","last_change":"2024-03-11T22:26:49.194654+0000","last_active":"2024-03-11T22:30:55.401409+0000","last_peered":"2024-03-11T22:30:55.401409+0000","last_clean":"2024-03-11T22:30:55.401409+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:30:55.401409+0000","last_undegraded":"2024-03-11T22:30:55.401409+0000","last_fullsized":"2024-03-11T22:30:55.401409+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:49.194569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:49.194569+0000","last_clean_scrub_stamp":"2024-03-11T22:26:49.194569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"479'18","reported_seq":1100,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:31:01.916356+0000","last_change":"2024-03-11T22:31:01.916414+0000","last_active":"2024-03-11T22:31:01.916356+0000","last_peered":"2024-03-11T22:31:01.916356+0000","last_clean":"2024-03-11T22:31:01.916356+0000","last_became_active":"2024-03-11T22:30:16.929794+0000","last_became_peered":"2024-03-11T22:30:16.929794+0000","last_unstale":"2024-03-11T22:31:01.916356+0000","last_undegraded":"2024-03-11T22:31:01.916356+0000","last_fullsized":"2024-03-11T22:31:01.916356+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'18","last_scrub_stamp":"2024-03-11T22:31:01.916274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:31:01.916274+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:11.535025+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1022,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.671990+0000","last_change":"2024-03-11T22:28:33.573448+0000","last_active":"2024-03-11T22:30:55.671990+0000","last_peered":"2024-03-11T22:30:55.671990+0000","last_clean":"2024-03-11T22:30:55.671990+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:30:55.671990+0000","last_undegraded":"2024-03-11T22:30:55.671990+0000","last_fullsized":"2024-03-11T22:30:55.671990+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:28:33.573346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:28:33.573346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1019,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401611+0000","last_change":"2024-03-11T22:30:03.168689+0000","last_active":"2024-03-11T22:30:55.401611+0000","last_peered":"2024-03-11T22:30:55.401611+0000","last_clean":"2024-03-11T22:30:55.401611+0000","last_became_active":"2024-03-11T22:29:04.731305+0000","last_became_peered":"2024-03-11T22:29:04.731305+0000","last_unstale":"2024-03-11T22:30:55.401611+0000","last_undegraded":"2024-03-11T22:30:55.401611+0000","last_fullsized":"2024-03-11T22:30:55.401611+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:03.168564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:30:03.168564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:32.143726+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":952,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672984+0000","last_change":"2024-03-11T22:29:52.451412+0000","last_active":"2024-03-11T22:30:55.672984+0000","last_peered":"2024-03-11T22:30:55.672984+0000","last_clean":"2024-03-11T22:30:55.672984+0000","last_became_active":"2024-03-11T22:10:35.415237+0000","last_became_peered":"2024-03-11T22:10:35.415237+0000","last_unstale":"2024-03-11T22:30:55.672984+0000","last_undegraded":"2024-03-11T22:30:55.672984+0000","last_fullsized":"2024-03-11T22:30:55.672984+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:52.451186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:29:52.451186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:30:59.862313+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"478'25","reported_seq":1080,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690854+0000","last_change":"2024-03-11T22:30:01.530506+0000","last_active":"2024-03-11T22:30:55.690854+0000","last_peered":"2024-03-11T22:30:55.690854+0000","last_clean":"2024-03-11T22:30:55.690854+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:30:55.690854+0000","last_undegraded":"2024-03-11T22:30:55.690854+0000","last_fullsized":"2024-03-11T22:30:55.690854+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:30:01.530350+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_clean_scrub_stamp":"2024-03-11T22:30:01.530350+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:27.843442+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":792,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672072+0000","last_change":"2024-03-11T22:30:46.646058+0000","last_active":"2024-03-11T22:30:55.672072+0000","last_peered":"2024-03-11T22:30:55.672072+0000","last_clean":"2024-03-11T22:30:55.672072+0000","last_became_active":"2024-03-11T22:14:52.247275+0000","last_became_peered":"2024-03-11T22:14:52.247275+0000","last_unstale":"2024-03-11T22:30:55.672072+0000","last_undegraded":"2024-03-11T22:30:55.672072+0000","last_fullsized":"2024-03-11T22:30:55.672072+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:46.645943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:46.645943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:15.950611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1023,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392507+0000","last_change":"2024-03-11T22:30:16.883019+0000","last_active":"2024-03-11T22:30:55.392507+0000","last_peered":"2024-03-11T22:30:55.392507+0000","last_clean":"2024-03-11T22:30:55.392507+0000","last_became_active":"2024-03-11T22:29:04.700408+0000","last_became_peered":"2024-03-11T22:29:04.700408+0000","last_unstale":"2024-03-11T22:30:55.392507+0000","last_undegraded":"2024-03-11T22:30:55.392507+0000","last_fullsized":"2024-03-11T22:30:55.392507+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:16.882883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:16.882883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:40.426358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1036,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705535+0000","last_change":"2024-03-11T22:30:49.480356+0000","last_active":"2024-03-11T22:30:54.705535+0000","last_peered":"2024-03-11T22:30:54.705535+0000","last_clean":"2024-03-11T22:30:54.705535+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:30:54.705535+0000","last_undegraded":"2024-03-11T22:30:54.705535+0000","last_fullsized":"2024-03-11T22:30:54.705535+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:49.480247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:30:49.480247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:59.538719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"479'25","reported_seq":1049,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401791+0000","last_change":"2024-03-11T22:26:12.149477+0000","last_active":"2024-03-11T22:30:55.401791+0000","last_peered":"2024-03-11T22:30:55.401791+0000","last_clean":"2024-03-11T22:30:55.401791+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:30:55.401791+0000","last_undegraded":"2024-03-11T22:30:55.401791+0000","last_fullsized":"2024-03-11T22:30:55.401791+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'19","last_scrub_stamp":"2024-03-11T22:26:12.149360+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:26:12.149360+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1026,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.689912+0000","last_change":"2024-03-11T22:29:53.461268+0000","last_active":"2024-03-11T22:30:55.689912+0000","last_peered":"2024-03-11T22:30:55.689912+0000","last_clean":"2024-03-11T22:30:55.689912+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:30:55.689912+0000","last_undegraded":"2024-03-11T22:30:55.689912+0000","last_fullsized":"2024-03-11T22:30:55.689912+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:53.461031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:50.030272+0000","last_clean_scrub_stamp":"2024-03-11T22:29:53.461031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:14.083004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"527'12","reported_seq":1316,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391941+0000","last_change":"2024-03-11T22:30:34.882548+0000","last_active":"2024-03-11T22:30:55.391941+0000","last_peered":"2024-03-11T22:30:55.391941+0000","last_clean":"2024-03-11T22:30:55.391941+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:30:55.391941+0000","last_undegraded":"2024-03-11T22:30:55.391941+0000","last_fullsized":"2024-03-11T22:30:55.391941+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'12","last_scrub_stamp":"2024-03-11T22:30:34.882462+0000","last_deep_scrub":"527'12","last_deep_scrub_stamp":"2024-03-11T22:30:34.882462+0000","last_clean_scrub_stamp":"2024-03-11T22:30:34.882462+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:45.890672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":798,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704582+0000","last_change":"2024-03-11T22:30:18.817369+0000","last_active":"2024-03-11T22:30:54.704582+0000","last_peered":"2024-03-11T22:30:54.704582+0000","last_clean":"2024-03-11T22:30:54.704582+0000","last_became_active":"2024-03-11T22:14:52.243136+0000","last_became_peered":"2024-03-11T22:14:52.243136+0000","last_unstale":"2024-03-11T22:30:54.704582+0000","last_undegraded":"2024-03-11T22:30:54.704582+0000","last_fullsized":"2024-03-11T22:30:54.704582+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:18.817224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:48.490757+0000","last_clean_scrub_stamp":"2024-03-11T22:30:18.817224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:31.548740+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"479'30","reported_seq":1102,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:57.859164+0000","last_change":"2024-03-11T22:30:57.859196+0000","last_active":"2024-03-11T22:30:57.859164+0000","last_peered":"2024-03-11T22:30:57.859164+0000","last_clean":"2024-03-11T22:30:57.859164+0000","last_became_active":"2024-03-11T22:29:04.720501+0000","last_became_peered":"2024-03-11T22:29:04.720501+0000","last_unstale":"2024-03-11T22:30:57.859164+0000","last_undegraded":"2024-03-11T22:30:57.859164+0000","last_fullsized":"2024-03-11T22:30:57.859164+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-03-11T22:30:57.859090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:57.859090+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:14.351210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":822,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705361+0000","last_change":"2024-03-11T22:30:53.455852+0000","last_active":"2024-03-11T22:30:54.705361+0000","last_peered":"2024-03-11T22:30:54.705361+0000","last_clean":"2024-03-11T22:30:54.705361+0000","last_became_active":"2024-03-11T22:20:29.415019+0000","last_became_peered":"2024-03-11T22:20:29.415019+0000","last_unstale":"2024-03-11T22:30:54.705361+0000","last_undegraded":"2024-03-11T22:30:54.705361+0000","last_fullsized":"2024-03-11T22:30:54.705361+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:53.455748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:51.034868+0000","last_clean_scrub_stamp":"2024-03-11T22:30:53.455748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:55.363500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"488'10","reported_seq":1307,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:58.624409+0000","last_change":"2024-03-11T22:30:08.228206+0000","last_active":"2024-03-11T22:30:58.624409+0000","last_peered":"2024-03-11T22:30:58.624409+0000","last_clean":"2024-03-11T22:30:58.624409+0000","last_became_active":"2024-03-11T22:29:04.787739+0000","last_became_peered":"2024-03-11T22:29:04.787739+0000","last_unstale":"2024-03-11T22:30:58.624409+0000","last_undegraded":"2024-03-11T22:30:58.624409+0000","last_fullsized":"2024-03-11T22:30:58.624409+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'10","last_scrub_stamp":"2024-03-11T22:30:08.228075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:08.228075+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:10.466725+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1006,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690302+0000","last_change":"2024-03-11T22:30:21.303542+0000","last_active":"2024-03-11T22:30:55.690302+0000","last_peered":"2024-03-11T22:30:55.690302+0000","last_clean":"2024-03-11T22:30:55.690302+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:30:55.690302+0000","last_undegraded":"2024-03-11T22:30:55.690302+0000","last_fullsized":"2024-03-11T22:30:55.690302+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:21.303440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:21.303440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:36.442252+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"479'27","reported_seq":886,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672384+0000","last_change":"2024-03-11T22:30:40.594125+0000","last_active":"2024-03-11T22:30:55.672384+0000","last_peered":"2024-03-11T22:30:55.672384+0000","last_clean":"2024-03-11T22:30:55.672384+0000","last_became_active":"2024-03-11T22:29:04.720174+0000","last_became_peered":"2024-03-11T22:29:04.720174+0000","last_unstale":"2024-03-11T22:30:55.672384+0000","last_undegraded":"2024-03-11T22:30:55.672384+0000","last_fullsized":"2024-03-11T22:30:55.672384+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-03-11T22:30:40.594016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.594016+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:55.638323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":817,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:58.615968+0000","last_change":"2024-03-11T22:30:58.616010+0000","last_active":"2024-03-11T22:30:58.615968+0000","last_peered":"2024-03-11T22:30:58.615968+0000","last_clean":"2024-03-11T22:30:58.615968+0000","last_became_active":"2024-03-11T22:20:29.420867+0000","last_became_peered":"2024-03-11T22:20:29.420867+0000","last_unstale":"2024-03-11T22:30:58.615968+0000","last_undegraded":"2024-03-11T22:30:58.615968+0000","last_fullsized":"2024-03-11T22:30:58.615968+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:30:58.615860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:58.615860+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:14.311608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":125,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705457+0000","last_change":"2024-03-11T22:30:40.510945+0000","last_active":"2024-03-11T22:30:54.705457+0000","last_peered":"2024-03-11T22:30:54.705457+0000","last_clean":"2024-03-11T22:30:54.705457+0000","last_became_active":"2024-03-11T22:29:04.723400+0000","last_became_peered":"2024-03-11T22:29:04.723400+0000","last_unstale":"2024-03-11T22:30:54.705457+0000","last_undegraded":"2024-03-11T22:30:54.705457+0000","last_fullsized":"2024-03-11T22:30:54.705457+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:40.510823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.510823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:09.057333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":987,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391290+0000","last_change":"2024-03-11T22:29:50.712811+0000","last_active":"2024-03-11T22:30:55.391290+0000","last_peered":"2024-03-11T22:30:55.391290+0000","last_clean":"2024-03-11T22:30:55.391290+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:30:55.391290+0000","last_undegraded":"2024-03-11T22:30:55.391290+0000","last_fullsized":"2024-03-11T22:30:55.391290+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:50.712416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:29:50.712416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:10.140704+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"478'25","reported_seq":132,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705098+0000","last_change":"2024-03-11T22:30:19.687640+0000","last_active":"2024-03-11T22:30:54.705098+0000","last_peered":"2024-03-11T22:30:54.705098+0000","last_clean":"2024-03-11T22:30:54.705098+0000","last_became_active":"2024-03-11T22:29:04.730485+0000","last_became_peered":"2024-03-11T22:29:04.730485+0000","last_unstale":"2024-03-11T22:30:54.705098+0000","last_undegraded":"2024-03-11T22:30:54.705098+0000","last_fullsized":"2024-03-11T22:30:54.705098+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:30:19.687537+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:30:19.687537+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:19.724923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":950,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392008+0000","last_change":"2024-03-11T22:30:12.898819+0000","last_active":"2024-03-11T22:30:55.392008+0000","last_peered":"2024-03-11T22:30:55.392008+0000","last_clean":"2024-03-11T22:30:55.392008+0000","last_became_active":"2024-03-11T22:20:29.422747+0000","last_became_peered":"2024-03-11T22:20:29.422747+0000","last_unstale":"2024-03-11T22:30:55.392008+0000","last_undegraded":"2024-03-11T22:30:55.392008+0000","last_fullsized":"2024-03-11T22:30:55.392008+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:12.898649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:12.898649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.975372+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1011,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391622+0000","last_change":"2024-03-11T22:30:15.836456+0000","last_active":"2024-03-11T22:30:55.391622+0000","last_peered":"2024-03-11T22:30:55.391622+0000","last_clean":"2024-03-11T22:30:55.391622+0000","last_became_active":"2024-03-11T22:29:04.700352+0000","last_became_peered":"2024-03-11T22:29:04.700352+0000","last_unstale":"2024-03-11T22:30:55.391622+0000","last_undegraded":"2024-03-11T22:30:55.391622+0000","last_fullsized":"2024-03-11T22:30:55.391622+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:15.836335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:30:15.836335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:21.846380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1014,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.689829+0000","last_change":"2024-03-11T22:30:48.240248+0000","last_active":"2024-03-11T22:30:55.689829+0000","last_peered":"2024-03-11T22:30:55.689829+0000","last_clean":"2024-03-11T22:30:55.689829+0000","last_became_active":"2024-03-11T22:29:04.707420+0000","last_became_peered":"2024-03-11T22:29:04.707420+0000","last_unstale":"2024-03-11T22:30:55.689829+0000","last_undegraded":"2024-03-11T22:30:55.689829+0000","last_fullsized":"2024-03-11T22:30:55.689829+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:48.240128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:09.057970+0000","last_clean_scrub_stamp":"2024-03-11T22:30:48.240128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:12.685122+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"480'28","reported_seq":1075,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391729+0000","last_change":"2024-03-11T22:29:51.655446+0000","last_active":"2024-03-11T22:30:55.391729+0000","last_peered":"2024-03-11T22:30:55.391729+0000","last_clean":"2024-03-11T22:30:55.391729+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:30:55.391729+0000","last_undegraded":"2024-03-11T22:30:55.391729+0000","last_fullsized":"2024-03-11T22:30:55.391729+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'28","last_scrub_stamp":"2024-03-11T22:29:51.655136+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:29:51.655136+0000","objects_scrubbed":5,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1039,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401613+0000","last_change":"2024-03-11T22:30:20.359726+0000","last_active":"2024-03-11T22:30:55.401613+0000","last_peered":"2024-03-11T22:30:55.401613+0000","last_clean":"2024-03-11T22:30:55.401613+0000","last_became_active":"2024-03-11T22:29:04.722573+0000","last_became_peered":"2024-03-11T22:29:04.722573+0000","last_unstale":"2024-03-11T22:30:55.401613+0000","last_undegraded":"2024-03-11T22:30:55.401613+0000","last_fullsized":"2024-03-11T22:30:55.401613+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:20.359623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:20.359623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:46.022838+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":791,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:31:01.713376+0000","last_change":"2024-03-11T22:31:01.713396+0000","last_active":"2024-03-11T22:31:01.713376+0000","last_peered":"2024-03-11T22:31:01.713376+0000","last_clean":"2024-03-11T22:31:01.713376+0000","last_became_active":"2024-03-11T22:14:52.247864+0000","last_became_peered":"2024-03-11T22:14:52.247864+0000","last_unstale":"2024-03-11T22:31:01.713376+0000","last_undegraded":"2024-03-11T22:31:01.713376+0000","last_fullsized":"2024-03-11T22:31:01.713376+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:01.713312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:42.175852+0000","last_clean_scrub_stamp":"2024-03-11T22:31:01.713312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:03.355991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":995,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391915+0000","last_change":"2024-03-11T22:30:53.091266+0000","last_active":"2024-03-11T22:30:55.391915+0000","last_peered":"2024-03-11T22:30:55.391915+0000","last_clean":"2024-03-11T22:30:55.391915+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:30:55.391915+0000","last_undegraded":"2024-03-11T22:30:55.391915+0000","last_fullsized":"2024-03-11T22:30:55.391915+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:53.091163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:53.091163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:16.620952+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"478'12","reported_seq":1057,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391233+0000","last_change":"2024-03-11T22:29:04.843107+0000","last_active":"2024-03-11T22:30:55.391233+0000","last_peered":"2024-03-11T22:30:55.391233+0000","last_clean":"2024-03-11T22:30:55.391233+0000","last_became_active":"2024-03-11T22:29:04.734207+0000","last_became_peered":"2024-03-11T22:29:04.734207+0000","last_unstale":"2024-03-11T22:30:55.391233+0000","last_undegraded":"2024-03-11T22:30:55.391233+0000","last_fullsized":"2024-03-11T22:30:55.391233+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'8","last_scrub_stamp":"2024-03-11T22:26:50.833149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:50.833149+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1006,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690611+0000","last_change":"2024-03-11T22:26:05.787798+0000","last_active":"2024-03-11T22:30:55.690611+0000","last_peered":"2024-03-11T22:30:55.690611+0000","last_clean":"2024-03-11T22:30:55.690611+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:30:55.690611+0000","last_undegraded":"2024-03-11T22:30:55.690611+0000","last_fullsized":"2024-03-11T22:30:55.690611+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:05.787543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:05.787543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":600,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672075+0000","last_change":"2024-03-11T22:30:03.392102+0000","last_active":"2024-03-11T22:30:55.672075+0000","last_peered":"2024-03-11T22:30:55.672075+0000","last_clean":"2024-03-11T22:30:55.672075+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:30:55.672075+0000","last_undegraded":"2024-03-11T22:30:55.672075+0000","last_fullsized":"2024-03-11T22:30:55.672075+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:03.391983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:03.391983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:28.249700+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":970,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705070+0000","last_change":"2024-03-11T22:30:01.668808+0000","last_active":"2024-03-11T22:30:54.705070+0000","last_peered":"2024-03-11T22:30:54.705070+0000","last_clean":"2024-03-11T22:30:54.705070+0000","last_became_active":"2024-03-11T22:10:35.410474+0000","last_became_peered":"2024-03-11T22:10:35.410474+0000","last_unstale":"2024-03-11T22:30:54.705070+0000","last_undegraded":"2024-03-11T22:30:54.705070+0000","last_fullsized":"2024-03-11T22:30:54.705070+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:01.668594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:01.668594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:07.076683+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"480'27","reported_seq":1106,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673172+0000","last_change":"2024-03-11T22:30:43.611613+0000","last_active":"2024-03-11T22:30:55.673172+0000","last_peered":"2024-03-11T22:30:55.673172+0000","last_clean":"2024-03-11T22:30:55.673172+0000","last_became_active":"2024-03-11T22:29:04.720148+0000","last_became_peered":"2024-03-11T22:29:04.720148+0000","last_unstale":"2024-03-11T22:30:55.673172+0000","last_undegraded":"2024-03-11T22:30:55.673172+0000","last_fullsized":"2024-03-11T22:30:55.673172+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'27","last_scrub_stamp":"2024-03-11T22:30:43.611513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:43.611513+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:00.303736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1019,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391232+0000","last_change":"2024-03-11T22:30:13.866705+0000","last_active":"2024-03-11T22:30:55.391232+0000","last_peered":"2024-03-11T22:30:55.391232+0000","last_clean":"2024-03-11T22:30:55.391232+0000","last_became_active":"2024-03-11T22:29:04.733742+0000","last_became_peered":"2024-03-11T22:29:04.733742+0000","last_unstale":"2024-03-11T22:30:55.391232+0000","last_undegraded":"2024-03-11T22:30:55.391232+0000","last_fullsized":"2024-03-11T22:30:55.391232+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:13.866615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:13.866615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:42.154858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1008,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690912+0000","last_change":"2024-03-11T22:26:43.777178+0000","last_active":"2024-03-11T22:30:55.690912+0000","last_peered":"2024-03-11T22:30:55.690912+0000","last_clean":"2024-03-11T22:30:55.690912+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:30:55.690912+0000","last_undegraded":"2024-03-11T22:30:55.690912+0000","last_fullsized":"2024-03-11T22:30:55.690912+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:43.777100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:26:43.777100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706579+0000","last_change":"2024-03-11T22:30:06.824568+0000","last_active":"2024-03-11T22:30:54.706579+0000","last_peered":"2024-03-11T22:30:54.706579+0000","last_clean":"2024-03-11T22:30:54.706579+0000","last_became_active":"2024-03-11T22:29:04.786119+0000","last_became_peered":"2024-03-11T22:29:04.786119+0000","last_unstale":"2024-03-11T22:30:54.706579+0000","last_undegraded":"2024-03-11T22:30:54.706579+0000","last_fullsized":"2024-03-11T22:30:54.706579+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:06.824468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:06.824468+0000","last_clean_scrub_stamp":"2024-03-11T22:30:06.824468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:29.075598+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"479'30","reported_seq":1093,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690716+0000","last_change":"2024-03-11T22:30:41.197575+0000","last_active":"2024-03-11T22:30:55.690716+0000","last_peered":"2024-03-11T22:30:55.690716+0000","last_clean":"2024-03-11T22:30:55.690716+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:30:55.690716+0000","last_undegraded":"2024-03-11T22:30:55.690716+0000","last_fullsized":"2024-03-11T22:30:55.690716+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-03-11T22:30:41.197457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:41.197457+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:01.110242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":126,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704469+0000","last_change":"2024-03-11T22:30:20.763816+0000","last_active":"2024-03-11T22:30:54.704469+0000","last_peered":"2024-03-11T22:30:54.704469+0000","last_clean":"2024-03-11T22:30:54.704469+0000","last_became_active":"2024-03-11T22:29:04.693762+0000","last_became_peered":"2024-03-11T22:29:04.693762+0000","last_unstale":"2024-03-11T22:30:54.704469+0000","last_undegraded":"2024-03-11T22:30:54.704469+0000","last_fullsized":"2024-03-11T22:30:54.704469+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:30:20.763628+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:30:20.763628+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:38.282863+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"315'11","reported_seq":1335,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:58.624600+0000","last_change":"2024-03-11T22:30:42.614513+0000","last_active":"2024-03-11T22:30:58.624600+0000","last_peered":"2024-03-11T22:30:58.624600+0000","last_clean":"2024-03-11T22:30:58.624600+0000","last_became_active":"2024-03-11T22:20:29.415781+0000","last_became_peered":"2024-03-11T22:20:29.415781+0000","last_unstale":"2024-03-11T22:30:58.624600+0000","last_undegraded":"2024-03-11T22:30:58.624600+0000","last_fullsized":"2024-03-11T22:30:58.624600+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-11T22:30:42.614396+0000","last_deep_scrub":"315'11","last_deep_scrub_stamp":"2024-03-11T22:20:54.741633+0000","last_clean_scrub_stamp":"2024-03-11T22:30:42.614396+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:56.150179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":956,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401482+0000","last_change":"2024-03-11T22:30:10.198091+0000","last_active":"2024-03-11T22:30:55.401482+0000","last_peered":"2024-03-11T22:30:55.401482+0000","last_clean":"2024-03-11T22:30:55.401482+0000","last_became_active":"2024-03-11T22:10:35.417379+0000","last_became_peered":"2024-03-11T22:10:35.417379+0000","last_unstale":"2024-03-11T22:30:55.401482+0000","last_undegraded":"2024-03-11T22:30:55.401482+0000","last_fullsized":"2024-03-11T22:30:55.401482+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-11T22:30:10.197974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:10.197974+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.414872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"480'24","reported_seq":1079,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:56.848659+0000","last_change":"2024-03-11T22:30:56.848699+0000","last_active":"2024-03-11T22:30:56.848659+0000","last_peered":"2024-03-11T22:30:56.848659+0000","last_clean":"2024-03-11T22:30:56.848659+0000","last_became_active":"2024-03-11T22:29:04.720341+0000","last_became_peered":"2024-03-11T22:29:04.720341+0000","last_unstale":"2024-03-11T22:30:56.848659+0000","last_undegraded":"2024-03-11T22:30:56.848659+0000","last_fullsized":"2024-03-11T22:30:56.848659+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'24","last_scrub_stamp":"2024-03-11T22:30:56.848590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:56.848590+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:58.594672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704997+0000","last_change":"2024-03-11T22:30:08.741018+0000","last_active":"2024-03-11T22:30:54.704997+0000","last_peered":"2024-03-11T22:30:54.704997+0000","last_clean":"2024-03-11T22:30:54.704997+0000","last_became_active":"2024-03-11T22:29:04.725678+0000","last_became_peered":"2024-03-11T22:29:04.725678+0000","last_unstale":"2024-03-11T22:30:54.704997+0000","last_undegraded":"2024-03-11T22:30:54.704997+0000","last_fullsized":"2024-03-11T22:30:54.704997+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:08.740926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:08.740926+0000","last_clean_scrub_stamp":"2024-03-11T22:30:08.740926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:21.051131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":765,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392090+0000","last_change":"2024-03-11T22:30:46.780258+0000","last_active":"2024-03-11T22:30:55.392090+0000","last_peered":"2024-03-11T22:30:55.392090+0000","last_clean":"2024-03-11T22:30:55.392090+0000","last_became_active":"2024-03-11T22:14:52.233215+0000","last_became_peered":"2024-03-11T22:14:52.233215+0000","last_unstale":"2024-03-11T22:30:55.392090+0000","last_undegraded":"2024-03-11T22:30:55.392090+0000","last_fullsized":"2024-03-11T22:30:55.392090+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:46.780169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:46.780169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:02.985331+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":988,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392290+0000","last_change":"2024-03-11T22:26:37.062133+0000","last_active":"2024-03-11T22:30:55.392290+0000","last_peered":"2024-03-11T22:30:55.392290+0000","last_clean":"2024-03-11T22:30:55.392290+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:30:55.392290+0000","last_undegraded":"2024-03-11T22:30:55.392290+0000","last_fullsized":"2024-03-11T22:30:55.392290+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:37.062022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:37.062022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"479'39","reported_seq":1148,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705296+0000","last_change":"2024-03-11T22:30:21.635123+0000","last_active":"2024-03-11T22:30:54.705296+0000","last_peered":"2024-03-11T22:30:54.705296+0000","last_clean":"2024-03-11T22:30:54.705296+0000","last_became_active":"2024-03-11T22:29:04.721324+0000","last_became_peered":"2024-03-11T22:29:04.721324+0000","last_unstale":"2024-03-11T22:30:54.705296+0000","last_undegraded":"2024-03-11T22:30:54.705296+0000","last_fullsized":"2024-03-11T22:30:54.705296+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'39","last_scrub_stamp":"2024-03-11T22:30:21.634960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:21.634960+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:39.506214+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1025,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.691304+0000","last_change":"2024-03-11T22:30:40.231188+0000","last_active":"2024-03-11T22:30:55.691304+0000","last_peered":"2024-03-11T22:30:55.691304+0000","last_clean":"2024-03-11T22:30:55.691304+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:30:55.691304+0000","last_undegraded":"2024-03-11T22:30:55.691304+0000","last_fullsized":"2024-03-11T22:30:55.691304+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:40.231100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:30:40.231100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:49.098484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.706564+0000","last_change":"2024-03-11T22:29:55.548541+0000","last_active":"2024-03-11T22:30:54.706564+0000","last_peered":"2024-03-11T22:30:54.706564+0000","last_clean":"2024-03-11T22:30:54.706564+0000","last_became_active":"2024-03-11T22:29:04.724856+0000","last_became_peered":"2024-03-11T22:29:04.724856+0000","last_unstale":"2024-03-11T22:30:54.706564+0000","last_undegraded":"2024-03-11T22:30:54.706564+0000","last_fullsized":"2024-03-11T22:30:54.706564+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:55.546853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:29:55.546853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:25.517836+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":790,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.673097+0000","last_change":"2024-03-11T22:30:17.307828+0000","last_active":"2024-03-11T22:30:55.673097+0000","last_peered":"2024-03-11T22:30:55.673097+0000","last_clean":"2024-03-11T22:30:55.673097+0000","last_became_active":"2024-03-11T22:14:52.240161+0000","last_became_peered":"2024-03-11T22:14:52.240161+0000","last_unstale":"2024-03-11T22:30:55.673097+0000","last_undegraded":"2024-03-11T22:30:55.673097+0000","last_fullsized":"2024-03-11T22:30:55.673097+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:17.307729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:17.307729+0000","last_clean_scrub_stamp":"2024-03-11T22:30:17.307729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:24.381369+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"478'11","reported_seq":869,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391398+0000","last_change":"2024-03-11T22:30:02.742323+0000","last_active":"2024-03-11T22:30:55.391398+0000","last_peered":"2024-03-11T22:30:55.391398+0000","last_clean":"2024-03-11T22:30:55.391398+0000","last_became_active":"2024-03-11T22:29:04.698810+0000","last_became_peered":"2024-03-11T22:29:04.698810+0000","last_unstale":"2024-03-11T22:30:55.391398+0000","last_undegraded":"2024-03-11T22:30:55.391398+0000","last_fullsized":"2024-03-11T22:30:55.391398+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'11","last_scrub_stamp":"2024-03-11T22:30:02.742197+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:30:02.742197+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:26.951844+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1027,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401634+0000","last_change":"2024-03-11T22:30:15.199268+0000","last_active":"2024-03-11T22:30:55.401634+0000","last_peered":"2024-03-11T22:30:55.401634+0000","last_clean":"2024-03-11T22:30:55.401634+0000","last_became_active":"2024-03-11T22:29:04.721192+0000","last_became_peered":"2024-03-11T22:29:04.721192+0000","last_unstale":"2024-03-11T22:30:55.401634+0000","last_undegraded":"2024-03-11T22:30:55.401634+0000","last_fullsized":"2024-03-11T22:30:55.401634+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:15.199165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:30:15.199165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:43.996210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.705264+0000","last_change":"2024-03-11T22:30:43.546407+0000","last_active":"2024-03-11T22:30:54.705264+0000","last_peered":"2024-03-11T22:30:54.705264+0000","last_clean":"2024-03-11T22:30:54.705264+0000","last_became_active":"2024-03-11T22:29:04.726418+0000","last_became_peered":"2024-03-11T22:29:04.726418+0000","last_unstale":"2024-03-11T22:30:54.705264+0000","last_undegraded":"2024-03-11T22:30:54.705264+0000","last_fullsized":"2024-03-11T22:30:54.705264+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:43.546317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:09.575187+0000","last_clean_scrub_stamp":"2024-03-11T22:30:43.546317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:10.181400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401239+0000","last_change":"2024-03-11T22:30:51.393304+0000","last_active":"2024-03-11T22:30:55.401239+0000","last_peered":"2024-03-11T22:30:55.401239+0000","last_clean":"2024-03-11T22:30:55.401239+0000","last_became_active":"2024-03-11T22:29:04.721807+0000","last_became_peered":"2024-03-11T22:29:04.721807+0000","last_unstale":"2024-03-11T22:30:55.401239+0000","last_undegraded":"2024-03-11T22:30:55.401239+0000","last_fullsized":"2024-03-11T22:30:55.401239+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:51.393206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:30:51.393206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:52.157560+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"479'21","reported_seq":826,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.873642+0000","last_change":"2024-03-11T22:30:55.873677+0000","last_active":"2024-03-11T22:30:55.873642+0000","last_peered":"2024-03-11T22:30:55.873642+0000","last_clean":"2024-03-11T22:30:55.873642+0000","last_became_active":"2024-03-11T22:14:52.237007+0000","last_became_peered":"2024-03-11T22:14:52.237007+0000","last_unstale":"2024-03-11T22:30:55.873642+0000","last_undegraded":"2024-03-11T22:30:55.873642+0000","last_fullsized":"2024-03-11T22:30:55.873642+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'21","last_scrub_stamp":"2024-03-11T22:30:55.873576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:30:55.873576+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:32:18.938926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401601+0000","last_change":"2024-03-11T22:29:53.126868+0000","last_active":"2024-03-11T22:30:55.401601+0000","last_peered":"2024-03-11T22:30:55.401601+0000","last_clean":"2024-03-11T22:30:55.401601+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:30:55.401601+0000","last_undegraded":"2024-03-11T22:30:55.401601+0000","last_fullsized":"2024-03-11T22:30:55.401601+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:29:53.126769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:29:53.126769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:12.911808+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1019,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401307+0000","last_change":"2024-03-11T22:30:16.197266+0000","last_active":"2024-03-11T22:30:55.401307+0000","last_peered":"2024-03-11T22:30:55.401307+0000","last_clean":"2024-03-11T22:30:55.401307+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:30:55.401307+0000","last_undegraded":"2024-03-11T22:30:55.401307+0000","last_fullsized":"2024-03-11T22:30:55.401307+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:16.197172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:30:16.197172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.268824+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1006,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690161+0000","last_change":"2024-03-11T22:29:04.708231+0000","last_active":"2024-03-11T22:30:55.690161+0000","last_peered":"2024-03-11T22:30:55.690161+0000","last_clean":"2024-03-11T22:30:55.690161+0000","last_became_active":"2024-03-11T22:29:04.707444+0000","last_became_peered":"2024-03-11T22:29:04.707444+0000","last_unstale":"2024-03-11T22:30:55.690161+0000","last_undegraded":"2024-03-11T22:30:55.690161+0000","last_fullsized":"2024-03-11T22:30:55.690161+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:37.738562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:37.738562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"480'39","reported_seq":1056,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391884+0000","last_change":"2024-03-11T22:26:38.945645+0000","last_active":"2024-03-11T22:30:55.391884+0000","last_peered":"2024-03-11T22:30:55.391884+0000","last_clean":"2024-03-11T22:30:55.391884+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:30:55.391884+0000","last_undegraded":"2024-03-11T22:30:55.391884+0000","last_fullsized":"2024-03-11T22:30:55.391884+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'27","last_scrub_stamp":"2024-03-11T22:26:38.945505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:26:38.945505+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"68'1","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.392257+0000","last_change":"2024-03-11T22:26:39.902683+0000","last_active":"2024-03-11T22:30:55.392257+0000","last_peered":"2024-03-11T22:30:55.392257+0000","last_clean":"2024-03-11T22:30:55.392257+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:30:55.392257+0000","last_undegraded":"2024-03-11T22:30:55.392257+0000","last_fullsized":"2024-03-11T22:30:55.392257+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:26:39.902588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:26:39.902588+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":803,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672111+0000","last_change":"2024-03-11T22:30:15.364972+0000","last_active":"2024-03-11T22:30:55.672111+0000","last_peered":"2024-03-11T22:30:55.672111+0000","last_clean":"2024-03-11T22:30:55.672111+0000","last_became_active":"2024-03-11T22:29:04.719011+0000","last_became_peered":"2024-03-11T22:29:04.719011+0000","last_unstale":"2024-03-11T22:30:55.672111+0000","last_undegraded":"2024-03-11T22:30:55.672111+0000","last_fullsized":"2024-03-11T22:30:55.672111+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:15.364881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:15.364881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:18.215173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":998,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.391581+0000","last_change":"2024-03-11T22:26:34.021098+0000","last_active":"2024-03-11T22:30:55.391581+0000","last_peered":"2024-03-11T22:30:55.391581+0000","last_clean":"2024-03-11T22:30:55.391581+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:30:55.391581+0000","last_undegraded":"2024-03-11T22:30:55.391581+0000","last_fullsized":"2024-03-11T22:30:55.391581+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:34.020992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:34.020992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"478'25","reported_seq":869,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.672223+0000","last_change":"2024-03-11T22:29:57.388072+0000","last_active":"2024-03-11T22:30:55.672223+0000","last_peered":"2024-03-11T22:30:55.672223+0000","last_clean":"2024-03-11T22:30:55.672223+0000","last_became_active":"2024-03-11T22:29:04.718521+0000","last_became_peered":"2024-03-11T22:29:04.718521+0000","last_unstale":"2024-03-11T22:30:55.672223+0000","last_undegraded":"2024-03-11T22:30:55.672223+0000","last_fullsized":"2024-03-11T22:30:55.672223+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:29:57.387926+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:29:57.387926+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:05.421282+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1015,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.690302+0000","last_change":"2024-03-11T22:30:04.486340+0000","last_active":"2024-03-11T22:30:55.690302+0000","last_peered":"2024-03-11T22:30:55.690302+0000","last_clean":"2024-03-11T22:30:55.690302+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:30:55.690302+0000","last_undegraded":"2024-03-11T22:30:55.690302+0000","last_fullsized":"2024-03-11T22:30:55.690302+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:04.486254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:30:04.486254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:17.389356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":122,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:54.704753+0000","last_change":"2024-03-11T22:30:41.486573+0000","last_active":"2024-03-11T22:30:54.704753+0000","last_peered":"2024-03-11T22:30:54.704753+0000","last_clean":"2024-03-11T22:30:54.704753+0000","last_became_active":"2024-03-11T22:29:04.701716+0000","last_became_peered":"2024-03-11T22:29:04.701716+0000","last_unstale":"2024-03-11T22:30:54.704753+0000","last_undegraded":"2024-03-11T22:30:54.704753+0000","last_fullsized":"2024-03-11T22:30:54.704753+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:30:41.486494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:30:41.486494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:31:43.179038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":785,"reported_epoch":538,"state":"active+clean","last_fresh":"2024-03-11T22:30:55.401118+0000","last_change":"2024-03-11T22:29:04.718466+0000","last_active":"2024-03-11T22:30:55.401118+0000","last_peered":"2024-03-11T22:30:55.401118+0000","last_clean":"2024-03-11T22:30:55.401118+0000","last_became_active":"2024-03-11T22:29:04.718122+0000","last_became_peered":"2024-03-11T22:29:04.718122+0000","last_unstale":"2024-03-11T22:30:55.401118+0000","last_undegraded":"2024-03-11T22:30:55.401118+0000","last_fullsized":"2024-03-11T22:30:55.401118+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:26:10.104642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:26:10.104642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":307,"num_read_kb":302,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":6574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1035,"ondisk_log_size":1035,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005331,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":845772,"kb_used_data":2436,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92899380,"statfs":{"total":95995035648,"available":95128965120,"internally_reserved":0,"allocated":2494464,"data_stored":1674408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":863566283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332376,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":691320,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93053832,"statfs":{"total":95995035648,"available":95287123968,"internally_reserved":0,"allocated":2146304,"data_stored":1346538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":705755590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575153,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":751072,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92994080,"statfs":{"total":95995035648,"available":95225937920,"internally_reserved":0,"allocated":2187264,"data_stored":1346596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738679,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957900,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92787252,"statfs":{"total":95995035648,"available":95014146048,"internally_reserved":0,"allocated":2162688,"data_stored":1346060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411629,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":647160,"kb_used_data":2352,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93097992,"statfs":{"total":95995035648,"available":95332343808,"internally_reserved":0,"allocated":2408448,"data_stored":1609052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149990,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":778016,"kb_used_data":2712,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92967136,"statfs":{"total":95995035648,"available":95198347264,"internally_reserved":0,"allocated":2777088,"data_stored":1937412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":793901511},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921059,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":714960,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":93030192,"statfs":{"total":95995035648,"available":95262916608,"internally_reserved":0,"allocated":2170880,"data_stored":1346269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":729938374},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659420,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":796040,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92949112,"statfs":{"total":95995035648,"available":95179890688,"internally_reserved":0,"allocated":2752512,"data_stored":1936317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":812382662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:31:08.147 INFO:tasks.thrashosds.thrasher:Setting 5.1f to [6, 0, 2, 3, 7, 5] 2024-03-11T22:31:08.147 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.1f', '6', '0', '2', '3', '7', '5'] 2024-03-11T22:31:08.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pg-upmap-items 5.1f 6 0 2 3 7 5 2024-03-11T22:31:08.395 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:08.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:08 smithi045 ceph-mon[30178]: 6.12 scrub starts 2024-03-11T22:31:08.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:08 smithi045 ceph-mon[30178]: 6.12 scrub ok 2024-03-11T22:31:08.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:08 smithi045 ceph-mon[30178]: from='client.25823 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:31:08.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:08 smithi045 ceph-mon[30178]: 5.0 scrub starts 2024-03-11T22:31:08.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:08 smithi045 ceph-mon[30178]: 5.0 scrub ok 2024-03-11T22:31:08.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:08 smithi045 ceph-mon[39715]: 6.12 scrub starts 2024-03-11T22:31:08.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:08 smithi045 ceph-mon[39715]: 6.12 scrub ok 2024-03-11T22:31:08.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:08 smithi045 ceph-mon[39715]: from='client.25823 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:31:08.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:08 smithi045 ceph-mon[39715]: 5.0 scrub starts 2024-03-11T22:31:08.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:08 smithi045 ceph-mon[39715]: 5.0 scrub ok 2024-03-11T22:31:09.040 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:08 smithi144 ceph-mon[33984]: 6.12 scrub starts 2024-03-11T22:31:09.040 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:08 smithi144 ceph-mon[33984]: 6.12 scrub ok 2024-03-11T22:31:09.040 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:08 smithi144 ceph-mon[33984]: from='client.25823 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:31:09.040 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:08 smithi144 ceph-mon[33984]: 5.0 scrub starts 2024-03-11T22:31:09.040 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:08 smithi144 ceph-mon[33984]: 5.0 scrub ok 2024-03-11T22:31:10.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[30178]: 1.5 deep-scrub starts 2024-03-11T22:31:10.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[30178]: 1.5 deep-scrub ok 2024-03-11T22:31:10.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[30178]: pgmap v1112: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:31:10.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[30178]: 3.a scrub starts 2024-03-11T22:31:10.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[30178]: 3.a scrub ok 2024-03-11T22:31:10.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[30178]: 4.7 deep-scrub starts 2024-03-11T22:31:10.006 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[30178]: 4.7 deep-scrub ok 2024-03-11T22:31:10.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:31:10.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:31:10.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[39715]: 1.5 deep-scrub starts 2024-03-11T22:31:10.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[39715]: 1.5 deep-scrub ok 2024-03-11T22:31:10.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[39715]: pgmap v1112: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:31:10.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[39715]: 3.a scrub starts 2024-03-11T22:31:10.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[39715]: 3.a scrub ok 2024-03-11T22:31:10.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[39715]: 4.7 deep-scrub starts 2024-03-11T22:31:10.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[39715]: 4.7 deep-scrub ok 2024-03-11T22:31:10.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:31:10.007 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:09 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:31:10.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:09 smithi144 ceph-mon[33984]: 1.5 deep-scrub starts 2024-03-11T22:31:10.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:09 smithi144 ceph-mon[33984]: 1.5 deep-scrub ok 2024-03-11T22:31:10.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:09 smithi144 ceph-mon[33984]: pgmap v1112: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:31:10.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:09 smithi144 ceph-mon[33984]: 3.a scrub starts 2024-03-11T22:31:10.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:09 smithi144 ceph-mon[33984]: 3.a scrub ok 2024-03-11T22:31:10.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:09 smithi144 ceph-mon[33984]: 4.7 deep-scrub starts 2024-03-11T22:31:10.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:09 smithi144 ceph-mon[33984]: 4.7 deep-scrub ok 2024-03-11T22:31:10.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:09 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:31:10.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:09 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:31:10.766 INFO:teuthology.orchestra.run.smithi045.stderr:set 5.1f pg_upmap_items mapping to [6->0,2->3,7->5] 2024-03-11T22:31:11.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[30178]: 6.1a scrub starts 2024-03-11T22:31:11.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[30178]: 6.1a scrub ok 2024-03-11T22:31:11.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[30178]: osdmap e539: 8 total, 8 up, 8 in 2024-03-11T22:31:11.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[30178]: 6.2 scrub starts 2024-03-11T22:31:11.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[30178]: 6.2 scrub ok 2024-03-11T22:31:11.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[30178]: 4.b scrub starts 2024-03-11T22:31:11.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[30178]: 4.b scrub ok 2024-03-11T22:31:11.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3930158060' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [6, 0, 2, 3, 7, 5]}]: dispatch 2024-03-11T22:31:11.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[39715]: 6.1a scrub starts 2024-03-11T22:31:11.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[39715]: 6.1a scrub ok 2024-03-11T22:31:11.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[39715]: osdmap e539: 8 total, 8 up, 8 in 2024-03-11T22:31:11.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[39715]: 6.2 scrub starts 2024-03-11T22:31:11.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[39715]: 6.2 scrub ok 2024-03-11T22:31:11.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[39715]: 4.b scrub starts 2024-03-11T22:31:11.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[39715]: 4.b scrub ok 2024-03-11T22:31:11.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:10 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3930158060' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [6, 0, 2, 3, 7, 5]}]: dispatch 2024-03-11T22:31:11.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:10 smithi144 ceph-mon[33984]: 6.1a scrub starts 2024-03-11T22:31:11.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:10 smithi144 ceph-mon[33984]: 6.1a scrub ok 2024-03-11T22:31:11.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:10 smithi144 ceph-mon[33984]: osdmap e539: 8 total, 8 up, 8 in 2024-03-11T22:31:11.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:10 smithi144 ceph-mon[33984]: 6.2 scrub starts 2024-03-11T22:31:11.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:10 smithi144 ceph-mon[33984]: 6.2 scrub ok 2024-03-11T22:31:11.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:10 smithi144 ceph-mon[33984]: 4.b scrub starts 2024-03-11T22:31:11.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:10 smithi144 ceph-mon[33984]: 4.b scrub ok 2024-03-11T22:31:11.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:10 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3930158060' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [6, 0, 2, 3, 7, 5]}]: dispatch 2024-03-11T22:31:12.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:11 smithi045 ceph-mon[30178]: pgmap v1114: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:31:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:11 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3930158060' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [6, 0, 2, 3, 7, 5]}]': finished 2024-03-11T22:31:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:11 smithi045 ceph-mon[30178]: osdmap e540: 8 total, 8 up, 8 in 2024-03-11T22:31:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:11 smithi045 ceph-mon[30178]: 1.2 scrub starts 2024-03-11T22:31:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:11 smithi045 ceph-mon[30178]: 1.2 scrub ok 2024-03-11T22:31:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:11 smithi045 ceph-mon[39715]: pgmap v1114: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:31:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:11 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3930158060' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [6, 0, 2, 3, 7, 5]}]': finished 2024-03-11T22:31:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:11 smithi045 ceph-mon[39715]: osdmap e540: 8 total, 8 up, 8 in 2024-03-11T22:31:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:11 smithi045 ceph-mon[39715]: 1.2 scrub starts 2024-03-11T22:31:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:11 smithi045 ceph-mon[39715]: 1.2 scrub ok 2024-03-11T22:31:12.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:11 smithi144 ceph-mon[33984]: pgmap v1114: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:31:12.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:11 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3930158060' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.1f", "id": [6, 0, 2, 3, 7, 5]}]': finished 2024-03-11T22:31:12.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:11 smithi144 ceph-mon[33984]: osdmap e540: 8 total, 8 up, 8 in 2024-03-11T22:31:12.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:11 smithi144 ceph-mon[33984]: 1.2 scrub starts 2024-03-11T22:31:12.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:11 smithi144 ceph-mon[33984]: 1.2 scrub ok 2024-03-11T22:31:13.037 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:31:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:31:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:31:13.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:12 smithi045 ceph-mon[39715]: osdmap e541: 8 total, 8 up, 8 in 2024-03-11T22:31:13.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:12 smithi045 ceph-mon[30178]: osdmap e541: 8 total, 8 up, 8 in 2024-03-11T22:31:13.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:12 smithi144 ceph-mon[33984]: osdmap e541: 8 total, 8 up, 8 in 2024-03-11T22:31:14.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:13 smithi045 ceph-mon[39715]: pgmap v1117: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:14.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:13 smithi045 ceph-mon[39715]: 6.18 scrub starts 2024-03-11T22:31:14.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:13 smithi045 ceph-mon[39715]: 6.18 scrub ok 2024-03-11T22:31:14.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:13 smithi045 ceph-mon[30178]: pgmap v1117: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:14.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:13 smithi045 ceph-mon[30178]: 6.18 scrub starts 2024-03-11T22:31:14.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:13 smithi045 ceph-mon[30178]: 6.18 scrub ok 2024-03-11T22:31:14.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:13 smithi144 ceph-mon[33984]: pgmap v1117: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:14.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:13 smithi144 ceph-mon[33984]: 6.18 scrub starts 2024-03-11T22:31:14.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:13 smithi144 ceph-mon[33984]: 6.18 scrub ok 2024-03-11T22:31:15.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:14 smithi045 ceph-mon[39715]: 3.10 scrub starts 2024-03-11T22:31:15.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:14 smithi045 ceph-mon[39715]: 3.10 scrub ok 2024-03-11T22:31:15.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:14 smithi045 ceph-mon[39715]: 6.6 scrub starts 2024-03-11T22:31:15.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:14 smithi045 ceph-mon[39715]: 6.6 scrub ok 2024-03-11T22:31:15.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:14 smithi045 ceph-mon[30178]: 3.10 scrub starts 2024-03-11T22:31:15.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:14 smithi045 ceph-mon[30178]: 3.10 scrub ok 2024-03-11T22:31:15.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:14 smithi045 ceph-mon[30178]: 6.6 scrub starts 2024-03-11T22:31:15.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:14 smithi045 ceph-mon[30178]: 6.6 scrub ok 2024-03-11T22:31:15.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:14 smithi144 ceph-mon[33984]: 3.10 scrub starts 2024-03-11T22:31:15.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:14 smithi144 ceph-mon[33984]: 3.10 scrub ok 2024-03-11T22:31:15.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:14 smithi144 ceph-mon[33984]: 6.6 scrub starts 2024-03-11T22:31:15.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:14 smithi144 ceph-mon[33984]: 6.6 scrub ok 2024-03-11T22:31:16.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[30178]: 5.8 deep-scrub starts 2024-03-11T22:31:16.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[30178]: 5.8 deep-scrub ok 2024-03-11T22:31:16.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[30178]: 3.d scrub starts 2024-03-11T22:31:16.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[30178]: 3.d scrub ok 2024-03-11T22:31:16.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[30178]: pgmap v1118: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:31:16.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[30178]: 5.2 scrub starts 2024-03-11T22:31:16.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[30178]: 5.2 scrub ok 2024-03-11T22:31:16.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[39715]: 5.8 deep-scrub starts 2024-03-11T22:31:16.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[39715]: 5.8 deep-scrub ok 2024-03-11T22:31:16.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[39715]: 3.d scrub starts 2024-03-11T22:31:16.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[39715]: 3.d scrub ok 2024-03-11T22:31:16.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[39715]: pgmap v1118: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:31:16.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[39715]: 5.2 scrub starts 2024-03-11T22:31:16.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:15 smithi045 ceph-mon[39715]: 5.2 scrub ok 2024-03-11T22:31:16.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:15 smithi144 ceph-mon[33984]: 5.8 deep-scrub starts 2024-03-11T22:31:16.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:15 smithi144 ceph-mon[33984]: 5.8 deep-scrub ok 2024-03-11T22:31:16.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:15 smithi144 ceph-mon[33984]: 3.d scrub starts 2024-03-11T22:31:16.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:15 smithi144 ceph-mon[33984]: 3.d scrub ok 2024-03-11T22:31:16.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:15 smithi144 ceph-mon[33984]: pgmap v1118: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:31:16.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:15 smithi144 ceph-mon[33984]: 5.2 scrub starts 2024-03-11T22:31:16.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:15 smithi144 ceph-mon[33984]: 5.2 scrub ok 2024-03-11T22:31:16.586 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-11T22:31:16.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-03-11T22:31:16.855 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:16.977 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:16.977 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:16 smithi045 ceph-mon[39715]: osdmap e542: 8 total, 8 up, 8 in 2024-03-11T22:31:16.977 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:16.977 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:16 smithi045 ceph-mon[30178]: osdmap e542: 8 total, 8 up, 8 in 2024-03-11T22:31:17.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:17.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:16 smithi144 ceph-mon[33984]: osdmap e542: 8 total, 8 up, 8 in 2024-03-11T22:31:17.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:31:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:31:18.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:17 smithi045 ceph-mon[39715]: pgmap v1119: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 754 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:31:18.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:17 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:18.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:17 smithi045 ceph-mon[30178]: pgmap v1119: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 754 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:31:18.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:17 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:18.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:17 smithi144 ceph-mon[33984]: pgmap v1119: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 754 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:31:18.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:17 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:19.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:18 smithi144 ceph-mon[33984]: 5.1e scrub starts 2024-03-11T22:31:19.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:18 smithi144 ceph-mon[33984]: 5.1e scrub ok 2024-03-11T22:31:19.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:18 smithi045 ceph-mon[39715]: 5.1e scrub starts 2024-03-11T22:31:19.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:18 smithi045 ceph-mon[39715]: 5.1e scrub ok 2024-03-11T22:31:19.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:18 smithi045 ceph-mon[30178]: 5.1e scrub starts 2024-03-11T22:31:19.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:18 smithi045 ceph-mon[30178]: 5.1e scrub ok 2024-03-11T22:31:19.804 INFO:teuthology.orchestra.run.smithi045.stderr:set pool 1 pgp_num to 11 2024-03-11T22:31:20.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:19 smithi144 ceph-mon[33984]: 2.2 scrub starts 2024-03-11T22:31:20.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:19 smithi144 ceph-mon[33984]: 2.2 scrub ok 2024-03-11T22:31:20.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:19 smithi144 ceph-mon[33984]: pgmap v1121: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:31:20.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:19 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2813295821' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-11T22:31:20.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:19 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-11T22:31:20.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:31:20.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:19 smithi144 ceph-mon[33984]: 4.6 scrub starts 2024-03-11T22:31:20.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:19 smithi144 ceph-mon[33984]: 4.6 scrub ok 2024-03-11T22:31:20.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:31:20.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:31:20.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:19 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:31:20.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:19 smithi144 ceph-mon[33984]: 1.3 scrub starts 2024-03-11T22:31:20.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:19 smithi144 ceph-mon[33984]: 1.3 scrub ok 2024-03-11T22:31:20.169 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[39715]: 2.2 scrub starts 2024-03-11T22:31:20.169 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[39715]: 2.2 scrub ok 2024-03-11T22:31:20.169 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[39715]: pgmap v1121: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:31:20.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2813295821' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-11T22:31:20.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-11T22:31:20.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:31:20.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[39715]: 4.6 scrub starts 2024-03-11T22:31:20.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[39715]: 4.6 scrub ok 2024-03-11T22:31:20.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:31:20.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:31:20.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:31:20.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[39715]: 1.3 scrub starts 2024-03-11T22:31:20.170 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[39715]: 1.3 scrub ok 2024-03-11T22:31:20.170 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[30178]: 2.2 scrub starts 2024-03-11T22:31:20.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[30178]: 2.2 scrub ok 2024-03-11T22:31:20.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[30178]: pgmap v1121: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:31:20.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2813295821' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-11T22:31:20.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-11T22:31:20.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:31:20.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[30178]: 4.6 scrub starts 2024-03-11T22:31:20.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[30178]: 4.6 scrub ok 2024-03-11T22:31:20.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:31:20.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:31:20.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:31:20.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[30178]: 1.3 scrub starts 2024-03-11T22:31:20.171 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:19 smithi045 ceph-mon[30178]: 1.3 scrub ok 2024-03-11T22:31:20.632 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-11T22:31:20.883 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:21.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[30178]: 5.6 scrub starts 2024-03-11T22:31:21.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[30178]: 5.6 scrub ok 2024-03-11T22:31:21.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-11T22:31:21.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[30178]: osdmap e543: 8 total, 8 up, 8 in 2024-03-11T22:31:21.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[30178]: 4.0 scrub starts 2024-03-11T22:31:21.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[30178]: 4.0 scrub ok 2024-03-11T22:31:21.007 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[30178]: 3.0 scrub starts 2024-03-11T22:31:21.008 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[30178]: 3.0 scrub ok 2024-03-11T22:31:21.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[39715]: 5.6 scrub starts 2024-03-11T22:31:21.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[39715]: 5.6 scrub ok 2024-03-11T22:31:21.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-11T22:31:21.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[39715]: osdmap e543: 8 total, 8 up, 8 in 2024-03-11T22:31:21.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[39715]: 4.0 scrub starts 2024-03-11T22:31:21.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[39715]: 4.0 scrub ok 2024-03-11T22:31:21.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[39715]: 3.0 scrub starts 2024-03-11T22:31:21.008 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:20 smithi045 ceph-mon[39715]: 3.0 scrub ok 2024-03-11T22:31:21.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:20 smithi144 ceph-mon[33984]: 5.6 scrub starts 2024-03-11T22:31:21.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:20 smithi144 ceph-mon[33984]: 5.6 scrub ok 2024-03-11T22:31:21.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:20 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-11T22:31:21.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:20 smithi144 ceph-mon[33984]: osdmap e543: 8 total, 8 up, 8 in 2024-03-11T22:31:21.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:20 smithi144 ceph-mon[33984]: 4.0 scrub starts 2024-03-11T22:31:21.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:20 smithi144 ceph-mon[33984]: 4.0 scrub ok 2024-03-11T22:31:21.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:20 smithi144 ceph-mon[33984]: 3.0 scrub starts 2024-03-11T22:31:21.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:20 smithi144 ceph-mon[33984]: 3.0 scrub ok 2024-03-11T22:31:22.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:21 smithi045 ceph-mon[39715]: pgmap v1123: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:31:22.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:21 smithi045 ceph-mon[39715]: 6.15 scrub starts 2024-03-11T22:31:22.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:21 smithi045 ceph-mon[39715]: 6.15 scrub ok 2024-03-11T22:31:22.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:21 smithi045 ceph-mon[39715]: osdmap e544: 8 total, 8 up, 8 in 2024-03-11T22:31:22.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:21 smithi045 ceph-mon[30178]: pgmap v1123: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:31:22.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:21 smithi045 ceph-mon[30178]: 6.15 scrub starts 2024-03-11T22:31:22.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:21 smithi045 ceph-mon[30178]: 6.15 scrub ok 2024-03-11T22:31:22.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:21 smithi045 ceph-mon[30178]: osdmap e544: 8 total, 8 up, 8 in 2024-03-11T22:31:22.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:21 smithi144 ceph-mon[33984]: pgmap v1123: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-11T22:31:22.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:21 smithi144 ceph-mon[33984]: 6.15 scrub starts 2024-03-11T22:31:22.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:21 smithi144 ceph-mon[33984]: 6.15 scrub ok 2024-03-11T22:31:22.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:21 smithi144 ceph-mon[33984]: osdmap e544: 8 total, 8 up, 8 in 2024-03-11T22:31:22.660 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:31:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:31:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:31:22.912 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:22 smithi045 ceph-mon[39715]: 4.12 scrub starts 2024-03-11T22:31:22.912 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:22 smithi045 ceph-mon[39715]: 4.12 scrub ok 2024-03-11T22:31:22.912 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:22 smithi045 ceph-mon[30178]: 4.12 scrub starts 2024-03-11T22:31:22.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:22 smithi045 ceph-mon[30178]: 4.12 scrub ok 2024-03-11T22:31:22.921 INFO:teuthology.orchestra.run.smithi045.stdout:mon.a: {} 2024-03-11T22:31:22.932 INFO:teuthology.orchestra.run.smithi045.stdout:mon.b: {} 2024-03-11T22:31:22.942 INFO:teuthology.orchestra.run.smithi045.stdout:mon.c: {} 2024-03-11T22:31:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T22:31:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T22:31:22.943 INFO:teuthology.orchestra.run.smithi045.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T22:31:23.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:22 smithi144 ceph-mon[33984]: 4.12 scrub starts 2024-03-11T22:31:23.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:22 smithi144 ceph-mon[33984]: 4.12 scrub ok 2024-03-11T22:31:23.656 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T22:31:23.908 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:24.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:23 smithi045 ceph-mon[39715]: pgmap v1125: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:24.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:23 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2165686783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:31:24.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:23 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:31:24.029 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:23 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:31:24.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:23 smithi045 ceph-mon[30178]: pgmap v1125: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:24.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:23 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2165686783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:31:24.029 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:23 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:31:24.030 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:23 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:31:24.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:23 smithi144 ceph-mon[33984]: pgmap v1125: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:24.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:23 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2165686783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:31:24.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:23 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:31:24.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:23 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:31:25.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:24 smithi144 ceph-mon[33984]: osdmap e545: 8 total, 8 up, 8 in 2024-03-11T22:31:25.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:24 smithi144 ceph-mon[33984]: 3.3 scrub starts 2024-03-11T22:31:25.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:24 smithi144 ceph-mon[33984]: 3.3 scrub ok 2024-03-11T22:31:25.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:24 smithi144 ceph-mon[33984]: 3.1a scrub starts 2024-03-11T22:31:25.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:24 smithi144 ceph-mon[33984]: 3.1a scrub ok 2024-03-11T22:31:25.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:24 smithi045 ceph-mon[30178]: osdmap e545: 8 total, 8 up, 8 in 2024-03-11T22:31:25.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:24 smithi045 ceph-mon[30178]: 3.3 scrub starts 2024-03-11T22:31:25.205 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:24 smithi045 ceph-mon[30178]: 3.3 scrub ok 2024-03-11T22:31:25.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:24 smithi045 ceph-mon[30178]: 3.1a scrub starts 2024-03-11T22:31:25.206 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:24 smithi045 ceph-mon[30178]: 3.1a scrub ok 2024-03-11T22:31:25.206 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:24 smithi045 ceph-mon[39715]: osdmap e545: 8 total, 8 up, 8 in 2024-03-11T22:31:25.206 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:24 smithi045 ceph-mon[39715]: 3.3 scrub starts 2024-03-11T22:31:25.206 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:24 smithi045 ceph-mon[39715]: 3.3 scrub ok 2024-03-11T22:31:25.206 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:24 smithi045 ceph-mon[39715]: 3.1a scrub starts 2024-03-11T22:31:25.206 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:24 smithi045 ceph-mon[39715]: 3.1a scrub ok 2024-03-11T22:31:26.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:25 smithi144 ceph-mon[33984]: 6.d scrub starts 2024-03-11T22:31:26.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:25 smithi144 ceph-mon[33984]: 6.d scrub ok 2024-03-11T22:31:26.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:25 smithi144 ceph-mon[33984]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:26.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:25 smithi144 ceph-mon[33984]: 1.9 scrub starts 2024-03-11T22:31:26.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:25 smithi144 ceph-mon[33984]: 1.9 scrub ok 2024-03-11T22:31:26.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:25 smithi144 ceph-mon[33984]: 4.3 scrub starts 2024-03-11T22:31:26.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:25 smithi144 ceph-mon[33984]: 4.3 scrub ok 2024-03-11T22:31:26.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[30178]: 6.d scrub starts 2024-03-11T22:31:26.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[30178]: 6.d scrub ok 2024-03-11T22:31:26.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[30178]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:26.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[30178]: 1.9 scrub starts 2024-03-11T22:31:26.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[30178]: 1.9 scrub ok 2024-03-11T22:31:26.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[30178]: 4.3 scrub starts 2024-03-11T22:31:26.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[30178]: 4.3 scrub ok 2024-03-11T22:31:26.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[39715]: 6.d scrub starts 2024-03-11T22:31:26.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[39715]: 6.d scrub ok 2024-03-11T22:31:26.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[39715]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:26.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[39715]: 1.9 scrub starts 2024-03-11T22:31:26.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[39715]: 1.9 scrub ok 2024-03-11T22:31:26.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[39715]: 4.3 scrub starts 2024-03-11T22:31:26.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:25 smithi045 ceph-mon[39715]: 4.3 scrub ok 2024-03-11T22:31:26.662 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T22:31:27.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[39715]: 5.1d scrub starts 2024-03-11T22:31:27.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[39715]: 5.1d scrub ok 2024-03-11T22:31:27.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[39715]: 4.18 scrub starts 2024-03-11T22:31:27.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[39715]: 4.18 scrub ok 2024-03-11T22:31:27.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[39715]: 6.b scrub starts 2024-03-11T22:31:27.022 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1541267048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[39715]: 6.b scrub ok 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1541267048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[39715]: osdmap e546: 8 total, 8 up, 8 in 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[30178]: 5.1d scrub starts 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[30178]: 5.1d scrub ok 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[30178]: 4.18 scrub starts 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[30178]: 4.18 scrub ok 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[30178]: 6.b scrub starts 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1541267048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[30178]: 6.b scrub ok 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:31:27.023 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:31:27.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1541267048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:31:27.024 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:26 smithi045 ceph-mon[30178]: osdmap e546: 8 total, 8 up, 8 in 2024-03-11T22:31:27.100 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:31:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:31:27.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:26 smithi144 ceph-mon[33984]: 5.1d scrub starts 2024-03-11T22:31:27.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:26 smithi144 ceph-mon[33984]: 5.1d scrub ok 2024-03-11T22:31:27.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:26 smithi144 ceph-mon[33984]: 4.18 scrub starts 2024-03-11T22:31:27.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:26 smithi144 ceph-mon[33984]: 4.18 scrub ok 2024-03-11T22:31:27.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:26 smithi144 ceph-mon[33984]: 6.b scrub starts 2024-03-11T22:31:27.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:26 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1541267048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:31:27.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:26 smithi144 ceph-mon[33984]: 6.b scrub ok 2024-03-11T22:31:27.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:26 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:31:27.102 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:26 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:31:27.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:26 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1541267048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:31:27.103 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:26 smithi144 ceph-mon[33984]: osdmap e546: 8 total, 8 up, 8 in 2024-03-11T22:31:27.599 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-11T22:31:27.856 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:28.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:27 smithi144 ceph-mon[33984]: pgmap v1128: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 758 B/s rd, 0 op/s 2024-03-11T22:31:28.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:27 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:31:28.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:27 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:31:28.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:27 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:28.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:27 smithi045 ceph-mon[39715]: pgmap v1128: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 758 B/s rd, 0 op/s 2024-03-11T22:31:28.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:27 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:31:28.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:27 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:31:28.110 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:27 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:28.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:27 smithi045 ceph-mon[30178]: pgmap v1128: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 758 B/s rd, 0 op/s 2024-03-11T22:31:28.110 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:27 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:31:28.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:27 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:31:28.111 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:27 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:29.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:28 smithi045 ceph-mon[39715]: 1.1 scrub starts 2024-03-11T22:31:29.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:28 smithi045 ceph-mon[39715]: 1.1 scrub ok 2024-03-11T22:31:29.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:28 smithi045 ceph-mon[39715]: osdmap e547: 8 total, 8 up, 8 in 2024-03-11T22:31:29.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:28 smithi045 ceph-mon[39715]: 4.19 scrub starts 2024-03-11T22:31:29.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:28 smithi045 ceph-mon[39715]: 4.19 scrub ok 2024-03-11T22:31:29.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:28 smithi045 ceph-mon[30178]: 1.1 scrub starts 2024-03-11T22:31:29.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:28 smithi045 ceph-mon[30178]: 1.1 scrub ok 2024-03-11T22:31:29.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:28 smithi045 ceph-mon[30178]: osdmap e547: 8 total, 8 up, 8 in 2024-03-11T22:31:29.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:28 smithi045 ceph-mon[30178]: 4.19 scrub starts 2024-03-11T22:31:29.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:28 smithi045 ceph-mon[30178]: 4.19 scrub ok 2024-03-11T22:31:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:28 smithi144 ceph-mon[33984]: 1.1 scrub starts 2024-03-11T22:31:29.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:28 smithi144 ceph-mon[33984]: 1.1 scrub ok 2024-03-11T22:31:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:28 smithi144 ceph-mon[33984]: osdmap e547: 8 total, 8 up, 8 in 2024-03-11T22:31:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:28 smithi144 ceph-mon[33984]: 4.19 scrub starts 2024-03-11T22:31:29.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:28 smithi144 ceph-mon[33984]: 4.19 scrub ok 2024-03-11T22:31:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:29 smithi045 ceph-mon[30178]: 4.d scrub starts 2024-03-11T22:31:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:29 smithi045 ceph-mon[30178]: 4.d scrub ok 2024-03-11T22:31:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:29 smithi045 ceph-mon[30178]: pgmap v1131: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:29 smithi045 ceph-mon[30178]: 3.1e scrub starts 2024-03-11T22:31:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:29 smithi045 ceph-mon[30178]: 3.1e scrub ok 2024-03-11T22:31:30.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:29 smithi045 ceph-mon[30178]: 6.1d scrub starts 2024-03-11T22:31:30.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:29 smithi045 ceph-mon[39715]: 4.d scrub starts 2024-03-11T22:31:30.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:29 smithi045 ceph-mon[39715]: 4.d scrub ok 2024-03-11T22:31:30.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:29 smithi045 ceph-mon[39715]: pgmap v1131: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:30.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:29 smithi045 ceph-mon[39715]: 3.1e scrub starts 2024-03-11T22:31:30.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:29 smithi045 ceph-mon[39715]: 3.1e scrub ok 2024-03-11T22:31:30.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:29 smithi045 ceph-mon[39715]: 6.1d scrub starts 2024-03-11T22:31:30.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:29 smithi144 ceph-mon[33984]: 4.d scrub starts 2024-03-11T22:31:30.217 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:29 smithi144 ceph-mon[33984]: 4.d scrub ok 2024-03-11T22:31:30.217 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:29 smithi144 ceph-mon[33984]: pgmap v1131: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:30.217 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:29 smithi144 ceph-mon[33984]: 3.1e scrub starts 2024-03-11T22:31:30.217 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:29 smithi144 ceph-mon[33984]: 3.1e scrub ok 2024-03-11T22:31:30.217 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:29 smithi144 ceph-mon[33984]: 6.1d scrub starts 2024-03-11T22:31:30.853 INFO:teuthology.orchestra.run.smithi045.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-11T22:31:31.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:30 smithi144 ceph-mon[33984]: 5.9 scrub starts 2024-03-11T22:31:31.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:30 smithi144 ceph-mon[33984]: 5.9 scrub ok 2024-03-11T22:31:31.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:30 smithi144 ceph-mon[33984]: 6.1d scrub ok 2024-03-11T22:31:31.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:30 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/474615074' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:31:31.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:30 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:31:31.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:30 smithi045 ceph-mon[30178]: 5.9 scrub starts 2024-03-11T22:31:31.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:30 smithi045 ceph-mon[30178]: 5.9 scrub ok 2024-03-11T22:31:31.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:30 smithi045 ceph-mon[30178]: 6.1d scrub ok 2024-03-11T22:31:31.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:30 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/474615074' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:31:31.233 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:30 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:31:31.233 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:30 smithi045 ceph-mon[39715]: 5.9 scrub starts 2024-03-11T22:31:31.233 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:30 smithi045 ceph-mon[39715]: 5.9 scrub ok 2024-03-11T22:31:31.233 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:30 smithi045 ceph-mon[39715]: 6.1d scrub ok 2024-03-11T22:31:31.234 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:30 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/474615074' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:31:31.234 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:30 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T22:31:31.670 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T22:31:31.917 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:32.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:31 smithi045 ceph-mon[39715]: pgmap v1132: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 759 B/s rd, 0 op/s 2024-03-11T22:31:32.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:31 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:31:32.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:31 smithi045 ceph-mon[39715]: osdmap e548: 8 total, 8 up, 8 in 2024-03-11T22:31:32.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:31 smithi045 ceph-mon[39715]: 6.1b scrub starts 2024-03-11T22:31:32.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:31 smithi045 ceph-mon[39715]: 6.1b scrub ok 2024-03-11T22:31:32.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:31 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:32.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:31 smithi045 ceph-mon[30178]: pgmap v1132: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 759 B/s rd, 0 op/s 2024-03-11T22:31:32.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:31 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:31:32.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:31 smithi045 ceph-mon[30178]: osdmap e548: 8 total, 8 up, 8 in 2024-03-11T22:31:32.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:31 smithi045 ceph-mon[30178]: 6.1b scrub starts 2024-03-11T22:31:32.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:31 smithi045 ceph-mon[30178]: 6.1b scrub ok 2024-03-11T22:31:32.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:31 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:31 smithi144 ceph-mon[33984]: pgmap v1132: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 759 B/s rd, 0 op/s 2024-03-11T22:31:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:31 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T22:31:32.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:31 smithi144 ceph-mon[33984]: osdmap e548: 8 total, 8 up, 8 in 2024-03-11T22:31:32.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:31 smithi144 ceph-mon[33984]: 6.1b scrub starts 2024-03-11T22:31:32.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:31 smithi144 ceph-mon[33984]: 6.1b scrub ok 2024-03-11T22:31:32.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:31 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:32.786 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:31:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:31:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:31:33.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:32 smithi144 ceph-mon[33984]: 6.1f scrub starts 2024-03-11T22:31:33.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:32 smithi144 ceph-mon[33984]: 6.1f scrub ok 2024-03-11T22:31:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:32 smithi045 ceph-mon[30178]: 6.1f scrub starts 2024-03-11T22:31:33.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:32 smithi045 ceph-mon[30178]: 6.1f scrub ok 2024-03-11T22:31:33.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:32 smithi045 ceph-mon[39715]: 6.1f scrub starts 2024-03-11T22:31:33.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:32 smithi045 ceph-mon[39715]: 6.1f scrub ok 2024-03-11T22:31:33.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:33 smithi045 ceph-mon[39715]: 6.f scrub starts 2024-03-11T22:31:33.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:33 smithi045 ceph-mon[39715]: 6.f scrub ok 2024-03-11T22:31:33.939 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:33 smithi045 ceph-mon[39715]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:33.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:33 smithi045 ceph-mon[39715]: 4.5 scrub starts 2024-03-11T22:31:33.940 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:33 smithi045 ceph-mon[39715]: 4.5 scrub ok 2024-03-11T22:31:33.940 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:33 smithi045 ceph-mon[30178]: 6.f scrub starts 2024-03-11T22:31:33.940 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:33 smithi045 ceph-mon[30178]: 6.f scrub ok 2024-03-11T22:31:33.940 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:33 smithi045 ceph-mon[30178]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:33.940 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:33 smithi045 ceph-mon[30178]: 4.5 scrub starts 2024-03-11T22:31:33.940 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:33 smithi045 ceph-mon[30178]: 4.5 scrub ok 2024-03-11T22:31:34.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:33 smithi144 ceph-mon[33984]: 6.f scrub starts 2024-03-11T22:31:34.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:33 smithi144 ceph-mon[33984]: 6.f scrub ok 2024-03-11T22:31:34.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:33 smithi144 ceph-mon[33984]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:34.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:33 smithi144 ceph-mon[33984]: 4.5 scrub starts 2024-03-11T22:31:34.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:33 smithi144 ceph-mon[33984]: 4.5 scrub ok 2024-03-11T22:31:34.859 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T22:31:35.252 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:34 smithi144 ceph-mon[33984]: 6.a scrub starts 2024-03-11T22:31:35.252 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:34 smithi144 ceph-mon[33984]: 6.a scrub ok 2024-03-11T22:31:35.252 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:34 smithi144 ceph-mon[33984]: 6.c scrub starts 2024-03-11T22:31:35.252 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:34 smithi144 ceph-mon[33984]: 6.c scrub ok 2024-03-11T22:31:35.252 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:34 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/762708082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:31:35.260 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:34 smithi045 ceph-mon[39715]: 6.a scrub starts 2024-03-11T22:31:35.260 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:34 smithi045 ceph-mon[39715]: 6.a scrub ok 2024-03-11T22:31:35.260 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:34 smithi045 ceph-mon[39715]: 6.c scrub starts 2024-03-11T22:31:35.260 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:34 smithi045 ceph-mon[39715]: 6.c scrub ok 2024-03-11T22:31:35.260 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:34 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/762708082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:31:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:34 smithi045 ceph-mon[30178]: 6.a scrub starts 2024-03-11T22:31:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:34 smithi045 ceph-mon[30178]: 6.a scrub ok 2024-03-11T22:31:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:34 smithi045 ceph-mon[30178]: 6.c scrub starts 2024-03-11T22:31:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:34 smithi045 ceph-mon[30178]: 6.c scrub ok 2024-03-11T22:31:35.261 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:34 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/762708082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:31:35.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-11T22:31:35.952 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:35 smithi045 ceph-mon[39715]: pgmap v1135: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T22:31:36.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:35 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/762708082' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:31:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:35 smithi045 ceph-mon[39715]: osdmap e549: 8 total, 8 up, 8 in 2024-03-11T22:31:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:35 smithi045 ceph-mon[39715]: 4.10 scrub starts 2024-03-11T22:31:36.039 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:35 smithi045 ceph-mon[39715]: 4.10 scrub ok 2024-03-11T22:31:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:35 smithi045 ceph-mon[30178]: pgmap v1135: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T22:31:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:35 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/762708082' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:31:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:35 smithi045 ceph-mon[30178]: osdmap e549: 8 total, 8 up, 8 in 2024-03-11T22:31:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:35 smithi045 ceph-mon[30178]: 4.10 scrub starts 2024-03-11T22:31:36.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:35 smithi045 ceph-mon[30178]: 4.10 scrub ok 2024-03-11T22:31:36.271 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:35 smithi144 ceph-mon[33984]: pgmap v1135: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T22:31:36.271 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:35 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/762708082' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:31:36.272 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:35 smithi144 ceph-mon[33984]: osdmap e549: 8 total, 8 up, 8 in 2024-03-11T22:31:36.272 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:35 smithi144 ceph-mon[33984]: 4.10 scrub starts 2024-03-11T22:31:36.272 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:35 smithi144 ceph-mon[33984]: 4.10 scrub ok 2024-03-11T22:31:37.009 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:36 smithi144 ceph-mon[33984]: 6.17 deep-scrub starts 2024-03-11T22:31:37.009 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:36 smithi144 ceph-mon[33984]: 4.8 deep-scrub starts 2024-03-11T22:31:37.009 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:36 smithi144 ceph-mon[33984]: 4.8 deep-scrub ok 2024-03-11T22:31:37.009 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:36 smithi144 ceph-mon[33984]: 6.17 deep-scrub ok 2024-03-11T22:31:37.009 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:36 smithi144 ceph-mon[33984]: osdmap e550: 8 total, 8 up, 8 in 2024-03-11T22:31:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:36 smithi045 ceph-mon[39715]: 6.17 deep-scrub starts 2024-03-11T22:31:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:36 smithi045 ceph-mon[39715]: 4.8 deep-scrub starts 2024-03-11T22:31:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:36 smithi045 ceph-mon[39715]: 4.8 deep-scrub ok 2024-03-11T22:31:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:36 smithi045 ceph-mon[39715]: 6.17 deep-scrub ok 2024-03-11T22:31:37.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:36 smithi045 ceph-mon[39715]: osdmap e550: 8 total, 8 up, 8 in 2024-03-11T22:31:37.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:36 smithi045 ceph-mon[30178]: 6.17 deep-scrub starts 2024-03-11T22:31:37.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:36 smithi045 ceph-mon[30178]: 4.8 deep-scrub starts 2024-03-11T22:31:37.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:36 smithi045 ceph-mon[30178]: 4.8 deep-scrub ok 2024-03-11T22:31:37.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:36 smithi045 ceph-mon[30178]: 6.17 deep-scrub ok 2024-03-11T22:31:37.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:36 smithi045 ceph-mon[30178]: osdmap e550: 8 total, 8 up, 8 in 2024-03-11T22:31:37.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:31:37 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:31:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:37 smithi045 ceph-mon[30178]: 3.f scrub starts 2024-03-11T22:31:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:37 smithi045 ceph-mon[30178]: 3.f scrub ok 2024-03-11T22:31:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:37 smithi045 ceph-mon[30178]: pgmap v1137: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:31:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:37 smithi045 ceph-mon[30178]: 1.0 scrub starts 2024-03-11T22:31:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:37 smithi045 ceph-mon[30178]: 1.0 scrub ok 2024-03-11T22:31:38.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:37 smithi045 ceph-mon[39715]: 3.f scrub starts 2024-03-11T22:31:38.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:37 smithi045 ceph-mon[39715]: 3.f scrub ok 2024-03-11T22:31:38.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:37 smithi045 ceph-mon[39715]: pgmap v1137: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:31:38.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:37 smithi045 ceph-mon[39715]: 1.0 scrub starts 2024-03-11T22:31:38.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:37 smithi045 ceph-mon[39715]: 1.0 scrub ok 2024-03-11T22:31:38.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:37 smithi144 ceph-mon[33984]: 3.f scrub starts 2024-03-11T22:31:38.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:37 smithi144 ceph-mon[33984]: 3.f scrub ok 2024-03-11T22:31:38.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:37 smithi144 ceph-mon[33984]: pgmap v1137: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:31:38.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:37 smithi144 ceph-mon[33984]: 1.0 scrub starts 2024-03-11T22:31:38.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:37 smithi144 ceph-mon[33984]: 1.0 scrub ok 2024-03-11T22:31:38.873 INFO:teuthology.orchestra.run.smithi045.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-11T22:31:39.178 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:39.179 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[30178]: 5.e scrub starts 2024-03-11T22:31:39.179 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[30178]: 5.e scrub ok 2024-03-11T22:31:39.179 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[30178]: 5.18 scrub starts 2024-03-11T22:31:39.179 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[30178]: 5.18 scrub ok 2024-03-11T22:31:39.179 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2221127631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:31:39.179 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[30178]: 4.2 scrub starts 2024-03-11T22:31:39.179 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[30178]: 4.2 scrub ok 2024-03-11T22:31:39.179 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:39.179 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[39715]: 5.e scrub starts 2024-03-11T22:31:39.179 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[39715]: 5.e scrub ok 2024-03-11T22:31:39.180 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[39715]: 5.18 scrub starts 2024-03-11T22:31:39.180 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[39715]: 5.18 scrub ok 2024-03-11T22:31:39.180 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2221127631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:31:39.180 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[39715]: 4.2 scrub starts 2024-03-11T22:31:39.180 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:38 smithi045 ceph-mon[39715]: 4.2 scrub ok 2024-03-11T22:31:39.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:38 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:39.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:38 smithi144 ceph-mon[33984]: 5.e scrub starts 2024-03-11T22:31:39.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:38 smithi144 ceph-mon[33984]: 5.e scrub ok 2024-03-11T22:31:39.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:38 smithi144 ceph-mon[33984]: 5.18 scrub starts 2024-03-11T22:31:39.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:38 smithi144 ceph-mon[33984]: 5.18 scrub ok 2024-03-11T22:31:39.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:38 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2221127631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T22:31:39.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:38 smithi144 ceph-mon[33984]: 4.2 scrub starts 2024-03-11T22:31:39.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:38 smithi144 ceph-mon[33984]: 4.2 scrub ok 2024-03-11T22:31:39.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T22:31:39.900 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:40.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[39715]: 3.12 scrub starts 2024-03-11T22:31:40.143 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[39715]: 3.12 scrub ok 2024-03-11T22:31:40.144 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[39715]: pgmap v1139: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T22:31:40.144 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:31:40.144 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:31:40.144 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2221127631' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:31:40.144 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[39715]: osdmap e551: 8 total, 8 up, 8 in 2024-03-11T22:31:40.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[30178]: 3.12 scrub starts 2024-03-11T22:31:40.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[30178]: 3.12 scrub ok 2024-03-11T22:31:40.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[30178]: pgmap v1139: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T22:31:40.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:31:40.144 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:31:40.145 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2221127631' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:31:40.145 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:39 smithi045 ceph-mon[30178]: osdmap e551: 8 total, 8 up, 8 in 2024-03-11T22:31:40.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:39 smithi144 ceph-mon[33984]: 3.12 scrub starts 2024-03-11T22:31:40.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:39 smithi144 ceph-mon[33984]: 3.12 scrub ok 2024-03-11T22:31:40.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:39 smithi144 ceph-mon[33984]: pgmap v1139: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T22:31:40.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:39 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:31:40.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:39 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:31:40.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:39 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2221127631' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T22:31:40.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:39 smithi144 ceph-mon[33984]: osdmap e551: 8 total, 8 up, 8 in 2024-03-11T22:31:41.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:40 smithi045 ceph-mon[30178]: 3.5 deep-scrub starts 2024-03-11T22:31:41.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:40 smithi045 ceph-mon[30178]: 3.5 deep-scrub ok 2024-03-11T22:31:41.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:40 smithi045 ceph-mon[30178]: 6.5 scrub starts 2024-03-11T22:31:41.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:40 smithi045 ceph-mon[30178]: 6.5 scrub ok 2024-03-11T22:31:41.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:40 smithi045 ceph-mon[39715]: 3.5 deep-scrub starts 2024-03-11T22:31:41.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:40 smithi045 ceph-mon[39715]: 3.5 deep-scrub ok 2024-03-11T22:31:41.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:40 smithi045 ceph-mon[39715]: 6.5 scrub starts 2024-03-11T22:31:41.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:40 smithi045 ceph-mon[39715]: 6.5 scrub ok 2024-03-11T22:31:41.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:40 smithi144 ceph-mon[33984]: 3.5 deep-scrub starts 2024-03-11T22:31:41.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:40 smithi144 ceph-mon[33984]: 3.5 deep-scrub ok 2024-03-11T22:31:41.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:40 smithi144 ceph-mon[33984]: 6.5 scrub starts 2024-03-11T22:31:41.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:40 smithi144 ceph-mon[33984]: 6.5 scrub ok 2024-03-11T22:31:42.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:41 smithi045 ceph-mon[39715]: 5.15 scrub starts 2024-03-11T22:31:42.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:41 smithi045 ceph-mon[39715]: 5.15 scrub ok 2024-03-11T22:31:42.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:41 smithi045 ceph-mon[39715]: pgmap v1141: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:42.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:41 smithi045 ceph-mon[39715]: 1.4 scrub starts 2024-03-11T22:31:42.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:41 smithi045 ceph-mon[39715]: 1.4 scrub ok 2024-03-11T22:31:42.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:41 smithi045 ceph-mon[39715]: osdmap e552: 8 total, 8 up, 8 in 2024-03-11T22:31:42.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:41 smithi045 ceph-mon[30178]: 5.15 scrub starts 2024-03-11T22:31:42.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:41 smithi045 ceph-mon[30178]: 5.15 scrub ok 2024-03-11T22:31:42.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:41 smithi045 ceph-mon[30178]: pgmap v1141: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:42.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:41 smithi045 ceph-mon[30178]: 1.4 scrub starts 2024-03-11T22:31:42.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:41 smithi045 ceph-mon[30178]: 1.4 scrub ok 2024-03-11T22:31:42.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:41 smithi045 ceph-mon[30178]: osdmap e552: 8 total, 8 up, 8 in 2024-03-11T22:31:42.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:41 smithi144 ceph-mon[33984]: 5.15 scrub starts 2024-03-11T22:31:42.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:41 smithi144 ceph-mon[33984]: 5.15 scrub ok 2024-03-11T22:31:42.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:41 smithi144 ceph-mon[33984]: pgmap v1141: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:42.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:41 smithi144 ceph-mon[33984]: 1.4 scrub starts 2024-03-11T22:31:42.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:41 smithi144 ceph-mon[33984]: 1.4 scrub ok 2024-03-11T22:31:42.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:41 smithi144 ceph-mon[33984]: osdmap e552: 8 total, 8 up, 8 in 2024-03-11T22:31:42.889 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:31:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:31:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:31:42.890 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T22:31:43.231 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:42 smithi045 ceph-mon[39715]: 6.1e deep-scrub starts 2024-03-11T22:31:43.231 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:42 smithi045 ceph-mon[39715]: 6.1e deep-scrub ok 2024-03-11T22:31:43.231 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:42 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1377020713' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:31:43.232 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:42 smithi045 ceph-mon[30178]: 6.1e deep-scrub starts 2024-03-11T22:31:43.232 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:42 smithi045 ceph-mon[30178]: 6.1e deep-scrub ok 2024-03-11T22:31:43.232 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:42 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1377020713' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:31:43.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:42 smithi144 ceph-mon[33984]: 6.1e deep-scrub starts 2024-03-11T22:31:43.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:42 smithi144 ceph-mon[33984]: 6.1e deep-scrub ok 2024-03-11T22:31:43.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:42 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1377020713' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:31:43.691 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-11T22:31:43.941 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:44.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:43 smithi045 ceph-mon[39715]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:44.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:43 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1377020713' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:31:44.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:43 smithi045 ceph-mon[39715]: osdmap e553: 8 total, 8 up, 8 in 2024-03-11T22:31:44.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:43 smithi045 ceph-mon[39715]: 3.1d scrub starts 2024-03-11T22:31:44.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:43 smithi045 ceph-mon[39715]: 3.1d scrub ok 2024-03-11T22:31:44.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:43 smithi045 ceph-mon[30178]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:44.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:43 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1377020713' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:31:44.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:43 smithi045 ceph-mon[30178]: osdmap e553: 8 total, 8 up, 8 in 2024-03-11T22:31:44.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:43 smithi045 ceph-mon[30178]: 3.1d scrub starts 2024-03-11T22:31:44.039 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:43 smithi045 ceph-mon[30178]: 3.1d scrub ok 2024-03-11T22:31:44.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:43 smithi144 ceph-mon[33984]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:44.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:43 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1377020713' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:31:44.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:43 smithi144 ceph-mon[33984]: osdmap e553: 8 total, 8 up, 8 in 2024-03-11T22:31:44.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:43 smithi144 ceph-mon[33984]: 3.1d scrub starts 2024-03-11T22:31:44.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:43 smithi144 ceph-mon[33984]: 3.1d scrub ok 2024-03-11T22:31:45.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[39715]: 3.1f scrub starts 2024-03-11T22:31:45.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[39715]: 3.1f scrub ok 2024-03-11T22:31:45.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[39715]: 4.11 scrub starts 2024-03-11T22:31:45.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[39715]: 4.11 scrub ok 2024-03-11T22:31:45.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[39715]: 5.1b scrub starts 2024-03-11T22:31:45.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[39715]: 5.1b scrub ok 2024-03-11T22:31:45.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[39715]: 3.e scrub starts 2024-03-11T22:31:45.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[39715]: 3.e scrub ok 2024-03-11T22:31:45.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[30178]: 3.1f scrub starts 2024-03-11T22:31:45.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[30178]: 3.1f scrub ok 2024-03-11T22:31:45.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[30178]: 4.11 scrub starts 2024-03-11T22:31:45.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[30178]: 4.11 scrub ok 2024-03-11T22:31:45.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[30178]: 5.1b scrub starts 2024-03-11T22:31:45.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[30178]: 5.1b scrub ok 2024-03-11T22:31:45.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[30178]: 3.e scrub starts 2024-03-11T22:31:45.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:44 smithi045 ceph-mon[30178]: 3.e scrub ok 2024-03-11T22:31:45.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:44 smithi144 ceph-mon[33984]: 3.1f scrub starts 2024-03-11T22:31:45.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:44 smithi144 ceph-mon[33984]: 3.1f scrub ok 2024-03-11T22:31:45.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:44 smithi144 ceph-mon[33984]: 4.11 scrub starts 2024-03-11T22:31:45.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:44 smithi144 ceph-mon[33984]: 4.11 scrub ok 2024-03-11T22:31:45.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:44 smithi144 ceph-mon[33984]: 5.1b scrub starts 2024-03-11T22:31:45.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:44 smithi144 ceph-mon[33984]: 5.1b scrub ok 2024-03-11T22:31:45.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:44 smithi144 ceph-mon[33984]: 3.e scrub starts 2024-03-11T22:31:45.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:44 smithi144 ceph-mon[33984]: 3.e scrub ok 2024-03-11T22:31:45.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:45 smithi045 ceph-mon[30178]: 4.1c scrub starts 2024-03-11T22:31:45.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:45 smithi045 ceph-mon[30178]: 4.1c scrub ok 2024-03-11T22:31:45.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:45 smithi045 ceph-mon[30178]: pgmap v1145: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:45.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:45 smithi045 ceph-mon[30178]: 5.11 scrub starts 2024-03-11T22:31:45.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:45 smithi045 ceph-mon[30178]: 5.11 scrub ok 2024-03-11T22:31:45.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:45 smithi045 ceph-mon[39715]: 4.1c scrub starts 2024-03-11T22:31:45.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:45 smithi045 ceph-mon[39715]: 4.1c scrub ok 2024-03-11T22:31:45.962 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:45 smithi045 ceph-mon[39715]: pgmap v1145: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:45.963 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:45 smithi045 ceph-mon[39715]: 5.11 scrub starts 2024-03-11T22:31:45.963 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:45 smithi045 ceph-mon[39715]: 5.11 scrub ok 2024-03-11T22:31:46.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:45 smithi144 ceph-mon[33984]: 4.1c scrub starts 2024-03-11T22:31:46.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:45 smithi144 ceph-mon[33984]: 4.1c scrub ok 2024-03-11T22:31:46.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:45 smithi144 ceph-mon[33984]: pgmap v1145: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:46.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:45 smithi144 ceph-mon[33984]: 5.11 scrub starts 2024-03-11T22:31:46.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:45 smithi144 ceph-mon[33984]: 5.11 scrub ok 2024-03-11T22:31:46.668 INFO:teuthology.orchestra.run.smithi045.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-11T22:31:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[30178]: 2.1 scrub starts 2024-03-11T22:31:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[30178]: 2.1 scrub ok 2024-03-11T22:31:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[30178]: 4.16 scrub starts 2024-03-11T22:31:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[30178]: 4.16 scrub ok 2024-03-11T22:31:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[30178]: 6.1c deep-scrub starts 2024-03-11T22:31:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[30178]: 6.1c deep-scrub ok 2024-03-11T22:31:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3631902393' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:31:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[30178]: 3.19 scrub starts 2024-03-11T22:31:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[30178]: 3.19 scrub ok 2024-03-11T22:31:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3631902393' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:31:47.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[30178]: osdmap e554: 8 total, 8 up, 8 in 2024-03-11T22:31:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[39715]: 2.1 scrub starts 2024-03-11T22:31:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[39715]: 2.1 scrub ok 2024-03-11T22:31:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[39715]: 4.16 scrub starts 2024-03-11T22:31:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[39715]: 4.16 scrub ok 2024-03-11T22:31:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[39715]: 6.1c deep-scrub starts 2024-03-11T22:31:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[39715]: 6.1c deep-scrub ok 2024-03-11T22:31:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3631902393' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:31:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[39715]: 3.19 scrub starts 2024-03-11T22:31:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[39715]: 3.19 scrub ok 2024-03-11T22:31:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3631902393' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:31:47.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:46 smithi045 ceph-mon[39715]: osdmap e554: 8 total, 8 up, 8 in 2024-03-11T22:31:47.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:31:47 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:31:47.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:46 smithi144 ceph-mon[33984]: 2.1 scrub starts 2024-03-11T22:31:47.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:46 smithi144 ceph-mon[33984]: 2.1 scrub ok 2024-03-11T22:31:47.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:46 smithi144 ceph-mon[33984]: 4.16 scrub starts 2024-03-11T22:31:47.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:46 smithi144 ceph-mon[33984]: 4.16 scrub ok 2024-03-11T22:31:47.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:46 smithi144 ceph-mon[33984]: 6.1c deep-scrub starts 2024-03-11T22:31:47.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:46 smithi144 ceph-mon[33984]: 6.1c deep-scrub ok 2024-03-11T22:31:47.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:46 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3631902393' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T22:31:47.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:46 smithi144 ceph-mon[33984]: 3.19 scrub starts 2024-03-11T22:31:47.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:46 smithi144 ceph-mon[33984]: 3.19 scrub ok 2024-03-11T22:31:47.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:31:47.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:46 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3631902393' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-11T22:31:47.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:46 smithi144 ceph-mon[33984]: osdmap e554: 8 total, 8 up, 8 in 2024-03-11T22:31:47.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T22:31:47.697 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:48.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[39715]: 5.5 scrub starts 2024-03-11T22:31:48.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[39715]: 5.5 scrub ok 2024-03-11T22:31:48.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[39715]: pgmap v1146: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 666 B/s rd, 0 op/s 2024-03-11T22:31:48.112 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[39715]: 4.a scrub starts 2024-03-11T22:31:48.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[39715]: 4.a scrub ok 2024-03-11T22:31:48.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[39715]: 5.d scrub starts 2024-03-11T22:31:48.113 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[39715]: 5.d scrub ok 2024-03-11T22:31:48.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[30178]: 5.5 scrub starts 2024-03-11T22:31:48.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[30178]: 5.5 scrub ok 2024-03-11T22:31:48.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[30178]: pgmap v1146: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 666 B/s rd, 0 op/s 2024-03-11T22:31:48.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[30178]: 4.a scrub starts 2024-03-11T22:31:48.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[30178]: 4.a scrub ok 2024-03-11T22:31:48.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[30178]: 5.d scrub starts 2024-03-11T22:31:48.113 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:47 smithi045 ceph-mon[30178]: 5.d scrub ok 2024-03-11T22:31:48.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:47 smithi144 ceph-mon[33984]: 5.5 scrub starts 2024-03-11T22:31:48.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:47 smithi144 ceph-mon[33984]: 5.5 scrub ok 2024-03-11T22:31:48.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:47 smithi144 ceph-mon[33984]: pgmap v1146: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 666 B/s rd, 0 op/s 2024-03-11T22:31:48.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:47 smithi144 ceph-mon[33984]: 4.a scrub starts 2024-03-11T22:31:48.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:47 smithi144 ceph-mon[33984]: 4.a scrub ok 2024-03-11T22:31:48.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:47 smithi144 ceph-mon[33984]: 5.d scrub starts 2024-03-11T22:31:48.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:47 smithi144 ceph-mon[33984]: 5.d scrub ok 2024-03-11T22:31:49.196 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:48 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:49.196 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:48 smithi045 ceph-mon[39715]: 4.4 scrub starts 2024-03-11T22:31:49.196 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:48 smithi045 ceph-mon[39715]: 4.4 scrub ok 2024-03-11T22:31:49.196 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:48 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:49.196 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:48 smithi045 ceph-mon[30178]: 4.4 scrub starts 2024-03-11T22:31:49.197 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:48 smithi045 ceph-mon[30178]: 4.4 scrub ok 2024-03-11T22:31:49.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:48 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:49.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:48 smithi144 ceph-mon[33984]: 4.4 scrub starts 2024-03-11T22:31:49.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:48 smithi144 ceph-mon[33984]: 4.4 scrub ok 2024-03-11T22:31:49.915 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T22:31:50.243 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:49 smithi045 ceph-mon[39715]: pgmap v1148: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:31:50.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:49 smithi045 ceph-mon[39715]: 6.11 scrub starts 2024-03-11T22:31:50.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:49 smithi045 ceph-mon[39715]: 6.11 scrub ok 2024-03-11T22:31:50.244 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:49 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2848279617' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:31:50.244 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:49 smithi045 ceph-mon[30178]: pgmap v1148: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:31:50.245 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:49 smithi045 ceph-mon[30178]: 6.11 scrub starts 2024-03-11T22:31:50.245 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:49 smithi045 ceph-mon[30178]: 6.11 scrub ok 2024-03-11T22:31:50.245 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:49 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2848279617' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:31:50.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:49 smithi144 ceph-mon[33984]: pgmap v1148: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:31:50.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:49 smithi144 ceph-mon[33984]: 6.11 scrub starts 2024-03-11T22:31:50.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:49 smithi144 ceph-mon[33984]: 6.11 scrub ok 2024-03-11T22:31:50.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:49 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2848279617' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:31:50.562 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-11T22:31:50.815 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:51.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:50 smithi045 ceph-mon[30178]: 5.16 scrub starts 2024-03-11T22:31:51.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:50 smithi045 ceph-mon[30178]: 5.16 scrub ok 2024-03-11T22:31:51.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:50 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2848279617' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:31:51.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:50 smithi045 ceph-mon[30178]: osdmap e555: 8 total, 8 up, 8 in 2024-03-11T22:31:51.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:50 smithi045 ceph-mon[30178]: 3.15 scrub starts 2024-03-11T22:31:51.255 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:50 smithi045 ceph-mon[30178]: 3.15 scrub ok 2024-03-11T22:31:51.255 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:50 smithi045 ceph-mon[39715]: 5.16 scrub starts 2024-03-11T22:31:51.255 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:50 smithi045 ceph-mon[39715]: 5.16 scrub ok 2024-03-11T22:31:51.256 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:50 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2848279617' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:31:51.256 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:50 smithi045 ceph-mon[39715]: osdmap e555: 8 total, 8 up, 8 in 2024-03-11T22:31:51.256 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:50 smithi045 ceph-mon[39715]: 3.15 scrub starts 2024-03-11T22:31:51.256 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:50 smithi045 ceph-mon[39715]: 3.15 scrub ok 2024-03-11T22:31:51.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:50 smithi144 ceph-mon[33984]: 5.16 scrub starts 2024-03-11T22:31:51.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:50 smithi144 ceph-mon[33984]: 5.16 scrub ok 2024-03-11T22:31:51.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:50 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2848279617' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:31:51.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:50 smithi144 ceph-mon[33984]: osdmap e555: 8 total, 8 up, 8 in 2024-03-11T22:31:51.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:50 smithi144 ceph-mon[33984]: 3.15 scrub starts 2024-03-11T22:31:51.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:50 smithi144 ceph-mon[33984]: 3.15 scrub ok 2024-03-11T22:31:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[39715]: 3.1 deep-scrub starts 2024-03-11T22:31:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[39715]: 3.1 deep-scrub ok 2024-03-11T22:31:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[39715]: pgmap v1150: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 667 B/s rd, 0 op/s 2024-03-11T22:31:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:31:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:31:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[39715]: osdmap e556: 8 total, 8 up, 8 in 2024-03-11T22:31:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[39715]: 5.b scrub starts 2024-03-11T22:31:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[39715]: 5.b scrub ok 2024-03-11T22:31:52.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:31:52.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:31:52.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[30178]: 3.1 deep-scrub starts 2024-03-11T22:31:52.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[30178]: 3.1 deep-scrub ok 2024-03-11T22:31:52.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[30178]: pgmap v1150: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 667 B/s rd, 0 op/s 2024-03-11T22:31:52.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:31:52.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:31:52.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[30178]: osdmap e556: 8 total, 8 up, 8 in 2024-03-11T22:31:52.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[30178]: 5.b scrub starts 2024-03-11T22:31:52.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[30178]: 5.b scrub ok 2024-03-11T22:31:52.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:31:52.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:51 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:31:52.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:51 smithi144 ceph-mon[33984]: 3.1 deep-scrub starts 2024-03-11T22:31:52.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:51 smithi144 ceph-mon[33984]: 3.1 deep-scrub ok 2024-03-11T22:31:52.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:51 smithi144 ceph-mon[33984]: pgmap v1150: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 667 B/s rd, 0 op/s 2024-03-11T22:31:52.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:51 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:31:52.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:51 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:31:52.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:51 smithi144 ceph-mon[33984]: osdmap e556: 8 total, 8 up, 8 in 2024-03-11T22:31:52.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:51 smithi144 ceph-mon[33984]: 5.b scrub starts 2024-03-11T22:31:52.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:51 smithi144 ceph-mon[33984]: 5.b scrub ok 2024-03-11T22:31:52.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:51 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:31:52.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:51 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:31:52.787 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:31:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:31:52.934 INFO:teuthology.orchestra.run.smithi045.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-11T22:31:53.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[39715]: osdmap e557: 8 total, 8 up, 8 in 2024-03-11T22:31:53.284 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[39715]: 6.8 scrub starts 2024-03-11T22:31:53.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[39715]: 6.8 scrub ok 2024-03-11T22:31:53.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[39715]: 5.19 scrub starts 2024-03-11T22:31:53.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[39715]: 5.19 scrub ok 2024-03-11T22:31:53.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/526252509' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:31:53.285 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:31:53.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[30178]: osdmap e557: 8 total, 8 up, 8 in 2024-03-11T22:31:53.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[30178]: 6.8 scrub starts 2024-03-11T22:31:53.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[30178]: 6.8 scrub ok 2024-03-11T22:31:53.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[30178]: 5.19 scrub starts 2024-03-11T22:31:53.285 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[30178]: 5.19 scrub ok 2024-03-11T22:31:53.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/526252509' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:31:53.286 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:52 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:31:53.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:52 smithi144 ceph-mon[33984]: osdmap e557: 8 total, 8 up, 8 in 2024-03-11T22:31:53.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:52 smithi144 ceph-mon[33984]: 6.8 scrub starts 2024-03-11T22:31:53.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:52 smithi144 ceph-mon[33984]: 6.8 scrub ok 2024-03-11T22:31:53.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:52 smithi144 ceph-mon[33984]: 5.19 scrub starts 2024-03-11T22:31:53.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:52 smithi144 ceph-mon[33984]: 5.19 scrub ok 2024-03-11T22:31:53.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:52 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/526252509' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:31:53.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:52 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T22:31:53.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T22:31:53.934 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:54.191 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:53 smithi045 ceph-mon[39715]: pgmap v1153: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:54.191 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:53 smithi045 ceph-mon[39715]: 4.e scrub starts 2024-03-11T22:31:54.191 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:53 smithi045 ceph-mon[39715]: 4.e scrub ok 2024-03-11T22:31:54.191 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:53 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:31:54.191 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:53 smithi045 ceph-mon[39715]: osdmap e558: 8 total, 8 up, 8 in 2024-03-11T22:31:54.191 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:53 smithi045 ceph-mon[30178]: pgmap v1153: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:54.192 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:53 smithi045 ceph-mon[30178]: 4.e scrub starts 2024-03-11T22:31:54.192 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:53 smithi045 ceph-mon[30178]: 4.e scrub ok 2024-03-11T22:31:54.192 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:53 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:31:54.192 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:53 smithi045 ceph-mon[30178]: osdmap e558: 8 total, 8 up, 8 in 2024-03-11T22:31:54.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:53 smithi144 ceph-mon[33984]: pgmap v1153: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:31:54.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:53 smithi144 ceph-mon[33984]: 4.e scrub starts 2024-03-11T22:31:54.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:53 smithi144 ceph-mon[33984]: 4.e scrub ok 2024-03-11T22:31:54.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:53 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-11T22:31:54.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:53 smithi144 ceph-mon[33984]: osdmap e558: 8 total, 8 up, 8 in 2024-03-11T22:31:55.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:54 smithi045 ceph-mon[39715]: 5.7 deep-scrub starts 2024-03-11T22:31:55.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:54 smithi045 ceph-mon[39715]: 5.7 deep-scrub ok 2024-03-11T22:31:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:54 smithi045 ceph-mon[30178]: 5.7 deep-scrub starts 2024-03-11T22:31:55.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:54 smithi045 ceph-mon[30178]: 5.7 deep-scrub ok 2024-03-11T22:31:55.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:54 smithi144 ceph-mon[33984]: 5.7 deep-scrub starts 2024-03-11T22:31:55.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:54 smithi144 ceph-mon[33984]: 5.7 deep-scrub ok 2024-03-11T22:31:56.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:55 smithi045 ceph-mon[30178]: pgmap v1155: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:31:56.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:55 smithi045 ceph-mon[30178]: 6.4 scrub starts 2024-03-11T22:31:56.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:55 smithi045 ceph-mon[30178]: 6.4 scrub ok 2024-03-11T22:31:56.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:55 smithi045 ceph-mon[39715]: pgmap v1155: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:31:56.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:55 smithi045 ceph-mon[39715]: 6.4 scrub starts 2024-03-11T22:31:56.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:55 smithi045 ceph-mon[39715]: 6.4 scrub ok 2024-03-11T22:31:56.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:55 smithi144 ceph-mon[33984]: pgmap v1155: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T22:31:56.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:55 smithi144 ceph-mon[33984]: 6.4 scrub starts 2024-03-11T22:31:56.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:55 smithi144 ceph-mon[33984]: 6.4 scrub ok 2024-03-11T22:31:56.956 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T22:31:57.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:56 smithi045 ceph-mon[30178]: 6.16 scrub starts 2024-03-11T22:31:57.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:56 smithi045 ceph-mon[30178]: 6.16 scrub ok 2024-03-11T22:31:57.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:56 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/345293892' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:31:57.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:56 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:31:57.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:56 smithi045 ceph-mon[30178]: 5.13 scrub starts 2024-03-11T22:31:57.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:56 smithi045 ceph-mon[30178]: 5.13 scrub ok 2024-03-11T22:31:57.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:56 smithi045 ceph-mon[39715]: 6.16 scrub starts 2024-03-11T22:31:57.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:56 smithi045 ceph-mon[39715]: 6.16 scrub ok 2024-03-11T22:31:57.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:56 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/345293892' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:31:57.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:56 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:31:57.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:56 smithi045 ceph-mon[39715]: 5.13 scrub starts 2024-03-11T22:31:57.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:56 smithi045 ceph-mon[39715]: 5.13 scrub ok 2024-03-11T22:31:57.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:31:57 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:31:57.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:56 smithi144 ceph-mon[33984]: 6.16 scrub starts 2024-03-11T22:31:57.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:56 smithi144 ceph-mon[33984]: 6.16 scrub ok 2024-03-11T22:31:57.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:56 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/345293892' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:31:57.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:56 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:31:57.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:56 smithi144 ceph-mon[33984]: 5.13 scrub starts 2024-03-11T22:31:57.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:56 smithi144 ceph-mon[33984]: 5.13 scrub ok 2024-03-11T22:31:57.821 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-11T22:31:58.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:57 smithi045 ceph-mon[30178]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:58.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:57 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:31:58.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:57 smithi045 ceph-mon[30178]: osdmap e559: 8 total, 8 up, 8 in 2024-03-11T22:31:58.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:57 smithi045 ceph-mon[39715]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:58.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:57 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:31:58.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:57 smithi045 ceph-mon[39715]: osdmap e559: 8 total, 8 up, 8 in 2024-03-11T22:31:58.076 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:31:58.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:57 smithi144 ceph-mon[33984]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:31:58.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:57 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:31:58.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:57 smithi144 ceph-mon[33984]: osdmap e559: 8 total, 8 up, 8 in 2024-03-11T22:31:59.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:58 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:59.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:58 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:31:59.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:58 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:00.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:59 smithi045 ceph-mon[39715]: 6.9 scrub starts 2024-03-11T22:32:00.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:59 smithi045 ceph-mon[39715]: pgmap v1158: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-03-11T22:32:00.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:59 smithi045 ceph-mon[39715]: 6.9 scrub ok 2024-03-11T22:32:00.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:59 smithi045 ceph-mon[39715]: 4.c scrub starts 2024-03-11T22:32:00.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:31:59 smithi045 ceph-mon[39715]: 4.c scrub ok 2024-03-11T22:32:00.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:59 smithi045 ceph-mon[30178]: 6.9 scrub starts 2024-03-11T22:32:00.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:59 smithi045 ceph-mon[30178]: pgmap v1158: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-03-11T22:32:00.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:59 smithi045 ceph-mon[30178]: 6.9 scrub ok 2024-03-11T22:32:00.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:59 smithi045 ceph-mon[30178]: 4.c scrub starts 2024-03-11T22:32:00.289 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:31:59 smithi045 ceph-mon[30178]: 4.c scrub ok 2024-03-11T22:32:00.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:59 smithi144 ceph-mon[33984]: 6.9 scrub starts 2024-03-11T22:32:00.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:59 smithi144 ceph-mon[33984]: pgmap v1158: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-03-11T22:32:00.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:59 smithi144 ceph-mon[33984]: 6.9 scrub ok 2024-03-11T22:32:00.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:59 smithi144 ceph-mon[33984]: 4.c scrub starts 2024-03-11T22:32:00.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:31:59 smithi144 ceph-mon[33984]: 4.c scrub ok 2024-03-11T22:32:00.972 INFO:teuthology.orchestra.run.smithi045.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-11T22:32:01.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:00 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/966312574' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:32:01.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:00 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:32:01.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:00 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/966312574' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:32:01.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:00 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:32:01.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:00 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/966312574' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:32:01.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:00 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T22:32:01.813 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T22:32:02.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[39715]: 3.1c scrub starts 2024-03-11T22:32:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[39715]: 3.1c scrub ok 2024-03-11T22:32:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[39715]: pgmap v1159: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:32:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:32:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[39715]: osdmap e560: 8 total, 8 up, 8 in 2024-03-11T22:32:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[39715]: 3.7 scrub starts 2024-03-11T22:32:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[39715]: 3.7 scrub ok 2024-03-11T22:32:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[39715]: 6.14 scrub starts 2024-03-11T22:32:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[39715]: 6.14 scrub ok 2024-03-11T22:32:02.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:02.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[30178]: 3.1c scrub starts 2024-03-11T22:32:02.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[30178]: 3.1c scrub ok 2024-03-11T22:32:02.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[30178]: pgmap v1159: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:32:02.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:32:02.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[30178]: osdmap e560: 8 total, 8 up, 8 in 2024-03-11T22:32:02.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[30178]: 3.7 scrub starts 2024-03-11T22:32:02.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[30178]: 3.7 scrub ok 2024-03-11T22:32:02.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[30178]: 6.14 scrub starts 2024-03-11T22:32:02.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[30178]: 6.14 scrub ok 2024-03-11T22:32:02.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:02.074 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:02.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:01 smithi144 ceph-mon[33984]: 3.1c scrub starts 2024-03-11T22:32:02.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:01 smithi144 ceph-mon[33984]: 3.1c scrub ok 2024-03-11T22:32:02.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:01 smithi144 ceph-mon[33984]: pgmap v1159: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T22:32:02.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:01 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-11T22:32:02.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:01 smithi144 ceph-mon[33984]: osdmap e560: 8 total, 8 up, 8 in 2024-03-11T22:32:02.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:01 smithi144 ceph-mon[33984]: 3.7 scrub starts 2024-03-11T22:32:02.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:01 smithi144 ceph-mon[33984]: 3.7 scrub ok 2024-03-11T22:32:02.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:01 smithi144 ceph-mon[33984]: 6.14 scrub starts 2024-03-11T22:32:02.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:01 smithi144 ceph-mon[33984]: 6.14 scrub ok 2024-03-11T22:32:02.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:02.787 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:32:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:32:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:32:03.265 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:02 smithi045 ceph-mon[39715]: osdmap e561: 8 total, 8 up, 8 in 2024-03-11T22:32:03.265 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:02 smithi045 ceph-mon[39715]: 3.b deep-scrub starts 2024-03-11T22:32:03.265 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:02 smithi045 ceph-mon[39715]: 1.5 scrub starts 2024-03-11T22:32:03.265 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:02 smithi045 ceph-mon[39715]: 3.b deep-scrub ok 2024-03-11T22:32:03.266 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:02 smithi045 ceph-mon[39715]: 1.5 scrub ok 2024-03-11T22:32:03.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:02 smithi045 ceph-mon[30178]: osdmap e561: 8 total, 8 up, 8 in 2024-03-11T22:32:03.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:02 smithi045 ceph-mon[30178]: 3.b deep-scrub starts 2024-03-11T22:32:03.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:02 smithi045 ceph-mon[30178]: 1.5 scrub starts 2024-03-11T22:32:03.266 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:02 smithi045 ceph-mon[30178]: 3.b deep-scrub ok 2024-03-11T22:32:03.267 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:02 smithi045 ceph-mon[30178]: 1.5 scrub ok 2024-03-11T22:32:03.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:02 smithi144 ceph-mon[33984]: osdmap e561: 8 total, 8 up, 8 in 2024-03-11T22:32:03.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:02 smithi144 ceph-mon[33984]: 3.b deep-scrub starts 2024-03-11T22:32:03.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:02 smithi144 ceph-mon[33984]: 1.5 scrub starts 2024-03-11T22:32:03.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:02 smithi144 ceph-mon[33984]: 3.b deep-scrub ok 2024-03-11T22:32:03.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:02 smithi144 ceph-mon[33984]: 1.5 scrub ok 2024-03-11T22:32:04.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:03 smithi045 ceph-mon[39715]: 3.1b scrub starts 2024-03-11T22:32:04.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:03 smithi045 ceph-mon[39715]: 3.1b scrub ok 2024-03-11T22:32:04.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:03 smithi045 ceph-mon[39715]: pgmap v1162: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:32:04.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:03 smithi045 ceph-mon[39715]: 1.7 deep-scrub starts 2024-03-11T22:32:04.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:03 smithi045 ceph-mon[39715]: 1.7 deep-scrub ok 2024-03-11T22:32:04.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:03 smithi045 ceph-mon[30178]: 3.1b scrub starts 2024-03-11T22:32:04.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:03 smithi045 ceph-mon[30178]: 3.1b scrub ok 2024-03-11T22:32:04.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:03 smithi045 ceph-mon[30178]: pgmap v1162: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:32:04.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:03 smithi045 ceph-mon[30178]: 1.7 deep-scrub starts 2024-03-11T22:32:04.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:03 smithi045 ceph-mon[30178]: 1.7 deep-scrub ok 2024-03-11T22:32:04.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:03 smithi144 ceph-mon[33984]: 3.1b scrub starts 2024-03-11T22:32:04.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:03 smithi144 ceph-mon[33984]: 3.1b scrub ok 2024-03-11T22:32:04.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:03 smithi144 ceph-mon[33984]: pgmap v1162: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T22:32:04.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:03 smithi144 ceph-mon[33984]: 1.7 deep-scrub starts 2024-03-11T22:32:04.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:03 smithi144 ceph-mon[33984]: 1.7 deep-scrub ok 2024-03-11T22:32:04.987 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T22:32:05.251 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:04 smithi144 ceph-mon[33984]: 3.6 scrub starts 2024-03-11T22:32:05.251 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:04 smithi144 ceph-mon[33984]: 3.6 scrub ok 2024-03-11T22:32:05.251 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:04 smithi144 ceph-mon[33984]: 4.1f scrub starts 2024-03-11T22:32:05.251 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:04 smithi144 ceph-mon[33984]: 4.1f scrub ok 2024-03-11T22:32:05.252 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:04 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1534924450' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:32:05.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:04 smithi045 ceph-mon[39715]: 3.6 scrub starts 2024-03-11T22:32:05.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:04 smithi045 ceph-mon[39715]: 3.6 scrub ok 2024-03-11T22:32:05.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:04 smithi045 ceph-mon[39715]: 4.1f scrub starts 2024-03-11T22:32:05.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:04 smithi045 ceph-mon[39715]: 4.1f scrub ok 2024-03-11T22:32:05.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:04 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1534924450' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:32:05.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:04 smithi045 ceph-mon[30178]: 3.6 scrub starts 2024-03-11T22:32:05.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:04 smithi045 ceph-mon[30178]: 3.6 scrub ok 2024-03-11T22:32:05.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:04 smithi045 ceph-mon[30178]: 4.1f scrub starts 2024-03-11T22:32:05.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:04 smithi045 ceph-mon[30178]: 4.1f scrub ok 2024-03-11T22:32:05.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:04 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1534924450' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:32:05.867 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-11T22:32:06.122 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:05 smithi045 ceph-mon[39715]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 672 B/s rd, 0 op/s 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:05 smithi045 ceph-mon[39715]: 5.17 scrub starts 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:05 smithi045 ceph-mon[39715]: 5.17 scrub ok 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:05 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1534924450' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:32:06.243 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:05 smithi045 ceph-mon[39715]: osdmap e562: 8 total, 8 up, 8 in 2024-03-11T22:32:06.244 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:05 smithi045 ceph-mon[30178]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 672 B/s rd, 0 op/s 2024-03-11T22:32:06.244 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:05 smithi045 ceph-mon[30178]: 5.17 scrub starts 2024-03-11T22:32:06.244 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:05 smithi045 ceph-mon[30178]: 5.17 scrub ok 2024-03-11T22:32:06.244 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:05 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1534924450' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:32:06.244 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:05 smithi045 ceph-mon[30178]: osdmap e562: 8 total, 8 up, 8 in 2024-03-11T22:32:06.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:05 smithi144 ceph-mon[33984]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 672 B/s rd, 0 op/s 2024-03-11T22:32:06.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:05 smithi144 ceph-mon[33984]: 5.17 scrub starts 2024-03-11T22:32:06.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:05 smithi144 ceph-mon[33984]: 5.17 scrub ok 2024-03-11T22:32:06.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:05 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1534924450' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:32:06.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:05 smithi144 ceph-mon[33984]: osdmap e562: 8 total, 8 up, 8 in 2024-03-11T22:32:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:06 smithi045 ceph-mon[30178]: 1.a scrub starts 2024-03-11T22:32:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:06 smithi045 ceph-mon[30178]: 1.a scrub ok 2024-03-11T22:32:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:06 smithi045 ceph-mon[30178]: 5.3 scrub starts 2024-03-11T22:32:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:06 smithi045 ceph-mon[30178]: 5.3 scrub ok 2024-03-11T22:32:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:06 smithi045 ceph-mon[30178]: 3.4 scrub starts 2024-03-11T22:32:07.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:06 smithi045 ceph-mon[30178]: 3.4 scrub ok 2024-03-11T22:32:07.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:06 smithi045 ceph-mon[39715]: 1.a scrub starts 2024-03-11T22:32:07.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:06 smithi045 ceph-mon[39715]: 1.a scrub ok 2024-03-11T22:32:07.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:06 smithi045 ceph-mon[39715]: 5.3 scrub starts 2024-03-11T22:32:07.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:06 smithi045 ceph-mon[39715]: 5.3 scrub ok 2024-03-11T22:32:07.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:06 smithi045 ceph-mon[39715]: 3.4 scrub starts 2024-03-11T22:32:07.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:06 smithi045 ceph-mon[39715]: 3.4 scrub ok 2024-03-11T22:32:07.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:32:07 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:32:07.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:06 smithi144 ceph-mon[33984]: 1.a scrub starts 2024-03-11T22:32:07.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:06 smithi144 ceph-mon[33984]: 1.a scrub ok 2024-03-11T22:32:07.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:06 smithi144 ceph-mon[33984]: 5.3 scrub starts 2024-03-11T22:32:07.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:06 smithi144 ceph-mon[33984]: 5.3 scrub ok 2024-03-11T22:32:07.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:06 smithi144 ceph-mon[33984]: 3.4 scrub starts 2024-03-11T22:32:07.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:06 smithi144 ceph-mon[33984]: 3.4 scrub ok 2024-03-11T22:32:08.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:07 smithi045 ceph-mon[39715]: pgmap v1165: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T22:32:08.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:07 smithi045 ceph-mon[39715]: 3.18 scrub starts 2024-03-11T22:32:08.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:07 smithi045 ceph-mon[39715]: 3.18 scrub ok 2024-03-11T22:32:08.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:07 smithi045 ceph-mon[39715]: 5.4 scrub starts 2024-03-11T22:32:08.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:07 smithi045 ceph-mon[39715]: 5.4 scrub ok 2024-03-11T22:32:08.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:07 smithi045 ceph-mon[30178]: pgmap v1165: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T22:32:08.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:07 smithi045 ceph-mon[30178]: 3.18 scrub starts 2024-03-11T22:32:08.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:07 smithi045 ceph-mon[30178]: 3.18 scrub ok 2024-03-11T22:32:08.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:07 smithi045 ceph-mon[30178]: 5.4 scrub starts 2024-03-11T22:32:08.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:07 smithi045 ceph-mon[30178]: 5.4 scrub ok 2024-03-11T22:32:08.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:07 smithi144 ceph-mon[33984]: pgmap v1165: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T22:32:08.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:07 smithi144 ceph-mon[33984]: 3.18 scrub starts 2024-03-11T22:32:08.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:07 smithi144 ceph-mon[33984]: 3.18 scrub ok 2024-03-11T22:32:08.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:07 smithi144 ceph-mon[33984]: 5.4 scrub starts 2024-03-11T22:32:08.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:07 smithi144 ceph-mon[33984]: 5.4 scrub ok 2024-03-11T22:32:08.997 INFO:teuthology.orchestra.run.smithi045.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-11T22:32:09.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:09.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[39715]: 4.1e scrub starts 2024-03-11T22:32:09.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[39715]: 4.1e scrub ok 2024-03-11T22:32:09.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[39715]: 4.1b scrub starts 2024-03-11T22:32:09.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[39715]: 4.1b scrub ok 2024-03-11T22:32:09.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2953828757' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:32:09.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:32:09.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[39715]: 4.f scrub starts 2024-03-11T22:32:09.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[39715]: 4.f scrub ok 2024-03-11T22:32:09.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:09.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[30178]: 4.1e scrub starts 2024-03-11T22:32:09.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[30178]: 4.1e scrub ok 2024-03-11T22:32:09.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[30178]: 4.1b scrub starts 2024-03-11T22:32:09.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[30178]: 4.1b scrub ok 2024-03-11T22:32:09.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2953828757' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:32:09.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:32:09.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[30178]: 4.f scrub starts 2024-03-11T22:32:09.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:08 smithi045 ceph-mon[30178]: 4.f scrub ok 2024-03-11T22:32:09.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:08 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:09.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:08 smithi144 ceph-mon[33984]: 4.1e scrub starts 2024-03-11T22:32:09.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:08 smithi144 ceph-mon[33984]: 4.1e scrub ok 2024-03-11T22:32:09.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:08 smithi144 ceph-mon[33984]: 4.1b scrub starts 2024-03-11T22:32:09.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:08 smithi144 ceph-mon[33984]: 4.1b scrub ok 2024-03-11T22:32:09.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:08 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2953828757' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:32:09.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:08 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T22:32:09.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:08 smithi144 ceph-mon[33984]: 4.f scrub starts 2024-03-11T22:32:09.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:08 smithi144 ceph-mon[33984]: 4.f scrub ok 2024-03-11T22:32:09.720 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T22:32:09.968 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:10.273 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[39715]: 5.1f deep-scrub starts 2024-03-11T22:32:10.273 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[39715]: 5.1f deep-scrub ok 2024-03-11T22:32:10.273 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[39715]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:10.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[39715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:32:10.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[39715]: osdmap e563: 8 total, 8 up, 8 in 2024-03-11T22:32:10.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[39715]: 4.13 scrub starts 2024-03-11T22:32:10.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[39715]: 4.13 scrub ok 2024-03-11T22:32:10.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[39715]: 6.1 scrub starts 2024-03-11T22:32:10.274 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[39715]: 6.1 scrub ok 2024-03-11T22:32:10.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[30178]: 5.1f deep-scrub starts 2024-03-11T22:32:10.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[30178]: 5.1f deep-scrub ok 2024-03-11T22:32:10.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[30178]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:10.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[30178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:32:10.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[30178]: osdmap e563: 8 total, 8 up, 8 in 2024-03-11T22:32:10.274 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[30178]: 4.13 scrub starts 2024-03-11T22:32:10.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[30178]: 4.13 scrub ok 2024-03-11T22:32:10.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[30178]: 6.1 scrub starts 2024-03-11T22:32:10.275 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:09 smithi045 ceph-mon[30178]: 6.1 scrub ok 2024-03-11T22:32:10.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:09 smithi144 ceph-mon[33984]: 5.1f deep-scrub starts 2024-03-11T22:32:10.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:09 smithi144 ceph-mon[33984]: 5.1f deep-scrub ok 2024-03-11T22:32:10.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:09 smithi144 ceph-mon[33984]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:10.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:09 smithi144 ceph-mon[33984]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T22:32:10.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:09 smithi144 ceph-mon[33984]: osdmap e563: 8 total, 8 up, 8 in 2024-03-11T22:32:10.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:09 smithi144 ceph-mon[33984]: 4.13 scrub starts 2024-03-11T22:32:10.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:09 smithi144 ceph-mon[33984]: 4.13 scrub ok 2024-03-11T22:32:10.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:09 smithi144 ceph-mon[33984]: 6.1 scrub starts 2024-03-11T22:32:10.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:09 smithi144 ceph-mon[33984]: 6.1 scrub ok 2024-03-11T22:32:11.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:10 smithi045 ceph-mon[30178]: 3.13 scrub starts 2024-03-11T22:32:11.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:10 smithi045 ceph-mon[30178]: 3.13 scrub ok 2024-03-11T22:32:11.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:10 smithi045 ceph-mon[39715]: 3.13 scrub starts 2024-03-11T22:32:11.099 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:10 smithi045 ceph-mon[39715]: 3.13 scrub ok 2024-03-11T22:32:11.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:10 smithi144 ceph-mon[33984]: 3.13 scrub starts 2024-03-11T22:32:11.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:10 smithi144 ceph-mon[33984]: 3.13 scrub ok 2024-03-11T22:32:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[39715]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T22:32:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[39715]: 3.11 scrub starts 2024-03-11T22:32:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[39715]: 3.11 scrub ok 2024-03-11T22:32:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:32:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:32:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[39715]: 2.0 scrub starts 2024-03-11T22:32:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[39715]: 2.0 scrub ok 2024-03-11T22:32:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[39715]: osdmap e564: 8 total, 8 up, 8 in 2024-03-11T22:32:12.038 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3141140675' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:32:12.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[30178]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T22:32:12.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[30178]: 3.11 scrub starts 2024-03-11T22:32:12.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[30178]: 3.11 scrub ok 2024-03-11T22:32:12.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:32:12.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:32:12.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[30178]: 2.0 scrub starts 2024-03-11T22:32:12.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[30178]: 2.0 scrub ok 2024-03-11T22:32:12.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[30178]: osdmap e564: 8 total, 8 up, 8 in 2024-03-11T22:32:12.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:11 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3141140675' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:32:12.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:11 smithi144 ceph-mon[33984]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T22:32:12.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:11 smithi144 ceph-mon[33984]: 3.11 scrub starts 2024-03-11T22:32:12.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:11 smithi144 ceph-mon[33984]: 3.11 scrub ok 2024-03-11T22:32:12.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:11 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:32:12.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:11 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:32:12.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:11 smithi144 ceph-mon[33984]: 2.0 scrub starts 2024-03-11T22:32:12.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:11 smithi144 ceph-mon[33984]: 2.0 scrub ok 2024-03-11T22:32:12.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:11 smithi144 ceph-mon[33984]: osdmap e564: 8 total, 8 up, 8 in 2024-03-11T22:32:12.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:11 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3141140675' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:32:12.676 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T22:32:13.008 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:32:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:32:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:32:13.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:13 smithi045 ceph-mon[39715]: 6.12 scrub starts 2024-03-11T22:32:13.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:13 smithi045 ceph-mon[39715]: 6.12 scrub ok 2024-03-11T22:32:13.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:13 smithi045 ceph-mon[39715]: 5.12 scrub starts 2024-03-11T22:32:13.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:13 smithi045 ceph-mon[39715]: 5.12 scrub ok 2024-03-11T22:32:13.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:13 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3141140675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:32:13.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:13 smithi045 ceph-mon[39715]: osdmap e565: 8 total, 8 up, 8 in 2024-03-11T22:32:13.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:13 smithi045 ceph-mon[30178]: 6.12 scrub starts 2024-03-11T22:32:13.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:13 smithi045 ceph-mon[30178]: 6.12 scrub ok 2024-03-11T22:32:13.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:13 smithi045 ceph-mon[30178]: 5.12 scrub starts 2024-03-11T22:32:13.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:13 smithi045 ceph-mon[30178]: 5.12 scrub ok 2024-03-11T22:32:13.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:13 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3141140675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:32:13.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:13 smithi045 ceph-mon[30178]: osdmap e565: 8 total, 8 up, 8 in 2024-03-11T22:32:13.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:13 smithi144 ceph-mon[33984]: 6.12 scrub starts 2024-03-11T22:32:13.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:13 smithi144 ceph-mon[33984]: 6.12 scrub ok 2024-03-11T22:32:13.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:13 smithi144 ceph-mon[33984]: 5.12 scrub starts 2024-03-11T22:32:13.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:13 smithi144 ceph-mon[33984]: 5.12 scrub ok 2024-03-11T22:32:13.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:13 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3141140675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:32:13.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:13 smithi144 ceph-mon[33984]: osdmap e565: 8 total, 8 up, 8 in 2024-03-11T22:32:13.399 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-11T22:32:13.651 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:14.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[39715]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:14.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[39715]: 5.14 deep-scrub starts 2024-03-11T22:32:14.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[39715]: 5.14 deep-scrub ok 2024-03-11T22:32:14.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[39715]: 4.1a scrub starts 2024-03-11T22:32:14.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[39715]: 4.1a scrub ok 2024-03-11T22:32:14.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[39715]: 5.c deep-scrub starts 2024-03-11T22:32:14.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[39715]: 5.c deep-scrub ok 2024-03-11T22:32:14.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[30178]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:14.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[30178]: 5.14 deep-scrub starts 2024-03-11T22:32:14.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[30178]: 5.14 deep-scrub ok 2024-03-11T22:32:14.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[30178]: 4.1a scrub starts 2024-03-11T22:32:14.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[30178]: 4.1a scrub ok 2024-03-11T22:32:14.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[30178]: 5.c deep-scrub starts 2024-03-11T22:32:14.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:14 smithi045 ceph-mon[30178]: 5.c deep-scrub ok 2024-03-11T22:32:14.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:14 smithi144 ceph-mon[33984]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:14.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:14 smithi144 ceph-mon[33984]: 5.14 deep-scrub starts 2024-03-11T22:32:14.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:14 smithi144 ceph-mon[33984]: 5.14 deep-scrub ok 2024-03-11T22:32:14.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:14 smithi144 ceph-mon[33984]: 4.1a scrub starts 2024-03-11T22:32:14.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:14 smithi144 ceph-mon[33984]: 4.1a scrub ok 2024-03-11T22:32:14.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:14 smithi144 ceph-mon[33984]: 5.c deep-scrub starts 2024-03-11T22:32:14.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:14 smithi144 ceph-mon[33984]: 5.c deep-scrub ok 2024-03-11T22:32:15.269 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:15 smithi045 ceph-mon[39715]: 3.8 scrub starts 2024-03-11T22:32:15.269 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:15 smithi045 ceph-mon[39715]: 3.8 scrub ok 2024-03-11T22:32:15.269 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:15 smithi045 ceph-mon[39715]: pgmap v1172: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:32:15.270 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:15 smithi045 ceph-mon[30178]: 3.8 scrub starts 2024-03-11T22:32:15.270 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:15 smithi045 ceph-mon[30178]: 3.8 scrub ok 2024-03-11T22:32:15.270 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:15 smithi045 ceph-mon[30178]: pgmap v1172: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:32:15.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:15 smithi144 ceph-mon[33984]: 3.8 scrub starts 2024-03-11T22:32:15.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:15 smithi144 ceph-mon[33984]: 3.8 scrub ok 2024-03-11T22:32:15.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:15 smithi144 ceph-mon[33984]: pgmap v1172: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:32:16.026 INFO:teuthology.orchestra.run.smithi045.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-11T22:32:16.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:16 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2564511531' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:32:16.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:16 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2564511531' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:32:16.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:16 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2564511531' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T22:32:16.781 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T22:32:17.031 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:17.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[39715]: 5.1c scrub starts 2024-03-11T22:32:17.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[39715]: 5.1c scrub ok 2024-03-11T22:32:17.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2564511531' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:32:17.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[39715]: osdmap e566: 8 total, 8 up, 8 in 2024-03-11T22:32:17.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[39715]: 3.2 scrub starts 2024-03-11T22:32:17.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[39715]: 3.2 scrub ok 2024-03-11T22:32:17.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[39715]: pgmap v1174: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:32:17.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:17.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[30178]: 5.1c scrub starts 2024-03-11T22:32:17.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[30178]: 5.1c scrub ok 2024-03-11T22:32:17.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2564511531' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:32:17.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[30178]: osdmap e566: 8 total, 8 up, 8 in 2024-03-11T22:32:17.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[30178]: 3.2 scrub starts 2024-03-11T22:32:17.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[30178]: 3.2 scrub ok 2024-03-11T22:32:17.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[30178]: pgmap v1174: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:32:17.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:17 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:17.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:32:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:32:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:17 smithi144 ceph-mon[33984]: 5.1c scrub starts 2024-03-11T22:32:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:17 smithi144 ceph-mon[33984]: 5.1c scrub ok 2024-03-11T22:32:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:17 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2564511531' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-11T22:32:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:17 smithi144 ceph-mon[33984]: osdmap e566: 8 total, 8 up, 8 in 2024-03-11T22:32:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:17 smithi144 ceph-mon[33984]: 3.2 scrub starts 2024-03-11T22:32:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:17 smithi144 ceph-mon[33984]: 3.2 scrub ok 2024-03-11T22:32:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:17 smithi144 ceph-mon[33984]: pgmap v1174: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:32:17.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:17 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:18.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[30178]: 3.16 deep-scrub starts 2024-03-11T22:32:18.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[30178]: 3.16 deep-scrub ok 2024-03-11T22:32:18.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[30178]: 3.a deep-scrub starts 2024-03-11T22:32:18.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[30178]: 3.a deep-scrub ok 2024-03-11T22:32:18.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[30178]: 4.1 scrub starts 2024-03-11T22:32:18.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[30178]: 4.1 scrub ok 2024-03-11T22:32:18.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:18.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[30178]: 4.1d scrub starts 2024-03-11T22:32:18.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[30178]: 4.1d scrub ok 2024-03-11T22:32:18.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[39715]: 3.16 deep-scrub starts 2024-03-11T22:32:18.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[39715]: 3.16 deep-scrub ok 2024-03-11T22:32:18.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[39715]: 3.a deep-scrub starts 2024-03-11T22:32:18.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[39715]: 3.a deep-scrub ok 2024-03-11T22:32:18.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[39715]: 4.1 scrub starts 2024-03-11T22:32:18.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[39715]: 4.1 scrub ok 2024-03-11T22:32:18.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:18.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[39715]: 4.1d scrub starts 2024-03-11T22:32:18.289 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:18 smithi045 ceph-mon[39715]: 4.1d scrub ok 2024-03-11T22:32:18.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:18 smithi144 ceph-mon[33984]: 3.16 deep-scrub starts 2024-03-11T22:32:18.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:18 smithi144 ceph-mon[33984]: 3.16 deep-scrub ok 2024-03-11T22:32:18.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:18 smithi144 ceph-mon[33984]: 3.a deep-scrub starts 2024-03-11T22:32:18.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:18 smithi144 ceph-mon[33984]: 3.a deep-scrub ok 2024-03-11T22:32:18.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:18 smithi144 ceph-mon[33984]: 4.1 scrub starts 2024-03-11T22:32:18.352 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:18 smithi144 ceph-mon[33984]: 4.1 scrub ok 2024-03-11T22:32:18.352 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:18 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:18.352 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:18 smithi144 ceph-mon[33984]: 4.1d scrub starts 2024-03-11T22:32:18.352 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:18 smithi144 ceph-mon[33984]: 4.1d scrub ok 2024-03-11T22:32:19.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:19 smithi045 ceph-mon[30178]: 5.f scrub starts 2024-03-11T22:32:19.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:19 smithi045 ceph-mon[30178]: 5.f scrub ok 2024-03-11T22:32:19.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:19 smithi045 ceph-mon[30178]: 6.10 scrub starts 2024-03-11T22:32:19.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:19 smithi045 ceph-mon[30178]: 6.10 scrub ok 2024-03-11T22:32:19.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:19 smithi045 ceph-mon[30178]: pgmap v1175: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:32:19.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:19 smithi045 ceph-mon[39715]: 5.f scrub starts 2024-03-11T22:32:19.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:19 smithi045 ceph-mon[39715]: 5.f scrub ok 2024-03-11T22:32:19.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:19 smithi045 ceph-mon[39715]: 6.10 scrub starts 2024-03-11T22:32:19.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:19 smithi045 ceph-mon[39715]: 6.10 scrub ok 2024-03-11T22:32:19.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:19 smithi045 ceph-mon[39715]: pgmap v1175: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:32:19.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:19 smithi144 ceph-mon[33984]: 5.f scrub starts 2024-03-11T22:32:19.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:19 smithi144 ceph-mon[33984]: 5.f scrub ok 2024-03-11T22:32:19.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:19 smithi144 ceph-mon[33984]: 6.10 scrub starts 2024-03-11T22:32:19.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:19 smithi144 ceph-mon[33984]: 6.10 scrub ok 2024-03-11T22:32:19.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:19 smithi144 ceph-mon[33984]: pgmap v1175: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:32:20.044 INFO:teuthology.orchestra.run.smithi045.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T22:32:20.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[39715]: 6.7 deep-scrub starts 2024-03-11T22:32:20.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[39715]: 6.7 deep-scrub ok 2024-03-11T22:32:20.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[39715]: 5.1a scrub starts 2024-03-11T22:32:20.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[39715]: 5.1a scrub ok 2024-03-11T22:32:20.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3910757738' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:32:20.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[39715]: 1.2 scrub starts 2024-03-11T22:32:20.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[39715]: 1.2 scrub ok 2024-03-11T22:32:20.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:32:20.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[30178]: 6.7 deep-scrub starts 2024-03-11T22:32:20.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[30178]: 6.7 deep-scrub ok 2024-03-11T22:32:20.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[30178]: 5.1a scrub starts 2024-03-11T22:32:20.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[30178]: 5.1a scrub ok 2024-03-11T22:32:20.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3910757738' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:32:20.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[30178]: 1.2 scrub starts 2024-03-11T22:32:20.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[30178]: 1.2 scrub ok 2024-03-11T22:32:20.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:20 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:32:20.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:20 smithi144 ceph-mon[33984]: 6.7 deep-scrub starts 2024-03-11T22:32:20.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:20 smithi144 ceph-mon[33984]: 6.7 deep-scrub ok 2024-03-11T22:32:20.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:20 smithi144 ceph-mon[33984]: 5.1a scrub starts 2024-03-11T22:32:20.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:20 smithi144 ceph-mon[33984]: 5.1a scrub ok 2024-03-11T22:32:20.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:20 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3910757738' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:32:20.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:20 smithi144 ceph-mon[33984]: 1.2 scrub starts 2024-03-11T22:32:20.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:20 smithi144 ceph-mon[33984]: 1.2 scrub ok 2024-03-11T22:32:20.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:20 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:32:20.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-11T22:32:21.079 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:21.201 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:21 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3910757738' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:32:21.201 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:21 smithi045 ceph-mon[39715]: osdmap e567: 8 total, 8 up, 8 in 2024-03-11T22:32:21.201 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:21 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:32:21.201 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:21 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:32:21.201 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:21 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:32:21.202 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:21 smithi045 ceph-mon[39715]: pgmap v1177: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 648 B/s rd, 0 op/s 2024-03-11T22:32:21.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:21 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3910757738' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:32:21.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:21 smithi045 ceph-mon[30178]: osdmap e567: 8 total, 8 up, 8 in 2024-03-11T22:32:21.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:21 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:32:21.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:21 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:32:21.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:21 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:32:21.202 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:21 smithi045 ceph-mon[30178]: pgmap v1177: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 648 B/s rd, 0 op/s 2024-03-11T22:32:21.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:21 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3910757738' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:32:21.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:21 smithi144 ceph-mon[33984]: osdmap e567: 8 total, 8 up, 8 in 2024-03-11T22:32:21.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:21 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:32:21.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:21 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:32:21.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:21 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:32:21.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:21 smithi144 ceph-mon[33984]: pgmap v1177: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 648 B/s rd, 0 op/s 2024-03-11T22:32:22.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[30178]: 1.6 deep-scrub starts 2024-03-11T22:32:22.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[30178]: 1.6 deep-scrub ok 2024-03-11T22:32:22.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[30178]: 4.17 scrub starts 2024-03-11T22:32:22.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[30178]: 4.17 scrub ok 2024-03-11T22:32:22.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[30178]: 5.a scrub starts 2024-03-11T22:32:22.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[30178]: 5.a scrub ok 2024-03-11T22:32:22.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[30178]: osdmap e568: 8 total, 8 up, 8 in 2024-03-11T22:32:22.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[39715]: 1.6 deep-scrub starts 2024-03-11T22:32:22.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[39715]: 1.6 deep-scrub ok 2024-03-11T22:32:22.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[39715]: 4.17 scrub starts 2024-03-11T22:32:22.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[39715]: 4.17 scrub ok 2024-03-11T22:32:22.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[39715]: 5.a scrub starts 2024-03-11T22:32:22.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[39715]: 5.a scrub ok 2024-03-11T22:32:22.288 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-mon[39715]: osdmap e568: 8 total, 8 up, 8 in 2024-03-11T22:32:22.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:22 smithi144 ceph-mon[33984]: 1.6 deep-scrub starts 2024-03-11T22:32:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:22 smithi144 ceph-mon[33984]: 1.6 deep-scrub ok 2024-03-11T22:32:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:22 smithi144 ceph-mon[33984]: 4.17 scrub starts 2024-03-11T22:32:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:22 smithi144 ceph-mon[33984]: 4.17 scrub ok 2024-03-11T22:32:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:22 smithi144 ceph-mon[33984]: 5.a scrub starts 2024-03-11T22:32:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:22 smithi144 ceph-mon[33984]: 5.a scrub ok 2024-03-11T22:32:22.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:22 smithi144 ceph-mon[33984]: osdmap e568: 8 total, 8 up, 8 in 2024-03-11T22:32:22.630 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:32:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:32:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:32:23.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:23 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:32:23.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:23 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:32:23.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:23 smithi144 ceph-mon[33984]: 6.0 scrub starts 2024-03-11T22:32:23.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:23 smithi144 ceph-mon[33984]: 4.14 scrub starts 2024-03-11T22:32:23.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:23 smithi144 ceph-mon[33984]: 4.14 scrub ok 2024-03-11T22:32:23.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:23 smithi144 ceph-mon[33984]: 6.0 scrub ok 2024-03-11T22:32:23.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:23 smithi144 ceph-mon[33984]: pgmap v1179: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:32:23.536 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:32:23.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:32:23.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[30178]: 6.0 scrub starts 2024-03-11T22:32:23.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[30178]: 4.14 scrub starts 2024-03-11T22:32:23.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[30178]: 4.14 scrub ok 2024-03-11T22:32:23.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[30178]: 6.0 scrub ok 2024-03-11T22:32:23.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[30178]: pgmap v1179: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:32:23.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:32:23.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:32:23.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[39715]: 6.0 scrub starts 2024-03-11T22:32:23.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[39715]: 4.14 scrub starts 2024-03-11T22:32:23.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[39715]: 4.14 scrub ok 2024-03-11T22:32:23.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[39715]: 6.0 scrub ok 2024-03-11T22:32:23.538 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:23 smithi045 ceph-mon[39715]: pgmap v1179: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T22:32:24.059 INFO:teuthology.orchestra.run.smithi045.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-11T22:32:24.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:24 smithi144 ceph-mon[33984]: 6.19 deep-scrub starts 2024-03-11T22:32:24.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:24 smithi144 ceph-mon[33984]: 6.19 deep-scrub ok 2024-03-11T22:32:24.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:24 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1988083664' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:32:24.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:24 smithi144 ceph-mon[33984]: 6.13 scrub starts 2024-03-11T22:32:24.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:24 smithi045 ceph-mon[39715]: 6.19 deep-scrub starts 2024-03-11T22:32:24.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:24 smithi045 ceph-mon[39715]: 6.19 deep-scrub ok 2024-03-11T22:32:24.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:24 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1988083664' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:32:24.429 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:24 smithi045 ceph-mon[39715]: 6.13 scrub starts 2024-03-11T22:32:24.429 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:24 smithi045 ceph-mon[30178]: 6.19 deep-scrub starts 2024-03-11T22:32:24.429 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:24 smithi045 ceph-mon[30178]: 6.19 deep-scrub ok 2024-03-11T22:32:24.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:24 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1988083664' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T22:32:24.430 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:24 smithi045 ceph-mon[30178]: 6.13 scrub starts 2024-03-11T22:32:24.879 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-11T22:32:24.879 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-11T22:32:24.879 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-11T22:32:24.879 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-11T22:32:24.879 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T22:32:24.879 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:32:25.140 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:25.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[39715]: 6.13 scrub ok 2024-03-11T22:32:25.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1988083664' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:32:25.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[39715]: osdmap e569: 8 total, 8 up, 8 in 2024-03-11T22:32:25.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[39715]: 4.b deep-scrub starts 2024-03-11T22:32:25.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[39715]: 4.b deep-scrub ok 2024-03-11T22:32:25.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[39715]: 5.0 scrub starts 2024-03-11T22:32:25.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[39715]: 5.0 scrub ok 2024-03-11T22:32:25.261 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[39715]: pgmap v1181: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:32:25.262 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[30178]: 6.13 scrub ok 2024-03-11T22:32:25.262 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1988083664' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:32:25.262 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[30178]: osdmap e569: 8 total, 8 up, 8 in 2024-03-11T22:32:25.262 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[30178]: 4.b deep-scrub starts 2024-03-11T22:32:25.262 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[30178]: 4.b deep-scrub ok 2024-03-11T22:32:25.262 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[30178]: 5.0 scrub starts 2024-03-11T22:32:25.262 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[30178]: 5.0 scrub ok 2024-03-11T22:32:25.262 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:25 smithi045 ceph-mon[30178]: pgmap v1181: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:32:25.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:25 smithi144 ceph-mon[33984]: 6.13 scrub ok 2024-03-11T22:32:25.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:25 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1988083664' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-11T22:32:25.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:25 smithi144 ceph-mon[33984]: osdmap e569: 8 total, 8 up, 8 in 2024-03-11T22:32:25.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:25 smithi144 ceph-mon[33984]: 4.b deep-scrub starts 2024-03-11T22:32:25.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:25 smithi144 ceph-mon[33984]: 4.b deep-scrub ok 2024-03-11T22:32:25.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:25 smithi144 ceph-mon[33984]: 5.0 scrub starts 2024-03-11T22:32:25.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:25 smithi144 ceph-mon[33984]: 5.0 scrub ok 2024-03-11T22:32:25.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:25 smithi144 ceph-mon[33984]: pgmap v1181: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:32:26.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:26 smithi144 ceph-mon[33984]: 5.2 scrub starts 2024-03-11T22:32:26.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:26 smithi144 ceph-mon[33984]: 5.2 scrub ok 2024-03-11T22:32:26.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:26 smithi144 ceph-mon[33984]: osdmap e570: 8 total, 8 up, 8 in 2024-03-11T22:32:26.447 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:26 smithi045 ceph-mon[39715]: 5.2 scrub starts 2024-03-11T22:32:26.481 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:26 smithi045 ceph-mon[39715]: 5.2 scrub ok 2024-03-11T22:32:26.481 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:26 smithi045 ceph-mon[39715]: osdmap e570: 8 total, 8 up, 8 in 2024-03-11T22:32:26.481 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:26 smithi045 ceph-mon[30178]: 5.2 scrub starts 2024-03-11T22:32:26.481 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:26 smithi045 ceph-mon[30178]: 5.2 scrub ok 2024-03-11T22:32:26.481 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:26 smithi045 ceph-mon[30178]: osdmap e570: 8 total, 8 up, 8 in 2024-03-11T22:32:27.248 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:32:27.248 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":570,"fsid":"f75aafa4-dff2-11ee-95c8-87774f69a715","created":"2024-03-11T22:02:51.075940+0000","modified":"2024-03-11T22:32:25.048196+0000","last_up_change":"2024-03-11T22:08:17.992246+0000","last_in_change":"2024-03-11T22:29:03.667666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:06:12.903386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":438,"last_epoch_started":437,"last_epoch_clean":437,"source_version":"93'130","target_version":"93'130"},"last_change":"543","last_force_op_resend":"0","last_force_op_resend_prenautilus":"438","last_force_op_resend_preluminous":"436","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-11T22:08:31.014060+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-11T22:08:31.591167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-11T22:08:33.606516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T22:08:35.315522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T22:08:37.383856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55a4e633-f2a6-4d27-b10c-de1c0f87cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":540,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6803","nonce":2216353820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6805","nonce":2216353820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6809","nonce":2216353820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6807","nonce":2216353820}]},"public_addr":"172.21.15.45:6803/2216353820","cluster_addr":"172.21.15.45:6805/2216353820","heartbeat_back_addr":"172.21.15.45:6809/2216353820","heartbeat_front_addr":"172.21.15.45:6807/2216353820","state":["exists","up"]},{"osd":1,"uuid":"60274eeb-591a-43fe-a579-f13d13631e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":515,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6811","nonce":1241486091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6813","nonce":1241486091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6817","nonce":1241486091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6815","nonce":1241486091}]},"public_addr":"172.21.15.45:6811/1241486091","cluster_addr":"172.21.15.45:6813/1241486091","heartbeat_back_addr":"172.21.15.45:6817/1241486091","heartbeat_front_addr":"172.21.15.45:6815/1241486091","state":["exists","up"]},{"osd":2,"uuid":"15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":504,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6819","nonce":4251983518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6821","nonce":4251983518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6825","nonce":4251983518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6823","nonce":4251983518}]},"public_addr":"172.21.15.45:6819/4251983518","cluster_addr":"172.21.15.45:6821/4251983518","heartbeat_back_addr":"172.21.15.45:6825/4251983518","heartbeat_front_addr":"172.21.15.45:6823/4251983518","state":["exists","up"]},{"osd":3,"uuid":"e51a65f3-96f6-478c-8d26-df1ace4a5058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6827","nonce":3178755830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6829","nonce":3178755830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6833","nonce":3178755830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6831","nonce":3178755830}]},"public_addr":"172.21.15.45:6827/3178755830","cluster_addr":"172.21.15.45:6829/3178755830","heartbeat_back_addr":"172.21.15.45:6833/3178755830","heartbeat_front_addr":"172.21.15.45:6831/3178755830","state":["exists","up"]},{"osd":4,"uuid":"32d0b71f-c696-4677-bdd3-b88a04c97604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":560,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6801","nonce":912745159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6803","nonce":912745159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6807","nonce":912745159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6805","nonce":912745159}]},"public_addr":"172.21.15.144:6801/912745159","cluster_addr":"172.21.15.144:6803/912745159","heartbeat_back_addr":"172.21.15.144:6807/912745159","heartbeat_front_addr":"172.21.15.144:6805/912745159","state":["exists","up"]},{"osd":5,"uuid":"f46b91af-d67d-404c-aad1-75348e35d1c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6809","nonce":3162869174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6811","nonce":3162869174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6815","nonce":3162869174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6813","nonce":3162869174}]},"public_addr":"172.21.15.144:6809/3162869174","cluster_addr":"172.21.15.144:6811/3162869174","heartbeat_back_addr":"172.21.15.144:6815/3162869174","heartbeat_front_addr":"172.21.15.144:6813/3162869174","state":["exists","up"]},{"osd":6,"uuid":"970b128d-cd56-4dff-8345-3196b46c1257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6817","nonce":3439446288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6819","nonce":3439446288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6823","nonce":3439446288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6821","nonce":3439446288}]},"public_addr":"172.21.15.144:6817/3439446288","cluster_addr":"172.21.15.144:6819/3439446288","heartbeat_back_addr":"172.21.15.144:6823/3439446288","heartbeat_front_addr":"172.21.15.144:6821/3439446288","state":["exists","up"]},{"osd":7,"uuid":"40100aaf-830d-4137-b037-5cf167cba308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":569,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6825","nonce":2371064429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6827","nonce":2371064429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6831","nonce":2371064429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6829","nonce":2371064429}]},"public_addr":"172.21.15.144:6825/2371064429","cluster_addr":"172.21.15.144:6827/2371064429","heartbeat_back_addr":"172.21.15.144:6831/2371064429","heartbeat_front_addr":"172.21.15.144:6829/2371064429","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:31:26.138596+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:31:22.924358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:31:51.604504+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:32:21.519681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:32:11.393252+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:31:26.892794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:31:38.691207+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:31:50.785450+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.e","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"5.1f","mappings":[{"from":6,"to":0},{"from":7,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/1901139892":"2024-03-12T22:09:14.021723+0000","172.21.15.45:0/1110781721":"2024-03-12T22:03:52.557006+0000","172.21.15.45:6800/2299156110":"2024-03-12T22:03:25.087713+0000","172.21.15.45:0/2331445880":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6801/4256719031":"2024-03-12T22:03:52.557006+0000","172.21.15.45:6801/2299156110":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6800/3882489981":"2024-03-12T22:09:14.021723+0000","172.21.15.45:0/272906373":"2024-03-12T22:09:14.021723+0000","172.21.15.45:0/11898002":"2024-03-12T22:09:14.021723+0000","172.21.15.45:0/2777740355":"2024-03-12T22:03:25.087713+0000","172.21.15.45:0/1903379086":"2024-03-12T22:09:14.021723+0000","172.21.15.45:0/926341889":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6800/4256719031":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/2513993410":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/2584267737":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/3509006862":"2024-03-12T22:09:14.021723+0000","172.21.15.45:6801/3882489981":"2024-03-12T22:09:14.021723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:32:27.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:32:27 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:32:27.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:27 smithi144 ceph-mon[33984]: 3.c deep-scrub starts 2024-03-11T22:32:27.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:27 smithi144 ceph-mon[33984]: 3.c deep-scrub ok 2024-03-11T22:32:27.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:27 smithi144 ceph-mon[33984]: 1.2 scrub starts 2024-03-11T22:32:27.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:27 smithi144 ceph-mon[33984]: 1.2 scrub ok 2024-03-11T22:32:27.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:27 smithi144 ceph-mon[33984]: 1.8 scrub starts 2024-03-11T22:32:27.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:27 smithi144 ceph-mon[33984]: 1.8 scrub ok 2024-03-11T22:32:27.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:27 smithi144 ceph-mon[33984]: pgmap v1183: 142 pgs: 142 active+clean; 609 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s wr, 1 op/s; 13 KiB/s, 0 objects/s recovering 2024-03-11T22:32:27.536 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[39715]: 3.c deep-scrub starts 2024-03-11T22:32:27.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[39715]: 3.c deep-scrub ok 2024-03-11T22:32:27.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[39715]: 1.2 scrub starts 2024-03-11T22:32:27.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[39715]: 1.2 scrub ok 2024-03-11T22:32:27.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[39715]: 1.8 scrub starts 2024-03-11T22:32:27.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[39715]: 1.8 scrub ok 2024-03-11T22:32:27.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[39715]: pgmap v1183: 142 pgs: 142 active+clean; 609 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s wr, 1 op/s; 13 KiB/s, 0 objects/s recovering 2024-03-11T22:32:27.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[30178]: 3.c deep-scrub starts 2024-03-11T22:32:27.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[30178]: 3.c deep-scrub ok 2024-03-11T22:32:27.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[30178]: 1.2 scrub starts 2024-03-11T22:32:27.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[30178]: 1.2 scrub ok 2024-03-11T22:32:27.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[30178]: 1.8 scrub starts 2024-03-11T22:32:27.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[30178]: 1.8 scrub ok 2024-03-11T22:32:27.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:27 smithi045 ceph-mon[30178]: pgmap v1183: 142 pgs: 142 active+clean; 609 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s wr, 1 op/s; 13 KiB/s, 0 objects/s recovering 2024-03-11T22:32:28.016 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T22:32:28.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:32:28.274 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:28.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:28.286 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[39715]: 1.0 scrub starts 2024-03-11T22:32:28.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[39715]: 1.0 scrub ok 2024-03-11T22:32:28.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:32:28.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:32:28.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3146925373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:32:28.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[39715]: 1.3 scrub starts 2024-03-11T22:32:28.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[39715]: 1.3 scrub ok 2024-03-11T22:32:28.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:28.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[30178]: 1.0 scrub starts 2024-03-11T22:32:28.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[30178]: 1.0 scrub ok 2024-03-11T22:32:28.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:32:28.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:32:28.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3146925373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:32:28.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[30178]: 1.3 scrub starts 2024-03-11T22:32:28.288 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:28 smithi045 ceph-mon[30178]: 1.3 scrub ok 2024-03-11T22:32:28.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:28 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:28.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:28 smithi144 ceph-mon[33984]: 1.0 scrub starts 2024-03-11T22:32:28.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:28 smithi144 ceph-mon[33984]: 1.0 scrub ok 2024-03-11T22:32:28.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:28 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:32:28.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:28 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:32:28.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:28 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3146925373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:32:28.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:28 smithi144 ceph-mon[33984]: 1.3 scrub starts 2024-03-11T22:32:28.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:28 smithi144 ceph-mon[33984]: 1.3 scrub ok 2024-03-11T22:32:29.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:29 smithi144 ceph-mon[33984]: 3.9 scrub starts 2024-03-11T22:32:29.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:29 smithi144 ceph-mon[33984]: 3.9 scrub ok 2024-03-11T22:32:29.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:29 smithi144 ceph-mon[33984]: osdmap e571: 8 total, 8 up, 8 in 2024-03-11T22:32:29.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:29 smithi144 ceph-mon[33984]: 3.d deep-scrub starts 2024-03-11T22:32:29.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:29 smithi144 ceph-mon[33984]: 3.d deep-scrub ok 2024-03-11T22:32:29.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:29 smithi144 ceph-mon[33984]: 5.8 scrub starts 2024-03-11T22:32:29.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:29 smithi144 ceph-mon[33984]: 5.8 scrub ok 2024-03-11T22:32:29.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:29 smithi144 ceph-mon[33984]: pgmap v1185: 142 pgs: 142 active+clean; 609 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s wr, 1 op/s; 13 KiB/s, 0 objects/s recovering 2024-03-11T22:32:29.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[39715]: 3.9 scrub starts 2024-03-11T22:32:29.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[39715]: 3.9 scrub ok 2024-03-11T22:32:29.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[39715]: osdmap e571: 8 total, 8 up, 8 in 2024-03-11T22:32:29.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[39715]: 3.d deep-scrub starts 2024-03-11T22:32:29.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[39715]: 3.d deep-scrub ok 2024-03-11T22:32:29.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[39715]: 5.8 scrub starts 2024-03-11T22:32:29.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[39715]: 5.8 scrub ok 2024-03-11T22:32:29.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[39715]: pgmap v1185: 142 pgs: 142 active+clean; 609 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s wr, 1 op/s; 13 KiB/s, 0 objects/s recovering 2024-03-11T22:32:29.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[30178]: 3.9 scrub starts 2024-03-11T22:32:29.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[30178]: 3.9 scrub ok 2024-03-11T22:32:29.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[30178]: osdmap e571: 8 total, 8 up, 8 in 2024-03-11T22:32:29.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[30178]: 3.d deep-scrub starts 2024-03-11T22:32:29.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[30178]: 3.d deep-scrub ok 2024-03-11T22:32:29.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[30178]: 5.8 scrub starts 2024-03-11T22:32:29.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[30178]: 5.8 scrub ok 2024-03-11T22:32:29.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:29 smithi045 ceph-mon[30178]: pgmap v1185: 142 pgs: 142 active+clean; 609 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 18 KiB/s wr, 1 op/s; 13 KiB/s, 0 objects/s recovering 2024-03-11T22:32:30.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:30 smithi144 ceph-mon[33984]: 3.10 scrub starts 2024-03-11T22:32:30.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:30 smithi144 ceph-mon[33984]: 3.10 scrub ok 2024-03-11T22:32:30.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:30 smithi144 ceph-mon[33984]: 4.9 scrub starts 2024-03-11T22:32:30.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:30 smithi144 ceph-mon[33984]: 4.9 scrub ok 2024-03-11T22:32:30.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:30 smithi144 ceph-mon[33984]: 5.10 deep-scrub starts 2024-03-11T22:32:30.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:30 smithi144 ceph-mon[33984]: 5.10 deep-scrub ok 2024-03-11T22:32:30.478 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:32:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":571,"fsid":"f75aafa4-dff2-11ee-95c8-87774f69a715","created":"2024-03-11T22:02:51.075940+0000","modified":"2024-03-11T22:32:28.058748+0000","last_up_change":"2024-03-11T22:08:17.992246+0000","last_in_change":"2024-03-11T22:29:03.667666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:06:12.903386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":438,"last_epoch_started":437,"last_epoch_clean":437,"source_version":"93'130","target_version":"93'130"},"last_change":"543","last_force_op_resend":"0","last_force_op_resend_prenautilus":"438","last_force_op_resend_preluminous":"436","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-11T22:08:31.014060+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-11T22:08:31.591167+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-11T22:08:33.606516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T22:08:35.315522+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T22:08:37.383856+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55a4e633-f2a6-4d27-b10c-de1c0f87cf32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":540,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6803","nonce":2216353820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6805","nonce":2216353820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6809","nonce":2216353820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":2216353820},{"type":"v1","addr":"172.21.15.45:6807","nonce":2216353820}]},"public_addr":"172.21.15.45:6803/2216353820","cluster_addr":"172.21.15.45:6805/2216353820","heartbeat_back_addr":"172.21.15.45:6809/2216353820","heartbeat_front_addr":"172.21.15.45:6807/2216353820","state":["exists","up"]},{"osd":1,"uuid":"60274eeb-591a-43fe-a579-f13d13631e14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":515,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6811","nonce":1241486091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6813","nonce":1241486091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6817","nonce":1241486091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1241486091},{"type":"v1","addr":"172.21.15.45:6815","nonce":1241486091}]},"public_addr":"172.21.15.45:6811/1241486091","cluster_addr":"172.21.15.45:6813/1241486091","heartbeat_back_addr":"172.21.15.45:6817/1241486091","heartbeat_front_addr":"172.21.15.45:6815/1241486091","state":["exists","up"]},{"osd":2,"uuid":"15b4dfea-7b2b-4d9d-bf21-42ad2a280f7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":504,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6819","nonce":4251983518}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6821","nonce":4251983518}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6825","nonce":4251983518}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":4251983518},{"type":"v1","addr":"172.21.15.45:6823","nonce":4251983518}]},"public_addr":"172.21.15.45:6819/4251983518","cluster_addr":"172.21.15.45:6821/4251983518","heartbeat_back_addr":"172.21.15.45:6825/4251983518","heartbeat_front_addr":"172.21.15.45:6823/4251983518","state":["exists","up"]},{"osd":3,"uuid":"e51a65f3-96f6-478c-8d26-df1ace4a5058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6827","nonce":3178755830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6829","nonce":3178755830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6833","nonce":3178755830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":3178755830},{"type":"v1","addr":"172.21.15.45:6831","nonce":3178755830}]},"public_addr":"172.21.15.45:6827/3178755830","cluster_addr":"172.21.15.45:6829/3178755830","heartbeat_back_addr":"172.21.15.45:6833/3178755830","heartbeat_front_addr":"172.21.15.45:6831/3178755830","state":["exists","up"]},{"osd":4,"uuid":"32d0b71f-c696-4677-bdd3-b88a04c97604","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":560,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6801","nonce":912745159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6803","nonce":912745159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6807","nonce":912745159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":912745159},{"type":"v1","addr":"172.21.15.144:6805","nonce":912745159}]},"public_addr":"172.21.15.144:6801/912745159","cluster_addr":"172.21.15.144:6803/912745159","heartbeat_back_addr":"172.21.15.144:6807/912745159","heartbeat_front_addr":"172.21.15.144:6805/912745159","state":["exists","up"]},{"osd":5,"uuid":"f46b91af-d67d-404c-aad1-75348e35d1c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6809","nonce":3162869174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6811","nonce":3162869174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6815","nonce":3162869174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3162869174},{"type":"v1","addr":"172.21.15.144:6813","nonce":3162869174}]},"public_addr":"172.21.15.144:6809/3162869174","cluster_addr":"172.21.15.144:6811/3162869174","heartbeat_back_addr":"172.21.15.144:6815/3162869174","heartbeat_front_addr":"172.21.15.144:6813/3162869174","state":["exists","up"]},{"osd":6,"uuid":"970b128d-cd56-4dff-8345-3196b46c1257","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":518,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6817","nonce":3439446288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6819","nonce":3439446288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6823","nonce":3439446288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":3439446288},{"type":"v1","addr":"172.21.15.144:6821","nonce":3439446288}]},"public_addr":"172.21.15.144:6817/3439446288","cluster_addr":"172.21.15.144:6819/3439446288","heartbeat_back_addr":"172.21.15.144:6823/3439446288","heartbeat_front_addr":"172.21.15.144:6821/3439446288","state":["exists","up"]},{"osd":7,"uuid":"40100aaf-830d-4137-b037-5cf167cba308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":569,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6825","nonce":2371064429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6827","nonce":2371064429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6831","nonce":2371064429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":2371064429},{"type":"v1","addr":"172.21.15.144:6829","nonce":2371064429}]},"public_addr":"172.21.15.144:6825/2371064429","cluster_addr":"172.21.15.144:6827/2371064429","heartbeat_back_addr":"172.21.15.144:6831/2371064429","heartbeat_front_addr":"172.21.15.144:6829/2371064429","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:32:27.150591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:31:22.924358+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:31:51.604504+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:32:21.519681+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:32:11.393252+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:31:26.892794+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:31:38.691207+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:31:50.785450+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.e","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"5.1f","mappings":[{"from":6,"to":0},{"from":7,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/1901139892":"2024-03-12T22:09:14.021723+0000","172.21.15.45:0/1110781721":"2024-03-12T22:03:52.557006+0000","172.21.15.45:6800/2299156110":"2024-03-12T22:03:25.087713+0000","172.21.15.45:0/2331445880":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6801/4256719031":"2024-03-12T22:03:52.557006+0000","172.21.15.45:6801/2299156110":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6800/3882489981":"2024-03-12T22:09:14.021723+0000","172.21.15.45:0/272906373":"2024-03-12T22:09:14.021723+0000","172.21.15.45:0/11898002":"2024-03-12T22:09:14.021723+0000","172.21.15.45:0/2777740355":"2024-03-12T22:03:25.087713+0000","172.21.15.45:0/1903379086":"2024-03-12T22:09:14.021723+0000","172.21.15.45:0/926341889":"2024-03-12T22:03:25.087713+0000","172.21.15.45:6800/4256719031":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/2513993410":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/2584267737":"2024-03-12T22:03:52.557006+0000","172.21.15.45:0/3509006862":"2024-03-12T22:09:14.021723+0000","172.21.15.45:6801/3882489981":"2024-03-12T22:09:14.021723+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:32:30.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:30 smithi045 ceph-mon[30178]: 3.10 scrub starts 2024-03-11T22:32:30.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:30 smithi045 ceph-mon[30178]: 3.10 scrub ok 2024-03-11T22:32:30.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:30 smithi045 ceph-mon[30178]: 4.9 scrub starts 2024-03-11T22:32:30.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:30 smithi045 ceph-mon[30178]: 4.9 scrub ok 2024-03-11T22:32:30.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:30 smithi045 ceph-mon[30178]: 5.10 deep-scrub starts 2024-03-11T22:32:30.479 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:30 smithi045 ceph-mon[30178]: 5.10 deep-scrub ok 2024-03-11T22:32:30.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:30 smithi045 ceph-mon[39715]: 3.10 scrub starts 2024-03-11T22:32:30.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:30 smithi045 ceph-mon[39715]: 3.10 scrub ok 2024-03-11T22:32:30.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:30 smithi045 ceph-mon[39715]: 4.9 scrub starts 2024-03-11T22:32:30.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:30 smithi045 ceph-mon[39715]: 4.9 scrub ok 2024-03-11T22:32:30.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:30 smithi045 ceph-mon[39715]: 5.10 deep-scrub starts 2024-03-11T22:32:30.479 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:30 smithi045 ceph-mon[39715]: 5.10 deep-scrub ok 2024-03-11T22:32:31.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:31 smithi045 ceph-mon[30178]: 6.e deep-scrub starts 2024-03-11T22:32:31.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:31 smithi045 ceph-mon[30178]: 6.e deep-scrub ok 2024-03-11T22:32:31.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:31 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3737674749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:32:31.115 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:31 smithi045 ceph-mon[30178]: pgmap v1186: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:31 smithi045 ceph-mon[39715]: 6.e deep-scrub starts 2024-03-11T22:32:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:31 smithi045 ceph-mon[39715]: 6.e deep-scrub ok 2024-03-11T22:32:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:31 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3737674749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:32:31.115 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:31 smithi045 ceph-mon[39715]: pgmap v1186: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:31.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T22:32:31.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T22:32:31.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T22:32:31.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T22:32:31.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T22:32:31.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T22:32:31.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T22:32:31.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T22:32:31.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:31 smithi144 ceph-mon[33984]: 6.e deep-scrub starts 2024-03-11T22:32:31.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:31 smithi144 ceph-mon[33984]: 6.e deep-scrub ok 2024-03-11T22:32:31.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:31 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3737674749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:32:31.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:31 smithi144 ceph-mon[33984]: pgmap v1186: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:31.709 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:31.710 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:31.711 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:31.713 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:31.714 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:31.717 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:32.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:32 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:32.137 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:32 smithi045 ceph-mon[39715]: osdmap e572: 8 total, 8 up, 8 in 2024-03-11T22:32:32.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:32 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:32.137 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:32 smithi045 ceph-mon[30178]: osdmap e572: 8 total, 8 up, 8 in 2024-03-11T22:32:32.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:32 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:32.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:32 smithi144 ceph-mon[33984]: osdmap e572: 8 total, 8 up, 8 in 2024-03-11T22:32:32.786 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:32:32 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:32:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:32:33.212 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:33 smithi045 ceph-mon[39715]: 4.15 scrub starts 2024-03-11T22:32:33.213 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:33 smithi045 ceph-mon[39715]: 4.15 scrub ok 2024-03-11T22:32:33.213 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:33 smithi045 ceph-mon[39715]: 6.b scrub starts 2024-03-11T22:32:33.213 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:33 smithi045 ceph-mon[39715]: 6.b scrub ok 2024-03-11T22:32:33.213 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:33 smithi045 ceph-mon[39715]: pgmap v1188: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:33.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:33 smithi045 ceph-mon[30178]: 4.15 scrub starts 2024-03-11T22:32:33.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:33 smithi045 ceph-mon[30178]: 4.15 scrub ok 2024-03-11T22:32:33.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:33 smithi045 ceph-mon[30178]: 6.b scrub starts 2024-03-11T22:32:33.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:33 smithi045 ceph-mon[30178]: 6.b scrub ok 2024-03-11T22:32:33.213 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:33 smithi045 ceph-mon[30178]: pgmap v1188: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:33.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:33 smithi144 ceph-mon[33984]: 4.15 scrub starts 2024-03-11T22:32:33.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:33 smithi144 ceph-mon[33984]: 4.15 scrub ok 2024-03-11T22:32:33.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:33 smithi144 ceph-mon[33984]: 6.b scrub starts 2024-03-11T22:32:33.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:33 smithi144 ceph-mon[33984]: 6.b scrub ok 2024-03-11T22:32:33.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:33 smithi144 ceph-mon[33984]: pgmap v1188: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:34.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:34 smithi045 ceph-mon[39715]: 6.3 scrub starts 2024-03-11T22:32:34.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:34 smithi045 ceph-mon[39715]: 3.17 scrub starts 2024-03-11T22:32:34.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:34 smithi045 ceph-mon[39715]: 3.17 scrub ok 2024-03-11T22:32:34.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:34 smithi045 ceph-mon[39715]: 6.3 scrub ok 2024-03-11T22:32:34.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:34 smithi045 ceph-mon[30178]: 6.3 scrub starts 2024-03-11T22:32:34.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:34 smithi045 ceph-mon[30178]: 3.17 scrub starts 2024-03-11T22:32:34.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:34 smithi045 ceph-mon[30178]: 3.17 scrub ok 2024-03-11T22:32:34.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:34 smithi045 ceph-mon[30178]: 6.3 scrub ok 2024-03-11T22:32:34.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:34 smithi144 ceph-mon[33984]: 6.3 scrub starts 2024-03-11T22:32:34.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:34 smithi144 ceph-mon[33984]: 3.17 scrub starts 2024-03-11T22:32:34.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:34 smithi144 ceph-mon[33984]: 3.17 scrub ok 2024-03-11T22:32:34.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:34 smithi144 ceph-mon[33984]: 6.3 scrub ok 2024-03-11T22:32:35.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[39715]: 2.2 scrub starts 2024-03-11T22:32:35.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[39715]: 2.2 scrub ok 2024-03-11T22:32:35.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[39715]: 3.0 scrub starts 2024-03-11T22:32:35.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[39715]: 3.0 scrub ok 2024-03-11T22:32:35.259 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[39715]: 6.1a scrub starts 2024-03-11T22:32:35.260 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[39715]: 6.1a scrub ok 2024-03-11T22:32:35.260 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[39715]: pgmap v1189: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[30178]: 2.2 scrub starts 2024-03-11T22:32:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[30178]: 2.2 scrub ok 2024-03-11T22:32:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[30178]: 3.0 scrub starts 2024-03-11T22:32:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[30178]: 3.0 scrub ok 2024-03-11T22:32:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[30178]: 6.1a scrub starts 2024-03-11T22:32:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[30178]: 6.1a scrub ok 2024-03-11T22:32:35.260 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:35 smithi045 ceph-mon[30178]: pgmap v1189: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:35.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:35 smithi144 ceph-mon[33984]: 2.2 scrub starts 2024-03-11T22:32:35.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:35 smithi144 ceph-mon[33984]: 2.2 scrub ok 2024-03-11T22:32:35.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:35 smithi144 ceph-mon[33984]: 3.0 scrub starts 2024-03-11T22:32:35.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:35 smithi144 ceph-mon[33984]: 3.0 scrub ok 2024-03-11T22:32:35.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:35 smithi144 ceph-mon[33984]: 6.1a scrub starts 2024-03-11T22:32:35.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:35 smithi144 ceph-mon[33984]: 6.1a scrub ok 2024-03-11T22:32:35.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:35 smithi144 ceph-mon[33984]: pgmap v1189: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:36.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:36 smithi045 ceph-mon[39715]: 5.1d scrub starts 2024-03-11T22:32:36.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:36 smithi045 ceph-mon[39715]: 5.1d scrub ok 2024-03-11T22:32:36.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:36 smithi045 ceph-mon[30178]: 5.1d scrub starts 2024-03-11T22:32:36.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:36 smithi045 ceph-mon[30178]: 5.1d scrub ok 2024-03-11T22:32:36.350 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:36 smithi144 ceph-mon[33984]: 5.1d scrub starts 2024-03-11T22:32:36.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:36 smithi144 ceph-mon[33984]: 5.1d scrub ok 2024-03-11T22:32:37.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:32:37 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:32:37.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:37 smithi144 ceph-mon[33984]: 4.5 deep-scrub starts 2024-03-11T22:32:37.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:37 smithi144 ceph-mon[33984]: 4.5 deep-scrub ok 2024-03-11T22:32:37.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:37 smithi144 ceph-mon[33984]: pgmap v1190: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:37.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:37 smithi144 ceph-mon[33984]: 3.14 scrub starts 2024-03-11T22:32:37.351 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:37 smithi144 ceph-mon[33984]: 3.14 scrub ok 2024-03-11T22:32:37.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:37 smithi045 ceph-mon[30178]: 4.5 deep-scrub starts 2024-03-11T22:32:37.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:37 smithi045 ceph-mon[30178]: 4.5 deep-scrub ok 2024-03-11T22:32:37.361 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:37 smithi045 ceph-mon[30178]: pgmap v1190: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:37.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:37 smithi045 ceph-mon[30178]: 3.14 scrub starts 2024-03-11T22:32:37.362 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:37 smithi045 ceph-mon[30178]: 3.14 scrub ok 2024-03-11T22:32:37.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:37 smithi045 ceph-mon[39715]: 4.5 deep-scrub starts 2024-03-11T22:32:37.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:37 smithi045 ceph-mon[39715]: 4.5 deep-scrub ok 2024-03-11T22:32:37.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:37 smithi045 ceph-mon[39715]: pgmap v1190: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:37.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:37 smithi045 ceph-mon[39715]: 3.14 scrub starts 2024-03-11T22:32:37.362 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:37 smithi045 ceph-mon[39715]: 3.14 scrub ok 2024-03-11T22:32:38.287 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:38 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:38.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:38 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:38.601 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:38 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:39.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[39715]: 4.3 scrub starts 2024-03-11T22:32:39.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[39715]: 4.3 scrub ok 2024-03-11T22:32:39.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[39715]: pgmap v1191: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:39.379 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[39715]: 4.7 scrub starts 2024-03-11T22:32:39.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[39715]: 4.7 scrub ok 2024-03-11T22:32:39.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[39715]: 5.1 scrub starts 2024-03-11T22:32:39.380 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[39715]: 5.1 scrub ok 2024-03-11T22:32:39.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[30178]: 4.3 scrub starts 2024-03-11T22:32:39.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[30178]: 4.3 scrub ok 2024-03-11T22:32:39.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[30178]: pgmap v1191: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:39.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[30178]: 4.7 scrub starts 2024-03-11T22:32:39.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[30178]: 4.7 scrub ok 2024-03-11T22:32:39.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[30178]: 5.1 scrub starts 2024-03-11T22:32:39.380 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:39 smithi045 ceph-mon[30178]: 5.1 scrub ok 2024-03-11T22:32:39.600 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:39 smithi144 ceph-mon[33984]: 4.3 scrub starts 2024-03-11T22:32:39.601 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:39 smithi144 ceph-mon[33984]: 4.3 scrub ok 2024-03-11T22:32:39.601 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:39 smithi144 ceph-mon[33984]: pgmap v1191: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-03-11T22:32:39.601 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:39 smithi144 ceph-mon[33984]: 4.7 scrub starts 2024-03-11T22:32:39.601 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:39 smithi144 ceph-mon[33984]: 4.7 scrub ok 2024-03-11T22:32:39.601 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:39 smithi144 ceph-mon[33984]: 5.1 scrub starts 2024-03-11T22:32:39.601 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:39 smithi144 ceph-mon[33984]: 5.1 scrub ok 2024-03-11T22:32:40.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:40 smithi045 ceph-mon[39715]: 4.19 scrub starts 2024-03-11T22:32:40.530 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:40 smithi045 ceph-mon[39715]: 4.19 scrub ok 2024-03-11T22:32:40.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:40 smithi045 ceph-mon[30178]: 4.19 scrub starts 2024-03-11T22:32:40.530 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:40 smithi045 ceph-mon[30178]: 4.19 scrub ok 2024-03-11T22:32:40.600 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:40 smithi144 ceph-mon[33984]: 4.19 scrub starts 2024-03-11T22:32:40.600 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:40 smithi144 ceph-mon[33984]: 4.19 scrub ok 2024-03-11T22:32:41.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:41 smithi045 ceph-mon[39715]: pgmap v1192: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:32:41.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:41 smithi045 ceph-mon[39715]: 3.3 scrub starts 2024-03-11T22:32:41.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:41 smithi045 ceph-mon[39715]: 3.3 scrub ok 2024-03-11T22:32:41.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:41 smithi045 ceph-mon[39715]: 4.0 scrub starts 2024-03-11T22:32:41.537 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:41 smithi045 ceph-mon[39715]: 4.0 scrub ok 2024-03-11T22:32:41.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:41 smithi045 ceph-mon[30178]: pgmap v1192: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:32:41.537 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:41 smithi045 ceph-mon[30178]: 3.3 scrub starts 2024-03-11T22:32:41.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:41 smithi045 ceph-mon[30178]: 3.3 scrub ok 2024-03-11T22:32:41.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:41 smithi045 ceph-mon[30178]: 4.0 scrub starts 2024-03-11T22:32:41.538 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:41 smithi045 ceph-mon[30178]: 4.0 scrub ok 2024-03-11T22:32:41.601 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:41 smithi144 ceph-mon[33984]: pgmap v1192: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:32:41.601 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:41 smithi144 ceph-mon[33984]: 3.3 scrub starts 2024-03-11T22:32:41.601 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:41 smithi144 ceph-mon[33984]: 3.3 scrub ok 2024-03-11T22:32:41.601 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:41 smithi144 ceph-mon[33984]: 4.0 scrub starts 2024-03-11T22:32:41.601 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:41 smithi144 ceph-mon[33984]: 4.0 scrub ok 2024-03-11T22:32:42.136 INFO:teuthology.orchestra.run.smithi045.stdout:141733921078 2024-03-11T22:32:42.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T22:32:42.682 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:32:42 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:32:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:32:42.967 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:42 smithi045 ceph-mon[39715]: 6.d scrub starts 2024-03-11T22:32:42.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:42 smithi045 ceph-mon[39715]: 6.d scrub ok 2024-03-11T22:32:42.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:42 smithi045 ceph-mon[39715]: 5.6 scrub starts 2024-03-11T22:32:42.968 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:42 smithi045 ceph-mon[39715]: 5.6 scrub ok 2024-03-11T22:32:42.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:42 smithi045 ceph-mon[30178]: 6.d scrub starts 2024-03-11T22:32:42.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:42 smithi045 ceph-mon[30178]: 6.d scrub ok 2024-03-11T22:32:42.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:42 smithi045 ceph-mon[30178]: 5.6 scrub starts 2024-03-11T22:32:42.968 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:42 smithi045 ceph-mon[30178]: 5.6 scrub ok 2024-03-11T22:32:43.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:42 smithi144 ceph-mon[33984]: 6.d scrub starts 2024-03-11T22:32:43.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:42 smithi144 ceph-mon[33984]: 6.d scrub ok 2024-03-11T22:32:43.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:42 smithi144 ceph-mon[33984]: 5.6 scrub starts 2024-03-11T22:32:43.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:42 smithi144 ceph-mon[33984]: 5.6 scrub ok 2024-03-11T22:32:44.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:43 smithi045 ceph-mon[39715]: pgmap v1193: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 939 B/s rd, 0 op/s 2024-03-11T22:32:44.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:43 smithi045 ceph-mon[39715]: 6.15 scrub starts 2024-03-11T22:32:44.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:43 smithi045 ceph-mon[39715]: 6.15 scrub ok 2024-03-11T22:32:44.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:43 smithi045 ceph-mon[39715]: 6.1b scrub starts 2024-03-11T22:32:44.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:43 smithi045 ceph-mon[39715]: 5.1e scrub starts 2024-03-11T22:32:44.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:43 smithi045 ceph-mon[39715]: 5.1e scrub ok 2024-03-11T22:32:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:43 smithi045 ceph-mon[30178]: pgmap v1193: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 939 B/s rd, 0 op/s 2024-03-11T22:32:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:43 smithi045 ceph-mon[30178]: 6.15 scrub starts 2024-03-11T22:32:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:43 smithi045 ceph-mon[30178]: 6.15 scrub ok 2024-03-11T22:32:44.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:43 smithi045 ceph-mon[30178]: 6.1b scrub starts 2024-03-11T22:32:44.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:43 smithi045 ceph-mon[30178]: 5.1e scrub starts 2024-03-11T22:32:44.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:43 smithi045 ceph-mon[30178]: 5.1e scrub ok 2024-03-11T22:32:44.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:43 smithi144 ceph-mon[33984]: pgmap v1193: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 939 B/s rd, 0 op/s 2024-03-11T22:32:44.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:43 smithi144 ceph-mon[33984]: 6.15 scrub starts 2024-03-11T22:32:44.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:43 smithi144 ceph-mon[33984]: 6.15 scrub ok 2024-03-11T22:32:44.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:43 smithi144 ceph-mon[33984]: 6.1b scrub starts 2024-03-11T22:32:44.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:43 smithi144 ceph-mon[33984]: 5.1e scrub starts 2024-03-11T22:32:44.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:43 smithi144 ceph-mon[33984]: 5.1e scrub ok 2024-03-11T22:32:45.013 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:44 smithi045 ceph-mon[39715]: 6.1b scrub ok 2024-03-11T22:32:45.013 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:44 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:32:45.013 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:44 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:32:45.013 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:44 smithi045 ceph-mon[30178]: 6.1b scrub ok 2024-03-11T22:32:45.014 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:44 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:32:45.014 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:44 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:32:45.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:44 smithi144 ceph-mon[33984]: 6.1b scrub ok 2024-03-11T22:32:45.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:44 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:32:45.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:44 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:32:46.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:45 smithi045 ceph-mon[39715]: pgmap v1194: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:32:46.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:45 smithi045 ceph-mon[39715]: osdmap e573: 8 total, 8 up, 8 in 2024-03-11T22:32:46.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:45 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:32:46.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:45 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:32:46.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:45 smithi045 ceph-mon[30178]: pgmap v1194: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:32:46.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:45 smithi045 ceph-mon[30178]: osdmap e573: 8 total, 8 up, 8 in 2024-03-11T22:32:46.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:45 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:32:46.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:45 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:32:46.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:45 smithi144 ceph-mon[33984]: pgmap v1194: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T22:32:46.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:45 smithi144 ceph-mon[33984]: osdmap e573: 8 total, 8 up, 8 in 2024-03-11T22:32:46.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:45 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:32:46.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:45 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:32:46.544 INFO:teuthology.orchestra.run.smithi045.stdout:55834575174 2024-03-11T22:32:46.545 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T22:32:46.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:46 smithi045 ceph-mon[30178]: osdmap e574: 8 total, 8 up, 8 in 2024-03-11T22:32:46.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:46 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:46.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:46 smithi045 ceph-mon[39715]: osdmap e574: 8 total, 8 up, 8 in 2024-03-11T22:32:46.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:46 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:46.939 INFO:teuthology.orchestra.run.smithi045.stdout:34359738700 2024-03-11T22:32:46.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T22:32:47.067 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:46 smithi144 ceph-mon[33984]: osdmap e574: 8 total, 8 up, 8 in 2024-03-11T22:32:47.067 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:46 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:32:47.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:32:47 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:32:47.415 INFO:teuthology.orchestra.run.smithi045.stdout:111669150011 2024-03-11T22:32:47.416 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T22:32:47.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:47 smithi045 ceph-mon[39715]: pgmap v1197: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:47.956 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:47 smithi045 ceph-mon[30178]: pgmap v1197: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:48.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:47 smithi144 ceph-mon[33984]: pgmap v1197: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:48.928 INFO:teuthology.orchestra.run.smithi045.stdout:261993005352 2024-03-11T22:32:48.929 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T22:32:48.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[39715]: 6.2 scrub starts 2024-03-11T22:32:48.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[39715]: 6.2 scrub ok 2024-03-11T22:32:48.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:48.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[39715]: 1.1 scrub starts 2024-03-11T22:32:48.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[39715]: 1.1 scrub ok 2024-03-11T22:32:48.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[39715]: 6.6 scrub starts 2024-03-11T22:32:48.956 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[39715]: 6.6 scrub ok 2024-03-11T22:32:48.956 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[30178]: 6.2 scrub starts 2024-03-11T22:32:48.957 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[30178]: 6.2 scrub ok 2024-03-11T22:32:48.957 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:48.957 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[30178]: 1.1 scrub starts 2024-03-11T22:32:48.957 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[30178]: 1.1 scrub ok 2024-03-11T22:32:48.957 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[30178]: 6.6 scrub starts 2024-03-11T22:32:48.957 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:48 smithi045 ceph-mon[30178]: 6.6 scrub ok 2024-03-11T22:32:49.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:48 smithi144 ceph-mon[33984]: 6.2 scrub starts 2024-03-11T22:32:49.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:48 smithi144 ceph-mon[33984]: 6.2 scrub ok 2024-03-11T22:32:49.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:48 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:49.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:48 smithi144 ceph-mon[33984]: 1.1 scrub starts 2024-03-11T22:32:49.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:48 smithi144 ceph-mon[33984]: 1.1 scrub ok 2024-03-11T22:32:49.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:48 smithi144 ceph-mon[33984]: 6.6 scrub starts 2024-03-11T22:32:49.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:48 smithi144 ceph-mon[33984]: 6.6 scrub ok 2024-03-11T22:32:49.541 INFO:teuthology.orchestra.run.smithi045.stdout:77309411649 2024-03-11T22:32:49.541 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T22:32:49.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:49 smithi045 ceph-mon[39715]: pgmap v1198: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-03-11T22:32:49.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:49 smithi045 ceph-mon[39715]: 6.18 scrub starts 2024-03-11T22:32:49.961 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:49 smithi045 ceph-mon[39715]: 6.18 scrub ok 2024-03-11T22:32:49.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:49 smithi045 ceph-mon[30178]: pgmap v1198: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-03-11T22:32:49.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:49 smithi045 ceph-mon[30178]: 6.18 scrub starts 2024-03-11T22:32:49.962 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:49 smithi045 ceph-mon[30178]: 6.18 scrub ok 2024-03-11T22:32:49.988 INFO:teuthology.orchestra.run.smithi045.stdout:219043332396 2024-03-11T22:32:49.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T22:32:50.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:49 smithi144 ceph-mon[33984]: pgmap v1198: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-03-11T22:32:50.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:49 smithi144 ceph-mon[33984]: 6.18 scrub starts 2024-03-11T22:32:50.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:49 smithi144 ceph-mon[33984]: 6.18 scrub ok 2024-03-11T22:32:50.354 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:50.355 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:50.979 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:50 smithi045 ceph-mon[30178]: 3.1a scrub starts 2024-03-11T22:32:50.979 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:50 smithi045 ceph-mon[30178]: 3.1a scrub ok 2024-03-11T22:32:50.979 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:50 smithi045 ceph-mon[30178]: 4.6 scrub starts 2024-03-11T22:32:50.979 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:50 smithi045 ceph-mon[30178]: 4.6 scrub ok 2024-03-11T22:32:50.979 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:50 smithi045 ceph-mon[39715]: 3.1a scrub starts 2024-03-11T22:32:50.980 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:50 smithi045 ceph-mon[39715]: 3.1a scrub ok 2024-03-11T22:32:50.980 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:50 smithi045 ceph-mon[39715]: 4.6 scrub starts 2024-03-11T22:32:50.980 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:50 smithi045 ceph-mon[39715]: 4.6 scrub ok 2024-03-11T22:32:51.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:50 smithi144 ceph-mon[33984]: 3.1a scrub starts 2024-03-11T22:32:51.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:50 smithi144 ceph-mon[33984]: 3.1a scrub ok 2024-03-11T22:32:51.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:50 smithi144 ceph-mon[33984]: 4.6 scrub starts 2024-03-11T22:32:51.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:50 smithi144 ceph-mon[33984]: 4.6 scrub ok 2024-03-11T22:32:51.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:51 smithi045 ceph-mon[30178]: pgmap v1199: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:51.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:51 smithi045 ceph-mon[30178]: 4.16 scrub starts 2024-03-11T22:32:51.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:51 smithi045 ceph-mon[30178]: 4.16 scrub ok 2024-03-11T22:32:51.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:51 smithi045 ceph-mon[30178]: 4.12 scrub starts 2024-03-11T22:32:51.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:51 smithi045 ceph-mon[30178]: 4.12 scrub ok 2024-03-11T22:32:51.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:51 smithi045 ceph-mon[39715]: pgmap v1199: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:51.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:51 smithi045 ceph-mon[39715]: 4.16 scrub starts 2024-03-11T22:32:51.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:51 smithi045 ceph-mon[39715]: 4.16 scrub ok 2024-03-11T22:32:51.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:51 smithi045 ceph-mon[39715]: 4.12 scrub starts 2024-03-11T22:32:51.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:51 smithi045 ceph-mon[39715]: 4.12 scrub ok 2024-03-11T22:32:52.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:51 smithi144 ceph-mon[33984]: pgmap v1199: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:52.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:51 smithi144 ceph-mon[33984]: 4.16 scrub starts 2024-03-11T22:32:52.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:51 smithi144 ceph-mon[33984]: 4.16 scrub ok 2024-03-11T22:32:52.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:51 smithi144 ceph-mon[33984]: 4.12 scrub starts 2024-03-11T22:32:52.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:51 smithi144 ceph-mon[33984]: 4.12 scrub ok 2024-03-11T22:32:52.915 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:32:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:32:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:32:52.915 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:52 smithi045 ceph-mon[39715]: 4.11 scrub starts 2024-03-11T22:32:52.915 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:52 smithi045 ceph-mon[39715]: 4.11 scrub ok 2024-03-11T22:32:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:52 smithi045 ceph-mon[30178]: 4.11 scrub starts 2024-03-11T22:32:52.916 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:52 smithi045 ceph-mon[30178]: 4.11 scrub ok 2024-03-11T22:32:53.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:52 smithi144 ceph-mon[33984]: 4.11 scrub starts 2024-03-11T22:32:53.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:52 smithi144 ceph-mon[33984]: 4.11 scrub ok 2024-03-11T22:32:53.370 INFO:teuthology.orchestra.run.smithi045.stdout:176093659441 2024-03-11T22:32:53.370 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T22:32:53.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[30178]: 5.11 scrub starts 2024-03-11T22:32:53.869 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[30178]: 5.11 scrub ok 2024-03-11T22:32:53.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[30178]: pgmap v1200: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:53.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[30178]: 6.8 scrub starts 2024-03-11T22:32:53.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[30178]: 6.8 scrub ok 2024-03-11T22:32:53.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[30178]: 6.c scrub starts 2024-03-11T22:32:53.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[30178]: 6.c scrub ok 2024-03-11T22:32:53.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[30178]: 4.2 scrub starts 2024-03-11T22:32:53.870 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[30178]: 4.2 scrub ok 2024-03-11T22:32:53.870 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[39715]: 5.11 scrub starts 2024-03-11T22:32:53.870 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[39715]: 5.11 scrub ok 2024-03-11T22:32:53.870 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[39715]: pgmap v1200: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:53.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[39715]: 6.8 scrub starts 2024-03-11T22:32:53.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[39715]: 6.8 scrub ok 2024-03-11T22:32:53.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[39715]: 6.c scrub starts 2024-03-11T22:32:53.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[39715]: 6.c scrub ok 2024-03-11T22:32:53.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[39715]: 4.2 scrub starts 2024-03-11T22:32:53.871 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:53 smithi045 ceph-mon[39715]: 4.2 scrub ok 2024-03-11T22:32:54.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:53 smithi144 ceph-mon[33984]: 5.11 scrub starts 2024-03-11T22:32:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:53 smithi144 ceph-mon[33984]: 5.11 scrub ok 2024-03-11T22:32:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:53 smithi144 ceph-mon[33984]: pgmap v1200: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:32:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:53 smithi144 ceph-mon[33984]: 6.8 scrub starts 2024-03-11T22:32:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:53 smithi144 ceph-mon[33984]: 6.8 scrub ok 2024-03-11T22:32:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:53 smithi144 ceph-mon[33984]: 6.c scrub starts 2024-03-11T22:32:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:53 smithi144 ceph-mon[33984]: 6.c scrub ok 2024-03-11T22:32:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:53 smithi144 ceph-mon[33984]: 4.2 scrub starts 2024-03-11T22:32:54.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:53 smithi144 ceph-mon[33984]: 4.2 scrub ok 2024-03-11T22:32:54.297 INFO:teuthology.orchestra.run.smithi045.stdout:141733921081 2024-03-11T22:32:54.549 INFO:teuthology.orchestra.run.smithi045.stdout:34359738701 2024-03-11T22:32:54.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[39715]: 4.18 scrub starts 2024-03-11T22:32:54.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[39715]: 4.18 scrub ok 2024-03-11T22:32:54.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[39715]: 6.1c scrub starts 2024-03-11T22:32:54.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[39715]: 6.1c scrub ok 2024-03-11T22:32:54.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[39715]: 6.1f deep-scrub starts 2024-03-11T22:32:54.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3911227078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T22:32:54.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[39715]: 6.1d scrub starts 2024-03-11T22:32:54.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3055071315' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:32:54.946 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[39715]: 6.1d scrub ok 2024-03-11T22:32:54.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[30178]: 4.18 scrub starts 2024-03-11T22:32:54.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[30178]: 4.18 scrub ok 2024-03-11T22:32:54.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[30178]: 6.1c scrub starts 2024-03-11T22:32:54.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[30178]: 6.1c scrub ok 2024-03-11T22:32:54.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[30178]: 6.1f deep-scrub starts 2024-03-11T22:32:54.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3911227078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T22:32:54.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[30178]: 6.1d scrub starts 2024-03-11T22:32:54.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3055071315' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:32:54.947 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:54 smithi045 ceph-mon[30178]: 6.1d scrub ok 2024-03-11T22:32:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:54.969 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:54.970 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:54.990 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733921078 got 141733921081 for osd.4 2024-03-11T22:32:54.991 DEBUG:teuthology.parallel:result is None 2024-03-11T22:32:55.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:54 smithi144 ceph-mon[33984]: 4.18 scrub starts 2024-03-11T22:32:55.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:54 smithi144 ceph-mon[33984]: 4.18 scrub ok 2024-03-11T22:32:55.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:54 smithi144 ceph-mon[33984]: 6.1c scrub starts 2024-03-11T22:32:55.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:54 smithi144 ceph-mon[33984]: 6.1c scrub ok 2024-03-11T22:32:55.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:54 smithi144 ceph-mon[33984]: 6.1f deep-scrub starts 2024-03-11T22:32:55.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:54 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3911227078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T22:32:55.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:54 smithi144 ceph-mon[33984]: 6.1d scrub starts 2024-03-11T22:32:55.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:54 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3055071315' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:32:55.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:54 smithi144 ceph-mon[33984]: 6.1d scrub ok 2024-03-11T22:32:55.247 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:56.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:55 smithi045 ceph-mon[30178]: 6.1f deep-scrub ok 2024-03-11T22:32:56.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:55 smithi045 ceph-mon[30178]: pgmap v1201: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:32:56.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:55 smithi045 ceph-mon[39715]: 6.1f deep-scrub ok 2024-03-11T22:32:56.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:55 smithi045 ceph-mon[39715]: pgmap v1201: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:32:56.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:55 smithi144 ceph-mon[33984]: 6.1f deep-scrub ok 2024-03-11T22:32:56.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:55 smithi144 ceph-mon[33984]: pgmap v1201: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:32:56.151 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738700 got 34359738701 for osd.0 2024-03-11T22:32:56.151 DEBUG:teuthology.parallel:result is None 2024-03-11T22:32:57.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:56 smithi045 ceph-mon[39715]: 5.5 scrub starts 2024-03-11T22:32:57.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:56 smithi045 ceph-mon[39715]: 5.5 scrub ok 2024-03-11T22:32:57.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:56 smithi045 ceph-mon[30178]: 5.5 scrub starts 2024-03-11T22:32:57.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:56 smithi045 ceph-mon[30178]: 5.5 scrub ok 2024-03-11T22:32:57.070 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:56 smithi144 ceph-mon[33984]: 5.5 scrub starts 2024-03-11T22:32:57.070 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:56 smithi144 ceph-mon[33984]: 5.5 scrub ok 2024-03-11T22:32:57.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:32:57 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:32:57.569 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:32:57.985 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[39715]: 4.d scrub starts 2024-03-11T22:32:57.986 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[39715]: 4.d scrub ok 2024-03-11T22:32:57.986 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[39715]: pgmap v1202: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-11T22:32:57.986 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[39715]: 4.1c scrub starts 2024-03-11T22:32:57.986 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[39715]: 4.1c scrub ok 2024-03-11T22:32:57.986 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:32:57.986 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:32:57.986 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[39715]: osdmap e575: 8 total, 8 up, 8 in 2024-03-11T22:32:57.986 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[39715]: 1.4 scrub starts 2024-03-11T22:32:57.986 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[39715]: 1.4 scrub ok 2024-03-11T22:32:57.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[30178]: 4.d scrub starts 2024-03-11T22:32:57.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[30178]: 4.d scrub ok 2024-03-11T22:32:57.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[30178]: pgmap v1202: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-11T22:32:57.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[30178]: 4.1c scrub starts 2024-03-11T22:32:57.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[30178]: 4.1c scrub ok 2024-03-11T22:32:57.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:32:57.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:32:57.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[30178]: osdmap e575: 8 total, 8 up, 8 in 2024-03-11T22:32:57.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[30178]: 1.4 scrub starts 2024-03-11T22:32:57.987 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:57 smithi045 ceph-mon[30178]: 1.4 scrub ok 2024-03-11T22:32:58.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:57 smithi144 ceph-mon[33984]: 4.d scrub starts 2024-03-11T22:32:58.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:57 smithi144 ceph-mon[33984]: 4.d scrub ok 2024-03-11T22:32:58.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:57 smithi144 ceph-mon[33984]: pgmap v1202: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 941 B/s rd, 0 op/s 2024-03-11T22:32:58.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:57 smithi144 ceph-mon[33984]: 4.1c scrub starts 2024-03-11T22:32:58.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:57 smithi144 ceph-mon[33984]: 4.1c scrub ok 2024-03-11T22:32:58.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:57 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:32:58.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:57 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:32:58.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:57 smithi144 ceph-mon[33984]: osdmap e575: 8 total, 8 up, 8 in 2024-03-11T22:32:58.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:57 smithi144 ceph-mon[33984]: 1.4 scrub starts 2024-03-11T22:32:58.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:57 smithi144 ceph-mon[33984]: 1.4 scrub ok 2024-03-11T22:32:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[39715]: 6.17 scrub starts 2024-03-11T22:32:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[39715]: 6.17 scrub ok 2024-03-11T22:32:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[39715]: 4.a scrub starts 2024-03-11T22:32:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[39715]: 4.a scrub ok 2024-03-11T22:32:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[39715]: 1.9 scrub starts 2024-03-11T22:32:59.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[39715]: 1.9 scrub ok 2024-03-11T22:32:59.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[30178]: 6.17 scrub starts 2024-03-11T22:32:59.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:59.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[30178]: 6.17 scrub ok 2024-03-11T22:32:59.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[30178]: 4.a scrub starts 2024-03-11T22:32:59.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[30178]: 4.a scrub ok 2024-03-11T22:32:59.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[30178]: 1.9 scrub starts 2024-03-11T22:32:59.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:58 smithi045 ceph-mon[30178]: 1.9 scrub ok 2024-03-11T22:32:59.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:58 smithi144 ceph-mon[33984]: 6.17 scrub starts 2024-03-11T22:32:59.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:58 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:32:59.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:58 smithi144 ceph-mon[33984]: 6.17 scrub ok 2024-03-11T22:32:59.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:58 smithi144 ceph-mon[33984]: 4.a scrub starts 2024-03-11T22:32:59.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:58 smithi144 ceph-mon[33984]: 4.a scrub ok 2024-03-11T22:32:59.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:58 smithi144 ceph-mon[33984]: 1.9 scrub starts 2024-03-11T22:32:59.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:58 smithi144 ceph-mon[33984]: 1.9 scrub ok 2024-03-11T22:33:00.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:59 smithi045 ceph-mon[39715]: pgmap v1204: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-11T22:33:00.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:59 smithi045 ceph-mon[39715]: 3.1e deep-scrub starts 2024-03-11T22:33:00.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:32:59 smithi045 ceph-mon[39715]: 3.1e deep-scrub ok 2024-03-11T22:33:00.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:59 smithi045 ceph-mon[30178]: pgmap v1204: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-11T22:33:00.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:59 smithi045 ceph-mon[30178]: 3.1e deep-scrub starts 2024-03-11T22:33:00.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:32:59 smithi045 ceph-mon[30178]: 3.1e deep-scrub ok 2024-03-11T22:33:00.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:59 smithi144 ceph-mon[33984]: pgmap v1204: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-11T22:33:00.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:59 smithi144 ceph-mon[33984]: 3.1e deep-scrub starts 2024-03-11T22:33:00.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:32:59 smithi144 ceph-mon[33984]: 3.1e deep-scrub ok 2024-03-11T22:33:00.722 INFO:teuthology.orchestra.run.smithi045.stdout:261993005354 2024-03-11T22:33:01.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[39715]: 3.1f scrub starts 2024-03-11T22:33:01.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[39715]: 3.1f scrub ok 2024-03-11T22:33:01.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[39715]: 5.9 scrub starts 2024-03-11T22:33:01.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[39715]: 5.9 scrub ok 2024-03-11T22:33:01.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[39715]: 4.10 scrub starts 2024-03-11T22:33:01.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[39715]: 4.10 scrub ok 2024-03-11T22:33:01.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[39715]: 6.4 deep-scrub starts 2024-03-11T22:33:01.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[39715]: 6.4 deep-scrub ok 2024-03-11T22:33:01.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/3287542744' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T22:33:01.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[30178]: 3.1f scrub starts 2024-03-11T22:33:01.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[30178]: 3.1f scrub ok 2024-03-11T22:33:01.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[30178]: 5.9 scrub starts 2024-03-11T22:33:01.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[30178]: 5.9 scrub ok 2024-03-11T22:33:01.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[30178]: 4.10 scrub starts 2024-03-11T22:33:01.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[30178]: 4.10 scrub ok 2024-03-11T22:33:01.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[30178]: 6.4 deep-scrub starts 2024-03-11T22:33:01.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[30178]: 6.4 deep-scrub ok 2024-03-11T22:33:01.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:00 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/3287542744' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T22:33:01.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:00 smithi144 ceph-mon[33984]: 3.1f scrub starts 2024-03-11T22:33:01.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:00 smithi144 ceph-mon[33984]: 3.1f scrub ok 2024-03-11T22:33:01.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:00 smithi144 ceph-mon[33984]: 5.9 scrub starts 2024-03-11T22:33:01.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:00 smithi144 ceph-mon[33984]: 5.9 scrub ok 2024-03-11T22:33:01.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:00 smithi144 ceph-mon[33984]: 4.10 scrub starts 2024-03-11T22:33:01.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:00 smithi144 ceph-mon[33984]: 4.10 scrub ok 2024-03-11T22:33:01.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:00 smithi144 ceph-mon[33984]: 6.4 deep-scrub starts 2024-03-11T22:33:01.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:00 smithi144 ceph-mon[33984]: 6.4 deep-scrub ok 2024-03-11T22:33:01.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:00 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/3287542744' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T22:33:01.912 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[39715]: pgmap v1205: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:33:01.912 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[39715]: 2.1 scrub starts 2024-03-11T22:33:01.912 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[39715]: 2.1 scrub ok 2024-03-11T22:33:01.912 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[39715]: 5.e scrub starts 2024-03-11T22:33:01.912 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[39715]: 5.e scrub ok 2024-03-11T22:33:01.912 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:01.912 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[39715]: osdmap e576: 8 total, 8 up, 8 in 2024-03-11T22:33:01.912 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[30178]: pgmap v1205: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:33:01.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[30178]: 2.1 scrub starts 2024-03-11T22:33:01.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[30178]: 2.1 scrub ok 2024-03-11T22:33:01.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[30178]: 5.e scrub starts 2024-03-11T22:33:01.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[30178]: 5.e scrub ok 2024-03-11T22:33:01.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:01.913 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:01 smithi045 ceph-mon[30178]: osdmap e576: 8 total, 8 up, 8 in 2024-03-11T22:33:02.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:01 smithi144 ceph-mon[33984]: pgmap v1205: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:33:02.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:01 smithi144 ceph-mon[33984]: 2.1 scrub starts 2024-03-11T22:33:02.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:01 smithi144 ceph-mon[33984]: 2.1 scrub ok 2024-03-11T22:33:02.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:01 smithi144 ceph-mon[33984]: 5.e scrub starts 2024-03-11T22:33:02.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:01 smithi144 ceph-mon[33984]: 5.e scrub ok 2024-03-11T22:33:02.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:01 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:02.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:01 smithi144 ceph-mon[33984]: osdmap e576: 8 total, 8 up, 8 in 2024-03-11T22:33:02.899 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:02 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:33:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:33:02.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:02 smithi045 ceph-mon[39715]: purged_snaps scrub starts 2024-03-11T22:33:02.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:02 smithi045 ceph-mon[39715]: purged_snaps scrub ok 2024-03-11T22:33:02.900 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:02 smithi045 ceph-mon[39715]: 6.f scrub starts 2024-03-11T22:33:02.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:02 smithi045 ceph-mon[30178]: purged_snaps scrub starts 2024-03-11T22:33:02.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:02 smithi045 ceph-mon[30178]: purged_snaps scrub ok 2024-03-11T22:33:02.900 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:02 smithi045 ceph-mon[30178]: 6.f scrub starts 2024-03-11T22:33:03.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:02 smithi144 ceph-mon[33984]: purged_snaps scrub starts 2024-03-11T22:33:03.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:02 smithi144 ceph-mon[33984]: purged_snaps scrub ok 2024-03-11T22:33:03.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:02 smithi144 ceph-mon[33984]: 6.f scrub starts 2024-03-11T22:33:04.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:03 smithi045 ceph-mon[39715]: 6.f scrub ok 2024-03-11T22:33:04.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:03 smithi045 ceph-mon[39715]: pgmap v1207: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:33:04.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:03 smithi045 ceph-mon[39715]: osdmap e577: 8 total, 8 up, 8 in 2024-03-11T22:33:04.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:03 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/384244947' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:33:04.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:03 smithi045 ceph-mon[30178]: 6.f scrub ok 2024-03-11T22:33:04.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:03 smithi045 ceph-mon[30178]: pgmap v1207: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:33:04.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:03 smithi045 ceph-mon[30178]: osdmap e577: 8 total, 8 up, 8 in 2024-03-11T22:33:04.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:03 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/384244947' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:33:04.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:03 smithi144 ceph-mon[33984]: 6.f scrub ok 2024-03-11T22:33:04.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:03 smithi144 ceph-mon[33984]: pgmap v1207: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:33:04.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:03 smithi144 ceph-mon[33984]: osdmap e577: 8 total, 8 up, 8 in 2024-03-11T22:33:04.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:03 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/384244947' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:33:04.330 INFO:teuthology.orchestra.run.smithi045.stdout:77309411653 2024-03-11T22:33:04.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:04 smithi045 ceph-mon[39715]: 6.1e scrub starts 2024-03-11T22:33:04.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:04 smithi045 ceph-mon[39715]: 6.1e scrub ok 2024-03-11T22:33:04.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:04 smithi045 ceph-mon[39715]: 5.19 scrub starts 2024-03-11T22:33:04.876 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:04 smithi045 ceph-mon[39715]: 5.19 scrub ok 2024-03-11T22:33:04.877 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:04 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/24570461' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T22:33:04.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:04 smithi045 ceph-mon[30178]: 6.1e scrub starts 2024-03-11T22:33:04.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:04 smithi045 ceph-mon[30178]: 6.1e scrub ok 2024-03-11T22:33:04.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:04 smithi045 ceph-mon[30178]: 5.19 scrub starts 2024-03-11T22:33:04.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:04 smithi045 ceph-mon[30178]: 5.19 scrub ok 2024-03-11T22:33:04.877 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:04 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/24570461' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T22:33:05.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:04 smithi144 ceph-mon[33984]: 6.1e scrub starts 2024-03-11T22:33:05.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:04 smithi144 ceph-mon[33984]: 6.1e scrub ok 2024-03-11T22:33:05.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:04 smithi144 ceph-mon[33984]: 5.19 scrub starts 2024-03-11T22:33:05.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:04 smithi144 ceph-mon[33984]: 5.19 scrub ok 2024-03-11T22:33:05.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:04 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/24570461' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T22:33:05.203 INFO:teuthology.orchestra.run.smithi045.stdout:55834575178 2024-03-11T22:33:05.555 INFO:teuthology.orchestra.run.smithi045.stdout:111669150015 2024-03-11T22:33:05.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:05 smithi045 ceph-mon[39715]: pgmap v1209: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-03-11T22:33:05.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:05 smithi045 ceph-mon[39715]: 3.f scrub starts 2024-03-11T22:33:05.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:05 smithi045 ceph-mon[39715]: 3.f scrub ok 2024-03-11T22:33:05.859 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:05 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2634529897' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:33:05.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:05 smithi045 ceph-mon[30178]: pgmap v1209: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-03-11T22:33:05.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:05 smithi045 ceph-mon[30178]: 3.f scrub starts 2024-03-11T22:33:05.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:05 smithi045 ceph-mon[30178]: 3.f scrub ok 2024-03-11T22:33:05.860 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:05 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2634529897' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:33:06.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:05 smithi144 ceph-mon[33984]: pgmap v1209: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-03-11T22:33:06.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:05 smithi144 ceph-mon[33984]: 3.f scrub starts 2024-03-11T22:33:06.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:05 smithi144 ceph-mon[33984]: 3.f scrub ok 2024-03-11T22:33:06.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:05 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2634529897' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:33:06.627 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005352 got 261993005354 for osd.7 2024-03-11T22:33:06.627 DEBUG:teuthology.parallel:result is None 2024-03-11T22:33:07.017 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[39715]: 6.9 scrub starts 2024-03-11T22:33:07.017 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[39715]: 5.18 scrub starts 2024-03-11T22:33:07.017 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[39715]: 5.18 scrub ok 2024-03-11T22:33:07.017 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[39715]: 6.9 scrub ok 2024-03-11T22:33:07.017 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/1826607429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T22:33:07.017 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/163184523' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T22:33:07.018 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[39715]: osdmap e578: 8 total, 8 up, 8 in 2024-03-11T22:33:07.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[30178]: 6.9 scrub starts 2024-03-11T22:33:07.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[30178]: 5.18 scrub starts 2024-03-11T22:33:07.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[30178]: 5.18 scrub ok 2024-03-11T22:33:07.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[30178]: 6.9 scrub ok 2024-03-11T22:33:07.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/1826607429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T22:33:07.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/163184523' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T22:33:07.018 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:06 smithi045 ceph-mon[30178]: osdmap e578: 8 total, 8 up, 8 in 2024-03-11T22:33:07.043 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575174 got 55834575178 for osd.1 2024-03-11T22:33:07.043 DEBUG:teuthology.parallel:result is None 2024-03-11T22:33:07.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:06 smithi144 ceph-mon[33984]: 6.9 scrub starts 2024-03-11T22:33:07.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:06 smithi144 ceph-mon[33984]: 5.18 scrub starts 2024-03-11T22:33:07.081 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:06 smithi144 ceph-mon[33984]: 5.18 scrub ok 2024-03-11T22:33:07.082 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:06 smithi144 ceph-mon[33984]: 6.9 scrub ok 2024-03-11T22:33:07.082 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:06 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/1826607429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T22:33:07.082 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:06 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/163184523' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T22:33:07.082 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:06 smithi144 ceph-mon[33984]: osdmap e578: 8 total, 8 up, 8 in 2024-03-11T22:33:07.125 INFO:teuthology.orchestra.run.smithi045.stdout:176093659444 2024-03-11T22:33:07.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:33:07 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:33:07.828 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150011 got 111669150015 for osd.3 2024-03-11T22:33:07.828 DEBUG:teuthology.parallel:result is None 2024-03-11T22:33:07.905 INFO:teuthology.orchestra.run.smithi045.stdout:219043332400 2024-03-11T22:33:08.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:07 smithi045 ceph-mon[39715]: pgmap v1210: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:33:08.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:07 smithi045 ceph-mon[30178]: pgmap v1210: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:33:08.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:07 smithi144 ceph-mon[33984]: pgmap v1210: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:33:08.552 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332396 got 219043332400 for osd.6 2024-03-11T22:33:08.552 DEBUG:teuthology.parallel:result is None 2024-03-11T22:33:08.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:08 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:33:08.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:08 smithi045 ceph-mon[30178]: 3.12 deep-scrub starts 2024-03-11T22:33:08.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:08 smithi045 ceph-mon[30178]: 3.12 deep-scrub ok 2024-03-11T22:33:08.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:08 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:33:08.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:08 smithi045 ceph-mon[39715]: 3.12 deep-scrub starts 2024-03-11T22:33:08.787 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:08 smithi045 ceph-mon[39715]: 3.12 deep-scrub ok 2024-03-11T22:33:08.971 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411649 got 77309411653 for osd.2 2024-03-11T22:33:08.972 DEBUG:teuthology.parallel:result is None 2024-03-11T22:33:09.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:08 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:33:09.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:08 smithi144 ceph-mon[33984]: 3.12 deep-scrub starts 2024-03-11T22:33:09.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:08 smithi144 ceph-mon[33984]: 3.12 deep-scrub ok 2024-03-11T22:33:09.405 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659441 got 176093659444 for osd.5 2024-03-11T22:33:09.406 DEBUG:teuthology.parallel:result is None 2024-03-11T22:33:09.406 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T22:33:09.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:33:09.658 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:33:10.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:09 smithi045 ceph-mon[39715]: pgmap v1212: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-03-11T22:33:10.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:09 smithi045 ceph-mon[30178]: pgmap v1212: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-03-11T22:33:10.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:09 smithi144 ceph-mon[33984]: pgmap v1212: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-03-11T22:33:10.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:10 smithi045 ceph-mon[39715]: 5.d deep-scrub starts 2024-03-11T22:33:10.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:10 smithi045 ceph-mon[39715]: 5.d deep-scrub ok 2024-03-11T22:33:10.865 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:10 smithi045 ceph-mon[39715]: 3.19 scrub starts 2024-03-11T22:33:10.866 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:10 smithi045 ceph-mon[39715]: 3.19 scrub ok 2024-03-11T22:33:10.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:10 smithi045 ceph-mon[30178]: 5.d deep-scrub starts 2024-03-11T22:33:10.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:10 smithi045 ceph-mon[30178]: 5.d deep-scrub ok 2024-03-11T22:33:10.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:10 smithi045 ceph-mon[30178]: 3.19 scrub starts 2024-03-11T22:33:10.866 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:10 smithi045 ceph-mon[30178]: 3.19 scrub ok 2024-03-11T22:33:11.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:10 smithi144 ceph-mon[33984]: 5.d deep-scrub starts 2024-03-11T22:33:11.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:10 smithi144 ceph-mon[33984]: 5.d deep-scrub ok 2024-03-11T22:33:11.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:10 smithi144 ceph-mon[33984]: 3.19 scrub starts 2024-03-11T22:33:11.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:10 smithi144 ceph-mon[33984]: 3.19 scrub ok 2024-03-11T22:33:11.725 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:33:11.728 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:33:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:11 smithi045 ceph-mon[39715]: 4.8 scrub starts 2024-03-11T22:33:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:11 smithi045 ceph-mon[39715]: 4.8 scrub ok 2024-03-11T22:33:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:11 smithi045 ceph-mon[39715]: pgmap v1213: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:33:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:11 smithi045 ceph-mon[39715]: 5.b scrub starts 2024-03-11T22:33:12.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:11 smithi045 ceph-mon[39715]: 5.b scrub ok 2024-03-11T22:33:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:11 smithi045 ceph-mon[30178]: 4.8 scrub starts 2024-03-11T22:33:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:11 smithi045 ceph-mon[30178]: 4.8 scrub ok 2024-03-11T22:33:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:11 smithi045 ceph-mon[30178]: pgmap v1213: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:33:12.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:11 smithi045 ceph-mon[30178]: 5.b scrub starts 2024-03-11T22:33:12.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:11 smithi045 ceph-mon[30178]: 5.b scrub ok 2024-03-11T22:33:12.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:11 smithi144 ceph-mon[33984]: 4.8 scrub starts 2024-03-11T22:33:12.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:11 smithi144 ceph-mon[33984]: 4.8 scrub ok 2024-03-11T22:33:12.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:11 smithi144 ceph-mon[33984]: pgmap v1213: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T22:33:12.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:11 smithi144 ceph-mon[33984]: 5.b scrub starts 2024-03-11T22:33:12.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:11 smithi144 ceph-mon[33984]: 5.b scrub ok 2024-03-11T22:33:12.486 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":1213,"stamp":"2024-03-11T22:33:10.569900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1701,"num_read_kb":1453,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":333942,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1151,"ondisk_log_size":1151,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6190216,"kb_used_data":22432,"kb_used_omap":12,"kb_used_meta":6167667,"kb_avail":743771000,"statfs":{"total":767960285184,"available":761621504000,"internally_reserved":0,"allocated":22970368,"data_stored":15002183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":6315691571},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001977"},"pg_stats":[{"pgid":"4.c","version":"479'35","reported_seq":752,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410325+0000","last_change":"2024-03-11T22:32:01.975419+0000","last_active":"2024-03-11T22:33:07.410325+0000","last_peered":"2024-03-11T22:33:07.410325+0000","last_clean":"2024-03-11T22:33:07.410325+0000","last_became_active":"2024-03-11T22:32:01.973558+0000","last_became_peered":"2024-03-11T22:32:01.973558+0000","last_unstale":"2024-03-11T22:33:07.410325+0000","last_undegraded":"2024-03-11T22:33:07.410325+0000","last_fullsized":"2024-03-11T22:33:07.410325+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'35","last_scrub_stamp":"2024-03-11T22:31:59.080346+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:31:59.080346+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1092,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338637+0000","last_change":"2024-03-11T22:32:02.018644+0000","last_active":"2024-03-11T22:33:03.338637+0000","last_peered":"2024-03-11T22:33:03.338637+0000","last_clean":"2024-03-11T22:33:03.338637+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:33:03.338637+0000","last_undegraded":"2024-03-11T22:33:03.338637+0000","last_fullsized":"2024-03-11T22:33:03.338637+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:02.018533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:02.018533+0000","last_clean_scrub_stamp":"2024-03-11T22:32:02.018533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:07.966734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":1431,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:02.745679+0000","last_change":"2024-03-11T22:31:47.227517+0000","last_active":"2024-03-11T22:33:02.745679+0000","last_peered":"2024-03-11T22:33:02.745679+0000","last_clean":"2024-03-11T22:33:02.745679+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:33:02.745679+0000","last_undegraded":"2024-03-11T22:33:02.745679+0000","last_fullsized":"2024-03-11T22:33:02.745679+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:31:47.227358+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:26:11.939962+0000","last_clean_scrub_stamp":"2024-03-11T22:31:47.227358+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:07.053005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":282,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409890+0000","last_change":"2024-03-11T22:32:29.837285+0000","last_active":"2024-03-11T22:33:07.409890+0000","last_peered":"2024-03-11T22:33:07.409890+0000","last_clean":"2024-03-11T22:33:07.409890+0000","last_became_active":"2024-03-11T22:32:01.980840+0000","last_became_peered":"2024-03-11T22:32:01.980840+0000","last_unstale":"2024-03-11T22:33:07.409890+0000","last_undegraded":"2024-03-11T22:33:07.409890+0000","last_fullsized":"2024-03-11T22:33:07.409890+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:29.837170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:29.837170+0000","last_clean_scrub_stamp":"2024-03-11T22:32:29.837170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:53.875012+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":883,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679543+0000","last_change":"2024-03-11T22:32:58.041419+0000","last_active":"2024-03-11T22:33:06.679543+0000","last_peered":"2024-03-11T22:33:06.679543+0000","last_clean":"2024-03-11T22:33:06.679543+0000","last_became_active":"2024-03-11T22:32:25.065015+0000","last_became_peered":"2024-03-11T22:32:25.065015+0000","last_unstale":"2024-03-11T22:33:06.679543+0000","last_undegraded":"2024-03-11T22:33:06.679543+0000","last_fullsized":"2024-03-11T22:33:06.679543+0000","mapping_epoch":569,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:58.041330+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:32:58.041330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:07.994370+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"479'53","reported_seq":845,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410516+0000","last_change":"2024-03-11T22:32:55.856605+0000","last_active":"2024-03-11T22:33:07.410516+0000","last_peered":"2024-03-11T22:33:07.410516+0000","last_clean":"2024-03-11T22:33:07.410516+0000","last_became_active":"2024-03-11T22:32:01.979928+0000","last_became_peered":"2024-03-11T22:32:01.979928+0000","last_unstale":"2024-03-11T22:33:07.410516+0000","last_undegraded":"2024-03-11T22:33:07.410516+0000","last_fullsized":"2024-03-11T22:33:07.410516+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'53","last_scrub_stamp":"2024-03-11T22:32:55.856488+0000","last_deep_scrub":"353'39","last_deep_scrub_stamp":"2024-03-11T22:23:05.461729+0000","last_clean_scrub_stamp":"2024-03-11T22:32:55.856488+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:59.165433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1137,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339588+0000","last_change":"2024-03-11T22:32:16.675251+0000","last_active":"2024-03-11T22:33:03.339588+0000","last_peered":"2024-03-11T22:33:03.339588+0000","last_clean":"2024-03-11T22:33:03.339588+0000","last_became_active":"2024-03-11T22:29:04.718290+0000","last_became_peered":"2024-03-11T22:29:04.718290+0000","last_unstale":"2024-03-11T22:33:03.339588+0000","last_undegraded":"2024-03-11T22:33:03.339588+0000","last_fullsized":"2024-03-11T22:33:03.339588+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:16.675138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:16.675138+0000","last_clean_scrub_stamp":"2024-03-11T22:32:16.675138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:46.593622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":211,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680611+0000","last_change":"2024-03-11T22:32:13.533581+0000","last_active":"2024-03-11T22:33:06.680611+0000","last_peered":"2024-03-11T22:33:06.680611+0000","last_clean":"2024-03-11T22:33:06.680611+0000","last_became_active":"2024-03-11T22:29:04.705661+0000","last_became_peered":"2024-03-11T22:29:04.705661+0000","last_unstale":"2024-03-11T22:33:06.680611+0000","last_undegraded":"2024-03-11T22:33:06.680611+0000","last_fullsized":"2024-03-11T22:33:06.680611+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:13.533500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:13.533500+0000","last_clean_scrub_stamp":"2024-03-11T22:32:13.533500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:16.724861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1123,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:02.746007+0000","last_change":"2024-03-11T22:33:01.969538+0000","last_active":"2024-03-11T22:33:02.746007+0000","last_peered":"2024-03-11T22:33:02.746007+0000","last_clean":"2024-03-11T22:33:02.746007+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:33:02.746007+0000","last_undegraded":"2024-03-11T22:33:02.746007+0000","last_fullsized":"2024-03-11T22:33:02.746007+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:01.969440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:33:01.969440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:28.434814+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":1424,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338447+0000","last_change":"2024-03-11T22:32:26.158635+0000","last_active":"2024-03-11T22:33:03.338447+0000","last_peered":"2024-03-11T22:33:03.338447+0000","last_clean":"2024-03-11T22:33:03.338447+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:33:03.338447+0000","last_undegraded":"2024-03-11T22:33:03.338447+0000","last_fullsized":"2024-03-11T22:33:03.338447+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:26.158500+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:30:57.479260+0000","last_clean_scrub_stamp":"2024-03-11T22:32:26.158500+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:55.322627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"516'32","reported_seq":884,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409545+0000","last_change":"2024-03-11T22:32:01.981837+0000","last_active":"2024-03-11T22:33:07.409545+0000","last_peered":"2024-03-11T22:33:07.409545+0000","last_clean":"2024-03-11T22:33:07.409545+0000","last_became_active":"2024-03-11T22:32:01.980333+0000","last_became_peered":"2024-03-11T22:32:01.980333+0000","last_unstale":"2024-03-11T22:33:07.409545+0000","last_undegraded":"2024-03-11T22:33:07.409545+0000","last_fullsized":"2024-03-11T22:33:07.409545+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'32","last_scrub_stamp":"2024-03-11T22:31:52.730391+0000","last_deep_scrub":"516'32","last_deep_scrub_stamp":"2024-03-11T22:30:10.404123+0000","last_clean_scrub_stamp":"2024-03-11T22:31:52.730391+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:08.489205+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":294,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408788+0000","last_change":"2024-03-11T22:32:27.854952+0000","last_active":"2024-03-11T22:33:07.408788+0000","last_peered":"2024-03-11T22:33:07.408788+0000","last_clean":"2024-03-11T22:33:07.408788+0000","last_became_active":"2024-03-11T22:32:01.978130+0000","last_became_peered":"2024-03-11T22:32:01.978130+0000","last_unstale":"2024-03-11T22:33:07.408788+0000","last_undegraded":"2024-03-11T22:33:07.408788+0000","last_fullsized":"2024-03-11T22:33:07.408788+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:27.854853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:27.854853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:31.282156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1093,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.340094+0000","last_change":"2024-03-11T22:32:17.258055+0000","last_active":"2024-03-11T22:33:03.340094+0000","last_peered":"2024-03-11T22:33:03.340094+0000","last_clean":"2024-03-11T22:33:03.340094+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:33:03.340094+0000","last_undegraded":"2024-03-11T22:33:03.340094+0000","last_fullsized":"2024-03-11T22:33:03.340094+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:17.257943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:17.257943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:19.039837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1080,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339075+0000","last_change":"2024-03-11T22:32:53.130648+0000","last_active":"2024-03-11T22:33:03.339075+0000","last_peered":"2024-03-11T22:33:03.339075+0000","last_clean":"2024-03-11T22:33:03.339075+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:33:03.339075+0000","last_undegraded":"2024-03-11T22:33:03.339075+0000","last_fullsized":"2024-03-11T22:33:03.339075+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:53.130551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:53.130551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:19.308993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"480'46","reported_seq":227,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.681135+0000","last_change":"2024-03-11T22:32:08.527853+0000","last_active":"2024-03-11T22:33:06.681135+0000","last_peered":"2024-03-11T22:33:06.681135+0000","last_clean":"2024-03-11T22:33:06.681135+0000","last_became_active":"2024-03-11T22:29:04.726632+0000","last_became_peered":"2024-03-11T22:29:04.726632+0000","last_unstale":"2024-03-11T22:33:06.681135+0000","last_undegraded":"2024-03-11T22:33:06.681135+0000","last_fullsized":"2024-03-11T22:33:06.681135+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'46","last_scrub_stamp":"2024-03-11T22:32:08.527758+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:32:08.527758+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:30.733421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1117,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338260+0000","last_change":"2024-03-11T22:32:14.066574+0000","last_active":"2024-03-11T22:33:03.338260+0000","last_peered":"2024-03-11T22:33:03.338260+0000","last_clean":"2024-03-11T22:33:03.338260+0000","last_became_active":"2024-03-11T22:29:04.701072+0000","last_became_peered":"2024-03-11T22:29:04.701072+0000","last_unstale":"2024-03-11T22:33:03.338260+0000","last_undegraded":"2024-03-11T22:33:03.338260+0000","last_fullsized":"2024-03-11T22:33:03.338260+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:14.066497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:14.066497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:20.415563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"561'10","reported_seq":384,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409824+0000","last_change":"2024-03-11T22:33:00.980576+0000","last_active":"2024-03-11T22:33:07.409824+0000","last_peered":"2024-03-11T22:33:07.409824+0000","last_clean":"2024-03-11T22:33:07.409824+0000","last_became_active":"2024-03-11T22:32:01.975280+0000","last_became_peered":"2024-03-11T22:32:01.975280+0000","last_unstale":"2024-03-11T22:33:07.409824+0000","last_undegraded":"2024-03-11T22:33:07.409824+0000","last_fullsized":"2024-03-11T22:33:07.409824+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"561'10","last_scrub_stamp":"2024-03-11T22:33:00.980403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:33:00.980403+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:26.719403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1101,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339202+0000","last_change":"2024-03-11T22:32:41.431748+0000","last_active":"2024-03-11T22:33:03.339202+0000","last_peered":"2024-03-11T22:33:03.339202+0000","last_clean":"2024-03-11T22:33:03.339202+0000","last_became_active":"2024-03-11T22:29:04.787650+0000","last_became_peered":"2024-03-11T22:29:04.787650+0000","last_unstale":"2024-03-11T22:33:03.339202+0000","last_undegraded":"2024-03-11T22:33:03.339202+0000","last_fullsized":"2024-03-11T22:33:03.339202+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:41.431610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:32:41.431610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:48.164708+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":1425,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338625+0000","last_change":"2024-03-11T22:32:05.248884+0000","last_active":"2024-03-11T22:33:03.338625+0000","last_peered":"2024-03-11T22:33:03.338625+0000","last_clean":"2024-03-11T22:33:03.338625+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:33:03.338625+0000","last_undegraded":"2024-03-11T22:33:03.338625+0000","last_fullsized":"2024-03-11T22:33:03.338625+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:05.248781+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:32:05.248781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:09.348541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":353,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408670+0000","last_change":"2024-03-11T22:32:38.683950+0000","last_active":"2024-03-11T22:33:07.408670+0000","last_peered":"2024-03-11T22:33:07.408670+0000","last_clean":"2024-03-11T22:33:07.408670+0000","last_became_active":"2024-03-11T22:32:01.969330+0000","last_became_peered":"2024-03-11T22:32:01.969330+0000","last_unstale":"2024-03-11T22:33:07.408670+0000","last_undegraded":"2024-03-11T22:33:07.408670+0000","last_fullsized":"2024-03-11T22:33:07.408670+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:38.683844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:38.683844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:48.398432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"479'42","reported_seq":1209,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338452+0000","last_change":"2024-03-11T22:32:41.170750+0000","last_active":"2024-03-11T22:33:03.338452+0000","last_peered":"2024-03-11T22:33:03.338452+0000","last_clean":"2024-03-11T22:33:03.338452+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:33:03.338452+0000","last_undegraded":"2024-03-11T22:33:03.338452+0000","last_fullsized":"2024-03-11T22:33:03.338452+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'42","last_scrub_stamp":"2024-03-11T22:32:41.170650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:41.170650+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:54.492505+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1105,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339744+0000","last_change":"2024-03-11T22:32:01.048534+0000","last_active":"2024-03-11T22:33:03.339744+0000","last_peered":"2024-03-11T22:33:03.339744+0000","last_clean":"2024-03-11T22:33:03.339744+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:33:03.339744+0000","last_undegraded":"2024-03-11T22:33:03.339744+0000","last_fullsized":"2024-03-11T22:33:03.339744+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:01.048412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:01.048412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:21.404669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":274,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409989+0000","last_change":"2024-03-11T22:32:46.771400+0000","last_active":"2024-03-11T22:33:07.409989+0000","last_peered":"2024-03-11T22:33:07.409989+0000","last_clean":"2024-03-11T22:33:07.409989+0000","last_became_active":"2024-03-11T22:32:01.979884+0000","last_became_peered":"2024-03-11T22:32:01.979884+0000","last_unstale":"2024-03-11T22:33:07.409989+0000","last_undegraded":"2024-03-11T22:33:07.409989+0000","last_fullsized":"2024-03-11T22:33:07.409989+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:46.771280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:46.771280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:49.352724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":567,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410041+0000","last_change":"2024-03-11T22:32:02.019354+0000","last_active":"2024-03-11T22:33:07.410041+0000","last_peered":"2024-03-11T22:33:07.410041+0000","last_clean":"2024-03-11T22:33:07.410041+0000","last_became_active":"2024-03-11T22:32:01.980016+0000","last_became_peered":"2024-03-11T22:32:01.980016+0000","last_unstale":"2024-03-11T22:33:07.410041+0000","last_undegraded":"2024-03-11T22:33:07.410041+0000","last_fullsized":"2024-03-11T22:33:07.410041+0000","mapping_epoch":560,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:02.019267+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:31:08.370869+0000","last_clean_scrub_stamp":"2024-03-11T22:32:02.019267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:28.470010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"488'10","reported_seq":1410,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338398+0000","last_change":"2024-03-11T22:32:24.135139+0000","last_active":"2024-03-11T22:33:03.338398+0000","last_peered":"2024-03-11T22:33:03.338398+0000","last_clean":"2024-03-11T22:33:03.338398+0000","last_became_active":"2024-03-11T22:29:04.721065+0000","last_became_peered":"2024-03-11T22:29:04.721065+0000","last_unstale":"2024-03-11T22:33:03.338398+0000","last_undegraded":"2024-03-11T22:33:03.338398+0000","last_fullsized":"2024-03-11T22:33:03.338398+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'10","last_scrub_stamp":"2024-03-11T22:32:24.135002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:24.135002+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:42.573609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"480'42","reported_seq":783,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408964+0000","last_change":"2024-03-11T22:32:16.969323+0000","last_active":"2024-03-11T22:33:07.408964+0000","last_peered":"2024-03-11T22:33:07.408964+0000","last_clean":"2024-03-11T22:33:07.408964+0000","last_became_active":"2024-03-11T22:32:01.972997+0000","last_became_peered":"2024-03-11T22:32:01.972997+0000","last_unstale":"2024-03-11T22:33:07.408964+0000","last_undegraded":"2024-03-11T22:33:07.408964+0000","last_fullsized":"2024-03-11T22:33:07.408964+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'42","last_scrub_stamp":"2024-03-11T22:32:16.969171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:16.969171+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:25.997062+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":1125,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409072+0000","last_change":"2024-03-11T22:32:03.364809+0000","last_active":"2024-03-11T22:33:07.409072+0000","last_peered":"2024-03-11T22:33:07.409072+0000","last_clean":"2024-03-11T22:33:07.409072+0000","last_became_active":"2024-03-11T22:29:04.718160+0000","last_became_peered":"2024-03-11T22:29:04.718160+0000","last_unstale":"2024-03-11T22:33:07.409072+0000","last_undegraded":"2024-03-11T22:33:07.409072+0000","last_fullsized":"2024-03-11T22:33:07.409072+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:32:03.364707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:03.364707+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:13.141927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":100,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680306+0000","last_change":"2024-03-11T22:32:33.072989+0000","last_active":"2024-03-11T22:33:06.680306+0000","last_peered":"2024-03-11T22:33:06.680306+0000","last_clean":"2024-03-11T22:33:06.680306+0000","last_became_active":"2024-03-11T22:32:25.067050+0000","last_became_peered":"2024-03-11T22:32:25.067050+0000","last_unstale":"2024-03-11T22:33:06.680306+0000","last_undegraded":"2024-03-11T22:33:06.680306+0000","last_fullsized":"2024-03-11T22:33:06.680306+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:33.072870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:32:33.072870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:48.865391+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":1460,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338908+0000","last_change":"2024-03-11T22:32:57.118065+0000","last_active":"2024-03-11T22:33:03.338908+0000","last_peered":"2024-03-11T22:33:03.338908+0000","last_clean":"2024-03-11T22:33:03.338908+0000","last_became_active":"2024-03-11T22:26:53.557811+0000","last_became_peered":"2024-03-11T22:26:53.557811+0000","last_unstale":"2024-03-11T22:33:03.338908+0000","last_undegraded":"2024-03-11T22:33:03.338908+0000","last_fullsized":"2024-03-11T22:33:03.338908+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:57.117970+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:32:57.117970+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:07.382758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"378'9","reported_seq":1400,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408698+0000","last_change":"2024-03-11T22:32:05.398376+0000","last_active":"2024-03-11T22:33:07.408698+0000","last_peered":"2024-03-11T22:33:07.408698+0000","last_clean":"2024-03-11T22:33:07.408698+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:33:07.408698+0000","last_undegraded":"2024-03-11T22:33:07.408698+0000","last_fullsized":"2024-03-11T22:33:07.408698+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9","last_scrub_stamp":"2024-03-11T22:32:05.398270+0000","last_deep_scrub":"378'9","last_deep_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_clean_scrub_stamp":"2024-03-11T22:32:05.398270+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:33.174332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'30","reported_seq":231,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680387+0000","last_change":"2024-03-11T22:32:53.523093+0000","last_active":"2024-03-11T22:33:06.680387+0000","last_peered":"2024-03-11T22:33:06.680387+0000","last_clean":"2024-03-11T22:33:06.680387+0000","last_became_active":"2024-03-11T22:29:04.727100+0000","last_became_peered":"2024-03-11T22:29:04.727100+0000","last_unstale":"2024-03-11T22:33:06.680387+0000","last_undegraded":"2024-03-11T22:33:06.680387+0000","last_fullsized":"2024-03-11T22:33:06.680387+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-03-11T22:32:53.522986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:53.522986+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:03.105583+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1102,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339545+0000","last_change":"2024-03-11T22:31:39.253559+0000","last_active":"2024-03-11T22:33:03.339545+0000","last_peered":"2024-03-11T22:33:03.339545+0000","last_clean":"2024-03-11T22:33:03.339545+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:33:03.339545+0000","last_undegraded":"2024-03-11T22:33:03.339545+0000","last_fullsized":"2024-03-11T22:33:03.339545+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:39.253450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:39.253450+0000","last_clean_scrub_stamp":"2024-03-11T22:31:39.253450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:04.606413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1093,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408041+0000","last_change":"2024-03-11T22:32:22.164266+0000","last_active":"2024-03-11T22:33:07.408041+0000","last_peered":"2024-03-11T22:33:07.408041+0000","last_clean":"2024-03-11T22:33:07.408041+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:33:07.408041+0000","last_undegraded":"2024-03-11T22:33:07.408041+0000","last_fullsized":"2024-03-11T22:33:07.408041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:22.164173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:22.164173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:38.086539+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":1375,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339460+0000","last_change":"2024-03-11T22:32:02.731217+0000","last_active":"2024-03-11T22:33:03.339460+0000","last_peered":"2024-03-11T22:33:03.339460+0000","last_clean":"2024-03-11T22:33:03.339460+0000","last_became_active":"2024-03-11T22:26:41.661375+0000","last_became_peered":"2024-03-11T22:26:41.661375+0000","last_unstale":"2024-03-11T22:33:03.339460+0000","last_undegraded":"2024-03-11T22:33:03.339460+0000","last_fullsized":"2024-03-11T22:33:03.339460+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":423,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:02.731093+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:32:02.731093+0000","last_clean_scrub_stamp":"2024-03-11T22:32:02.731093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:31.044844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1117,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339134+0000","last_change":"2024-03-11T22:32:24.561932+0000","last_active":"2024-03-11T22:33:03.339134+0000","last_peered":"2024-03-11T22:33:03.339134+0000","last_clean":"2024-03-11T22:33:03.339134+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:33:03.339134+0000","last_undegraded":"2024-03-11T22:33:03.339134+0000","last_fullsized":"2024-03-11T22:33:03.339134+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:24.561799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:32:24.561799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:47.605538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"479'47","reported_seq":1174,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.407977+0000","last_change":"2024-03-11T22:32:38.133530+0000","last_active":"2024-03-11T22:33:07.407977+0000","last_peered":"2024-03-11T22:33:07.407977+0000","last_clean":"2024-03-11T22:33:07.407977+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:33:07.407977+0000","last_undegraded":"2024-03-11T22:33:07.407977+0000","last_fullsized":"2024-03-11T22:33:07.407977+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'47","last_scrub_stamp":"2024-03-11T22:32:38.133415+0000","last_deep_scrub":"429'37","last_deep_scrub_stamp":"2024-03-11T22:26:47.889156+0000","last_clean_scrub_stamp":"2024-03-11T22:32:38.133415+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:56.303809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":211,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680763+0000","last_change":"2024-03-11T22:32:06.558954+0000","last_active":"2024-03-11T22:33:06.680763+0000","last_peered":"2024-03-11T22:33:06.680763+0000","last_clean":"2024-03-11T22:33:06.680763+0000","last_became_active":"2024-03-11T22:29:04.722813+0000","last_became_peered":"2024-03-11T22:29:04.722813+0000","last_unstale":"2024-03-11T22:33:06.680763+0000","last_undegraded":"2024-03-11T22:33:06.680763+0000","last_fullsized":"2024-03-11T22:33:06.680763+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:06.558793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:04.706082+0000","last_clean_scrub_stamp":"2024-03-11T22:32:06.558793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:34.974395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":211,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679794+0000","last_change":"2024-03-11T22:32:09.603736+0000","last_active":"2024-03-11T22:33:06.679794+0000","last_peered":"2024-03-11T22:33:06.679794+0000","last_clean":"2024-03-11T22:33:06.679794+0000","last_became_active":"2024-03-11T22:29:04.729455+0000","last_became_peered":"2024-03-11T22:29:04.729455+0000","last_unstale":"2024-03-11T22:33:06.679794+0000","last_undegraded":"2024-03-11T22:33:06.679794+0000","last_fullsized":"2024-03-11T22:33:06.679794+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:09.603590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:09.603590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:23.940374+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":1350,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339438+0000","last_change":"2024-03-11T22:32:20.617366+0000","last_active":"2024-03-11T22:33:03.339438+0000","last_peered":"2024-03-11T22:33:03.339438+0000","last_clean":"2024-03-11T22:33:03.339438+0000","last_became_active":"2024-03-11T22:26:45.688609+0000","last_became_peered":"2024-03-11T22:26:45.688609+0000","last_unstale":"2024-03-11T22:33:03.339438+0000","last_undegraded":"2024-03-11T22:33:03.339438+0000","last_fullsized":"2024-03-11T22:33:03.339438+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:20.617283+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:32:20.617283+0000","last_clean_scrub_stamp":"2024-03-11T22:32:20.617283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:33.162195+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1106,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.340145+0000","last_change":"2024-03-11T22:31:53.264975+0000","last_active":"2024-03-11T22:33:03.340145+0000","last_peered":"2024-03-11T22:33:03.340145+0000","last_clean":"2024-03-11T22:33:03.340145+0000","last_became_active":"2024-03-11T22:29:04.718259+0000","last_became_peered":"2024-03-11T22:29:04.718259+0000","last_unstale":"2024-03-11T22:33:03.340145+0000","last_undegraded":"2024-03-11T22:33:03.340145+0000","last_fullsized":"2024-03-11T22:33:03.340145+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:53.264847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:53.264847+0000","last_clean_scrub_stamp":"2024-03-11T22:31:53.264847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:12.439345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"479'36","reported_seq":1202,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409068+0000","last_change":"2024-03-11T22:32:50.037636+0000","last_active":"2024-03-11T22:33:07.409068+0000","last_peered":"2024-03-11T22:33:07.409068+0000","last_clean":"2024-03-11T22:33:07.409068+0000","last_became_active":"2024-03-11T22:29:04.720793+0000","last_became_peered":"2024-03-11T22:29:04.720793+0000","last_unstale":"2024-03-11T22:33:07.409068+0000","last_undegraded":"2024-03-11T22:33:07.409068+0000","last_fullsized":"2024-03-11T22:33:07.409068+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'36","last_scrub_stamp":"2024-03-11T22:32:50.037523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:50.037523+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:59.885167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":277,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:09.132724+0000","last_change":"2024-03-11T22:32:25.071721+0000","last_active":"2024-03-11T22:33:09.132724+0000","last_peered":"2024-03-11T22:33:09.132724+0000","last_clean":"2024-03-11T22:33:09.132724+0000","last_became_active":"2024-03-11T22:32:25.070571+0000","last_became_peered":"2024-03-11T22:32:25.070571+0000","last_unstale":"2024-03-11T22:33:09.132724+0000","last_undegraded":"2024-03-11T22:33:09.132724+0000","last_fullsized":"2024-03-11T22:33:09.132724+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:32:11.608791+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:32:11.608791+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:21.449716+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":158,"num_read_kb":153,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1104,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408315+0000","last_change":"2024-03-11T22:31:50.414259+0000","last_active":"2024-03-11T22:33:07.408315+0000","last_peered":"2024-03-11T22:33:07.408315+0000","last_clean":"2024-03-11T22:33:07.408315+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:33:07.408315+0000","last_undegraded":"2024-03-11T22:33:07.408315+0000","last_fullsized":"2024-03-11T22:33:07.408315+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:50.414158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:50.414158+0000","last_clean_scrub_stamp":"2024-03-11T22:31:50.414158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:15.542430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":219,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680668+0000","last_change":"2024-03-11T22:33:00.528512+0000","last_active":"2024-03-11T22:33:06.680668+0000","last_peered":"2024-03-11T22:33:06.680668+0000","last_clean":"2024-03-11T22:33:06.680668+0000","last_became_active":"2024-03-11T22:29:04.696141+0000","last_became_peered":"2024-03-11T22:29:04.696141+0000","last_unstale":"2024-03-11T22:33:06.680668+0000","last_undegraded":"2024-03-11T22:33:06.680668+0000","last_fullsized":"2024-03-11T22:33:06.680668+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:00.528432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:33:00.528432+0000","last_clean_scrub_stamp":"2024-03-11T22:33:00.528432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:02.356742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":669,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680596+0000","last_change":"2024-03-11T22:32:27.554691+0000","last_active":"2024-03-11T22:33:06.680596+0000","last_peered":"2024-03-11T22:33:06.680596+0000","last_clean":"2024-03-11T22:33:06.680596+0000","last_became_active":"2024-03-11T22:29:37.643184+0000","last_became_peered":"2024-03-11T22:29:37.643184+0000","last_unstale":"2024-03-11T22:33:06.680596+0000","last_undegraded":"2024-03-11T22:33:06.680596+0000","last_fullsized":"2024-03-11T22:33:06.680596+0000","mapping_epoch":502,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:27.554598+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:28:24.623016+0000","last_clean_scrub_stamp":"2024-03-11T22:32:27.554598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:34.300636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":517,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680481+0000","last_change":"2024-03-11T22:32:25.073943+0000","last_active":"2024-03-11T22:33:06.680481+0000","last_peered":"2024-03-11T22:33:06.680481+0000","last_clean":"2024-03-11T22:33:06.680481+0000","last_became_active":"2024-03-11T22:32:25.072113+0000","last_became_peered":"2024-03-11T22:32:25.072113+0000","last_unstale":"2024-03-11T22:33:06.680481+0000","last_undegraded":"2024-03-11T22:33:06.680481+0000","last_fullsized":"2024-03-11T22:33:06.680481+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:07.436607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:32:07.436607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:15.312033+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"514'48","reported_seq":1482,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339203+0000","last_change":"2024-03-11T22:32:35.488420+0000","last_active":"2024-03-11T22:33:03.339203+0000","last_peered":"2024-03-11T22:33:03.339203+0000","last_clean":"2024-03-11T22:33:03.339203+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:33:03.339203+0000","last_undegraded":"2024-03-11T22:33:03.339203+0000","last_fullsized":"2024-03-11T22:33:03.339203+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"514'48","last_scrub_stamp":"2024-03-11T22:32:35.488293+0000","last_deep_scrub":"514'48","last_deep_scrub_stamp":"2024-03-11T22:32:35.488293+0000","last_clean_scrub_stamp":"2024-03-11T22:32:35.488293+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:43.188761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1093,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338581+0000","last_change":"2024-03-11T22:32:16.099056+0000","last_active":"2024-03-11T22:33:03.338581+0000","last_peered":"2024-03-11T22:33:03.338581+0000","last_clean":"2024-03-11T22:33:03.338581+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:33:03.338581+0000","last_undegraded":"2024-03-11T22:33:03.338581+0000","last_fullsized":"2024-03-11T22:33:03.338581+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:16.098966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:32:16.098966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:38.304085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":872,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679575+0000","last_change":"2024-03-11T22:32:27.076843+0000","last_active":"2024-03-11T22:33:06.679575+0000","last_peered":"2024-03-11T22:33:06.679575+0000","last_clean":"2024-03-11T22:33:06.679575+0000","last_became_active":"2024-03-11T22:32:25.067408+0000","last_became_peered":"2024-03-11T22:32:25.067408+0000","last_unstale":"2024-03-11T22:33:06.679575+0000","last_undegraded":"2024-03-11T22:33:06.679575+0000","last_fullsized":"2024-03-11T22:33:06.679575+0000","mapping_epoch":569,"log_start":"93'130","ondisk_log_start":"93'130","created":19,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:27.076730+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:30:06.478921+0000","last_clean_scrub_stamp":"2024-03-11T22:32:27.076730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:55.973112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1090,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338218+0000","last_change":"2024-03-11T22:32:18.431130+0000","last_active":"2024-03-11T22:33:03.338218+0000","last_peered":"2024-03-11T22:33:03.338218+0000","last_clean":"2024-03-11T22:33:03.338218+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:33:03.338218+0000","last_undegraded":"2024-03-11T22:33:03.338218+0000","last_fullsized":"2024-03-11T22:33:03.338218+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:18.430994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:18.430994+0000","last_clean_scrub_stamp":"2024-03-11T22:32:18.430994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:36.072787+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1128,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:02.745926+0000","last_change":"2024-03-11T22:32:41.459780+0000","last_active":"2024-03-11T22:33:02.745926+0000","last_peered":"2024-03-11T22:33:02.745926+0000","last_clean":"2024-03-11T22:33:02.745926+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:33:02.745926+0000","last_undegraded":"2024-03-11T22:33:02.745926+0000","last_fullsized":"2024-03-11T22:33:02.745926+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:41.459705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:41.459705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:03.995612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"538'26","reported_seq":236,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.681088+0000","last_change":"2024-03-11T22:32:38.583950+0000","last_active":"2024-03-11T22:33:06.681088+0000","last_peered":"2024-03-11T22:33:06.681088+0000","last_clean":"2024-03-11T22:33:06.681088+0000","last_became_active":"2024-03-11T22:29:04.705980+0000","last_became_peered":"2024-03-11T22:29:04.705980+0000","last_unstale":"2024-03-11T22:33:06.681088+0000","last_undegraded":"2024-03-11T22:33:06.681088+0000","last_fullsized":"2024-03-11T22:33:06.681088+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'26","last_scrub_stamp":"2024-03-11T22:32:38.583829+0000","last_deep_scrub":"538'26","last_deep_scrub_stamp":"2024-03-11T22:31:09.564609+0000","last_clean_scrub_stamp":"2024-03-11T22:32:38.583829+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:40.430137+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1153,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:02.743858+0000","last_change":"2024-03-11T22:33:00.767985+0000","last_active":"2024-03-11T22:33:02.743858+0000","last_peered":"2024-03-11T22:33:02.743858+0000","last_clean":"2024-03-11T22:33:02.743858+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:33:02.743858+0000","last_undegraded":"2024-03-11T22:33:02.743858+0000","last_fullsized":"2024-03-11T22:33:02.743858+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:00.767847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:33:00.767847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:11.597370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":220,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679440+0000","last_change":"2024-03-11T22:32:33.662740+0000","last_active":"2024-03-11T22:33:06.679440+0000","last_peered":"2024-03-11T22:33:06.679440+0000","last_clean":"2024-03-11T22:33:06.679440+0000","last_became_active":"2024-03-11T22:29:04.713225+0000","last_became_peered":"2024-03-11T22:29:04.713225+0000","last_unstale":"2024-03-11T22:33:06.679440+0000","last_undegraded":"2024-03-11T22:33:06.679440+0000","last_fullsized":"2024-03-11T22:33:06.679440+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:33.662663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:32:33.662663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:47.377747+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":92,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680681+0000","last_change":"2024-03-11T22:32:25.069671+0000","last_active":"2024-03-11T22:33:06.680681+0000","last_peered":"2024-03-11T22:33:06.680681+0000","last_clean":"2024-03-11T22:33:06.680681+0000","last_became_active":"2024-03-11T22:32:25.066872+0000","last_became_peered":"2024-03-11T22:32:25.066872+0000","last_unstale":"2024-03-11T22:33:06.680681+0000","last_undegraded":"2024-03-11T22:33:06.680681+0000","last_fullsized":"2024-03-11T22:33:06.680681+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:39.812558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:31:39.812558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":880,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679701+0000","last_change":"2024-03-11T22:32:26.052079+0000","last_active":"2024-03-11T22:33:06.679701+0000","last_peered":"2024-03-11T22:33:06.679701+0000","last_clean":"2024-03-11T22:33:06.679701+0000","last_became_active":"2024-03-11T22:32:25.063983+0000","last_became_peered":"2024-03-11T22:32:25.063983+0000","last_unstale":"2024-03-11T22:33:06.679701+0000","last_undegraded":"2024-03-11T22:33:06.679701+0000","last_fullsized":"2024-03-11T22:33:06.679701+0000","mapping_epoch":569,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:26.051959+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:32:26.051959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:44.466239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1116,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408649+0000","last_change":"2024-03-11T22:32:55.954802+0000","last_active":"2024-03-11T22:33:07.408649+0000","last_peered":"2024-03-11T22:33:07.408649+0000","last_clean":"2024-03-11T22:33:07.408649+0000","last_became_active":"2024-03-11T22:29:04.707688+0000","last_became_peered":"2024-03-11T22:29:04.707688+0000","last_unstale":"2024-03-11T22:33:07.408649+0000","last_undegraded":"2024-03-11T22:33:07.408649+0000","last_fullsized":"2024-03-11T22:33:07.408649+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:55.954694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:05.507462+0000","last_clean_scrub_stamp":"2024-03-11T22:32:55.954694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:23.210941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"520'55","reported_seq":191,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339091+0000","last_change":"2024-03-11T22:31:48.173854+0000","last_active":"2024-03-11T22:33:03.339091+0000","last_peered":"2024-03-11T22:33:03.339091+0000","last_clean":"2024-03-11T22:33:03.339091+0000","last_became_active":"2024-03-11T22:30:16.930211+0000","last_became_peered":"2024-03-11T22:30:16.930211+0000","last_unstale":"2024-03-11T22:33:03.339091+0000","last_undegraded":"2024-03-11T22:33:03.339091+0000","last_fullsized":"2024-03-11T22:33:03.339091+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'55","last_scrub_stamp":"2024-03-11T22:31:48.173730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:31:48.173730+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":295,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409352+0000","last_change":"2024-03-11T22:32:40.645171+0000","last_active":"2024-03-11T22:33:07.409352+0000","last_peered":"2024-03-11T22:33:07.409352+0000","last_clean":"2024-03-11T22:33:07.409352+0000","last_became_active":"2024-03-11T22:32:01.973398+0000","last_became_peered":"2024-03-11T22:32:01.973398+0000","last_unstale":"2024-03-11T22:33:07.409352+0000","last_undegraded":"2024-03-11T22:33:07.409352+0000","last_fullsized":"2024-03-11T22:33:07.409352+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:40.645073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:40.645073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:51.820533+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":1142,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338809+0000","last_change":"2024-03-11T22:32:33.326977+0000","last_active":"2024-03-11T22:33:03.338809+0000","last_peered":"2024-03-11T22:33:03.338809+0000","last_clean":"2024-03-11T22:33:03.338809+0000","last_became_active":"2024-03-11T22:29:04.718809+0000","last_became_peered":"2024-03-11T22:29:04.718809+0000","last_unstale":"2024-03-11T22:33:03.338809+0000","last_undegraded":"2024-03-11T22:33:03.338809+0000","last_fullsized":"2024-03-11T22:33:03.338809+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:32:33.326892+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:32:33.326892+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:48.990981+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1093,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339497+0000","last_change":"2024-03-11T22:32:48.191134+0000","last_active":"2024-03-11T22:33:03.339497+0000","last_peered":"2024-03-11T22:33:03.339497+0000","last_clean":"2024-03-11T22:33:03.339497+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:33:03.339497+0000","last_undegraded":"2024-03-11T22:33:03.339497+0000","last_fullsized":"2024-03-11T22:33:03.339497+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:48.191008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:32:48.191008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:04.826416+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":1440,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338513+0000","last_change":"2024-03-11T22:32:47.311220+0000","last_active":"2024-03-11T22:33:03.338513+0000","last_peered":"2024-03-11T22:33:03.338513+0000","last_clean":"2024-03-11T22:33:03.338513+0000","last_became_active":"2024-03-11T22:26:33.607145+0000","last_became_peered":"2024-03-11T22:26:33.607145+0000","last_unstale":"2024-03-11T22:33:03.338513+0000","last_undegraded":"2024-03-11T22:33:03.338513+0000","last_fullsized":"2024-03-11T22:33:03.338513+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:47.311113+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:32:47.311113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:53.558931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"569'25","reported_seq":1185,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408660+0000","last_change":"2024-03-11T22:32:24.073606+0000","last_active":"2024-03-11T22:33:07.408660+0000","last_peered":"2024-03-11T22:33:07.408660+0000","last_clean":"2024-03-11T22:33:07.408660+0000","last_became_active":"2024-03-11T22:29:04.707867+0000","last_became_peered":"2024-03-11T22:29:04.707867+0000","last_unstale":"2024-03-11T22:33:07.408660+0000","last_undegraded":"2024-03-11T22:33:07.408660+0000","last_fullsized":"2024-03-11T22:33:07.408660+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'25","last_scrub_stamp":"2024-03-11T22:32:24.073486+0000","last_deep_scrub":"569'25","last_deep_scrub_stamp":"2024-03-11T22:32:24.073486+0000","last_clean_scrub_stamp":"2024-03-11T22:32:24.073486+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:49.739916+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1101,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339534+0000","last_change":"2024-03-11T22:32:25.206526+0000","last_active":"2024-03-11T22:33:03.339534+0000","last_peered":"2024-03-11T22:33:03.339534+0000","last_clean":"2024-03-11T22:33:03.339534+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:33:03.339534+0000","last_undegraded":"2024-03-11T22:33:03.339534+0000","last_fullsized":"2024-03-11T22:33:03.339534+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:25.206432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:25.206432+0000","last_clean_scrub_stamp":"2024-03-11T22:32:25.206432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:26.184339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1128,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:02.743751+0000","last_change":"2024-03-11T22:32:21.381860+0000","last_active":"2024-03-11T22:33:02.743751+0000","last_peered":"2024-03-11T22:33:02.743751+0000","last_clean":"2024-03-11T22:33:02.743751+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:33:02.743751+0000","last_undegraded":"2024-03-11T22:33:02.743751+0000","last_fullsized":"2024-03-11T22:33:02.743751+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:21.381769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:32:21.381769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:25.903405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1112,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408422+0000","last_change":"2024-03-11T22:33:06.055241+0000","last_active":"2024-03-11T22:33:07.408422+0000","last_peered":"2024-03-11T22:33:07.408422+0000","last_clean":"2024-03-11T22:33:07.408422+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:33:07.408422+0000","last_undegraded":"2024-03-11T22:33:07.408422+0000","last_fullsized":"2024-03-11T22:33:07.408422+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:06.055130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:20.400349+0000","last_clean_scrub_stamp":"2024-03-11T22:33:06.055130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:26.656002+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"480'51","reported_seq":1304,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338991+0000","last_change":"2024-03-11T22:32:57.544585+0000","last_active":"2024-03-11T22:33:03.338991+0000","last_peered":"2024-03-11T22:33:03.338991+0000","last_clean":"2024-03-11T22:33:03.338991+0000","last_became_active":"2024-03-11T22:29:04.718797+0000","last_became_peered":"2024-03-11T22:29:04.718797+0000","last_unstale":"2024-03-11T22:33:03.338991+0000","last_undegraded":"2024-03-11T22:33:03.338991+0000","last_fullsized":"2024-03-11T22:33:03.338991+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'51","last_scrub_stamp":"2024-03-11T22:32:57.544466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:57.544466+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:14.331227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":510,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679985+0000","last_change":"2024-03-11T22:32:28.091584+0000","last_active":"2024-03-11T22:33:06.679985+0000","last_peered":"2024-03-11T22:33:06.679985+0000","last_clean":"2024-03-11T22:33:06.679985+0000","last_became_active":"2024-03-11T22:32:25.072199+0000","last_became_peered":"2024-03-11T22:32:25.072199+0000","last_unstale":"2024-03-11T22:33:06.679985+0000","last_undegraded":"2024-03-11T22:33:06.679985+0000","last_fullsized":"2024-03-11T22:33:06.679985+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:28.091484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:28.091484+0000","last_clean_scrub_stamp":"2024-03-11T22:32:28.091484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:42.050317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1119,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:02.744751+0000","last_change":"2024-03-11T22:31:51.335247+0000","last_active":"2024-03-11T22:33:02.744751+0000","last_peered":"2024-03-11T22:33:02.744751+0000","last_clean":"2024-03-11T22:33:02.744751+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:33:02.744751+0000","last_undegraded":"2024-03-11T22:33:02.744751+0000","last_fullsized":"2024-03-11T22:33:02.744751+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:51.335152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:31:51.335152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:07.159936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":525,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679465+0000","last_change":"2024-03-11T22:32:52.940775+0000","last_active":"2024-03-11T22:33:06.679465+0000","last_peered":"2024-03-11T22:33:06.679465+0000","last_clean":"2024-03-11T22:33:06.679465+0000","last_became_active":"2024-03-11T22:32:25.063927+0000","last_became_peered":"2024-03-11T22:32:25.063927+0000","last_unstale":"2024-03-11T22:33:06.679465+0000","last_undegraded":"2024-03-11T22:33:06.679465+0000","last_fullsized":"2024-03-11T22:33:06.679465+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:52.940688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:47.568500+0000","last_clean_scrub_stamp":"2024-03-11T22:32:52.940688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:19.412757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1140,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:02.743326+0000","last_change":"2024-03-11T22:32:28.355320+0000","last_active":"2024-03-11T22:33:02.743326+0000","last_peered":"2024-03-11T22:33:02.743326+0000","last_clean":"2024-03-11T22:33:02.743326+0000","last_became_active":"2024-03-11T22:29:04.718513+0000","last_became_peered":"2024-03-11T22:29:04.718513+0000","last_unstale":"2024-03-11T22:33:02.743326+0000","last_undegraded":"2024-03-11T22:33:02.743326+0000","last_fullsized":"2024-03-11T22:33:02.743326+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:28.355239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:14.325174+0000","last_clean_scrub_stamp":"2024-03-11T22:32:28.355239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:50.655394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"478'24","reported_seq":420,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409219+0000","last_change":"2024-03-11T22:32:28.812733+0000","last_active":"2024-03-11T22:33:07.409219+0000","last_peered":"2024-03-11T22:33:07.409219+0000","last_clean":"2024-03-11T22:33:07.409219+0000","last_became_active":"2024-03-11T22:32:01.975548+0000","last_became_peered":"2024-03-11T22:32:01.975548+0000","last_unstale":"2024-03-11T22:33:07.409219+0000","last_undegraded":"2024-03-11T22:33:07.409219+0000","last_fullsized":"2024-03-11T22:33:07.409219+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'24","last_scrub_stamp":"2024-03-11T22:32:28.812617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:28.812617+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:46.520206+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":343,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680063+0000","last_change":"2024-03-11T22:32:25.070248+0000","last_active":"2024-03-11T22:33:06.680063+0000","last_peered":"2024-03-11T22:33:06.680063+0000","last_clean":"2024-03-11T22:33:06.680063+0000","last_became_active":"2024-03-11T22:32:25.069890+0000","last_became_peered":"2024-03-11T22:32:25.069890+0000","last_unstale":"2024-03-11T22:33:06.680063+0000","last_undegraded":"2024-03-11T22:33:06.680063+0000","last_fullsized":"2024-03-11T22:33:06.680063+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:44.529411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:31:44.529411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:06.856542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":1123,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339155+0000","last_change":"2024-03-11T22:32:32.188940+0000","last_active":"2024-03-11T22:33:03.339155+0000","last_peered":"2024-03-11T22:33:03.339155+0000","last_clean":"2024-03-11T22:33:03.339155+0000","last_became_active":"2024-03-11T22:29:04.698531+0000","last_became_peered":"2024-03-11T22:29:04.698531+0000","last_unstale":"2024-03-11T22:33:03.339155+0000","last_undegraded":"2024-03-11T22:33:03.339155+0000","last_fullsized":"2024-03-11T22:33:03.339155+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:32:32.188824+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:32:32.188824+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:01.594253+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"570'12","reported_seq":676,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680230+0000","last_change":"2024-03-11T22:33:00.059789+0000","last_active":"2024-03-11T22:33:06.680230+0000","last_peered":"2024-03-11T22:33:06.680230+0000","last_clean":"2024-03-11T22:33:06.680230+0000","last_became_active":"2024-03-11T22:32:25.070745+0000","last_became_peered":"2024-03-11T22:32:25.070745+0000","last_unstale":"2024-03-11T22:33:06.680230+0000","last_undegraded":"2024-03-11T22:33:06.680230+0000","last_fullsized":"2024-03-11T22:33:06.680230+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"570'12","last_scrub_stamp":"2024-03-11T22:33:00.059686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:33:00.059686+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:16.629600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'46","reported_seq":1210,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338270+0000","last_change":"2024-03-11T22:31:35.314856+0000","last_active":"2024-03-11T22:33:03.338270+0000","last_peered":"2024-03-11T22:33:03.338270+0000","last_clean":"2024-03-11T22:33:03.338270+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:33:03.338270+0000","last_undegraded":"2024-03-11T22:33:03.338270+0000","last_fullsized":"2024-03-11T22:33:03.338270+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'46","last_scrub_stamp":"2024-03-11T22:31:35.314705+0000","last_deep_scrub":"479'46","last_deep_scrub_stamp":"2024-03-11T22:31:35.314705+0000","last_clean_scrub_stamp":"2024-03-11T22:31:35.314705+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"432'4","reported_seq":626,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680697+0000","last_change":"2024-03-11T22:33:05.047160+0000","last_active":"2024-03-11T22:33:06.680697+0000","last_peered":"2024-03-11T22:33:06.680697+0000","last_clean":"2024-03-11T22:33:06.680697+0000","last_became_active":"2024-03-11T22:32:25.065375+0000","last_became_peered":"2024-03-11T22:32:25.065375+0000","last_unstale":"2024-03-11T22:33:06.680697+0000","last_undegraded":"2024-03-11T22:33:06.680697+0000","last_fullsized":"2024-03-11T22:33:06.680697+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'4","last_scrub_stamp":"2024-03-11T22:33:05.047051+0000","last_deep_scrub":"432'4","last_deep_scrub_stamp":"2024-03-11T22:26:50.847349+0000","last_clean_scrub_stamp":"2024-03-11T22:33:05.047051+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:30.850741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1082,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338973+0000","last_change":"2024-03-11T22:31:33.403335+0000","last_active":"2024-03-11T22:33:03.338973+0000","last_peered":"2024-03-11T22:33:03.338973+0000","last_clean":"2024-03-11T22:33:03.338973+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:33:03.338973+0000","last_undegraded":"2024-03-11T22:33:03.338973+0000","last_fullsized":"2024-03-11T22:33:03.338973+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:33.403232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:49.194569+0000","last_clean_scrub_stamp":"2024-03-11T22:31:33.403232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"479'18","reported_seq":1185,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339877+0000","last_change":"2024-03-11T22:32:21.175430+0000","last_active":"2024-03-11T22:33:03.339877+0000","last_peered":"2024-03-11T22:33:03.339877+0000","last_clean":"2024-03-11T22:33:03.339877+0000","last_became_active":"2024-03-11T22:30:16.929794+0000","last_became_peered":"2024-03-11T22:30:16.929794+0000","last_unstale":"2024-03-11T22:33:03.339877+0000","last_undegraded":"2024-03-11T22:33:03.339877+0000","last_fullsized":"2024-03-11T22:33:03.339877+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'18","last_scrub_stamp":"2024-03-11T22:32:21.175319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:21.175319+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:49.790907+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1116,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338158+0000","last_change":"2024-03-11T22:32:28.569731+0000","last_active":"2024-03-11T22:33:03.338158+0000","last_peered":"2024-03-11T22:33:03.338158+0000","last_clean":"2024-03-11T22:33:03.338158+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:33:03.338158+0000","last_undegraded":"2024-03-11T22:33:03.338158+0000","last_fullsized":"2024-03-11T22:33:03.338158+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:28.569588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:32:28.569588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:47.860375+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1105,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339303+0000","last_change":"2024-03-11T22:31:49.196645+0000","last_active":"2024-03-11T22:33:03.339303+0000","last_peered":"2024-03-11T22:33:03.339303+0000","last_clean":"2024-03-11T22:33:03.339303+0000","last_became_active":"2024-03-11T22:29:04.731305+0000","last_became_peered":"2024-03-11T22:29:04.731305+0000","last_unstale":"2024-03-11T22:33:03.339303+0000","last_undegraded":"2024-03-11T22:33:03.339303+0000","last_fullsized":"2024-03-11T22:33:03.339303+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:49.196546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:31:49.196546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":100,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680078+0000","last_change":"2024-03-11T22:32:43.017598+0000","last_active":"2024-03-11T22:33:06.680078+0000","last_peered":"2024-03-11T22:33:06.680078+0000","last_clean":"2024-03-11T22:33:06.680078+0000","last_became_active":"2024-03-11T22:32:25.066663+0000","last_became_peered":"2024-03-11T22:32:25.066663+0000","last_unstale":"2024-03-11T22:33:06.680078+0000","last_undegraded":"2024-03-11T22:33:06.680078+0000","last_fullsized":"2024-03-11T22:33:06.680078+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:43.017454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:32:43.017454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:59.300760+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"478'25","reported_seq":1177,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409195+0000","last_change":"2024-03-11T22:32:51.013986+0000","last_active":"2024-03-11T22:33:07.409195+0000","last_peered":"2024-03-11T22:33:07.409195+0000","last_clean":"2024-03-11T22:33:07.409195+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:33:07.409195+0000","last_undegraded":"2024-03-11T22:33:07.409195+0000","last_fullsized":"2024-03-11T22:33:07.409195+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:32:51.013089+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_clean_scrub_stamp":"2024-03-11T22:32:51.013089+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:03.501164+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":201,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680582+0000","last_change":"2024-03-11T22:32:25.070018+0000","last_active":"2024-03-11T22:33:06.680582+0000","last_peered":"2024-03-11T22:33:06.680582+0000","last_clean":"2024-03-11T22:33:06.680582+0000","last_became_active":"2024-03-11T22:32:25.067290+0000","last_became_peered":"2024-03-11T22:32:25.067290+0000","last_unstale":"2024-03-11T22:33:06.680582+0000","last_undegraded":"2024-03-11T22:33:06.680582+0000","last_fullsized":"2024-03-11T22:33:06.680582+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:11.015678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:11.015678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:29.113585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1108,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339841+0000","last_change":"2024-03-11T22:32:04.961686+0000","last_active":"2024-03-11T22:33:03.339841+0000","last_peered":"2024-03-11T22:33:03.339841+0000","last_clean":"2024-03-11T22:33:03.339841+0000","last_became_active":"2024-03-11T22:29:04.700408+0000","last_became_peered":"2024-03-11T22:29:04.700408+0000","last_unstale":"2024-03-11T22:33:03.339841+0000","last_undegraded":"2024-03-11T22:33:03.339841+0000","last_fullsized":"2024-03-11T22:33:03.339841+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:04.961581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:04.961581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:26.928726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1123,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:02.745299+0000","last_change":"2024-03-11T22:32:01.355414+0000","last_active":"2024-03-11T22:33:02.745299+0000","last_peered":"2024-03-11T22:33:02.745299+0000","last_clean":"2024-03-11T22:33:02.745299+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:33:02.745299+0000","last_undegraded":"2024-03-11T22:33:02.745299+0000","last_fullsized":"2024-03-11T22:33:02.745299+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:01.355283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:32:01.355283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:10.777489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"479'25","reported_seq":1143,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339815+0000","last_change":"2024-03-11T22:32:31.285963+0000","last_active":"2024-03-11T22:33:03.339815+0000","last_peered":"2024-03-11T22:33:03.339815+0000","last_clean":"2024-03-11T22:33:03.339815+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:33:03.339815+0000","last_undegraded":"2024-03-11T22:33:03.339815+0000","last_fullsized":"2024-03-11T22:33:03.339815+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'25","last_scrub_stamp":"2024-03-11T22:32:31.285804+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:32:31.285804+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:31.806256+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1123,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.900324+0000","last_change":"2024-03-11T22:33:07.900346+0000","last_active":"2024-03-11T22:33:07.900324+0000","last_peered":"2024-03-11T22:33:07.900324+0000","last_clean":"2024-03-11T22:33:07.900324+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:33:07.900324+0000","last_undegraded":"2024-03-11T22:33:07.900324+0000","last_fullsized":"2024-03-11T22:33:07.900324+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:07.900247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:33:07.900247+0000","last_clean_scrub_stamp":"2024-03-11T22:33:07.900247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:18.886364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"527'12","reported_seq":1426,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339335+0000","last_change":"2024-03-11T22:32:13.043961+0000","last_active":"2024-03-11T22:33:03.339335+0000","last_peered":"2024-03-11T22:33:03.339335+0000","last_clean":"2024-03-11T22:33:03.339335+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:33:03.339335+0000","last_undegraded":"2024-03-11T22:33:03.339335+0000","last_fullsized":"2024-03-11T22:33:03.339335+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'12","last_scrub_stamp":"2024-03-11T22:32:13.043826+0000","last_deep_scrub":"527'12","last_deep_scrub_stamp":"2024-03-11T22:32:13.043826+0000","last_clean_scrub_stamp":"2024-03-11T22:32:13.043826+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:22.042414+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":274,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410390+0000","last_change":"2024-03-11T22:32:57.070738+0000","last_active":"2024-03-11T22:33:07.410390+0000","last_peered":"2024-03-11T22:33:07.410390+0000","last_clean":"2024-03-11T22:33:07.410390+0000","last_became_active":"2024-03-11T22:32:01.980472+0000","last_became_peered":"2024-03-11T22:32:01.980472+0000","last_unstale":"2024-03-11T22:33:07.410390+0000","last_undegraded":"2024-03-11T22:33:07.410390+0000","last_fullsized":"2024-03-11T22:33:07.410390+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:57.070618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:35.357245+0000","last_clean_scrub_stamp":"2024-03-11T22:32:57.070618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:17.798863+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'30","reported_seq":1187,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339245+0000","last_change":"2024-03-11T22:32:22.146773+0000","last_active":"2024-03-11T22:33:03.339245+0000","last_peered":"2024-03-11T22:33:03.339245+0000","last_clean":"2024-03-11T22:33:03.339245+0000","last_became_active":"2024-03-11T22:29:04.720501+0000","last_became_peered":"2024-03-11T22:29:04.720501+0000","last_unstale":"2024-03-11T22:33:03.339245+0000","last_undegraded":"2024-03-11T22:33:03.339245+0000","last_fullsized":"2024-03-11T22:33:03.339245+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-03-11T22:32:22.146647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:22.146647+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:24.642238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":343,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679864+0000","last_change":"2024-03-11T22:32:25.066585+0000","last_active":"2024-03-11T22:33:06.679864+0000","last_peered":"2024-03-11T22:33:06.679864+0000","last_clean":"2024-03-11T22:33:06.679864+0000","last_became_active":"2024-03-11T22:32:25.066167+0000","last_became_peered":"2024-03-11T22:32:25.066167+0000","last_unstale":"2024-03-11T22:33:06.679864+0000","last_undegraded":"2024-03-11T22:33:06.679864+0000","last_fullsized":"2024-03-11T22:33:06.679864+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:10.023566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:51.034868+0000","last_clean_scrub_stamp":"2024-03-11T22:32:10.023566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:37.215399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"488'10","reported_seq":1417,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339482+0000","last_change":"2024-03-11T22:31:40.234937+0000","last_active":"2024-03-11T22:33:03.339482+0000","last_peered":"2024-03-11T22:33:03.339482+0000","last_clean":"2024-03-11T22:33:03.339482+0000","last_became_active":"2024-03-11T22:29:04.787739+0000","last_became_peered":"2024-03-11T22:29:04.787739+0000","last_unstale":"2024-03-11T22:33:03.339482+0000","last_undegraded":"2024-03-11T22:33:03.339482+0000","last_fullsized":"2024-03-11T22:33:03.339482+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'10","last_scrub_stamp":"2024-03-11T22:31:40.234834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:31:40.234834+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1095,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408978+0000","last_change":"2024-03-11T22:31:55.479021+0000","last_active":"2024-03-11T22:33:07.408978+0000","last_peered":"2024-03-11T22:33:07.408978+0000","last_clean":"2024-03-11T22:33:07.408978+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:33:07.408978+0000","last_undegraded":"2024-03-11T22:33:07.408978+0000","last_fullsized":"2024-03-11T22:33:07.408978+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:55.478922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:31:55.478922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:18.804848+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"479'27","reported_seq":794,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409764+0000","last_change":"2024-03-11T22:32:09.001879+0000","last_active":"2024-03-11T22:33:07.409764+0000","last_peered":"2024-03-11T22:33:07.409764+0000","last_clean":"2024-03-11T22:33:07.409764+0000","last_became_active":"2024-03-11T22:32:01.974003+0000","last_became_peered":"2024-03-11T22:32:01.974003+0000","last_unstale":"2024-03-11T22:33:07.409764+0000","last_undegraded":"2024-03-11T22:33:07.409764+0000","last_fullsized":"2024-03-11T22:33:07.409764+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-03-11T22:32:09.001751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:09.001751+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:37.170218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":300,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410063+0000","last_change":"2024-03-11T22:32:36.705524+0000","last_active":"2024-03-11T22:33:07.410063+0000","last_peered":"2024-03-11T22:33:07.410063+0000","last_clean":"2024-03-11T22:33:07.410063+0000","last_became_active":"2024-03-11T22:32:01.972673+0000","last_became_peered":"2024-03-11T22:32:01.972673+0000","last_unstale":"2024-03-11T22:33:07.410063+0000","last_undegraded":"2024-03-11T22:33:07.410063+0000","last_fullsized":"2024-03-11T22:33:07.410063+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:32:36.705401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:36.705401+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:54.053440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":214,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680487+0000","last_change":"2024-03-11T22:32:12.575956+0000","last_active":"2024-03-11T22:33:06.680487+0000","last_peered":"2024-03-11T22:33:06.680487+0000","last_clean":"2024-03-11T22:33:06.680487+0000","last_became_active":"2024-03-11T22:29:04.723400+0000","last_became_peered":"2024-03-11T22:29:04.723400+0000","last_unstale":"2024-03-11T22:33:06.680487+0000","last_undegraded":"2024-03-11T22:33:06.680487+0000","last_fullsized":"2024-03-11T22:33:06.680487+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:12.575860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:12.575860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:30.116230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1072,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338454+0000","last_change":"2024-03-11T22:31:49.185033+0000","last_active":"2024-03-11T22:33:03.338454+0000","last_peered":"2024-03-11T22:33:03.338454+0000","last_clean":"2024-03-11T22:33:03.338454+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:33:03.338454+0000","last_undegraded":"2024-03-11T22:33:03.338454+0000","last_fullsized":"2024-03-11T22:33:03.338454+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:49.184919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:31:49.184919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:08.918644+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"478'25","reported_seq":229,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680112+0000","last_change":"2024-03-11T22:32:51.610299+0000","last_active":"2024-03-11T22:33:06.680112+0000","last_peered":"2024-03-11T22:33:06.680112+0000","last_clean":"2024-03-11T22:33:06.680112+0000","last_became_active":"2024-03-11T22:29:04.730485+0000","last_became_peered":"2024-03-11T22:29:04.730485+0000","last_unstale":"2024-03-11T22:33:06.680112+0000","last_undegraded":"2024-03-11T22:33:06.680112+0000","last_fullsized":"2024-03-11T22:33:06.680112+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:32:51.610209+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:32:51.610209+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:06.971761+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":494,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679820+0000","last_change":"2024-03-11T22:32:25.065573+0000","last_active":"2024-03-11T22:33:06.679820+0000","last_peered":"2024-03-11T22:33:06.679820+0000","last_clean":"2024-03-11T22:33:06.679820+0000","last_became_active":"2024-03-11T22:32:25.064682+0000","last_became_peered":"2024-03-11T22:32:25.064682+0000","last_unstale":"2024-03-11T22:33:06.679820+0000","last_undegraded":"2024-03-11T22:33:06.679820+0000","last_fullsized":"2024-03-11T22:33:06.679820+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:50.162015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:31:50.162015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:13.107667+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1096,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338745+0000","last_change":"2024-03-11T22:31:56.101706+0000","last_active":"2024-03-11T22:33:03.338745+0000","last_peered":"2024-03-11T22:33:03.338745+0000","last_clean":"2024-03-11T22:33:03.338745+0000","last_became_active":"2024-03-11T22:29:04.700352+0000","last_became_peered":"2024-03-11T22:29:04.700352+0000","last_unstale":"2024-03-11T22:33:03.338745+0000","last_undegraded":"2024-03-11T22:33:03.338745+0000","last_fullsized":"2024-03-11T22:33:03.338745+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:56.101587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:31:56.101587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:12.452173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1104,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408033+0000","last_change":"2024-03-11T22:32:18.264189+0000","last_active":"2024-03-11T22:33:07.408033+0000","last_peered":"2024-03-11T22:33:07.408033+0000","last_clean":"2024-03-11T22:33:07.408033+0000","last_became_active":"2024-03-11T22:29:04.707420+0000","last_became_peered":"2024-03-11T22:29:04.707420+0000","last_unstale":"2024-03-11T22:33:07.408033+0000","last_undegraded":"2024-03-11T22:33:07.408033+0000","last_fullsized":"2024-03-11T22:33:07.408033+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:18.264067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:09.057970+0000","last_clean_scrub_stamp":"2024-03-11T22:32:18.264067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:40.145531+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"480'28","reported_seq":1168,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338657+0000","last_change":"2024-03-11T22:32:52.163857+0000","last_active":"2024-03-11T22:33:03.338657+0000","last_peered":"2024-03-11T22:33:03.338657+0000","last_clean":"2024-03-11T22:33:03.338657+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:33:03.338657+0000","last_undegraded":"2024-03-11T22:33:03.338657+0000","last_fullsized":"2024-03-11T22:33:03.338657+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'28","last_scrub_stamp":"2024-03-11T22:32:52.163741+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:32:52.163741+0000","objects_scrubbed":5,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:15.948914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1125,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339124+0000","last_change":"2024-03-11T22:32:16.283660+0000","last_active":"2024-03-11T22:33:03.339124+0000","last_peered":"2024-03-11T22:33:03.339124+0000","last_clean":"2024-03-11T22:33:03.339124+0000","last_became_active":"2024-03-11T22:29:04.722573+0000","last_became_peered":"2024-03-11T22:29:04.722573+0000","last_unstale":"2024-03-11T22:33:03.339124+0000","last_undegraded":"2024-03-11T22:33:03.339124+0000","last_fullsized":"2024-03-11T22:33:03.339124+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:16.283559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:16.283559+0000","last_clean_scrub_stamp":"2024-03-11T22:32:16.283559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:27.190217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":208,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680303+0000","last_change":"2024-03-11T22:32:29.046069+0000","last_active":"2024-03-11T22:33:06.680303+0000","last_peered":"2024-03-11T22:33:06.680303+0000","last_clean":"2024-03-11T22:33:06.680303+0000","last_became_active":"2024-03-11T22:32:25.066717+0000","last_became_peered":"2024-03-11T22:32:25.066717+0000","last_unstale":"2024-03-11T22:33:06.680303+0000","last_undegraded":"2024-03-11T22:33:06.680303+0000","last_fullsized":"2024-03-11T22:33:06.680303+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:29.045953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:29.045953+0000","last_clean_scrub_stamp":"2024-03-11T22:32:29.045953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:54.322587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1080,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339621+0000","last_change":"2024-03-11T22:32:23.283924+0000","last_active":"2024-03-11T22:33:03.339621+0000","last_peered":"2024-03-11T22:33:03.339621+0000","last_clean":"2024-03-11T22:33:03.339621+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:33:03.339621+0000","last_undegraded":"2024-03-11T22:33:03.339621+0000","last_fullsized":"2024-03-11T22:33:03.339621+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:23.283801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:23.283801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:32.124554+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"478'12","reported_seq":1150,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338151+0000","last_change":"2024-03-11T22:33:00.202526+0000","last_active":"2024-03-11T22:33:03.338151+0000","last_peered":"2024-03-11T22:33:03.338151+0000","last_clean":"2024-03-11T22:33:03.338151+0000","last_became_active":"2024-03-11T22:29:04.734207+0000","last_became_peered":"2024-03-11T22:29:04.734207+0000","last_unstale":"2024-03-11T22:33:03.338151+0000","last_undegraded":"2024-03-11T22:33:03.338151+0000","last_fullsized":"2024-03-11T22:33:03.338151+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'12","last_scrub_stamp":"2024-03-11T22:33:00.202402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:33:00.202402+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:29.698511+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1103,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409567+0000","last_change":"2024-03-11T22:32:33.018315+0000","last_active":"2024-03-11T22:33:07.409567+0000","last_peered":"2024-03-11T22:33:07.409567+0000","last_clean":"2024-03-11T22:33:07.409567+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:33:07.409567+0000","last_undegraded":"2024-03-11T22:33:07.409567+0000","last_fullsized":"2024-03-11T22:33:07.409567+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:33.018185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:04.134523+0000","last_clean_scrub_stamp":"2024-03-11T22:32:33.018185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:41.017610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":694,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338183+0000","last_change":"2024-03-11T22:32:52.517091+0000","last_active":"2024-03-11T22:33:03.338183+0000","last_peered":"2024-03-11T22:33:03.338183+0000","last_clean":"2024-03-11T22:33:03.338183+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:33:03.338183+0000","last_undegraded":"2024-03-11T22:33:03.338183+0000","last_fullsized":"2024-03-11T22:33:03.338183+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:52.516980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:52.516980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:11.666992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":92,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679743+0000","last_change":"2024-03-11T22:32:25.066370+0000","last_active":"2024-03-11T22:33:06.679743+0000","last_peered":"2024-03-11T22:33:06.679743+0000","last_clean":"2024-03-11T22:33:06.679743+0000","last_became_active":"2024-03-11T22:32:25.065512+0000","last_became_peered":"2024-03-11T22:32:25.065512+0000","last_unstale":"2024-03-11T22:33:06.679743+0000","last_undegraded":"2024-03-11T22:33:06.679743+0000","last_fullsized":"2024-03-11T22:33:06.679743+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:12.331743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:12.331743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:40.788117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"480'27","reported_seq":1192,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339343+0000","last_change":"2024-03-11T22:32:03.688836+0000","last_active":"2024-03-11T22:33:03.339343+0000","last_peered":"2024-03-11T22:33:03.339343+0000","last_clean":"2024-03-11T22:33:03.339343+0000","last_became_active":"2024-03-11T22:29:04.720148+0000","last_became_peered":"2024-03-11T22:29:04.720148+0000","last_unstale":"2024-03-11T22:33:03.339343+0000","last_undegraded":"2024-03-11T22:33:03.339343+0000","last_fullsized":"2024-03-11T22:33:03.339343+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'27","last_scrub_stamp":"2024-03-11T22:32:03.688734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:03.688734+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:16.351475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1104,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338123+0000","last_change":"2024-03-11T22:32:06.986662+0000","last_active":"2024-03-11T22:33:03.338123+0000","last_peered":"2024-03-11T22:33:03.338123+0000","last_clean":"2024-03-11T22:33:03.338123+0000","last_became_active":"2024-03-11T22:29:04.733742+0000","last_became_peered":"2024-03-11T22:29:04.733742+0000","last_unstale":"2024-03-11T22:33:03.338123+0000","last_undegraded":"2024-03-11T22:33:03.338123+0000","last_fullsized":"2024-03-11T22:33:03.338123+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:06.986570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:06.986570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:27.504039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1105,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409301+0000","last_change":"2024-03-11T22:32:43.220021+0000","last_active":"2024-03-11T22:33:07.409301+0000","last_peered":"2024-03-11T22:33:07.409301+0000","last_clean":"2024-03-11T22:33:07.409301+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:33:07.409301+0000","last_undegraded":"2024-03-11T22:33:07.409301+0000","last_fullsized":"2024-03-11T22:33:07.409301+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:43.219886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:43.219886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:11.167500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":219,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680894+0000","last_change":"2024-03-11T22:32:54.614518+0000","last_active":"2024-03-11T22:33:06.680894+0000","last_peered":"2024-03-11T22:33:06.680894+0000","last_clean":"2024-03-11T22:33:06.680894+0000","last_became_active":"2024-03-11T22:29:04.786119+0000","last_became_peered":"2024-03-11T22:29:04.786119+0000","last_unstale":"2024-03-11T22:33:06.680894+0000","last_undegraded":"2024-03-11T22:33:06.680894+0000","last_fullsized":"2024-03-11T22:33:06.680894+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:54.614382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:06.824468+0000","last_clean_scrub_stamp":"2024-03-11T22:32:54.614382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:07.388340+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"479'30","reported_seq":1182,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409188+0000","last_change":"2024-03-11T22:32:07.378996+0000","last_active":"2024-03-11T22:33:07.409188+0000","last_peered":"2024-03-11T22:33:07.409188+0000","last_clean":"2024-03-11T22:33:07.409188+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:33:07.409188+0000","last_undegraded":"2024-03-11T22:33:07.409188+0000","last_fullsized":"2024-03-11T22:33:07.409188+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-03-11T22:32:07.378892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:07.378892+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:22.849990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":215,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679640+0000","last_change":"2024-03-11T22:31:46.481014+0000","last_active":"2024-03-11T22:33:06.679640+0000","last_peered":"2024-03-11T22:33:06.679640+0000","last_clean":"2024-03-11T22:33:06.679640+0000","last_became_active":"2024-03-11T22:29:04.693762+0000","last_became_peered":"2024-03-11T22:29:04.693762+0000","last_unstale":"2024-03-11T22:33:06.679640+0000","last_undegraded":"2024-03-11T22:33:06.679640+0000","last_fullsized":"2024-03-11T22:33:06.679640+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:31:46.480926+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:31:46.480926+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"541'12","reported_seq":121,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408920+0000","last_change":"2024-03-11T22:32:08.427075+0000","last_active":"2024-03-11T22:33:07.408920+0000","last_peered":"2024-03-11T22:33:07.408920+0000","last_clean":"2024-03-11T22:33:07.408920+0000","last_became_active":"2024-03-11T22:31:11.769411+0000","last_became_peered":"2024-03-11T22:31:11.769411+0000","last_unstale":"2024-03-11T22:33:07.408920+0000","last_undegraded":"2024-03-11T22:33:07.408920+0000","last_fullsized":"2024-03-11T22:33:07.408920+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"541'12","last_scrub_stamp":"2024-03-11T22:32:08.426964+0000","last_deep_scrub":"541'12","last_deep_scrub_stamp":"2024-03-11T22:32:08.426964+0000","last_clean_scrub_stamp":"2024-03-11T22:32:08.426964+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:20.953228+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"553'2","reported_seq":1024,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680485+0000","last_change":"2024-03-11T22:32:53.990047+0000","last_active":"2024-03-11T22:33:06.680485+0000","last_peered":"2024-03-11T22:33:06.680485+0000","last_clean":"2024-03-11T22:33:06.680485+0000","last_became_active":"2024-03-11T22:32:25.072182+0000","last_became_peered":"2024-03-11T22:32:25.072182+0000","last_unstale":"2024-03-11T22:33:06.680485+0000","last_undegraded":"2024-03-11T22:33:06.680485+0000","last_fullsized":"2024-03-11T22:33:06.680485+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"553'2","last_scrub_stamp":"2024-03-11T22:32:53.989909+0000","last_deep_scrub":"553'2","last_deep_scrub_stamp":"2024-03-11T22:31:46.173315+0000","last_clean_scrub_stamp":"2024-03-11T22:32:53.989909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:59.823594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"480'24","reported_seq":1164,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339160+0000","last_change":"2024-03-11T22:32:17.060931+0000","last_active":"2024-03-11T22:33:03.339160+0000","last_peered":"2024-03-11T22:33:03.339160+0000","last_clean":"2024-03-11T22:33:03.339160+0000","last_became_active":"2024-03-11T22:29:04.720341+0000","last_became_peered":"2024-03-11T22:29:04.720341+0000","last_unstale":"2024-03-11T22:33:03.339160+0000","last_undegraded":"2024-03-11T22:33:03.339160+0000","last_fullsized":"2024-03-11T22:33:03.339160+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'24","last_scrub_stamp":"2024-03-11T22:32:17.060793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:17.060793+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:20.357668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":294,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410126+0000","last_change":"2024-03-11T22:32:48.857529+0000","last_active":"2024-03-11T22:33:07.410126+0000","last_peered":"2024-03-11T22:33:07.410126+0000","last_clean":"2024-03-11T22:33:07.410126+0000","last_became_active":"2024-03-11T22:32:01.978045+0000","last_became_peered":"2024-03-11T22:32:01.978045+0000","last_unstale":"2024-03-11T22:33:07.410126+0000","last_undegraded":"2024-03-11T22:33:07.410126+0000","last_fullsized":"2024-03-11T22:33:07.410126+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:48.857426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:08.740926+0000","last_clean_scrub_stamp":"2024-03-11T22:32:48.857426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:52.918396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":286,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410331+0000","last_change":"2024-03-11T22:32:15.970183+0000","last_active":"2024-03-11T22:33:07.410331+0000","last_peered":"2024-03-11T22:33:07.410331+0000","last_clean":"2024-03-11T22:33:07.410331+0000","last_became_active":"2024-03-11T22:32:01.980091+0000","last_became_peered":"2024-03-11T22:32:01.980091+0000","last_unstale":"2024-03-11T22:33:07.410331+0000","last_undegraded":"2024-03-11T22:33:07.410331+0000","last_fullsized":"2024-03-11T22:33:07.410331+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:15.970077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:15.970077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:24.278206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1081,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339596+0000","last_change":"2024-03-11T22:32:54.364298+0000","last_active":"2024-03-11T22:33:03.339596+0000","last_peered":"2024-03-11T22:33:03.339596+0000","last_clean":"2024-03-11T22:33:03.339596+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:33:03.339596+0000","last_undegraded":"2024-03-11T22:33:03.339596+0000","last_fullsized":"2024-03-11T22:33:03.339596+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:54.364192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:54.364192+0000","last_clean_scrub_stamp":"2024-03-11T22:32:54.364192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:55.788227+0000","scrub_duration":213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"479'39","reported_seq":1243,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:02.745001+0000","last_change":"2024-03-11T22:32:56.667313+0000","last_active":"2024-03-11T22:33:02.745001+0000","last_peered":"2024-03-11T22:33:02.745001+0000","last_clean":"2024-03-11T22:33:02.745001+0000","last_became_active":"2024-03-11T22:29:04.721324+0000","last_became_peered":"2024-03-11T22:29:04.721324+0000","last_unstale":"2024-03-11T22:33:02.745001+0000","last_undegraded":"2024-03-11T22:33:02.745001+0000","last_fullsized":"2024-03-11T22:33:02.745001+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'39","last_scrub_stamp":"2024-03-11T22:32:56.667210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:56.667210+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:09.653888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1114,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408545+0000","last_change":"2024-03-11T22:32:02.395766+0000","last_active":"2024-03-11T22:33:07.408545+0000","last_peered":"2024-03-11T22:33:07.408545+0000","last_clean":"2024-03-11T22:33:07.408545+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:33:07.408545+0000","last_undegraded":"2024-03-11T22:33:07.408545+0000","last_fullsized":"2024-03-11T22:33:07.408545+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:02.395678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:32:02.395678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:22.591865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":219,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680890+0000","last_change":"2024-03-11T22:32:35.581682+0000","last_active":"2024-03-11T22:33:06.680890+0000","last_peered":"2024-03-11T22:33:06.680890+0000","last_clean":"2024-03-11T22:33:06.680890+0000","last_became_active":"2024-03-11T22:29:04.724856+0000","last_became_peered":"2024-03-11T22:29:04.724856+0000","last_unstale":"2024-03-11T22:33:06.680890+0000","last_undegraded":"2024-03-11T22:33:06.680890+0000","last_fullsized":"2024-03-11T22:33:06.680890+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:35.581593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:35.581593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:53.240847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":283,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409653+0000","last_change":"2024-03-11T22:33:04.051479+0000","last_active":"2024-03-11T22:33:07.409653+0000","last_peered":"2024-03-11T22:33:07.409653+0000","last_clean":"2024-03-11T22:33:07.409653+0000","last_became_active":"2024-03-11T22:32:01.973161+0000","last_became_peered":"2024-03-11T22:32:01.973161+0000","last_unstale":"2024-03-11T22:33:07.409653+0000","last_undegraded":"2024-03-11T22:33:07.409653+0000","last_fullsized":"2024-03-11T22:33:07.409653+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:04.051367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:41.776548+0000","last_clean_scrub_stamp":"2024-03-11T22:33:04.051367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:23.828209+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"478'11","reported_seq":780,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409580+0000","last_change":"2024-03-11T22:32:08.050734+0000","last_active":"2024-03-11T22:33:07.409580+0000","last_peered":"2024-03-11T22:33:07.409580+0000","last_clean":"2024-03-11T22:33:07.409580+0000","last_became_active":"2024-03-11T22:32:01.972835+0000","last_became_peered":"2024-03-11T22:32:01.972835+0000","last_unstale":"2024-03-11T22:33:07.409580+0000","last_undegraded":"2024-03-11T22:33:07.409580+0000","last_fullsized":"2024-03-11T22:33:07.409580+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'11","last_scrub_stamp":"2024-03-11T22:32:08.050630+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:32:08.050630+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:15.787633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1112,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339438+0000","last_change":"2024-03-11T22:32:00.332632+0000","last_active":"2024-03-11T22:33:03.339438+0000","last_peered":"2024-03-11T22:33:03.339438+0000","last_clean":"2024-03-11T22:33:03.339438+0000","last_became_active":"2024-03-11T22:29:04.721192+0000","last_became_peered":"2024-03-11T22:29:04.721192+0000","last_unstale":"2024-03-11T22:33:03.339438+0000","last_undegraded":"2024-03-11T22:33:03.339438+0000","last_fullsized":"2024-03-11T22:33:03.339438+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:00.332516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:32:00.332516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:12.537196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":200,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679971+0000","last_change":"2024-03-11T22:32:25.070793+0000","last_active":"2024-03-11T22:33:06.679971+0000","last_peered":"2024-03-11T22:33:06.679971+0000","last_clean":"2024-03-11T22:33:06.679971+0000","last_became_active":"2024-03-11T22:32:25.069724+0000","last_became_peered":"2024-03-11T22:32:25.069724+0000","last_unstale":"2024-03-11T22:33:06.679971+0000","last_undegraded":"2024-03-11T22:33:06.679971+0000","last_fullsized":"2024-03-11T22:33:06.679971+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:18.974465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:09.575187+0000","last_clean_scrub_stamp":"2024-03-11T22:32:18.974465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:23.746670+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1101,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338688+0000","last_change":"2024-03-11T22:32:22.365025+0000","last_active":"2024-03-11T22:33:03.338688+0000","last_peered":"2024-03-11T22:33:03.338688+0000","last_clean":"2024-03-11T22:33:03.338688+0000","last_became_active":"2024-03-11T22:29:04.721807+0000","last_became_peered":"2024-03-11T22:29:04.721807+0000","last_unstale":"2024-03-11T22:33:03.338688+0000","last_undegraded":"2024-03-11T22:33:03.338688+0000","last_fullsized":"2024-03-11T22:33:03.338688+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:22.364881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:22.364881+0000","last_clean_scrub_stamp":"2024-03-11T22:32:22.364881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:50.855997+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"479'21","reported_seq":751,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410107+0000","last_change":"2024-03-11T22:32:13.061722+0000","last_active":"2024-03-11T22:33:07.410107+0000","last_peered":"2024-03-11T22:33:07.410107+0000","last_clean":"2024-03-11T22:33:07.410107+0000","last_became_active":"2024-03-11T22:32:01.966449+0000","last_became_peered":"2024-03-11T22:32:01.966449+0000","last_unstale":"2024-03-11T22:33:07.410107+0000","last_undegraded":"2024-03-11T22:33:07.410107+0000","last_fullsized":"2024-03-11T22:33:07.410107+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'21","last_scrub_stamp":"2024-03-11T22:32:13.061595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:13.061595+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:28.213044+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1100,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.339324+0000","last_change":"2024-03-11T22:31:43.180566+0000","last_active":"2024-03-11T22:33:03.339324+0000","last_peered":"2024-03-11T22:33:03.339324+0000","last_clean":"2024-03-11T22:33:03.339324+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:33:03.339324+0000","last_undegraded":"2024-03-11T22:33:03.339324+0000","last_fullsized":"2024-03-11T22:33:03.339324+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:43.180426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:31:43.180426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1104,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338588+0000","last_change":"2024-03-11T22:31:44.158242+0000","last_active":"2024-03-11T22:33:03.338588+0000","last_peered":"2024-03-11T22:33:03.338588+0000","last_clean":"2024-03-11T22:33:03.338588+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:33:03.338588+0000","last_undegraded":"2024-03-11T22:33:03.338588+0000","last_fullsized":"2024-03-11T22:33:03.338588+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:44.158149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:31:44.158149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1103,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408467+0000","last_change":"2024-03-11T22:32:49.083293+0000","last_active":"2024-03-11T22:33:07.408467+0000","last_peered":"2024-03-11T22:33:07.408467+0000","last_clean":"2024-03-11T22:33:07.408467+0000","last_became_active":"2024-03-11T22:29:04.707444+0000","last_became_peered":"2024-03-11T22:29:04.707444+0000","last_unstale":"2024-03-11T22:33:07.408467+0000","last_undegraded":"2024-03-11T22:33:07.408467+0000","last_fullsized":"2024-03-11T22:33:07.408467+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:49.083192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:49.083192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:06.742489+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"480'39","reported_seq":1149,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338693+0000","last_change":"2024-03-11T22:32:40.160594+0000","last_active":"2024-03-11T22:33:03.338693+0000","last_peered":"2024-03-11T22:33:03.338693+0000","last_clean":"2024-03-11T22:33:03.338693+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:33:03.338693+0000","last_undegraded":"2024-03-11T22:33:03.338693+0000","last_fullsized":"2024-03-11T22:33:03.338693+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-11T22:32:40.160489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:40.160489+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:06.390231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"575'2","reported_seq":1109,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338980+0000","last_change":"2024-03-11T22:32:59.178704+0000","last_active":"2024-03-11T22:33:03.338980+0000","last_peered":"2024-03-11T22:33:03.338980+0000","last_clean":"2024-03-11T22:33:03.338980+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:33:03.338980+0000","last_undegraded":"2024-03-11T22:33:03.338980+0000","last_fullsized":"2024-03-11T22:33:03.338980+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'2","last_scrub_stamp":"2024-03-11T22:32:59.178597+0000","last_deep_scrub":"575'2","last_deep_scrub_stamp":"2024-03-11T22:32:59.178597+0000","last_clean_scrub_stamp":"2024-03-11T22:32:59.178597+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:08.640395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":736,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408409+0000","last_change":"2024-03-11T22:33:05.946294+0000","last_active":"2024-03-11T22:33:07.408409+0000","last_peered":"2024-03-11T22:33:07.408409+0000","last_clean":"2024-03-11T22:33:07.408409+0000","last_became_active":"2024-03-11T22:32:01.975227+0000","last_became_peered":"2024-03-11T22:32:01.975227+0000","last_unstale":"2024-03-11T22:33:07.408409+0000","last_undegraded":"2024-03-11T22:33:07.408409+0000","last_fullsized":"2024-03-11T22:33:07.408409+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:05.946180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:33:05.946180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:25.507312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1091,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-11T22:33:03.338855+0000","last_change":"2024-03-11T22:32:43.271141+0000","last_active":"2024-03-11T22:33:03.338855+0000","last_peered":"2024-03-11T22:33:03.338855+0000","last_clean":"2024-03-11T22:33:03.338855+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:33:03.338855+0000","last_undegraded":"2024-03-11T22:33:03.338855+0000","last_fullsized":"2024-03-11T22:33:03.338855+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:43.271029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:43.271029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:56.249736+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"478'25","reported_seq":777,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408505+0000","last_change":"2024-03-11T22:32:52.823260+0000","last_active":"2024-03-11T22:33:07.408505+0000","last_peered":"2024-03-11T22:33:07.408505+0000","last_clean":"2024-03-11T22:33:07.408505+0000","last_became_active":"2024-03-11T22:32:01.973526+0000","last_became_peered":"2024-03-11T22:32:01.973526+0000","last_unstale":"2024-03-11T22:33:07.408505+0000","last_undegraded":"2024-03-11T22:33:07.408505+0000","last_fullsized":"2024-03-11T22:33:07.408505+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:32:52.823135+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:32:52.823135+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:57.389349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":30,"num_read_kb":24,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1112,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409357+0000","last_change":"2024-03-11T22:33:00.035171+0000","last_active":"2024-03-11T22:33:07.409357+0000","last_peered":"2024-03-11T22:33:07.409357+0000","last_clean":"2024-03-11T22:33:07.409357+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:33:07.409357+0000","last_undegraded":"2024-03-11T22:33:07.409357+0000","last_fullsized":"2024-03-11T22:33:07.409357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:00.035072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:33:00.035072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:10.449315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":219,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679909+0000","last_change":"2024-03-11T22:33:04.533400+0000","last_active":"2024-03-11T22:33:06.679909+0000","last_peered":"2024-03-11T22:33:06.679909+0000","last_clean":"2024-03-11T22:33:06.679909+0000","last_became_active":"2024-03-11T22:29:04.701716+0000","last_became_peered":"2024-03-11T22:29:04.701716+0000","last_unstale":"2024-03-11T22:33:06.679909+0000","last_undegraded":"2024-03-11T22:33:06.679909+0000","last_fullsized":"2024-03-11T22:33:06.679909+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:04.533317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:33:04.533317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:15.117209+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":725,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409831+0000","last_change":"2024-03-11T22:32:33.840657+0000","last_active":"2024-03-11T22:33:07.409831+0000","last_peered":"2024-03-11T22:33:07.409831+0000","last_clean":"2024-03-11T22:33:07.409831+0000","last_became_active":"2024-03-11T22:32:01.974072+0000","last_became_peered":"2024-03-11T22:32:01.974072+0000","last_unstale":"2024-03-11T22:33:07.409831+0000","last_undegraded":"2024-03-11T22:33:07.409831+0000","last_fullsized":"2024-03-11T22:33:07.409831+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:33.840544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:33.840544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:35.615389+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":158,"num_read_kb":153,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1529,"num_read_kb":1286,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":5804,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1044,"ondisk_log_size":1044,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005357,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":846256,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92898896,"statfs":{"total":95995035648,"available":95128469504,"internally_reserved":0,"allocated":2985984,"data_stored":1986710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":863566283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332401,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":691792,"kb_used_data":2564,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93053360,"statfs":{"total":95995035648,"available":95286640640,"internally_reserved":0,"allocated":2625536,"data_stored":1651063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":705755590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575179,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":751540,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92993612,"statfs":{"total":95995035648,"available":95225458688,"internally_reserved":0,"allocated":2666496,"data_stored":1651121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738705,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":958380,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92786772,"statfs":{"total":95995035648,"available":95013654528,"internally_reserved":0,"allocated":2654208,"data_stored":1658362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411654,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":647636,"kb_used_data":2820,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93097516,"statfs":{"total":95995035648,"available":95331856384,"internally_reserved":0,"allocated":2887680,"data_stored":1913577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150016,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":778492,"kb_used_data":3180,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92966660,"statfs":{"total":95995035648,"available":95197859840,"internally_reserved":0,"allocated":3256320,"data_stored":2241937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":793901511},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921085,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":719604,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93025548,"statfs":{"total":95995035648,"available":95258161152,"internally_reserved":0,"allocated":2662400,"data_stored":1658571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":734198214},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659445,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":796516,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92948636,"statfs":{"total":95995035648,"available":95179403264,"internally_reserved":0,"allocated":3231744,"data_stored":2240842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":812382662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:33:12.492 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:33:12.742 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:33:12.775 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:12 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:33:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:33:13.035 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:12 smithi045 ceph-mon[39715]: 3.1d scrub starts 2024-03-11T22:33:13.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:12 smithi045 ceph-mon[39715]: 3.1d scrub ok 2024-03-11T22:33:13.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:12 smithi045 ceph-mon[39715]: from='client.16227 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:33:13.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:12 smithi045 ceph-mon[39715]: 6.14 scrub starts 2024-03-11T22:33:13.036 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:12 smithi045 ceph-mon[39715]: 6.14 scrub ok 2024-03-11T22:33:13.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:12 smithi045 ceph-mon[30178]: 3.1d scrub starts 2024-03-11T22:33:13.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:12 smithi045 ceph-mon[30178]: 3.1d scrub ok 2024-03-11T22:33:13.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:12 smithi045 ceph-mon[30178]: from='client.16227 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:33:13.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:12 smithi045 ceph-mon[30178]: 6.14 scrub starts 2024-03-11T22:33:13.036 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:12 smithi045 ceph-mon[30178]: 6.14 scrub ok 2024-03-11T22:33:13.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:12 smithi144 ceph-mon[33984]: 3.1d scrub starts 2024-03-11T22:33:13.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:12 smithi144 ceph-mon[33984]: 3.1d scrub ok 2024-03-11T22:33:13.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:12 smithi144 ceph-mon[33984]: from='client.16227 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:33:13.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:12 smithi144 ceph-mon[33984]: 6.14 scrub starts 2024-03-11T22:33:13.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:12 smithi144 ceph-mon[33984]: 6.14 scrub ok 2024-03-11T22:33:13.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:13 smithi045 ceph-mon[39715]: pgmap v1214: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:33:13.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:13 smithi045 ceph-mon[39715]: 6.5 scrub starts 2024-03-11T22:33:13.999 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:13 smithi045 ceph-mon[39715]: 6.5 scrub ok 2024-03-11T22:33:13.999 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:13 smithi045 ceph-mon[30178]: pgmap v1214: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:33:14.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:13 smithi045 ceph-mon[30178]: 6.5 scrub starts 2024-03-11T22:33:14.000 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:13 smithi045 ceph-mon[30178]: 6.5 scrub ok 2024-03-11T22:33:14.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:13 smithi144 ceph-mon[33984]: pgmap v1214: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T22:33:14.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:13 smithi144 ceph-mon[33984]: 6.5 scrub starts 2024-03-11T22:33:14.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:13 smithi144 ceph-mon[33984]: 6.5 scrub ok 2024-03-11T22:33:14.770 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-03-11T22:33:14.772 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-03-11T22:33:15.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:14 smithi045 ceph-mon[39715]: 4.4 deep-scrub starts 2024-03-11T22:33:15.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:14 smithi045 ceph-mon[39715]: 4.4 deep-scrub ok 2024-03-11T22:33:15.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:14 smithi045 ceph-mon[39715]: 3.b scrub starts 2024-03-11T22:33:15.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:14 smithi045 ceph-mon[39715]: 3.b scrub ok 2024-03-11T22:33:15.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:14 smithi045 ceph-mon[30178]: 4.4 deep-scrub starts 2024-03-11T22:33:15.037 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:14 smithi045 ceph-mon[30178]: 4.4 deep-scrub ok 2024-03-11T22:33:15.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:14 smithi045 ceph-mon[30178]: 3.b scrub starts 2024-03-11T22:33:15.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:14 smithi045 ceph-mon[30178]: 3.b scrub ok 2024-03-11T22:33:15.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:14 smithi144 ceph-mon[33984]: 4.4 deep-scrub starts 2024-03-11T22:33:15.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:14 smithi144 ceph-mon[33984]: 4.4 deep-scrub ok 2024-03-11T22:33:15.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:14 smithi144 ceph-mon[33984]: 3.b scrub starts 2024-03-11T22:33:15.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:14 smithi144 ceph-mon[33984]: 3.b scrub ok 2024-03-11T22:33:15.496 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":1215,"stamp":"2024-03-11T22:33:14.571038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1701,"num_read_kb":1453,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":333942,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1151,"ondisk_log_size":1151,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6190260,"kb_used_data":22492,"kb_used_omap":12,"kb_used_meta":6167667,"kb_avail":743770956,"statfs":{"total":767960285184,"available":761621458944,"internally_reserved":0,"allocated":23031808,"data_stored":15041068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":6315691571},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002670"},"pg_stats":[{"pgid":"4.c","version":"479'35","reported_seq":752,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410325+0000","last_change":"2024-03-11T22:32:01.975419+0000","last_active":"2024-03-11T22:33:07.410325+0000","last_peered":"2024-03-11T22:33:07.410325+0000","last_clean":"2024-03-11T22:33:07.410325+0000","last_became_active":"2024-03-11T22:32:01.973558+0000","last_became_peered":"2024-03-11T22:32:01.973558+0000","last_unstale":"2024-03-11T22:33:07.410325+0000","last_undegraded":"2024-03-11T22:33:07.410325+0000","last_fullsized":"2024-03-11T22:33:07.410325+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'35","last_scrub_stamp":"2024-03-11T22:31:59.080346+0000","last_deep_scrub":"294'25","last_deep_scrub_stamp":"2024-03-11T22:19:41.508161+0000","last_clean_scrub_stamp":"2024-03-11T22:31:59.080346+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1094,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409708+0000","last_change":"2024-03-11T22:32:02.018644+0000","last_active":"2024-03-11T22:33:07.409708+0000","last_peered":"2024-03-11T22:33:07.409708+0000","last_clean":"2024-03-11T22:33:07.409708+0000","last_became_active":"2024-03-11T22:08:33.035507+0000","last_became_peered":"2024-03-11T22:08:33.035507+0000","last_unstale":"2024-03-11T22:33:07.409708+0000","last_undegraded":"2024-03-11T22:33:07.409708+0000","last_fullsized":"2024-03-11T22:33:07.409708+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:02.018533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:02.018533+0000","last_clean_scrub_stamp":"2024-03-11T22:32:02.018533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:07.966734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"91'9","reported_seq":1442,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:09.846131+0000","last_change":"2024-03-11T22:33:09.846157+0000","last_active":"2024-03-11T22:33:09.846131+0000","last_peered":"2024-03-11T22:33:09.846131+0000","last_clean":"2024-03-11T22:33:09.846131+0000","last_became_active":"2024-03-11T22:08:37.074691+0000","last_became_peered":"2024-03-11T22:08:37.074691+0000","last_unstale":"2024-03-11T22:33:09.846131+0000","last_undegraded":"2024-03-11T22:33:09.846131+0000","last_fullsized":"2024-03-11T22:33:09.846131+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"91'9","last_scrub_stamp":"2024-03-11T22:33:09.846067+0000","last_deep_scrub":"91'9","last_deep_scrub_stamp":"2024-03-11T22:33:09.846067+0000","last_clean_scrub_stamp":"2024-03-11T22:33:09.846067+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:15.475115+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":282,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409890+0000","last_change":"2024-03-11T22:32:29.837285+0000","last_active":"2024-03-11T22:33:07.409890+0000","last_peered":"2024-03-11T22:33:07.409890+0000","last_clean":"2024-03-11T22:33:07.409890+0000","last_became_active":"2024-03-11T22:32:01.980840+0000","last_became_peered":"2024-03-11T22:32:01.980840+0000","last_unstale":"2024-03-11T22:33:07.409890+0000","last_undegraded":"2024-03-11T22:33:07.409890+0000","last_fullsized":"2024-03-11T22:33:07.409890+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:29.837170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:29.837170+0000","last_clean_scrub_stamp":"2024-03-11T22:32:29.837170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:53.875012+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"93'130","reported_seq":883,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679543+0000","last_change":"2024-03-11T22:32:58.041419+0000","last_active":"2024-03-11T22:33:06.679543+0000","last_peered":"2024-03-11T22:33:06.679543+0000","last_clean":"2024-03-11T22:33:06.679543+0000","last_became_active":"2024-03-11T22:32:25.065015+0000","last_became_peered":"2024-03-11T22:32:25.065015+0000","last_unstale":"2024-03-11T22:33:06.679543+0000","last_undegraded":"2024-03-11T22:33:06.679543+0000","last_fullsized":"2024-03-11T22:33:06.679543+0000","mapping_epoch":569,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:58.041330+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:32:58.041330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:07.994370+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"479'53","reported_seq":845,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410516+0000","last_change":"2024-03-11T22:32:55.856605+0000","last_active":"2024-03-11T22:33:07.410516+0000","last_peered":"2024-03-11T22:33:07.410516+0000","last_clean":"2024-03-11T22:33:07.410516+0000","last_became_active":"2024-03-11T22:32:01.979928+0000","last_became_peered":"2024-03-11T22:32:01.979928+0000","last_unstale":"2024-03-11T22:33:07.410516+0000","last_undegraded":"2024-03-11T22:33:07.410516+0000","last_fullsized":"2024-03-11T22:33:07.410516+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'53","last_scrub_stamp":"2024-03-11T22:32:55.856488+0000","last_deep_scrub":"353'39","last_deep_scrub_stamp":"2024-03-11T22:23:05.461729+0000","last_clean_scrub_stamp":"2024-03-11T22:32:55.856488+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:59.165433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1139,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409047+0000","last_change":"2024-03-11T22:32:16.675251+0000","last_active":"2024-03-11T22:33:07.409047+0000","last_peered":"2024-03-11T22:33:07.409047+0000","last_clean":"2024-03-11T22:33:07.409047+0000","last_became_active":"2024-03-11T22:29:04.718290+0000","last_became_peered":"2024-03-11T22:29:04.718290+0000","last_unstale":"2024-03-11T22:33:07.409047+0000","last_undegraded":"2024-03-11T22:33:07.409047+0000","last_fullsized":"2024-03-11T22:33:07.409047+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:16.675138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:16.675138+0000","last_clean_scrub_stamp":"2024-03-11T22:32:16.675138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:46.593622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":211,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680611+0000","last_change":"2024-03-11T22:32:13.533581+0000","last_active":"2024-03-11T22:33:06.680611+0000","last_peered":"2024-03-11T22:33:06.680611+0000","last_clean":"2024-03-11T22:33:06.680611+0000","last_became_active":"2024-03-11T22:29:04.705661+0000","last_became_peered":"2024-03-11T22:29:04.705661+0000","last_unstale":"2024-03-11T22:33:06.680611+0000","last_undegraded":"2024-03-11T22:33:06.680611+0000","last_fullsized":"2024-03-11T22:33:06.680611+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:13.533500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:13.533500+0000","last_clean_scrub_stamp":"2024-03-11T22:32:13.533500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:16.724861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1125,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679671+0000","last_change":"2024-03-11T22:33:01.969538+0000","last_active":"2024-03-11T22:33:06.679671+0000","last_peered":"2024-03-11T22:33:06.679671+0000","last_clean":"2024-03-11T22:33:06.679671+0000","last_became_active":"2024-03-11T22:08:39.074367+0000","last_became_peered":"2024-03-11T22:08:39.074367+0000","last_unstale":"2024-03-11T22:33:06.679671+0000","last_undegraded":"2024-03-11T22:33:06.679671+0000","last_fullsized":"2024-03-11T22:33:06.679671+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:01.969440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:33:01.969440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:28.434814+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"93'130","reported_seq":1426,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364131+0000","last_change":"2024-03-11T22:32:26.158635+0000","last_active":"2024-03-11T22:33:07.364131+0000","last_peered":"2024-03-11T22:33:07.364131+0000","last_clean":"2024-03-11T22:33:07.364131+0000","last_became_active":"2024-03-11T22:18:10.290055+0000","last_became_peered":"2024-03-11T22:18:10.290055+0000","last_unstale":"2024-03-11T22:33:07.364131+0000","last_undegraded":"2024-03-11T22:33:07.364131+0000","last_fullsized":"2024-03-11T22:33:07.364131+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:26.158500+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:30:57.479260+0000","last_clean_scrub_stamp":"2024-03-11T22:32:26.158500+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":70,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:55.322627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"516'32","reported_seq":884,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409545+0000","last_change":"2024-03-11T22:32:01.981837+0000","last_active":"2024-03-11T22:33:07.409545+0000","last_peered":"2024-03-11T22:33:07.409545+0000","last_clean":"2024-03-11T22:33:07.409545+0000","last_became_active":"2024-03-11T22:32:01.980333+0000","last_became_peered":"2024-03-11T22:32:01.980333+0000","last_unstale":"2024-03-11T22:33:07.409545+0000","last_undegraded":"2024-03-11T22:33:07.409545+0000","last_fullsized":"2024-03-11T22:33:07.409545+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"516'32","last_scrub_stamp":"2024-03-11T22:31:52.730391+0000","last_deep_scrub":"516'32","last_deep_scrub_stamp":"2024-03-11T22:30:10.404123+0000","last_clean_scrub_stamp":"2024-03-11T22:31:52.730391+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:08.489205+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":294,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408788+0000","last_change":"2024-03-11T22:32:27.854952+0000","last_active":"2024-03-11T22:33:07.408788+0000","last_peered":"2024-03-11T22:33:07.408788+0000","last_clean":"2024-03-11T22:33:07.408788+0000","last_became_active":"2024-03-11T22:32:01.978130+0000","last_became_peered":"2024-03-11T22:32:01.978130+0000","last_unstale":"2024-03-11T22:33:07.408788+0000","last_undegraded":"2024-03-11T22:33:07.408788+0000","last_fullsized":"2024-03-11T22:33:07.408788+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:27.854853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:27.854853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:31.282156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1095,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.365111+0000","last_change":"2024-03-11T22:32:17.258055+0000","last_active":"2024-03-11T22:33:07.365111+0000","last_peered":"2024-03-11T22:33:07.365111+0000","last_clean":"2024-03-11T22:33:07.365111+0000","last_became_active":"2024-03-11T22:08:37.081883+0000","last_became_peered":"2024-03-11T22:08:37.081883+0000","last_unstale":"2024-03-11T22:33:07.365111+0000","last_undegraded":"2024-03-11T22:33:07.365111+0000","last_fullsized":"2024-03-11T22:33:07.365111+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:17.257943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:17.257943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:19.039837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1082,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409057+0000","last_change":"2024-03-11T22:32:53.130648+0000","last_active":"2024-03-11T22:33:07.409057+0000","last_peered":"2024-03-11T22:33:07.409057+0000","last_clean":"2024-03-11T22:33:07.409057+0000","last_became_active":"2024-03-11T22:08:39.076948+0000","last_became_peered":"2024-03-11T22:08:39.076948+0000","last_unstale":"2024-03-11T22:33:07.409057+0000","last_undegraded":"2024-03-11T22:33:07.409057+0000","last_fullsized":"2024-03-11T22:33:07.409057+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:53.130551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:53.130551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:19.308993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"480'46","reported_seq":227,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.681135+0000","last_change":"2024-03-11T22:32:08.527853+0000","last_active":"2024-03-11T22:33:06.681135+0000","last_peered":"2024-03-11T22:33:06.681135+0000","last_clean":"2024-03-11T22:33:06.681135+0000","last_became_active":"2024-03-11T22:29:04.726632+0000","last_became_peered":"2024-03-11T22:29:04.726632+0000","last_unstale":"2024-03-11T22:33:06.681135+0000","last_undegraded":"2024-03-11T22:33:06.681135+0000","last_fullsized":"2024-03-11T22:33:06.681135+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'46","last_scrub_stamp":"2024-03-11T22:32:08.527758+0000","last_deep_scrub":"223'22","last_deep_scrub_stamp":"2024-03-11T22:17:25.782099+0000","last_clean_scrub_stamp":"2024-03-11T22:32:08.527758+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:30.733421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1119,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408602+0000","last_change":"2024-03-11T22:32:14.066574+0000","last_active":"2024-03-11T22:33:07.408602+0000","last_peered":"2024-03-11T22:33:07.408602+0000","last_clean":"2024-03-11T22:33:07.408602+0000","last_became_active":"2024-03-11T22:29:04.701072+0000","last_became_peered":"2024-03-11T22:29:04.701072+0000","last_unstale":"2024-03-11T22:33:07.408602+0000","last_undegraded":"2024-03-11T22:33:07.408602+0000","last_fullsized":"2024-03-11T22:33:07.408602+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:14.066497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:14.066497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:20.415563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"561'10","reported_seq":384,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409824+0000","last_change":"2024-03-11T22:33:00.980576+0000","last_active":"2024-03-11T22:33:07.409824+0000","last_peered":"2024-03-11T22:33:07.409824+0000","last_clean":"2024-03-11T22:33:07.409824+0000","last_became_active":"2024-03-11T22:32:01.975280+0000","last_became_peered":"2024-03-11T22:32:01.975280+0000","last_unstale":"2024-03-11T22:33:07.409824+0000","last_undegraded":"2024-03-11T22:33:07.409824+0000","last_fullsized":"2024-03-11T22:33:07.409824+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"561'10","last_scrub_stamp":"2024-03-11T22:33:00.980403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:33:00.980403+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:26.719403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1103,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364634+0000","last_change":"2024-03-11T22:32:41.431748+0000","last_active":"2024-03-11T22:33:07.364634+0000","last_peered":"2024-03-11T22:33:07.364634+0000","last_clean":"2024-03-11T22:33:07.364634+0000","last_became_active":"2024-03-11T22:29:04.787650+0000","last_became_peered":"2024-03-11T22:29:04.787650+0000","last_unstale":"2024-03-11T22:33:07.364634+0000","last_undegraded":"2024-03-11T22:33:07.364634+0000","last_fullsized":"2024-03-11T22:33:07.364634+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:41.431610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:23.033964+0000","last_clean_scrub_stamp":"2024-03-11T22:32:41.431610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:48.164708+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"93'130","reported_seq":1427,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.363933+0000","last_change":"2024-03-11T22:32:05.248884+0000","last_active":"2024-03-11T22:33:07.363933+0000","last_peered":"2024-03-11T22:33:07.363933+0000","last_clean":"2024-03-11T22:33:07.363933+0000","last_became_active":"2024-03-11T22:18:03.323848+0000","last_became_peered":"2024-03-11T22:18:03.323848+0000","last_unstale":"2024-03-11T22:33:07.363933+0000","last_undegraded":"2024-03-11T22:33:07.363933+0000","last_fullsized":"2024-03-11T22:33:07.363933+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:05.248781+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:32:05.248781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:09.348541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":353,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408670+0000","last_change":"2024-03-11T22:32:38.683950+0000","last_active":"2024-03-11T22:33:07.408670+0000","last_peered":"2024-03-11T22:33:07.408670+0000","last_clean":"2024-03-11T22:33:07.408670+0000","last_became_active":"2024-03-11T22:32:01.969330+0000","last_became_peered":"2024-03-11T22:32:01.969330+0000","last_unstale":"2024-03-11T22:33:07.408670+0000","last_undegraded":"2024-03-11T22:33:07.408670+0000","last_fullsized":"2024-03-11T22:33:07.408670+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:38.683844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:38.683844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:48.398432+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"479'42","reported_seq":1211,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409525+0000","last_change":"2024-03-11T22:32:41.170750+0000","last_active":"2024-03-11T22:33:07.409525+0000","last_peered":"2024-03-11T22:33:07.409525+0000","last_clean":"2024-03-11T22:33:07.409525+0000","last_became_active":"2024-03-11T22:20:29.424902+0000","last_became_peered":"2024-03-11T22:20:29.424902+0000","last_unstale":"2024-03-11T22:33:07.409525+0000","last_undegraded":"2024-03-11T22:33:07.409525+0000","last_fullsized":"2024-03-11T22:33:07.409525+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'42","last_scrub_stamp":"2024-03-11T22:32:41.170650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:41.170650+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:54.492505+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1107,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409325+0000","last_change":"2024-03-11T22:32:01.048534+0000","last_active":"2024-03-11T22:33:07.409325+0000","last_peered":"2024-03-11T22:33:07.409325+0000","last_clean":"2024-03-11T22:33:07.409325+0000","last_became_active":"2024-03-11T22:20:29.433687+0000","last_became_peered":"2024-03-11T22:20:29.433687+0000","last_unstale":"2024-03-11T22:33:07.409325+0000","last_undegraded":"2024-03-11T22:33:07.409325+0000","last_fullsized":"2024-03-11T22:33:07.409325+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:01.048412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:01.048412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:21.404669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":274,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409989+0000","last_change":"2024-03-11T22:32:46.771400+0000","last_active":"2024-03-11T22:33:07.409989+0000","last_peered":"2024-03-11T22:33:07.409989+0000","last_clean":"2024-03-11T22:33:07.409989+0000","last_became_active":"2024-03-11T22:32:01.979884+0000","last_became_peered":"2024-03-11T22:32:01.979884+0000","last_unstale":"2024-03-11T22:33:07.409989+0000","last_undegraded":"2024-03-11T22:33:07.409989+0000","last_fullsized":"2024-03-11T22:33:07.409989+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:46.771280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:46.771280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:49.352724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"93'130","reported_seq":567,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410041+0000","last_change":"2024-03-11T22:32:02.019354+0000","last_active":"2024-03-11T22:33:07.410041+0000","last_peered":"2024-03-11T22:33:07.410041+0000","last_clean":"2024-03-11T22:33:07.410041+0000","last_became_active":"2024-03-11T22:32:01.980016+0000","last_became_peered":"2024-03-11T22:32:01.980016+0000","last_unstale":"2024-03-11T22:33:07.410041+0000","last_undegraded":"2024-03-11T22:33:07.410041+0000","last_fullsized":"2024-03-11T22:33:07.410041+0000","mapping_epoch":560,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:02.019267+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:31:08.370869+0000","last_clean_scrub_stamp":"2024-03-11T22:32:02.019267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:28.470010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"488'10","reported_seq":1413,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408773+0000","last_change":"2024-03-11T22:32:24.135139+0000","last_active":"2024-03-11T22:33:07.408773+0000","last_peered":"2024-03-11T22:33:07.408773+0000","last_clean":"2024-03-11T22:33:07.408773+0000","last_became_active":"2024-03-11T22:29:04.721065+0000","last_became_peered":"2024-03-11T22:29:04.721065+0000","last_unstale":"2024-03-11T22:33:07.408773+0000","last_undegraded":"2024-03-11T22:33:07.408773+0000","last_fullsized":"2024-03-11T22:33:07.408773+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'10","last_scrub_stamp":"2024-03-11T22:32:24.135002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:24.135002+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:42.573609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"480'42","reported_seq":783,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408964+0000","last_change":"2024-03-11T22:32:16.969323+0000","last_active":"2024-03-11T22:33:07.408964+0000","last_peered":"2024-03-11T22:33:07.408964+0000","last_clean":"2024-03-11T22:33:07.408964+0000","last_became_active":"2024-03-11T22:32:01.972997+0000","last_became_peered":"2024-03-11T22:32:01.972997+0000","last_unstale":"2024-03-11T22:33:07.408964+0000","last_undegraded":"2024-03-11T22:33:07.408964+0000","last_fullsized":"2024-03-11T22:33:07.408964+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'42","last_scrub_stamp":"2024-03-11T22:32:16.969171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:16.969171+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:25.997062+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":1125,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409072+0000","last_change":"2024-03-11T22:32:03.364809+0000","last_active":"2024-03-11T22:33:07.409072+0000","last_peered":"2024-03-11T22:33:07.409072+0000","last_clean":"2024-03-11T22:33:07.409072+0000","last_became_active":"2024-03-11T22:29:04.718160+0000","last_became_peered":"2024-03-11T22:29:04.718160+0000","last_unstale":"2024-03-11T22:33:07.409072+0000","last_undegraded":"2024-03-11T22:33:07.409072+0000","last_fullsized":"2024-03-11T22:33:07.409072+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:32:03.364707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:03.364707+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:13.141927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":100,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680306+0000","last_change":"2024-03-11T22:32:33.072989+0000","last_active":"2024-03-11T22:33:06.680306+0000","last_peered":"2024-03-11T22:33:06.680306+0000","last_clean":"2024-03-11T22:33:06.680306+0000","last_became_active":"2024-03-11T22:32:25.067050+0000","last_became_peered":"2024-03-11T22:32:25.067050+0000","last_unstale":"2024-03-11T22:33:06.680306+0000","last_undegraded":"2024-03-11T22:33:06.680306+0000","last_fullsized":"2024-03-11T22:33:06.680306+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:33.072870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:03.279112+0000","last_clean_scrub_stamp":"2024-03-11T22:32:33.072870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:48.865391+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"93'130","reported_seq":1462,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408880+0000","last_change":"2024-03-11T22:32:57.118065+0000","last_active":"2024-03-11T22:33:07.408880+0000","last_peered":"2024-03-11T22:33:07.408880+0000","last_clean":"2024-03-11T22:33:07.408880+0000","last_became_active":"2024-03-11T22:26:53.557811+0000","last_became_peered":"2024-03-11T22:26:53.557811+0000","last_unstale":"2024-03-11T22:33:07.408880+0000","last_undegraded":"2024-03-11T22:33:07.408880+0000","last_fullsized":"2024-03-11T22:33:07.408880+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:57.117970+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:32:57.117970+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":60,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:07.382758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"378'9","reported_seq":1401,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408743+0000","last_change":"2024-03-11T22:32:05.398376+0000","last_active":"2024-03-11T22:33:07.408743+0000","last_peered":"2024-03-11T22:33:07.408743+0000","last_clean":"2024-03-11T22:33:07.408743+0000","last_became_active":"2024-03-11T22:08:37.073822+0000","last_became_peered":"2024-03-11T22:08:37.073822+0000","last_unstale":"2024-03-11T22:33:07.408743+0000","last_undegraded":"2024-03-11T22:33:07.408743+0000","last_fullsized":"2024-03-11T22:33:07.408743+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"378'9","last_scrub_stamp":"2024-03-11T22:32:05.398270+0000","last_deep_scrub":"378'9","last_deep_scrub_stamp":"2024-03-11T22:24:54.792049+0000","last_clean_scrub_stamp":"2024-03-11T22:32:05.398270+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:33.174332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"479'30","reported_seq":231,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680387+0000","last_change":"2024-03-11T22:32:53.523093+0000","last_active":"2024-03-11T22:33:06.680387+0000","last_peered":"2024-03-11T22:33:06.680387+0000","last_clean":"2024-03-11T22:33:06.680387+0000","last_became_active":"2024-03-11T22:29:04.727100+0000","last_became_peered":"2024-03-11T22:29:04.727100+0000","last_unstale":"2024-03-11T22:33:06.680387+0000","last_undegraded":"2024-03-11T22:33:06.680387+0000","last_fullsized":"2024-03-11T22:33:06.680387+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-03-11T22:32:53.522986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:53.522986+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:03.105583+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1104,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364926+0000","last_change":"2024-03-11T22:31:39.253559+0000","last_active":"2024-03-11T22:33:07.364926+0000","last_peered":"2024-03-11T22:33:07.364926+0000","last_clean":"2024-03-11T22:33:07.364926+0000","last_became_active":"2024-03-11T22:08:33.299654+0000","last_became_peered":"2024-03-11T22:08:33.299654+0000","last_unstale":"2024-03-11T22:33:07.364926+0000","last_undegraded":"2024-03-11T22:33:07.364926+0000","last_fullsized":"2024-03-11T22:33:07.364926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:39.253450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:39.253450+0000","last_clean_scrub_stamp":"2024-03-11T22:31:39.253450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1093,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408041+0000","last_change":"2024-03-11T22:32:22.164266+0000","last_active":"2024-03-11T22:33:07.408041+0000","last_peered":"2024-03-11T22:33:07.408041+0000","last_clean":"2024-03-11T22:33:07.408041+0000","last_became_active":"2024-03-11T22:08:39.082813+0000","last_became_peered":"2024-03-11T22:08:39.082813+0000","last_unstale":"2024-03-11T22:33:07.408041+0000","last_undegraded":"2024-03-11T22:33:07.408041+0000","last_fullsized":"2024-03-11T22:33:07.408041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:22.164173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:22.164173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:38.086539+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"93'130","reported_seq":1377,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409074+0000","last_change":"2024-03-11T22:32:02.731217+0000","last_active":"2024-03-11T22:33:07.409074+0000","last_peered":"2024-03-11T22:33:07.409074+0000","last_clean":"2024-03-11T22:33:07.409074+0000","last_became_active":"2024-03-11T22:26:41.661375+0000","last_became_peered":"2024-03-11T22:26:41.661375+0000","last_unstale":"2024-03-11T22:33:07.409074+0000","last_undegraded":"2024-03-11T22:33:07.409074+0000","last_fullsized":"2024-03-11T22:33:07.409074+0000","mapping_epoch":312,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":423,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:02.731093+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:32:02.731093+0000","last_clean_scrub_stamp":"2024-03-11T22:32:02.731093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:31.044844+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1119,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408917+0000","last_change":"2024-03-11T22:32:24.561932+0000","last_active":"2024-03-11T22:33:07.408917+0000","last_peered":"2024-03-11T22:33:07.408917+0000","last_clean":"2024-03-11T22:33:07.408917+0000","last_became_active":"2024-03-11T22:08:37.074033+0000","last_became_peered":"2024-03-11T22:08:37.074033+0000","last_unstale":"2024-03-11T22:33:07.408917+0000","last_undegraded":"2024-03-11T22:33:07.408917+0000","last_fullsized":"2024-03-11T22:33:07.408917+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:24.561799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:40.863445+0000","last_clean_scrub_stamp":"2024-03-11T22:32:24.561799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:47.605538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"479'47","reported_seq":1174,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.407977+0000","last_change":"2024-03-11T22:32:38.133530+0000","last_active":"2024-03-11T22:33:07.407977+0000","last_peered":"2024-03-11T22:33:07.407977+0000","last_clean":"2024-03-11T22:33:07.407977+0000","last_became_active":"2024-03-11T22:08:35.086965+0000","last_became_peered":"2024-03-11T22:08:35.086965+0000","last_unstale":"2024-03-11T22:33:07.407977+0000","last_undegraded":"2024-03-11T22:33:07.407977+0000","last_fullsized":"2024-03-11T22:33:07.407977+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'47","last_scrub_stamp":"2024-03-11T22:32:38.133415+0000","last_deep_scrub":"429'37","last_deep_scrub_stamp":"2024-03-11T22:26:47.889156+0000","last_clean_scrub_stamp":"2024-03-11T22:32:38.133415+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:56.303809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":211,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680763+0000","last_change":"2024-03-11T22:32:06.558954+0000","last_active":"2024-03-11T22:33:06.680763+0000","last_peered":"2024-03-11T22:33:06.680763+0000","last_clean":"2024-03-11T22:33:06.680763+0000","last_became_active":"2024-03-11T22:29:04.722813+0000","last_became_peered":"2024-03-11T22:29:04.722813+0000","last_unstale":"2024-03-11T22:33:06.680763+0000","last_undegraded":"2024-03-11T22:33:06.680763+0000","last_fullsized":"2024-03-11T22:33:06.680763+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:06.558793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:04.706082+0000","last_clean_scrub_stamp":"2024-03-11T22:32:06.558793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:34.974395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":211,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679794+0000","last_change":"2024-03-11T22:32:09.603736+0000","last_active":"2024-03-11T22:33:06.679794+0000","last_peered":"2024-03-11T22:33:06.679794+0000","last_clean":"2024-03-11T22:33:06.679794+0000","last_became_active":"2024-03-11T22:29:04.729455+0000","last_became_peered":"2024-03-11T22:29:04.729455+0000","last_unstale":"2024-03-11T22:33:06.679794+0000","last_undegraded":"2024-03-11T22:33:06.679794+0000","last_fullsized":"2024-03-11T22:33:06.679794+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:09.603590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:09.603590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:23.940374+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"93'130","reported_seq":1352,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409480+0000","last_change":"2024-03-11T22:32:20.617366+0000","last_active":"2024-03-11T22:33:07.409480+0000","last_peered":"2024-03-11T22:33:07.409480+0000","last_clean":"2024-03-11T22:33:07.409480+0000","last_became_active":"2024-03-11T22:26:45.688609+0000","last_became_peered":"2024-03-11T22:26:45.688609+0000","last_unstale":"2024-03-11T22:33:07.409480+0000","last_undegraded":"2024-03-11T22:33:07.409480+0000","last_fullsized":"2024-03-11T22:33:07.409480+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:20.617283+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:32:20.617283+0000","last_clean_scrub_stamp":"2024-03-11T22:32:20.617283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:33.162195+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1108,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.365166+0000","last_change":"2024-03-11T22:31:53.264975+0000","last_active":"2024-03-11T22:33:07.365166+0000","last_peered":"2024-03-11T22:33:07.365166+0000","last_clean":"2024-03-11T22:33:07.365166+0000","last_became_active":"2024-03-11T22:29:04.718259+0000","last_became_peered":"2024-03-11T22:29:04.718259+0000","last_unstale":"2024-03-11T22:33:07.365166+0000","last_undegraded":"2024-03-11T22:33:07.365166+0000","last_fullsized":"2024-03-11T22:33:07.365166+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:53.264847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:53.264847+0000","last_clean_scrub_stamp":"2024-03-11T22:31:53.264847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:12.439345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"479'36","reported_seq":1202,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409068+0000","last_change":"2024-03-11T22:32:50.037636+0000","last_active":"2024-03-11T22:33:07.409068+0000","last_peered":"2024-03-11T22:33:07.409068+0000","last_clean":"2024-03-11T22:33:07.409068+0000","last_became_active":"2024-03-11T22:29:04.720793+0000","last_became_peered":"2024-03-11T22:29:04.720793+0000","last_unstale":"2024-03-11T22:33:07.409068+0000","last_undegraded":"2024-03-11T22:33:07.409068+0000","last_fullsized":"2024-03-11T22:33:07.409068+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'36","last_scrub_stamp":"2024-03-11T22:32:50.037523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:50.037523+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:59.885167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"95'6","reported_seq":277,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:09.132724+0000","last_change":"2024-03-11T22:32:25.071721+0000","last_active":"2024-03-11T22:33:09.132724+0000","last_peered":"2024-03-11T22:33:09.132724+0000","last_clean":"2024-03-11T22:33:09.132724+0000","last_became_active":"2024-03-11T22:32:25.070571+0000","last_became_peered":"2024-03-11T22:32:25.070571+0000","last_unstale":"2024-03-11T22:33:09.132724+0000","last_undegraded":"2024-03-11T22:33:09.132724+0000","last_fullsized":"2024-03-11T22:33:09.132724+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-03-11T22:32:11.608791+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-03-11T22:10:14.408533+0000","last_clean_scrub_stamp":"2024-03-11T22:32:11.608791+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:21.449716+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00040282099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":158,"num_read_kb":153,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1104,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408315+0000","last_change":"2024-03-11T22:31:50.414259+0000","last_active":"2024-03-11T22:33:07.408315+0000","last_peered":"2024-03-11T22:33:07.408315+0000","last_clean":"2024-03-11T22:33:07.408315+0000","last_became_active":"2024-03-11T22:08:33.035483+0000","last_became_peered":"2024-03-11T22:08:33.035483+0000","last_unstale":"2024-03-11T22:33:07.408315+0000","last_undegraded":"2024-03-11T22:33:07.408315+0000","last_fullsized":"2024-03-11T22:33:07.408315+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:50.414158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:50.414158+0000","last_clean_scrub_stamp":"2024-03-11T22:31:50.414158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:15.542430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":219,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680668+0000","last_change":"2024-03-11T22:33:00.528512+0000","last_active":"2024-03-11T22:33:06.680668+0000","last_peered":"2024-03-11T22:33:06.680668+0000","last_clean":"2024-03-11T22:33:06.680668+0000","last_became_active":"2024-03-11T22:29:04.696141+0000","last_became_peered":"2024-03-11T22:29:04.696141+0000","last_unstale":"2024-03-11T22:33:06.680668+0000","last_undegraded":"2024-03-11T22:33:06.680668+0000","last_fullsized":"2024-03-11T22:33:06.680668+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:00.528432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:33:00.528432+0000","last_clean_scrub_stamp":"2024-03-11T22:33:00.528432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:02.356742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"93'130","reported_seq":669,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680596+0000","last_change":"2024-03-11T22:32:27.554691+0000","last_active":"2024-03-11T22:33:06.680596+0000","last_peered":"2024-03-11T22:33:06.680596+0000","last_clean":"2024-03-11T22:33:06.680596+0000","last_became_active":"2024-03-11T22:29:37.643184+0000","last_became_peered":"2024-03-11T22:29:37.643184+0000","last_unstale":"2024-03-11T22:33:06.680596+0000","last_undegraded":"2024-03-11T22:33:06.680596+0000","last_fullsized":"2024-03-11T22:33:06.680596+0000","mapping_epoch":502,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":3,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:27.554598+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:28:24.623016+0000","last_clean_scrub_stamp":"2024-03-11T22:32:27.554598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:34.300636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":517,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680481+0000","last_change":"2024-03-11T22:32:25.073943+0000","last_active":"2024-03-11T22:33:06.680481+0000","last_peered":"2024-03-11T22:33:06.680481+0000","last_clean":"2024-03-11T22:33:06.680481+0000","last_became_active":"2024-03-11T22:32:25.072113+0000","last_became_peered":"2024-03-11T22:32:25.072113+0000","last_unstale":"2024-03-11T22:33:06.680481+0000","last_undegraded":"2024-03-11T22:33:06.680481+0000","last_fullsized":"2024-03-11T22:33:06.680481+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:07.436607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:10.320575+0000","last_clean_scrub_stamp":"2024-03-11T22:32:07.436607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:15.312033+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"514'48","reported_seq":1485,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409615+0000","last_change":"2024-03-11T22:32:35.488420+0000","last_active":"2024-03-11T22:33:07.409615+0000","last_peered":"2024-03-11T22:33:07.409615+0000","last_clean":"2024-03-11T22:33:07.409615+0000","last_became_active":"2024-03-11T22:08:35.085880+0000","last_became_peered":"2024-03-11T22:08:35.085880+0000","last_unstale":"2024-03-11T22:33:07.409615+0000","last_undegraded":"2024-03-11T22:33:07.409615+0000","last_fullsized":"2024-03-11T22:33:07.409615+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"514'48","last_scrub_stamp":"2024-03-11T22:32:35.488293+0000","last_deep_scrub":"514'48","last_deep_scrub_stamp":"2024-03-11T22:32:35.488293+0000","last_clean_scrub_stamp":"2024-03-11T22:32:35.488293+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:43.188761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1095,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409574+0000","last_change":"2024-03-11T22:32:16.099056+0000","last_active":"2024-03-11T22:33:07.409574+0000","last_peered":"2024-03-11T22:33:07.409574+0000","last_clean":"2024-03-11T22:33:07.409574+0000","last_became_active":"2024-03-11T22:08:33.033738+0000","last_became_peered":"2024-03-11T22:08:33.033738+0000","last_unstale":"2024-03-11T22:33:07.409574+0000","last_undegraded":"2024-03-11T22:33:07.409574+0000","last_fullsized":"2024-03-11T22:33:07.409574+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:16.098966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:34.038930+0000","last_clean_scrub_stamp":"2024-03-11T22:32:16.098966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:38.304085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"93'130","reported_seq":872,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679575+0000","last_change":"2024-03-11T22:32:27.076843+0000","last_active":"2024-03-11T22:33:06.679575+0000","last_peered":"2024-03-11T22:33:06.679575+0000","last_clean":"2024-03-11T22:33:06.679575+0000","last_became_active":"2024-03-11T22:32:25.067408+0000","last_became_peered":"2024-03-11T22:32:25.067408+0000","last_unstale":"2024-03-11T22:33:06.679575+0000","last_undegraded":"2024-03-11T22:33:06.679575+0000","last_fullsized":"2024-03-11T22:33:06.679575+0000","mapping_epoch":569,"log_start":"93'130","ondisk_log_start":"93'130","created":19,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:27.076730+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:30:06.478921+0000","last_clean_scrub_stamp":"2024-03-11T22:32:27.076730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:55.973112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1092,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364893+0000","last_change":"2024-03-11T22:32:18.431130+0000","last_active":"2024-03-11T22:33:07.364893+0000","last_peered":"2024-03-11T22:33:07.364893+0000","last_clean":"2024-03-11T22:33:07.364893+0000","last_became_active":"2024-03-11T22:08:39.072890+0000","last_became_peered":"2024-03-11T22:08:39.072890+0000","last_unstale":"2024-03-11T22:33:07.364893+0000","last_undegraded":"2024-03-11T22:33:07.364893+0000","last_fullsized":"2024-03-11T22:33:07.364893+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:18.430994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:18.430994+0000","last_clean_scrub_stamp":"2024-03-11T22:32:18.430994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:36.072787+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1130,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679418+0000","last_change":"2024-03-11T22:32:41.459780+0000","last_active":"2024-03-11T22:33:06.679418+0000","last_peered":"2024-03-11T22:33:06.679418+0000","last_clean":"2024-03-11T22:33:06.679418+0000","last_became_active":"2024-03-11T22:08:37.076226+0000","last_became_peered":"2024-03-11T22:08:37.076226+0000","last_unstale":"2024-03-11T22:33:06.679418+0000","last_undegraded":"2024-03-11T22:33:06.679418+0000","last_fullsized":"2024-03-11T22:33:06.679418+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:41.459705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:41.459705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:03.995612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"538'26","reported_seq":236,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.681088+0000","last_change":"2024-03-11T22:32:38.583950+0000","last_active":"2024-03-11T22:33:06.681088+0000","last_peered":"2024-03-11T22:33:06.681088+0000","last_clean":"2024-03-11T22:33:06.681088+0000","last_became_active":"2024-03-11T22:29:04.705980+0000","last_became_peered":"2024-03-11T22:29:04.705980+0000","last_unstale":"2024-03-11T22:33:06.681088+0000","last_undegraded":"2024-03-11T22:33:06.681088+0000","last_fullsized":"2024-03-11T22:33:06.681088+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"538'26","last_scrub_stamp":"2024-03-11T22:32:38.583829+0000","last_deep_scrub":"538'26","last_deep_scrub_stamp":"2024-03-11T22:31:09.564609+0000","last_clean_scrub_stamp":"2024-03-11T22:32:38.583829+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:40.430137+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1155,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.678839+0000","last_change":"2024-03-11T22:33:00.767985+0000","last_active":"2024-03-11T22:33:06.678839+0000","last_peered":"2024-03-11T22:33:06.678839+0000","last_clean":"2024-03-11T22:33:06.678839+0000","last_became_active":"2024-03-11T22:08:32.030666+0000","last_became_peered":"2024-03-11T22:08:32.030666+0000","last_unstale":"2024-03-11T22:33:06.678839+0000","last_undegraded":"2024-03-11T22:33:06.678839+0000","last_fullsized":"2024-03-11T22:33:06.678839+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:00.767847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.733239+0000","last_clean_scrub_stamp":"2024-03-11T22:33:00.767847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:11.597370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000872472,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":220,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679440+0000","last_change":"2024-03-11T22:32:33.662740+0000","last_active":"2024-03-11T22:33:06.679440+0000","last_peered":"2024-03-11T22:33:06.679440+0000","last_clean":"2024-03-11T22:33:06.679440+0000","last_became_active":"2024-03-11T22:29:04.713225+0000","last_became_peered":"2024-03-11T22:29:04.713225+0000","last_unstale":"2024-03-11T22:33:06.679440+0000","last_undegraded":"2024-03-11T22:33:06.679440+0000","last_fullsized":"2024-03-11T22:33:06.679440+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:33.662663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:19:46.586446+0000","last_clean_scrub_stamp":"2024-03-11T22:32:33.662663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:47.377747+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":92,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680681+0000","last_change":"2024-03-11T22:32:25.069671+0000","last_active":"2024-03-11T22:33:06.680681+0000","last_peered":"2024-03-11T22:33:06.680681+0000","last_clean":"2024-03-11T22:33:06.680681+0000","last_became_active":"2024-03-11T22:32:25.066872+0000","last_became_peered":"2024-03-11T22:32:25.066872+0000","last_unstale":"2024-03-11T22:33:06.680681+0000","last_undegraded":"2024-03-11T22:33:06.680681+0000","last_fullsized":"2024-03-11T22:33:06.680681+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:39.812558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:34.131530+0000","last_clean_scrub_stamp":"2024-03-11T22:31:39.812558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"93'130","reported_seq":880,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679701+0000","last_change":"2024-03-11T22:32:26.052079+0000","last_active":"2024-03-11T22:33:06.679701+0000","last_peered":"2024-03-11T22:33:06.679701+0000","last_clean":"2024-03-11T22:33:06.679701+0000","last_became_active":"2024-03-11T22:32:25.063983+0000","last_became_peered":"2024-03-11T22:32:25.063983+0000","last_unstale":"2024-03-11T22:33:06.679701+0000","last_undegraded":"2024-03-11T22:33:06.679701+0000","last_fullsized":"2024-03-11T22:33:06.679701+0000","mapping_epoch":569,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:26.051959+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:32:26.051959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:44.466239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1116,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408649+0000","last_change":"2024-03-11T22:32:55.954802+0000","last_active":"2024-03-11T22:33:07.408649+0000","last_peered":"2024-03-11T22:33:07.408649+0000","last_clean":"2024-03-11T22:33:07.408649+0000","last_became_active":"2024-03-11T22:29:04.707688+0000","last_became_peered":"2024-03-11T22:29:04.707688+0000","last_unstale":"2024-03-11T22:33:07.408649+0000","last_undegraded":"2024-03-11T22:33:07.408649+0000","last_fullsized":"2024-03-11T22:33:07.408649+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:55.954694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:05.507462+0000","last_clean_scrub_stamp":"2024-03-11T22:32:55.954694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:23.210941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"520'55","reported_seq":194,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364188+0000","last_change":"2024-03-11T22:31:48.173854+0000","last_active":"2024-03-11T22:33:07.364188+0000","last_peered":"2024-03-11T22:33:07.364188+0000","last_clean":"2024-03-11T22:33:07.364188+0000","last_became_active":"2024-03-11T22:30:16.930211+0000","last_became_peered":"2024-03-11T22:30:16.930211+0000","last_unstale":"2024-03-11T22:33:07.364188+0000","last_undegraded":"2024-03-11T22:33:07.364188+0000","last_fullsized":"2024-03-11T22:33:07.364188+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'55","last_scrub_stamp":"2024-03-11T22:31:48.173730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:31:48.173730+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":295,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409352+0000","last_change":"2024-03-11T22:32:40.645171+0000","last_active":"2024-03-11T22:33:07.409352+0000","last_peered":"2024-03-11T22:33:07.409352+0000","last_clean":"2024-03-11T22:33:07.409352+0000","last_became_active":"2024-03-11T22:32:01.973398+0000","last_became_peered":"2024-03-11T22:32:01.973398+0000","last_unstale":"2024-03-11T22:33:07.409352+0000","last_undegraded":"2024-03-11T22:33:07.409352+0000","last_fullsized":"2024-03-11T22:33:07.409352+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:40.645073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:40.645073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:51.820533+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"93'3","reported_seq":1144,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364297+0000","last_change":"2024-03-11T22:32:33.326977+0000","last_active":"2024-03-11T22:33:07.364297+0000","last_peered":"2024-03-11T22:33:07.364297+0000","last_clean":"2024-03-11T22:33:07.364297+0000","last_became_active":"2024-03-11T22:29:04.718809+0000","last_became_peered":"2024-03-11T22:29:04.718809+0000","last_unstale":"2024-03-11T22:33:07.364297+0000","last_undegraded":"2024-03-11T22:33:07.364297+0000","last_fullsized":"2024-03-11T22:33:07.364297+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'3","last_scrub_stamp":"2024-03-11T22:32:33.326892+0000","last_deep_scrub":"93'3","last_deep_scrub_stamp":"2024-03-11T22:10:00.177978+0000","last_clean_scrub_stamp":"2024-03-11T22:32:33.326892+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:48.990981+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055656499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1095,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409525+0000","last_change":"2024-03-11T22:32:48.191134+0000","last_active":"2024-03-11T22:33:07.409525+0000","last_peered":"2024-03-11T22:33:07.409525+0000","last_clean":"2024-03-11T22:33:07.409525+0000","last_became_active":"2024-03-11T22:20:29.434547+0000","last_became_peered":"2024-03-11T22:20:29.434547+0000","last_unstale":"2024-03-11T22:33:07.409525+0000","last_undegraded":"2024-03-11T22:33:07.409525+0000","last_fullsized":"2024-03-11T22:33:07.409525+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:48.191008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:12:45.439584+0000","last_clean_scrub_stamp":"2024-03-11T22:32:48.191008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:04.826416+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"93'130","reported_seq":1442,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.363966+0000","last_change":"2024-03-11T22:32:47.311220+0000","last_active":"2024-03-11T22:33:07.363966+0000","last_peered":"2024-03-11T22:33:07.363966+0000","last_clean":"2024-03-11T22:33:07.363966+0000","last_became_active":"2024-03-11T22:26:33.607145+0000","last_became_peered":"2024-03-11T22:26:33.607145+0000","last_unstale":"2024-03-11T22:33:07.363966+0000","last_undegraded":"2024-03-11T22:33:07.363966+0000","last_fullsized":"2024-03-11T22:33:07.363966+0000","mapping_epoch":149,"log_start":"93'130","ondisk_log_start":"93'130","created":144,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":4,"last_scrub":"93'130","last_scrub_stamp":"2024-03-11T22:32:47.311113+0000","last_deep_scrub":"93'130","last_deep_scrub_stamp":"2024-03-11T22:09:59.243159+0000","last_clean_scrub_stamp":"2024-03-11T22:32:47.311113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:53.558931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"569'25","reported_seq":1185,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408660+0000","last_change":"2024-03-11T22:32:24.073606+0000","last_active":"2024-03-11T22:33:07.408660+0000","last_peered":"2024-03-11T22:33:07.408660+0000","last_clean":"2024-03-11T22:33:07.408660+0000","last_became_active":"2024-03-11T22:29:04.707867+0000","last_became_peered":"2024-03-11T22:29:04.707867+0000","last_unstale":"2024-03-11T22:33:07.408660+0000","last_undegraded":"2024-03-11T22:33:07.408660+0000","last_fullsized":"2024-03-11T22:33:07.408660+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"569'25","last_scrub_stamp":"2024-03-11T22:32:24.073486+0000","last_deep_scrub":"569'25","last_deep_scrub_stamp":"2024-03-11T22:32:24.073486+0000","last_clean_scrub_stamp":"2024-03-11T22:32:24.073486+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:49.739916+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1103,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364544+0000","last_change":"2024-03-11T22:32:25.206526+0000","last_active":"2024-03-11T22:33:07.364544+0000","last_peered":"2024-03-11T22:33:07.364544+0000","last_clean":"2024-03-11T22:33:07.364544+0000","last_became_active":"2024-03-11T22:08:33.038465+0000","last_became_peered":"2024-03-11T22:08:33.038465+0000","last_unstale":"2024-03-11T22:33:07.364544+0000","last_undegraded":"2024-03-11T22:33:07.364544+0000","last_fullsized":"2024-03-11T22:33:07.364544+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:25.206432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:25.206432+0000","last_clean_scrub_stamp":"2024-03-11T22:32:25.206432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:26.184339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1130,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.678954+0000","last_change":"2024-03-11T22:32:21.381860+0000","last_active":"2024-03-11T22:33:06.678954+0000","last_peered":"2024-03-11T22:33:06.678954+0000","last_clean":"2024-03-11T22:33:06.678954+0000","last_became_active":"2024-03-11T22:08:37.073337+0000","last_became_peered":"2024-03-11T22:08:37.073337+0000","last_unstale":"2024-03-11T22:33:06.678954+0000","last_undegraded":"2024-03-11T22:33:06.678954+0000","last_fullsized":"2024-03-11T22:33:06.678954+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:21.381769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:36.289016+0000","last_clean_scrub_stamp":"2024-03-11T22:32:21.381769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:25.903405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1112,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408422+0000","last_change":"2024-03-11T22:33:06.055241+0000","last_active":"2024-03-11T22:33:07.408422+0000","last_peered":"2024-03-11T22:33:07.408422+0000","last_clean":"2024-03-11T22:33:07.408422+0000","last_became_active":"2024-03-11T22:20:29.417644+0000","last_became_peered":"2024-03-11T22:20:29.417644+0000","last_unstale":"2024-03-11T22:33:07.408422+0000","last_undegraded":"2024-03-11T22:33:07.408422+0000","last_fullsized":"2024-03-11T22:33:07.408422+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:06.055130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:20.400349+0000","last_clean_scrub_stamp":"2024-03-11T22:33:06.055130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:26.656002+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"480'51","reported_seq":1306,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408900+0000","last_change":"2024-03-11T22:32:57.544585+0000","last_active":"2024-03-11T22:33:07.408900+0000","last_peered":"2024-03-11T22:33:07.408900+0000","last_clean":"2024-03-11T22:33:07.408900+0000","last_became_active":"2024-03-11T22:29:04.718797+0000","last_became_peered":"2024-03-11T22:29:04.718797+0000","last_unstale":"2024-03-11T22:33:07.408900+0000","last_undegraded":"2024-03-11T22:33:07.408900+0000","last_fullsized":"2024-03-11T22:33:07.408900+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'51","last_scrub_stamp":"2024-03-11T22:32:57.544466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:57.544466+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:14.331227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":510,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679985+0000","last_change":"2024-03-11T22:32:28.091584+0000","last_active":"2024-03-11T22:33:06.679985+0000","last_peered":"2024-03-11T22:33:06.679985+0000","last_clean":"2024-03-11T22:33:06.679985+0000","last_became_active":"2024-03-11T22:32:25.072199+0000","last_became_peered":"2024-03-11T22:32:25.072199+0000","last_unstale":"2024-03-11T22:33:06.679985+0000","last_undegraded":"2024-03-11T22:33:06.679985+0000","last_fullsized":"2024-03-11T22:33:06.679985+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:28.091484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:28.091484+0000","last_clean_scrub_stamp":"2024-03-11T22:32:28.091484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:42.050317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1129,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:10.886603+0000","last_change":"2024-03-11T22:33:10.886644+0000","last_active":"2024-03-11T22:33:10.886603+0000","last_peered":"2024-03-11T22:33:10.886603+0000","last_clean":"2024-03-11T22:33:10.886603+0000","last_became_active":"2024-03-11T22:08:37.074097+0000","last_became_peered":"2024-03-11T22:08:37.074097+0000","last_unstale":"2024-03-11T22:33:10.886603+0000","last_undegraded":"2024-03-11T22:33:10.886603+0000","last_fullsized":"2024-03-11T22:33:10.886603+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:10.886521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:41.668508+0000","last_clean_scrub_stamp":"2024-03-11T22:33:10.886521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:25.100999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":525,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679465+0000","last_change":"2024-03-11T22:32:52.940775+0000","last_active":"2024-03-11T22:33:06.679465+0000","last_peered":"2024-03-11T22:33:06.679465+0000","last_clean":"2024-03-11T22:33:06.679465+0000","last_became_active":"2024-03-11T22:32:25.063927+0000","last_became_peered":"2024-03-11T22:32:25.063927+0000","last_unstale":"2024-03-11T22:33:06.679465+0000","last_undegraded":"2024-03-11T22:33:06.679465+0000","last_fullsized":"2024-03-11T22:33:06.679465+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:52.940688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:47.568500+0000","last_clean_scrub_stamp":"2024-03-11T22:32:52.940688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:19.412757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1142,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.678634+0000","last_change":"2024-03-11T22:32:28.355320+0000","last_active":"2024-03-11T22:33:06.678634+0000","last_peered":"2024-03-11T22:33:06.678634+0000","last_clean":"2024-03-11T22:33:06.678634+0000","last_became_active":"2024-03-11T22:29:04.718513+0000","last_became_peered":"2024-03-11T22:29:04.718513+0000","last_unstale":"2024-03-11T22:33:06.678634+0000","last_undegraded":"2024-03-11T22:33:06.678634+0000","last_fullsized":"2024-03-11T22:33:06.678634+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:28.355239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:14.325174+0000","last_clean_scrub_stamp":"2024-03-11T22:32:28.355239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:50.655394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"478'24","reported_seq":420,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409219+0000","last_change":"2024-03-11T22:32:28.812733+0000","last_active":"2024-03-11T22:33:07.409219+0000","last_peered":"2024-03-11T22:33:07.409219+0000","last_clean":"2024-03-11T22:33:07.409219+0000","last_became_active":"2024-03-11T22:32:01.975548+0000","last_became_peered":"2024-03-11T22:32:01.975548+0000","last_unstale":"2024-03-11T22:33:07.409219+0000","last_undegraded":"2024-03-11T22:33:07.409219+0000","last_fullsized":"2024-03-11T22:33:07.409219+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'24","last_scrub_stamp":"2024-03-11T22:32:28.812617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:28.812617+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:46.520206+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":343,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680063+0000","last_change":"2024-03-11T22:32:25.070248+0000","last_active":"2024-03-11T22:33:06.680063+0000","last_peered":"2024-03-11T22:33:06.680063+0000","last_clean":"2024-03-11T22:33:06.680063+0000","last_became_active":"2024-03-11T22:32:25.069890+0000","last_became_peered":"2024-03-11T22:32:25.069890+0000","last_unstale":"2024-03-11T22:33:06.680063+0000","last_undegraded":"2024-03-11T22:33:06.680063+0000","last_fullsized":"2024-03-11T22:33:06.680063+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:44.529411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:31:44.529411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:06.856542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"96'2","reported_seq":1125,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409949+0000","last_change":"2024-03-11T22:32:32.188940+0000","last_active":"2024-03-11T22:33:07.409949+0000","last_peered":"2024-03-11T22:33:07.409949+0000","last_clean":"2024-03-11T22:33:07.409949+0000","last_became_active":"2024-03-11T22:29:04.698531+0000","last_became_peered":"2024-03-11T22:29:04.698531+0000","last_unstale":"2024-03-11T22:33:07.409949+0000","last_undegraded":"2024-03-11T22:33:07.409949+0000","last_fullsized":"2024-03-11T22:33:07.409949+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'2","last_scrub_stamp":"2024-03-11T22:32:32.188824+0000","last_deep_scrub":"96'2","last_deep_scrub_stamp":"2024-03-11T22:10:31.703644+0000","last_clean_scrub_stamp":"2024-03-11T22:32:32.188824+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:01.594253+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"570'12","reported_seq":676,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680230+0000","last_change":"2024-03-11T22:33:00.059789+0000","last_active":"2024-03-11T22:33:06.680230+0000","last_peered":"2024-03-11T22:33:06.680230+0000","last_clean":"2024-03-11T22:33:06.680230+0000","last_became_active":"2024-03-11T22:32:25.070745+0000","last_became_peered":"2024-03-11T22:32:25.070745+0000","last_unstale":"2024-03-11T22:33:06.680230+0000","last_undegraded":"2024-03-11T22:33:06.680230+0000","last_fullsized":"2024-03-11T22:33:06.680230+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"570'12","last_scrub_stamp":"2024-03-11T22:33:00.059686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:33:00.059686+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:16.629600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"479'46","reported_seq":1220,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:10.373393+0000","last_change":"2024-03-11T22:33:10.373454+0000","last_active":"2024-03-11T22:33:10.373393+0000","last_peered":"2024-03-11T22:33:10.373393+0000","last_clean":"2024-03-11T22:33:10.373393+0000","last_became_active":"2024-03-11T22:20:29.412517+0000","last_became_peered":"2024-03-11T22:20:29.412517+0000","last_unstale":"2024-03-11T22:33:10.373393+0000","last_undegraded":"2024-03-11T22:33:10.373393+0000","last_fullsized":"2024-03-11T22:33:10.373393+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'46","last_scrub_stamp":"2024-03-11T22:33:10.373315+0000","last_deep_scrub":"479'46","last_deep_scrub_stamp":"2024-03-11T22:31:35.314705+0000","last_clean_scrub_stamp":"2024-03-11T22:33:10.373315+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:15.562715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"432'4","reported_seq":626,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680697+0000","last_change":"2024-03-11T22:33:05.047160+0000","last_active":"2024-03-11T22:33:06.680697+0000","last_peered":"2024-03-11T22:33:06.680697+0000","last_clean":"2024-03-11T22:33:06.680697+0000","last_became_active":"2024-03-11T22:32:25.065375+0000","last_became_peered":"2024-03-11T22:32:25.065375+0000","last_unstale":"2024-03-11T22:33:06.680697+0000","last_undegraded":"2024-03-11T22:33:06.680697+0000","last_fullsized":"2024-03-11T22:33:06.680697+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'4","last_scrub_stamp":"2024-03-11T22:33:05.047051+0000","last_deep_scrub":"432'4","last_deep_scrub_stamp":"2024-03-11T22:26:50.847349+0000","last_clean_scrub_stamp":"2024-03-11T22:33:05.047051+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:30.850741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1084,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364489+0000","last_change":"2024-03-11T22:31:33.403335+0000","last_active":"2024-03-11T22:33:07.364489+0000","last_peered":"2024-03-11T22:33:07.364489+0000","last_clean":"2024-03-11T22:33:07.364489+0000","last_became_active":"2024-03-11T22:08:39.079560+0000","last_became_peered":"2024-03-11T22:08:39.079560+0000","last_unstale":"2024-03-11T22:33:07.364489+0000","last_undegraded":"2024-03-11T22:33:07.364489+0000","last_fullsized":"2024-03-11T22:33:07.364489+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:33.403232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:49.194569+0000","last_clean_scrub_stamp":"2024-03-11T22:31:33.403232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"479'18","reported_seq":1187,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409780+0000","last_change":"2024-03-11T22:32:21.175430+0000","last_active":"2024-03-11T22:33:07.409780+0000","last_peered":"2024-03-11T22:33:07.409780+0000","last_clean":"2024-03-11T22:33:07.409780+0000","last_became_active":"2024-03-11T22:30:16.929794+0000","last_became_peered":"2024-03-11T22:30:16.929794+0000","last_unstale":"2024-03-11T22:33:07.409780+0000","last_undegraded":"2024-03-11T22:33:07.409780+0000","last_fullsized":"2024-03-11T22:33:07.409780+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'18","last_scrub_stamp":"2024-03-11T22:32:21.175319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:21.175319+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:49.790907+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1118,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408571+0000","last_change":"2024-03-11T22:32:28.569731+0000","last_active":"2024-03-11T22:33:07.408571+0000","last_peered":"2024-03-11T22:33:07.408571+0000","last_clean":"2024-03-11T22:33:07.408571+0000","last_became_active":"2024-03-11T22:08:33.035276+0000","last_became_peered":"2024-03-11T22:08:33.035276+0000","last_unstale":"2024-03-11T22:33:07.408571+0000","last_undegraded":"2024-03-11T22:33:07.408571+0000","last_fullsized":"2024-03-11T22:33:07.408571+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:28.569588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:00.499404+0000","last_clean_scrub_stamp":"2024-03-11T22:32:28.569588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:47.860375+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1107,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364602+0000","last_change":"2024-03-11T22:31:49.196645+0000","last_active":"2024-03-11T22:33:07.364602+0000","last_peered":"2024-03-11T22:33:07.364602+0000","last_clean":"2024-03-11T22:33:07.364602+0000","last_became_active":"2024-03-11T22:29:04.731305+0000","last_became_peered":"2024-03-11T22:29:04.731305+0000","last_unstale":"2024-03-11T22:33:07.364602+0000","last_undegraded":"2024-03-11T22:33:07.364602+0000","last_fullsized":"2024-03-11T22:33:07.364602+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:49.196546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:20:54.635309+0000","last_clean_scrub_stamp":"2024-03-11T22:31:49.196546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":100,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680078+0000","last_change":"2024-03-11T22:32:43.017598+0000","last_active":"2024-03-11T22:33:06.680078+0000","last_peered":"2024-03-11T22:33:06.680078+0000","last_clean":"2024-03-11T22:33:06.680078+0000","last_became_active":"2024-03-11T22:32:25.066663+0000","last_became_peered":"2024-03-11T22:32:25.066663+0000","last_unstale":"2024-03-11T22:33:06.680078+0000","last_undegraded":"2024-03-11T22:33:06.680078+0000","last_fullsized":"2024-03-11T22:33:06.680078+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:43.017454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:13:37.172700+0000","last_clean_scrub_stamp":"2024-03-11T22:32:43.017454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:59.300760+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"478'25","reported_seq":1177,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409195+0000","last_change":"2024-03-11T22:32:51.013986+0000","last_active":"2024-03-11T22:33:07.409195+0000","last_peered":"2024-03-11T22:33:07.409195+0000","last_clean":"2024-03-11T22:33:07.409195+0000","last_became_active":"2024-03-11T22:20:29.431872+0000","last_became_peered":"2024-03-11T22:20:29.431872+0000","last_unstale":"2024-03-11T22:33:07.409195+0000","last_undegraded":"2024-03-11T22:33:07.409195+0000","last_fullsized":"2024-03-11T22:33:07.409195+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:32:51.013089+0000","last_deep_scrub":"273'19","last_deep_scrub_stamp":"2024-03-11T22:23:53.926316+0000","last_clean_scrub_stamp":"2024-03-11T22:32:51.013089+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:03.501164+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":201,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680582+0000","last_change":"2024-03-11T22:32:25.070018+0000","last_active":"2024-03-11T22:33:06.680582+0000","last_peered":"2024-03-11T22:33:06.680582+0000","last_clean":"2024-03-11T22:33:06.680582+0000","last_became_active":"2024-03-11T22:32:25.067290+0000","last_became_peered":"2024-03-11T22:32:25.067290+0000","last_unstale":"2024-03-11T22:33:06.680582+0000","last_undegraded":"2024-03-11T22:33:06.680582+0000","last_fullsized":"2024-03-11T22:33:06.680582+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:11.015678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:11.015678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:29.113585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1110,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409567+0000","last_change":"2024-03-11T22:32:04.961686+0000","last_active":"2024-03-11T22:33:07.409567+0000","last_peered":"2024-03-11T22:33:07.409567+0000","last_clean":"2024-03-11T22:33:07.409567+0000","last_became_active":"2024-03-11T22:29:04.700408+0000","last_became_peered":"2024-03-11T22:29:04.700408+0000","last_unstale":"2024-03-11T22:33:07.409567+0000","last_undegraded":"2024-03-11T22:33:07.409567+0000","last_fullsized":"2024-03-11T22:33:07.409567+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:04.961581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:04.961581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:26.928726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1125,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679699+0000","last_change":"2024-03-11T22:32:01.355414+0000","last_active":"2024-03-11T22:33:06.679699+0000","last_peered":"2024-03-11T22:33:06.679699+0000","last_clean":"2024-03-11T22:33:06.679699+0000","last_became_active":"2024-03-11T22:08:39.070525+0000","last_became_peered":"2024-03-11T22:08:39.070525+0000","last_unstale":"2024-03-11T22:33:06.679699+0000","last_undegraded":"2024-03-11T22:33:06.679699+0000","last_fullsized":"2024-03-11T22:33:06.679699+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:01.355283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:17:27.980507+0000","last_clean_scrub_stamp":"2024-03-11T22:32:01.355283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:10.777489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"479'25","reported_seq":1145,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.363847+0000","last_change":"2024-03-11T22:32:31.285963+0000","last_active":"2024-03-11T22:33:07.363847+0000","last_peered":"2024-03-11T22:33:07.363847+0000","last_clean":"2024-03-11T22:33:07.363847+0000","last_became_active":"2024-03-11T22:08:35.086259+0000","last_became_peered":"2024-03-11T22:08:35.086259+0000","last_unstale":"2024-03-11T22:33:07.363847+0000","last_undegraded":"2024-03-11T22:33:07.363847+0000","last_fullsized":"2024-03-11T22:33:07.363847+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'25","last_scrub_stamp":"2024-03-11T22:32:31.285804+0000","last_deep_scrub":"188'13","last_deep_scrub_stamp":"2024-03-11T22:15:29.636639+0000","last_clean_scrub_stamp":"2024-03-11T22:32:31.285804+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:31.806256+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1123,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.900324+0000","last_change":"2024-03-11T22:33:07.900346+0000","last_active":"2024-03-11T22:33:07.900324+0000","last_peered":"2024-03-11T22:33:07.900324+0000","last_clean":"2024-03-11T22:33:07.900324+0000","last_became_active":"2024-03-11T22:20:29.416982+0000","last_became_peered":"2024-03-11T22:20:29.416982+0000","last_unstale":"2024-03-11T22:33:07.900324+0000","last_undegraded":"2024-03-11T22:33:07.900324+0000","last_fullsized":"2024-03-11T22:33:07.900324+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:07.900247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:33:07.900247+0000","last_clean_scrub_stamp":"2024-03-11T22:33:07.900247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:18.886364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"527'12","reported_seq":1429,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409494+0000","last_change":"2024-03-11T22:32:13.043961+0000","last_active":"2024-03-11T22:33:07.409494+0000","last_peered":"2024-03-11T22:33:07.409494+0000","last_clean":"2024-03-11T22:33:07.409494+0000","last_became_active":"2024-03-11T22:20:29.423415+0000","last_became_peered":"2024-03-11T22:20:29.423415+0000","last_unstale":"2024-03-11T22:33:07.409494+0000","last_undegraded":"2024-03-11T22:33:07.409494+0000","last_fullsized":"2024-03-11T22:33:07.409494+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'12","last_scrub_stamp":"2024-03-11T22:32:13.043826+0000","last_deep_scrub":"527'12","last_deep_scrub_stamp":"2024-03-11T22:32:13.043826+0000","last_clean_scrub_stamp":"2024-03-11T22:32:13.043826+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:22.042414+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":274,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410390+0000","last_change":"2024-03-11T22:32:57.070738+0000","last_active":"2024-03-11T22:33:07.410390+0000","last_peered":"2024-03-11T22:33:07.410390+0000","last_clean":"2024-03-11T22:33:07.410390+0000","last_became_active":"2024-03-11T22:32:01.980472+0000","last_became_peered":"2024-03-11T22:32:01.980472+0000","last_unstale":"2024-03-11T22:33:07.410390+0000","last_undegraded":"2024-03-11T22:33:07.410390+0000","last_fullsized":"2024-03-11T22:33:07.410390+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:57.070618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:35.357245+0000","last_clean_scrub_stamp":"2024-03-11T22:32:57.070618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:17.798863+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"479'30","reported_seq":1189,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409326+0000","last_change":"2024-03-11T22:32:22.146773+0000","last_active":"2024-03-11T22:33:07.409326+0000","last_peered":"2024-03-11T22:33:07.409326+0000","last_clean":"2024-03-11T22:33:07.409326+0000","last_became_active":"2024-03-11T22:29:04.720501+0000","last_became_peered":"2024-03-11T22:29:04.720501+0000","last_unstale":"2024-03-11T22:33:07.409326+0000","last_undegraded":"2024-03-11T22:33:07.409326+0000","last_fullsized":"2024-03-11T22:33:07.409326+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-03-11T22:32:22.146647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:22.146647+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:24.642238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":343,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679864+0000","last_change":"2024-03-11T22:32:25.066585+0000","last_active":"2024-03-11T22:33:06.679864+0000","last_peered":"2024-03-11T22:33:06.679864+0000","last_clean":"2024-03-11T22:33:06.679864+0000","last_became_active":"2024-03-11T22:32:25.066167+0000","last_became_peered":"2024-03-11T22:32:25.066167+0000","last_unstale":"2024-03-11T22:33:06.679864+0000","last_undegraded":"2024-03-11T22:33:06.679864+0000","last_fullsized":"2024-03-11T22:33:06.679864+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:10.023566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:51.034868+0000","last_clean_scrub_stamp":"2024-03-11T22:32:10.023566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:37.215399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"488'10","reported_seq":1420,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364870+0000","last_change":"2024-03-11T22:31:40.234937+0000","last_active":"2024-03-11T22:33:07.364870+0000","last_peered":"2024-03-11T22:33:07.364870+0000","last_clean":"2024-03-11T22:33:07.364870+0000","last_became_active":"2024-03-11T22:29:04.787739+0000","last_became_peered":"2024-03-11T22:29:04.787739+0000","last_unstale":"2024-03-11T22:33:07.364870+0000","last_undegraded":"2024-03-11T22:33:07.364870+0000","last_fullsized":"2024-03-11T22:33:07.364870+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'10","last_scrub_stamp":"2024-03-11T22:31:40.234834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:31:40.234834+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1095,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408978+0000","last_change":"2024-03-11T22:31:55.479021+0000","last_active":"2024-03-11T22:33:07.408978+0000","last_peered":"2024-03-11T22:33:07.408978+0000","last_clean":"2024-03-11T22:33:07.408978+0000","last_became_active":"2024-03-11T22:20:29.425260+0000","last_became_peered":"2024-03-11T22:20:29.425260+0000","last_unstale":"2024-03-11T22:33:07.408978+0000","last_undegraded":"2024-03-11T22:33:07.408978+0000","last_fullsized":"2024-03-11T22:33:07.408978+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:55.478922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:31:55.478922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:18.804848+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"479'27","reported_seq":794,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409764+0000","last_change":"2024-03-11T22:32:09.001879+0000","last_active":"2024-03-11T22:33:07.409764+0000","last_peered":"2024-03-11T22:33:07.409764+0000","last_clean":"2024-03-11T22:33:07.409764+0000","last_became_active":"2024-03-11T22:32:01.974003+0000","last_became_peered":"2024-03-11T22:32:01.974003+0000","last_unstale":"2024-03-11T22:33:07.409764+0000","last_undegraded":"2024-03-11T22:33:07.409764+0000","last_fullsized":"2024-03-11T22:33:07.409764+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'27","last_scrub_stamp":"2024-03-11T22:32:09.001751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:09.001751+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:37.170218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"68'1","reported_seq":300,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410063+0000","last_change":"2024-03-11T22:32:36.705524+0000","last_active":"2024-03-11T22:33:07.410063+0000","last_peered":"2024-03-11T22:33:07.410063+0000","last_clean":"2024-03-11T22:33:07.410063+0000","last_became_active":"2024-03-11T22:32:01.972673+0000","last_became_peered":"2024-03-11T22:32:01.972673+0000","last_unstale":"2024-03-11T22:33:07.410063+0000","last_undegraded":"2024-03-11T22:33:07.410063+0000","last_fullsized":"2024-03-11T22:33:07.410063+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-11T22:32:36.705401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:36.705401+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:54.053440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":214,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680487+0000","last_change":"2024-03-11T22:32:12.575956+0000","last_active":"2024-03-11T22:33:06.680487+0000","last_peered":"2024-03-11T22:33:06.680487+0000","last_clean":"2024-03-11T22:33:06.680487+0000","last_became_active":"2024-03-11T22:29:04.723400+0000","last_became_peered":"2024-03-11T22:29:04.723400+0000","last_unstale":"2024-03-11T22:33:06.680487+0000","last_undegraded":"2024-03-11T22:33:06.680487+0000","last_fullsized":"2024-03-11T22:33:06.680487+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:12.575860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:12.575860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:30.116230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1074,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408944+0000","last_change":"2024-03-11T22:31:49.185033+0000","last_active":"2024-03-11T22:33:07.408944+0000","last_peered":"2024-03-11T22:33:07.408944+0000","last_clean":"2024-03-11T22:33:07.408944+0000","last_became_active":"2024-03-11T22:08:39.080341+0000","last_became_peered":"2024-03-11T22:08:39.080341+0000","last_unstale":"2024-03-11T22:33:07.408944+0000","last_undegraded":"2024-03-11T22:33:07.408944+0000","last_fullsized":"2024-03-11T22:33:07.408944+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:49.184919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:31:49.184919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:08.918644+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"478'25","reported_seq":229,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680112+0000","last_change":"2024-03-11T22:32:51.610299+0000","last_active":"2024-03-11T22:33:06.680112+0000","last_peered":"2024-03-11T22:33:06.680112+0000","last_clean":"2024-03-11T22:33:06.680112+0000","last_became_active":"2024-03-11T22:29:04.730485+0000","last_became_peered":"2024-03-11T22:29:04.730485+0000","last_unstale":"2024-03-11T22:33:06.680112+0000","last_undegraded":"2024-03-11T22:33:06.680112+0000","last_fullsized":"2024-03-11T22:33:06.680112+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:32:51.610209+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-03-11T22:17:35.738211+0000","last_clean_scrub_stamp":"2024-03-11T22:32:51.610209+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:06.971761+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":494,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679820+0000","last_change":"2024-03-11T22:32:25.065573+0000","last_active":"2024-03-11T22:33:06.679820+0000","last_peered":"2024-03-11T22:33:06.679820+0000","last_clean":"2024-03-11T22:33:06.679820+0000","last_became_active":"2024-03-11T22:32:25.064682+0000","last_became_peered":"2024-03-11T22:32:25.064682+0000","last_unstale":"2024-03-11T22:33:06.679820+0000","last_undegraded":"2024-03-11T22:33:06.679820+0000","last_fullsized":"2024-03-11T22:33:06.679820+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:50.162015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:31:50.162015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:13.107667+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1098,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409808+0000","last_change":"2024-03-11T22:31:56.101706+0000","last_active":"2024-03-11T22:33:07.409808+0000","last_peered":"2024-03-11T22:33:07.409808+0000","last_clean":"2024-03-11T22:33:07.409808+0000","last_became_active":"2024-03-11T22:29:04.700352+0000","last_became_peered":"2024-03-11T22:29:04.700352+0000","last_unstale":"2024-03-11T22:33:07.409808+0000","last_undegraded":"2024-03-11T22:33:07.409808+0000","last_fullsized":"2024-03-11T22:33:07.409808+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:56.101587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:11:37.075868+0000","last_clean_scrub_stamp":"2024-03-11T22:31:56.101587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:12.452173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1104,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408033+0000","last_change":"2024-03-11T22:32:18.264189+0000","last_active":"2024-03-11T22:33:07.408033+0000","last_peered":"2024-03-11T22:33:07.408033+0000","last_clean":"2024-03-11T22:33:07.408033+0000","last_became_active":"2024-03-11T22:29:04.707420+0000","last_became_peered":"2024-03-11T22:29:04.707420+0000","last_unstale":"2024-03-11T22:33:07.408033+0000","last_undegraded":"2024-03-11T22:33:07.408033+0000","last_fullsized":"2024-03-11T22:33:07.408033+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:18.264067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:23:09.057970+0000","last_clean_scrub_stamp":"2024-03-11T22:32:18.264067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:40.145531+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"480'28","reported_seq":1170,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409184+0000","last_change":"2024-03-11T22:32:52.163857+0000","last_active":"2024-03-11T22:33:07.409184+0000","last_peered":"2024-03-11T22:33:07.409184+0000","last_clean":"2024-03-11T22:33:07.409184+0000","last_became_active":"2024-03-11T22:20:29.439747+0000","last_became_peered":"2024-03-11T22:20:29.439747+0000","last_unstale":"2024-03-11T22:33:07.409184+0000","last_undegraded":"2024-03-11T22:33:07.409184+0000","last_fullsized":"2024-03-11T22:33:07.409184+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'28","last_scrub_stamp":"2024-03-11T22:32:52.163741+0000","last_deep_scrub":"274'20","last_deep_scrub_stamp":"2024-03-11T22:18:42.135888+0000","last_clean_scrub_stamp":"2024-03-11T22:32:52.163741+0000","objects_scrubbed":5,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:15.948914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1127,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364698+0000","last_change":"2024-03-11T22:32:16.283660+0000","last_active":"2024-03-11T22:33:07.364698+0000","last_peered":"2024-03-11T22:33:07.364698+0000","last_clean":"2024-03-11T22:33:07.364698+0000","last_became_active":"2024-03-11T22:29:04.722573+0000","last_became_peered":"2024-03-11T22:29:04.722573+0000","last_unstale":"2024-03-11T22:33:07.364698+0000","last_undegraded":"2024-03-11T22:33:07.364698+0000","last_fullsized":"2024-03-11T22:33:07.364698+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:16.283559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:16.283559+0000","last_clean_scrub_stamp":"2024-03-11T22:32:16.283559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:27.190217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":208,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680303+0000","last_change":"2024-03-11T22:32:29.046069+0000","last_active":"2024-03-11T22:33:06.680303+0000","last_peered":"2024-03-11T22:33:06.680303+0000","last_clean":"2024-03-11T22:33:06.680303+0000","last_became_active":"2024-03-11T22:32:25.066717+0000","last_became_peered":"2024-03-11T22:32:25.066717+0000","last_unstale":"2024-03-11T22:33:06.680303+0000","last_undegraded":"2024-03-11T22:33:06.680303+0000","last_fullsized":"2024-03-11T22:33:06.680303+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:29.045953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:29.045953+0000","last_clean_scrub_stamp":"2024-03-11T22:32:29.045953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:54.322587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1082,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410014+0000","last_change":"2024-03-11T22:32:23.283924+0000","last_active":"2024-03-11T22:33:07.410014+0000","last_peered":"2024-03-11T22:33:07.410014+0000","last_clean":"2024-03-11T22:33:07.410014+0000","last_became_active":"2024-03-11T22:08:39.081511+0000","last_became_peered":"2024-03-11T22:08:39.081511+0000","last_unstale":"2024-03-11T22:33:07.410014+0000","last_undegraded":"2024-03-11T22:33:07.410014+0000","last_fullsized":"2024-03-11T22:33:07.410014+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:23.283801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:23.283801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:32.124554+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"478'12","reported_seq":1152,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408563+0000","last_change":"2024-03-11T22:33:00.202526+0000","last_active":"2024-03-11T22:33:07.408563+0000","last_peered":"2024-03-11T22:33:07.408563+0000","last_clean":"2024-03-11T22:33:07.408563+0000","last_became_active":"2024-03-11T22:29:04.734207+0000","last_became_peered":"2024-03-11T22:29:04.734207+0000","last_unstale":"2024-03-11T22:33:07.408563+0000","last_undegraded":"2024-03-11T22:33:07.408563+0000","last_fullsized":"2024-03-11T22:33:07.408563+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'12","last_scrub_stamp":"2024-03-11T22:33:00.202402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:33:00.202402+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:29.698511+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1103,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409567+0000","last_change":"2024-03-11T22:32:33.018315+0000","last_active":"2024-03-11T22:33:07.409567+0000","last_peered":"2024-03-11T22:33:07.409567+0000","last_clean":"2024-03-11T22:33:07.409567+0000","last_became_active":"2024-03-11T22:08:33.040570+0000","last_became_peered":"2024-03-11T22:08:33.040570+0000","last_unstale":"2024-03-11T22:33:07.409567+0000","last_undegraded":"2024-03-11T22:33:07.409567+0000","last_fullsized":"2024-03-11T22:33:07.409567+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:33.018185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:04.134523+0000","last_clean_scrub_stamp":"2024-03-11T22:32:33.018185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:41.017610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":696,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408668+0000","last_change":"2024-03-11T22:32:52.517091+0000","last_active":"2024-03-11T22:33:07.408668+0000","last_peered":"2024-03-11T22:33:07.408668+0000","last_clean":"2024-03-11T22:33:07.408668+0000","last_became_active":"2024-03-11T22:20:29.415755+0000","last_became_peered":"2024-03-11T22:20:29.415755+0000","last_unstale":"2024-03-11T22:33:07.408668+0000","last_undegraded":"2024-03-11T22:33:07.408668+0000","last_fullsized":"2024-03-11T22:33:07.408668+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:52.516980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:52.516980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:11.666992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":92,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679743+0000","last_change":"2024-03-11T22:32:25.066370+0000","last_active":"2024-03-11T22:33:06.679743+0000","last_peered":"2024-03-11T22:33:06.679743+0000","last_clean":"2024-03-11T22:33:06.679743+0000","last_became_active":"2024-03-11T22:32:25.065512+0000","last_became_peered":"2024-03-11T22:32:25.065512+0000","last_unstale":"2024-03-11T22:33:06.679743+0000","last_undegraded":"2024-03-11T22:33:06.679743+0000","last_fullsized":"2024-03-11T22:33:06.679743+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:12.331743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:12.331743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:40.788117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"480'27","reported_seq":1194,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409313+0000","last_change":"2024-03-11T22:32:03.688836+0000","last_active":"2024-03-11T22:33:07.409313+0000","last_peered":"2024-03-11T22:33:07.409313+0000","last_clean":"2024-03-11T22:33:07.409313+0000","last_became_active":"2024-03-11T22:29:04.720148+0000","last_became_peered":"2024-03-11T22:29:04.720148+0000","last_unstale":"2024-03-11T22:33:07.409313+0000","last_undegraded":"2024-03-11T22:33:07.409313+0000","last_fullsized":"2024-03-11T22:33:07.409313+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'27","last_scrub_stamp":"2024-03-11T22:32:03.688734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:03.688734+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:16.351475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1106,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408510+0000","last_change":"2024-03-11T22:32:06.986662+0000","last_active":"2024-03-11T22:33:07.408510+0000","last_peered":"2024-03-11T22:33:07.408510+0000","last_clean":"2024-03-11T22:33:07.408510+0000","last_became_active":"2024-03-11T22:29:04.733742+0000","last_became_peered":"2024-03-11T22:29:04.733742+0000","last_unstale":"2024-03-11T22:33:07.408510+0000","last_undegraded":"2024-03-11T22:33:07.408510+0000","last_fullsized":"2024-03-11T22:33:07.408510+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:06.986570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:32:06.986570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:27.504039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1105,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409301+0000","last_change":"2024-03-11T22:32:43.220021+0000","last_active":"2024-03-11T22:33:07.409301+0000","last_peered":"2024-03-11T22:33:07.409301+0000","last_clean":"2024-03-11T22:33:07.409301+0000","last_became_active":"2024-03-11T22:08:37.068945+0000","last_became_peered":"2024-03-11T22:08:37.068945+0000","last_unstale":"2024-03-11T22:33:07.409301+0000","last_undegraded":"2024-03-11T22:33:07.409301+0000","last_fullsized":"2024-03-11T22:33:07.409301+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:43.219886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:43.219886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:11.167500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":219,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680894+0000","last_change":"2024-03-11T22:32:54.614518+0000","last_active":"2024-03-11T22:33:06.680894+0000","last_peered":"2024-03-11T22:33:06.680894+0000","last_clean":"2024-03-11T22:33:06.680894+0000","last_became_active":"2024-03-11T22:29:04.786119+0000","last_became_peered":"2024-03-11T22:29:04.786119+0000","last_unstale":"2024-03-11T22:33:06.680894+0000","last_undegraded":"2024-03-11T22:33:06.680894+0000","last_fullsized":"2024-03-11T22:33:06.680894+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:54.614382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:06.824468+0000","last_clean_scrub_stamp":"2024-03-11T22:32:54.614382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:07.388340+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"479'30","reported_seq":1182,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409188+0000","last_change":"2024-03-11T22:32:07.378996+0000","last_active":"2024-03-11T22:33:07.409188+0000","last_peered":"2024-03-11T22:33:07.409188+0000","last_clean":"2024-03-11T22:33:07.409188+0000","last_became_active":"2024-03-11T22:20:29.431913+0000","last_became_peered":"2024-03-11T22:20:29.431913+0000","last_unstale":"2024-03-11T22:33:07.409188+0000","last_undegraded":"2024-03-11T22:33:07.409188+0000","last_fullsized":"2024-03-11T22:33:07.409188+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'30","last_scrub_stamp":"2024-03-11T22:32:07.378892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:07.378892+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:22.849990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"225'2","reported_seq":223,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:10.572540+0000","last_change":"2024-03-11T22:33:10.572570+0000","last_active":"2024-03-11T22:33:10.572540+0000","last_peered":"2024-03-11T22:33:10.572540+0000","last_clean":"2024-03-11T22:33:10.572540+0000","last_became_active":"2024-03-11T22:29:04.693762+0000","last_became_peered":"2024-03-11T22:29:04.693762+0000","last_unstale":"2024-03-11T22:33:10.572540+0000","last_undegraded":"2024-03-11T22:33:10.572540+0000","last_fullsized":"2024-03-11T22:33:10.572540+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"225'2","last_scrub_stamp":"2024-03-11T22:33:10.572474+0000","last_deep_scrub":"225'2","last_deep_scrub_stamp":"2024-03-11T22:17:36.687660+0000","last_clean_scrub_stamp":"2024-03-11T22:33:10.572474+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:10.591515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"541'12","reported_seq":122,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408975+0000","last_change":"2024-03-11T22:32:08.427075+0000","last_active":"2024-03-11T22:33:07.408975+0000","last_peered":"2024-03-11T22:33:07.408975+0000","last_clean":"2024-03-11T22:33:07.408975+0000","last_became_active":"2024-03-11T22:31:11.769411+0000","last_became_peered":"2024-03-11T22:31:11.769411+0000","last_unstale":"2024-03-11T22:33:07.408975+0000","last_undegraded":"2024-03-11T22:33:07.408975+0000","last_fullsized":"2024-03-11T22:33:07.408975+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"541'12","last_scrub_stamp":"2024-03-11T22:32:08.426964+0000","last_deep_scrub":"541'12","last_deep_scrub_stamp":"2024-03-11T22:32:08.426964+0000","last_clean_scrub_stamp":"2024-03-11T22:32:08.426964+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:20.953228+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"553'2","reported_seq":1024,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680485+0000","last_change":"2024-03-11T22:32:53.990047+0000","last_active":"2024-03-11T22:33:06.680485+0000","last_peered":"2024-03-11T22:33:06.680485+0000","last_clean":"2024-03-11T22:33:06.680485+0000","last_became_active":"2024-03-11T22:32:25.072182+0000","last_became_peered":"2024-03-11T22:32:25.072182+0000","last_unstale":"2024-03-11T22:33:06.680485+0000","last_undegraded":"2024-03-11T22:33:06.680485+0000","last_fullsized":"2024-03-11T22:33:06.680485+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"553'2","last_scrub_stamp":"2024-03-11T22:32:53.989909+0000","last_deep_scrub":"553'2","last_deep_scrub_stamp":"2024-03-11T22:31:46.173315+0000","last_clean_scrub_stamp":"2024-03-11T22:32:53.989909+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:59.823594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"480'24","reported_seq":1166,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409387+0000","last_change":"2024-03-11T22:32:17.060931+0000","last_active":"2024-03-11T22:33:07.409387+0000","last_peered":"2024-03-11T22:33:07.409387+0000","last_clean":"2024-03-11T22:33:07.409387+0000","last_became_active":"2024-03-11T22:29:04.720341+0000","last_became_peered":"2024-03-11T22:29:04.720341+0000","last_unstale":"2024-03-11T22:33:07.409387+0000","last_undegraded":"2024-03-11T22:33:07.409387+0000","last_fullsized":"2024-03-11T22:33:07.409387+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'24","last_scrub_stamp":"2024-03-11T22:32:17.060793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:17.060793+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:20.357668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":294,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410126+0000","last_change":"2024-03-11T22:32:48.857529+0000","last_active":"2024-03-11T22:33:07.410126+0000","last_peered":"2024-03-11T22:33:07.410126+0000","last_clean":"2024-03-11T22:33:07.410126+0000","last_became_active":"2024-03-11T22:32:01.978045+0000","last_became_peered":"2024-03-11T22:32:01.978045+0000","last_unstale":"2024-03-11T22:33:07.410126+0000","last_undegraded":"2024-03-11T22:33:07.410126+0000","last_fullsized":"2024-03-11T22:33:07.410126+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:48.857426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:30:08.740926+0000","last_clean_scrub_stamp":"2024-03-11T22:32:48.857426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:52.918396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":286,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410331+0000","last_change":"2024-03-11T22:32:15.970183+0000","last_active":"2024-03-11T22:33:07.410331+0000","last_peered":"2024-03-11T22:33:07.410331+0000","last_clean":"2024-03-11T22:33:07.410331+0000","last_became_active":"2024-03-11T22:32:01.980091+0000","last_became_peered":"2024-03-11T22:32:01.980091+0000","last_unstale":"2024-03-11T22:33:07.410331+0000","last_undegraded":"2024-03-11T22:33:07.410331+0000","last_fullsized":"2024-03-11T22:33:07.410331+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:15.970077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:15.970077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:24.278206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1083,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409170+0000","last_change":"2024-03-11T22:32:54.364298+0000","last_active":"2024-03-11T22:33:07.409170+0000","last_peered":"2024-03-11T22:33:07.409170+0000","last_clean":"2024-03-11T22:33:07.409170+0000","last_became_active":"2024-03-11T22:08:39.077934+0000","last_became_peered":"2024-03-11T22:08:39.077934+0000","last_unstale":"2024-03-11T22:33:07.409170+0000","last_undegraded":"2024-03-11T22:33:07.409170+0000","last_fullsized":"2024-03-11T22:33:07.409170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:54.364192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:54.364192+0000","last_clean_scrub_stamp":"2024-03-11T22:32:54.364192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:55.788227+0000","scrub_duration":213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"479'39","reported_seq":1245,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679441+0000","last_change":"2024-03-11T22:32:56.667313+0000","last_active":"2024-03-11T22:33:06.679441+0000","last_peered":"2024-03-11T22:33:06.679441+0000","last_clean":"2024-03-11T22:33:06.679441+0000","last_became_active":"2024-03-11T22:29:04.721324+0000","last_became_peered":"2024-03-11T22:29:04.721324+0000","last_unstale":"2024-03-11T22:33:06.679441+0000","last_undegraded":"2024-03-11T22:33:06.679441+0000","last_fullsized":"2024-03-11T22:33:06.679441+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'39","last_scrub_stamp":"2024-03-11T22:32:56.667210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:56.667210+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:09.653888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1114,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408545+0000","last_change":"2024-03-11T22:32:02.395766+0000","last_active":"2024-03-11T22:33:07.408545+0000","last_peered":"2024-03-11T22:33:07.408545+0000","last_clean":"2024-03-11T22:33:07.408545+0000","last_became_active":"2024-03-11T22:20:29.414189+0000","last_became_peered":"2024-03-11T22:20:29.414189+0000","last_unstale":"2024-03-11T22:33:07.408545+0000","last_undegraded":"2024-03-11T22:33:07.408545+0000","last_fullsized":"2024-03-11T22:33:07.408545+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:02.395678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:18:43.931797+0000","last_clean_scrub_stamp":"2024-03-11T22:32:02.395678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:22.591865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":219,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.680890+0000","last_change":"2024-03-11T22:32:35.581682+0000","last_active":"2024-03-11T22:33:06.680890+0000","last_peered":"2024-03-11T22:33:06.680890+0000","last_clean":"2024-03-11T22:33:06.680890+0000","last_became_active":"2024-03-11T22:29:04.724856+0000","last_became_peered":"2024-03-11T22:29:04.724856+0000","last_unstale":"2024-03-11T22:33:06.680890+0000","last_undegraded":"2024-03-11T22:33:06.680890+0000","last_fullsized":"2024-03-11T22:33:06.680890+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:35.581593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:32:35.581593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:53.240847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":283,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409653+0000","last_change":"2024-03-11T22:33:04.051479+0000","last_active":"2024-03-11T22:33:07.409653+0000","last_peered":"2024-03-11T22:33:07.409653+0000","last_clean":"2024-03-11T22:33:07.409653+0000","last_became_active":"2024-03-11T22:32:01.973161+0000","last_became_peered":"2024-03-11T22:32:01.973161+0000","last_unstale":"2024-03-11T22:33:07.409653+0000","last_undegraded":"2024-03-11T22:33:07.409653+0000","last_fullsized":"2024-03-11T22:33:07.409653+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:04.051367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:31:41.776548+0000","last_clean_scrub_stamp":"2024-03-11T22:33:04.051367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:23.828209+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"478'11","reported_seq":780,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409580+0000","last_change":"2024-03-11T22:32:08.050734+0000","last_active":"2024-03-11T22:33:07.409580+0000","last_peered":"2024-03-11T22:33:07.409580+0000","last_clean":"2024-03-11T22:33:07.409580+0000","last_became_active":"2024-03-11T22:32:01.972835+0000","last_became_peered":"2024-03-11T22:32:01.972835+0000","last_unstale":"2024-03-11T22:33:07.409580+0000","last_undegraded":"2024-03-11T22:33:07.409580+0000","last_fullsized":"2024-03-11T22:33:07.409580+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'11","last_scrub_stamp":"2024-03-11T22:32:08.050630+0000","last_deep_scrub":"95'7","last_deep_scrub_stamp":"2024-03-11T22:10:18.579896+0000","last_clean_scrub_stamp":"2024-03-11T22:32:08.050630+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:15.787633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1114,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364501+0000","last_change":"2024-03-11T22:32:00.332632+0000","last_active":"2024-03-11T22:33:07.364501+0000","last_peered":"2024-03-11T22:33:07.364501+0000","last_clean":"2024-03-11T22:33:07.364501+0000","last_became_active":"2024-03-11T22:29:04.721192+0000","last_became_peered":"2024-03-11T22:29:04.721192+0000","last_unstale":"2024-03-11T22:33:07.364501+0000","last_undegraded":"2024-03-11T22:33:07.364501+0000","last_fullsized":"2024-03-11T22:33:07.364501+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:00.332516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:14:30.716844+0000","last_clean_scrub_stamp":"2024-03-11T22:32:00.332516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:12.537196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":200,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679971+0000","last_change":"2024-03-11T22:32:25.070793+0000","last_active":"2024-03-11T22:33:06.679971+0000","last_peered":"2024-03-11T22:33:06.679971+0000","last_clean":"2024-03-11T22:33:06.679971+0000","last_became_active":"2024-03-11T22:32:25.069724+0000","last_became_peered":"2024-03-11T22:32:25.069724+0000","last_unstale":"2024-03-11T22:33:06.679971+0000","last_undegraded":"2024-03-11T22:33:06.679971+0000","last_fullsized":"2024-03-11T22:33:06.679971+0000","mapping_epoch":569,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:18.974465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:26:09.575187+0000","last_clean_scrub_stamp":"2024-03-11T22:32:18.974465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:23.746670+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1103,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364194+0000","last_change":"2024-03-11T22:32:22.365025+0000","last_active":"2024-03-11T22:33:07.364194+0000","last_peered":"2024-03-11T22:33:07.364194+0000","last_clean":"2024-03-11T22:33:07.364194+0000","last_became_active":"2024-03-11T22:29:04.721807+0000","last_became_peered":"2024-03-11T22:29:04.721807+0000","last_unstale":"2024-03-11T22:33:07.364194+0000","last_undegraded":"2024-03-11T22:33:07.364194+0000","last_fullsized":"2024-03-11T22:33:07.364194+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:22.364881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:32:22.364881+0000","last_clean_scrub_stamp":"2024-03-11T22:32:22.364881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:50.855997+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"479'21","reported_seq":751,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.410107+0000","last_change":"2024-03-11T22:32:13.061722+0000","last_active":"2024-03-11T22:33:07.410107+0000","last_peered":"2024-03-11T22:33:07.410107+0000","last_clean":"2024-03-11T22:33:07.410107+0000","last_became_active":"2024-03-11T22:32:01.966449+0000","last_became_peered":"2024-03-11T22:32:01.966449+0000","last_unstale":"2024-03-11T22:33:07.410107+0000","last_undegraded":"2024-03-11T22:33:07.410107+0000","last_fullsized":"2024-03-11T22:33:07.410107+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'21","last_scrub_stamp":"2024-03-11T22:32:13.061595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:13.061595+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:28.213044+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1102,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364707+0000","last_change":"2024-03-11T22:31:43.180566+0000","last_active":"2024-03-11T22:33:07.364707+0000","last_peered":"2024-03-11T22:33:07.364707+0000","last_clean":"2024-03-11T22:33:07.364707+0000","last_became_active":"2024-03-11T22:08:33.041130+0000","last_became_peered":"2024-03-11T22:08:33.041130+0000","last_unstale":"2024-03-11T22:33:07.364707+0000","last_undegraded":"2024-03-11T22:33:07.364707+0000","last_fullsized":"2024-03-11T22:33:07.364707+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:43.180426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:31:43.180426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1106,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.364066+0000","last_change":"2024-03-11T22:31:44.158242+0000","last_active":"2024-03-11T22:33:07.364066+0000","last_peered":"2024-03-11T22:33:07.364066+0000","last_clean":"2024-03-11T22:33:07.364066+0000","last_became_active":"2024-03-11T22:20:29.419619+0000","last_became_peered":"2024-03-11T22:20:29.419619+0000","last_unstale":"2024-03-11T22:33:07.364066+0000","last_undegraded":"2024-03-11T22:33:07.364066+0000","last_fullsized":"2024-03-11T22:33:07.364066+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:31:44.158149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:10:25.082500+0000","last_clean_scrub_stamp":"2024-03-11T22:31:44.158149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1103,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408467+0000","last_change":"2024-03-11T22:32:49.083293+0000","last_active":"2024-03-11T22:33:07.408467+0000","last_peered":"2024-03-11T22:33:07.408467+0000","last_clean":"2024-03-11T22:33:07.408467+0000","last_became_active":"2024-03-11T22:29:04.707444+0000","last_became_peered":"2024-03-11T22:29:04.707444+0000","last_unstale":"2024-03-11T22:33:07.408467+0000","last_undegraded":"2024-03-11T22:33:07.408467+0000","last_fullsized":"2024-03-11T22:33:07.408467+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:49.083192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:49.083192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:06.742489+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"480'39","reported_seq":1151,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408190+0000","last_change":"2024-03-11T22:32:40.160594+0000","last_active":"2024-03-11T22:33:07.408190+0000","last_peered":"2024-03-11T22:33:07.408190+0000","last_clean":"2024-03-11T22:33:07.408190+0000","last_became_active":"2024-03-11T22:08:35.060867+0000","last_became_peered":"2024-03-11T22:08:35.060867+0000","last_unstale":"2024-03-11T22:33:07.408190+0000","last_undegraded":"2024-03-11T22:33:07.408190+0000","last_fullsized":"2024-03-11T22:33:07.408190+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'39","last_scrub_stamp":"2024-03-11T22:32:40.160489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:34.037605+0000","last_clean_scrub_stamp":"2024-03-11T22:32:40.160489+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:06.390231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"575'2","reported_seq":1111,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409305+0000","last_change":"2024-03-11T22:32:59.178704+0000","last_active":"2024-03-11T22:33:07.409305+0000","last_peered":"2024-03-11T22:33:07.409305+0000","last_clean":"2024-03-11T22:33:07.409305+0000","last_became_active":"2024-03-11T22:08:33.300479+0000","last_became_peered":"2024-03-11T22:08:33.300479+0000","last_unstale":"2024-03-11T22:33:07.409305+0000","last_undegraded":"2024-03-11T22:33:07.409305+0000","last_fullsized":"2024-03-11T22:33:07.409305+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'2","last_scrub_stamp":"2024-03-11T22:32:59.178597+0000","last_deep_scrub":"575'2","last_deep_scrub_stamp":"2024-03-11T22:32:59.178597+0000","last_clean_scrub_stamp":"2024-03-11T22:32:59.178597+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:08.640395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":736,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408409+0000","last_change":"2024-03-11T22:33:05.946294+0000","last_active":"2024-03-11T22:33:07.408409+0000","last_peered":"2024-03-11T22:33:07.408409+0000","last_clean":"2024-03-11T22:33:07.408409+0000","last_became_active":"2024-03-11T22:32:01.975227+0000","last_became_peered":"2024-03-11T22:32:01.975227+0000","last_unstale":"2024-03-11T22:33:07.408409+0000","last_undegraded":"2024-03-11T22:33:07.408409+0000","last_fullsized":"2024-03-11T22:33:07.408409+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:05.946180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:33:05.946180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:25.507312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1093,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409915+0000","last_change":"2024-03-11T22:32:43.271141+0000","last_active":"2024-03-11T22:33:07.409915+0000","last_peered":"2024-03-11T22:33:07.409915+0000","last_clean":"2024-03-11T22:33:07.409915+0000","last_became_active":"2024-03-11T22:20:29.435904+0000","last_became_peered":"2024-03-11T22:20:29.435904+0000","last_unstale":"2024-03-11T22:33:07.409915+0000","last_undegraded":"2024-03-11T22:33:07.409915+0000","last_fullsized":"2024-03-11T22:33:07.409915+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:43.271029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:43.271029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:56.249736+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"478'25","reported_seq":777,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.408505+0000","last_change":"2024-03-11T22:32:52.823260+0000","last_active":"2024-03-11T22:33:07.408505+0000","last_peered":"2024-03-11T22:33:07.408505+0000","last_clean":"2024-03-11T22:33:07.408505+0000","last_became_active":"2024-03-11T22:32:01.973526+0000","last_became_peered":"2024-03-11T22:32:01.973526+0000","last_unstale":"2024-03-11T22:33:07.408505+0000","last_undegraded":"2024-03-11T22:33:07.408505+0000","last_fullsized":"2024-03-11T22:33:07.408505+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'25","last_scrub_stamp":"2024-03-11T22:32:52.823135+0000","last_deep_scrub":"211'13","last_deep_scrub_stamp":"2024-03-11T22:16:28.105152+0000","last_clean_scrub_stamp":"2024-03-11T22:32:52.823135+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:57.389349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":30,"num_read_kb":24,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":1112,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409357+0000","last_change":"2024-03-11T22:33:00.035171+0000","last_active":"2024-03-11T22:33:07.409357+0000","last_peered":"2024-03-11T22:33:07.409357+0000","last_clean":"2024-03-11T22:33:07.409357+0000","last_became_active":"2024-03-11T22:08:33.298188+0000","last_became_peered":"2024-03-11T22:08:33.298188+0000","last_unstale":"2024-03-11T22:33:07.409357+0000","last_undegraded":"2024-03-11T22:33:07.409357+0000","last_fullsized":"2024-03-11T22:33:07.409357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:00.035072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:32.020092+0000","last_clean_scrub_stamp":"2024-03-11T22:33:00.035072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:10.449315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":219,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:06.679909+0000","last_change":"2024-03-11T22:33:04.533400+0000","last_active":"2024-03-11T22:33:06.679909+0000","last_peered":"2024-03-11T22:33:06.679909+0000","last_clean":"2024-03-11T22:33:06.679909+0000","last_became_active":"2024-03-11T22:29:04.701716+0000","last_became_peered":"2024-03-11T22:29:04.701716+0000","last_unstale":"2024-03-11T22:33:06.679909+0000","last_undegraded":"2024-03-11T22:33:06.679909+0000","last_fullsized":"2024-03-11T22:33:06.679909+0000","mapping_epoch":485,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:33:04.533317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:36.051029+0000","last_clean_scrub_stamp":"2024-03-11T22:33:04.533317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:34:15.117209+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":725,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-11T22:33:07.409831+0000","last_change":"2024-03-11T22:32:33.840657+0000","last_active":"2024-03-11T22:33:07.409831+0000","last_peered":"2024-03-11T22:33:07.409831+0000","last_clean":"2024-03-11T22:33:07.409831+0000","last_became_active":"2024-03-11T22:32:01.974072+0000","last_became_peered":"2024-03-11T22:32:01.974072+0000","last_unstale":"2024-03-11T22:33:07.409831+0000","last_undegraded":"2024-03-11T22:33:07.409831+0000","last_fullsized":"2024-03-11T22:33:07.409831+0000","mapping_epoch":560,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":561,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:32:33.840544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:08:38.057455+0000","last_clean_scrub_stamp":"2024-03-11T22:32:33.840544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T22:33:35.615389+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":158,"num_read_kb":153,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1529,"num_read_kb":1286,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":5804,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1044,"ondisk_log_size":1044,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":61,"seq":261993005357,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":846256,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92898896,"statfs":{"total":95995035648,"available":95128469504,"internally_reserved":0,"allocated":2985984,"data_stored":1986710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":863566283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332402,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":691804,"kb_used_data":2576,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93053348,"statfs":{"total":95995035648,"available":95286628352,"internally_reserved":0,"allocated":2637824,"data_stored":1658840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":705755590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575180,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":751552,"kb_used_data":2616,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":92993600,"statfs":{"total":95995035648,"available":95225446400,"internally_reserved":0,"allocated":2678784,"data_stored":1658898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738706,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":958380,"kb_used_data":2592,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92786772,"statfs":{"total":95995035648,"available":95013654528,"internally_reserved":0,"allocated":2654208,"data_stored":1658362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411655,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":647640,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93097512,"statfs":{"total":95995035648,"available":95331852288,"internally_reserved":0,"allocated":2899968,"data_stored":1921354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150017,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":778500,"kb_used_data":3192,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92966652,"statfs":{"total":95995035648,"available":95197851648,"internally_reserved":0,"allocated":3268608,"data_stored":2249714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":793901511},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921085,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":719604,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93025548,"statfs":{"total":95995035648,"available":95258161152,"internally_reserved":0,"allocated":2662400,"data_stored":1658571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":734198214},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659446,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":796524,"kb_used_data":3168,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92948628,"statfs":{"total":95995035648,"available":95179395072,"internally_reserved":0,"allocated":3244032,"data_stored":2248619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":812382662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:33:15.501 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T22:33:15.501 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-11T22:33:15.754 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:33:16.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:15 smithi045 ceph-mon[39715]: pgmap v1215: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:33:16.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:15 smithi045 ceph-mon[39715]: from='client.16233 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:33:16.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:15 smithi045 ceph-mon[39715]: 3.6 scrub starts 2024-03-11T22:33:16.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:15 smithi045 ceph-mon[39715]: 3.6 scrub ok 2024-03-11T22:33:16.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:15 smithi045 ceph-mon[39715]: 3.e scrub starts 2024-03-11T22:33:16.037 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:15 smithi045 ceph-mon[39715]: 3.e scrub ok 2024-03-11T22:33:16.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:15 smithi045 ceph-mon[30178]: pgmap v1215: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:33:16.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:15 smithi045 ceph-mon[30178]: from='client.16233 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:33:16.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:15 smithi045 ceph-mon[30178]: 3.6 scrub starts 2024-03-11T22:33:16.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:15 smithi045 ceph-mon[30178]: 3.6 scrub ok 2024-03-11T22:33:16.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:15 smithi045 ceph-mon[30178]: 3.e scrub starts 2024-03-11T22:33:16.038 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:15 smithi045 ceph-mon[30178]: 3.e scrub ok 2024-03-11T22:33:16.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:15 smithi144 ceph-mon[33984]: pgmap v1215: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:33:16.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:15 smithi144 ceph-mon[33984]: from='client.16233 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:33:16.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:15 smithi144 ceph-mon[33984]: 3.6 scrub starts 2024-03-11T22:33:16.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:15 smithi144 ceph-mon[33984]: 3.6 scrub ok 2024-03-11T22:33:16.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:15 smithi144 ceph-mon[33984]: 3.e scrub starts 2024-03-11T22:33:16.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:15 smithi144 ceph-mon[33984]: 3.e scrub ok 2024-03-11T22:33:16.932 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:16 smithi045 ceph-mon[39715]: 6.11 scrub starts 2024-03-11T22:33:16.932 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:16 smithi045 ceph-mon[39715]: 6.11 scrub ok 2024-03-11T22:33:16.932 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:16 smithi045 ceph-mon[39715]: 3.1 deep-scrub starts 2024-03-11T22:33:16.932 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:16 smithi045 ceph-mon[39715]: 3.1 deep-scrub ok 2024-03-11T22:33:16.932 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:16 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:16.932 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:16 smithi045 ceph-mon[30178]: 6.11 scrub starts 2024-03-11T22:33:16.932 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:16 smithi045 ceph-mon[30178]: 6.11 scrub ok 2024-03-11T22:33:16.933 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:16 smithi045 ceph-mon[30178]: 3.1 deep-scrub starts 2024-03-11T22:33:16.933 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:16 smithi045 ceph-mon[30178]: 3.1 deep-scrub ok 2024-03-11T22:33:16.933 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:16 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:17.092 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:16 smithi144 ceph-mon[33984]: 6.11 scrub starts 2024-03-11T22:33:17.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:16 smithi144 ceph-mon[33984]: 6.11 scrub ok 2024-03-11T22:33:17.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:16 smithi144 ceph-mon[33984]: 3.1 deep-scrub starts 2024-03-11T22:33:17.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:16 smithi144 ceph-mon[33984]: 3.1 deep-scrub ok 2024-03-11T22:33:17.093 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:16 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:33:17.351 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:33:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug there is no tcmu-runner data available 2024-03-11T22:33:17.787 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:17] ENGINE Bus STOPPING 2024-03-11T22:33:17.787 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:33:17.787 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:17] ENGINE Bus STOPPED 2024-03-11T22:33:17.787 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:17] ENGINE Bus STARTING 2024-03-11T22:33:18.097 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:17] ENGINE Serving on http://:::9283 2024-03-11T22:33:18.097 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:17] ENGINE Bus STARTED 2024-03-11T22:33:18.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[39715]: pgmap v1216: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:33:18.097 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[39715]: 4.1f scrub starts 2024-03-11T22:33:18.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[39715]: 4.1f scrub ok 2024-03-11T22:33:18.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[39715]: 4.c scrub starts 2024-03-11T22:33:18.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[39715]: 4.c scrub ok 2024-03-11T22:33:18.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2355352326' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T22:33:18.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[39715]: from='client.? 172.21.15.45:0/2355352326' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-11T22:33:18.098 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:33:18.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[30178]: pgmap v1216: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:33:18.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[30178]: 4.1f scrub starts 2024-03-11T22:33:18.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[30178]: 4.1f scrub ok 2024-03-11T22:33:18.098 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[30178]: 4.c scrub starts 2024-03-11T22:33:18.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[30178]: 4.c scrub ok 2024-03-11T22:33:18.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2355352326' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T22:33:18.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[30178]: from='client.? 172.21.15.45:0/2355352326' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-11T22:33:18.099 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:17 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:33:18.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:17 smithi144 ceph-mon[33984]: pgmap v1216: 142 pgs: 142 active+clean; 582 KiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T22:33:18.100 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:17 smithi144 ceph-mon[33984]: 4.1f scrub starts 2024-03-11T22:33:18.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:17 smithi144 ceph-mon[33984]: 4.1f scrub ok 2024-03-11T22:33:18.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:17 smithi144 ceph-mon[33984]: 4.c scrub starts 2024-03-11T22:33:18.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:17 smithi144 ceph-mon[33984]: 4.c scrub ok 2024-03-11T22:33:18.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:17 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2355352326' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T22:33:18.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:17 smithi144 ceph-mon[33984]: from='client.? 172.21.15.45:0/2355352326' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-11T22:33:18.101 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:17 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:33:18.470 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T22:33:18.482 INFO:tasks.cephadm:Teardown begin 2024-03-11T22:33:18.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:33:18.542 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:33:18.600 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T22:33:18.600 DEBUG:teuthology.orchestra.run.smithi045:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:33:18.632 DEBUG:teuthology.orchestra.run.smithi144:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:33:18.649 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T22:33:18.649 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T22:33:18.649 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.a 2024-03-11T22:33:18.853 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:18 smithi144 ceph-mon[33984]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:33:18.854 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:18 smithi144 ceph-mon[33984]: 5.15 deep-scrub starts 2024-03-11T22:33:18.854 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:18 smithi144 ceph-mon[33984]: 5.15 deep-scrub ok 2024-03-11T22:33:18.854 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:18 smithi144 ceph-mon[33984]: 5.13 scrub starts 2024-03-11T22:33:18.854 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:18 smithi144 ceph-mon[33984]: 5.13 scrub ok 2024-03-11T22:33:18.854 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:18 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:33:18.854 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:18 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:33:18.854 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:18 smithi144 ceph-mon[33984]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:33:18.959 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[39715]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:33:18.959 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[39715]: 5.15 deep-scrub starts 2024-03-11T22:33:18.959 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[39715]: 5.15 deep-scrub ok 2024-03-11T22:33:18.959 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[39715]: 5.13 scrub starts 2024-03-11T22:33:18.959 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[39715]: 5.13 scrub ok 2024-03-11T22:33:18.959 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:33:18.959 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:33:18.959 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[39715]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:33:18.960 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:18 smithi045 systemd[1]: Stopping Ceph mon.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:33:18.960 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[30178]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T22:33:18.960 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[30178]: 5.15 deep-scrub starts 2024-03-11T22:33:18.960 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[30178]: 5.15 deep-scrub ok 2024-03-11T22:33:18.960 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[30178]: 5.13 scrub starts 2024-03-11T22:33:18.960 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[30178]: 5.13 scrub ok 2024-03-11T22:33:18.960 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:33:18.960 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:33:18.960 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:18 smithi045 ceph-mon[30178]: from='mgr.14565 172.21.15.45:0/472763283' entity='mgr.y' 2024-03-11T22:33:19.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a[30155]: 2024-03-11T22:33:19.073+0000 7f7c9e516700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:33:19.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a[30155]: 2024-03-11T22:33:19.073+0000 7f7c9e516700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T22:33:19.287 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:19 smithi045 podman[228569]: 2024-03-11 22:33:19.175652258 +0000 UTC m=+0.206532007 container died 8386a2a2886424ad728da9f8f0c981d7289bad54deb8f6d04e9573911b9361cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T22:33:19.287 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:19] ENGINE Bus STOPPING 2024-03-11T22:33:19.287 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:33:19.288 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:19] ENGINE Bus STOPPED 2024-03-11T22:33:19.288 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:19] ENGINE Bus STARTING 2024-03-11T22:33:19.288 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:19] ENGINE Serving on http://:::9283 2024-03-11T22:33:19.288 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:19 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:19] ENGINE Bus STARTED 2024-03-11T22:33:19.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:19 smithi045 podman[228569]: 2024-03-11 22:33:19.571379134 +0000 UTC m=+0.602258890 container cleanup 8386a2a2886424ad728da9f8f0c981d7289bad54deb8f6d04e9573911b9361cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-11T22:33:19.787 INFO:journalctl@ceph.mon.a.smithi045.stdout:Mar 11 22:33:19 smithi045 bash[228569]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-a 2024-03-11T22:33:19.975 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T22:33:20.032 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:33:20.032 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T22:33:20.033 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-11T22:33:20.033 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.c 2024-03-11T22:33:20.343 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:20 smithi045 systemd[1]: Stopping Ceph mon.c for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:33:20.787 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:20 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:20] ENGINE Bus STOPPING 2024-03-11T22:33:20.787 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:20 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:33:20.787 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:20 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:20] ENGINE Bus STOPPED 2024-03-11T22:33:20.787 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:20 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:20] ENGINE Bus STARTING 2024-03-11T22:33:20.787 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:20 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:20] ENGINE Serving on http://:::9283 2024-03-11T22:33:20.787 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:20 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: [11/Mar/2024:22:33:20] ENGINE Bus STARTED 2024-03-11T22:33:20.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:20 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-c[39692]: 2024-03-11T22:33:20.336+0000 7f4299905700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:33:20.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:20 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-c[39692]: 2024-03-11T22:33:20.336+0000 7f4299905700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T22:33:20.788 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:20 smithi045 podman[228715]: 2024-03-11 22:33:20.541874274 +0000 UTC m=+0.330365662 container died 28174753370cb1c38ce141290547f95bb7beb15efe5d84106e8696f50808903f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397) 2024-03-11T22:33:21.192 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:20 smithi045 podman[228715]: 2024-03-11 22:33:20.959699541 +0000 UTC m=+0.748190935 container cleanup 28174753370cb1c38ce141290547f95bb7beb15efe5d84106e8696f50808903f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-11T22:33:21.192 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:20 smithi045 bash[228715]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-c 2024-03-11T22:33:21.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T22:33:21.455 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:21 smithi045 podman[228745]: 2024-03-11 22:33:21.18621203 +0000 UTC m=+0.633615745 container remove 28174753370cb1c38ce141290547f95bb7beb15efe5d84106e8696f50808903f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-11T22:33:21.455 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:21 smithi045 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-11T22:33:21.455 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:21 smithi045 systemd[1]: Stopped Ceph mon.c for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:33:21.455 INFO:journalctl@ceph.mon.c.smithi045.stdout:Mar 11 22:33:21 smithi045 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.c.service: Consumed 23.198s CPU time. 2024-03-11T22:33:21.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:33:21.469 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-11T22:33:21.469 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-11T22:33:21.469 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.b 2024-03-11T22:33:21.850 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:21 smithi144 systemd[1]: Stopping Ceph mon.b for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:33:22.188 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:21 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-b[33961]: 2024-03-11T22:33:21.896+0000 7f7e961ab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:33:22.188 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:21 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-b[33961]: 2024-03-11T22:33:21.896+0000 7f7e961ab700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T22:33:22.188 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:22 smithi144 podman[162539]: 2024-03-11 22:33:22.182465457 +0000 UTC m=+0.418501065 container died 061bce8bb349978ea4d86db352a1757b4456c8edc34aa72cdb0ce79e893b847d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-b, RELEASE=squid-94a5397, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:33:22.851 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:22 smithi144 podman[162539]: 2024-03-11 22:33:22.694490396 +0000 UTC m=+0.930526039 container cleanup 061bce8bb349978ea4d86db352a1757b4456c8edc34aa72cdb0ce79e893b847d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-b, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, ceph=True) 2024-03-11T22:33:22.852 INFO:journalctl@ceph.mon.b.smithi144.stdout:Mar 11 22:33:22 smithi144 bash[162539]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mon-b 2024-03-11T22:33:23.037 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:22 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y[30436]: ::ffff:172.21.15.144 - - [11/Mar/2024:22:33:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:33:23.154 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T22:33:23.205 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:33:23.206 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-11T22:33:23.206 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T22:33:23.206 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mgr.y 2024-03-11T22:33:23.531 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:23 smithi045 systemd[1]: Stopping Ceph mgr.y for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:33:23.787 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:23 smithi045 podman[228858]: 2024-03-11 22:33:23.531455029 +0000 UTC m=+0.135649238 container died facc990c7c1b706faf5fd75546eda5969aa3204c32b4e9447d7e1f6d6a60668e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:33:23.851 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:33:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:33:23.717Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.45:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.45:8765: connect: connection refused" 2024-03-11T22:33:23.851 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:33:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:33:23.718Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.45:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.45:8765: connect: connection refused" 2024-03-11T22:33:23.851 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:33:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:33:23.722Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.45:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.45:8765: connect: connection refused" 2024-03-11T22:33:23.851 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:33:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:33:23.727Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.45:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.45:8765: connect: connection refused" 2024-03-11T22:33:24.280 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:24 smithi045 podman[228858]: 2024-03-11 22:33:24.014743178 +0000 UTC m=+0.618937428 container cleanup facc990c7c1b706faf5fd75546eda5969aa3204c32b4e9447d7e1f6d6a60668e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:33:24.280 INFO:journalctl@ceph.mgr.y.smithi045.stdout:Mar 11 22:33:24 smithi045 bash[228858]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-y 2024-03-11T22:33:24.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T22:33:24.477 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:33:24.477 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T22:33:24.477 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T22:33:24.478 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mgr.x 2024-03-11T22:33:24.829 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:33:24 smithi144 systemd[1]: Stopping Ceph mgr.x for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:33:25.101 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:33:24 smithi144 podman[162671]: 2024-03-11 22:33:24.823404107 +0000 UTC m=+0.161055271 container died 06fde1e7966f9ce709c1a16c4b306c8f0260420d51e20dc0c9648863328f6db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:33:25.524 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:33:25 smithi144 podman[162671]: 2024-03-11 22:33:25.241719809 +0000 UTC m=+0.579370979 container cleanup 06fde1e7966f9ce709c1a16c4b306c8f0260420d51e20dc0c9648863328f6db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-11T22:33:25.524 INFO:journalctl@ceph.mgr.x.smithi144.stdout:Mar 11 22:33:25 smithi144 bash[162671]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-mgr-x 2024-03-11T22:33:25.628 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T22:33:25.664 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:33:25.664 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T22:33:25.664 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T22:33:25.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.0 2024-03-11T22:33:26.037 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:25 smithi045 systemd[1]: Stopping Ceph osd.0 for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:33:26.537 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:26 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0[44097]: 2024-03-11T22:33:26.145+0000 7f0695c32700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:33:26.537 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:26 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0[44097]: 2024-03-11T22:33:26.145+0000 7f0695c32700 -1 osd.0 579 *** Got signal Terminated *** 2024-03-11T22:33:26.538 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:26 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0[44097]: 2024-03-11T22:33:26.145+0000 7f0695c32700 -1 osd.0 579 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:33:32.037 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:31 smithi045 podman[228990]: 2024-03-11 22:33:31.776878065 +0000 UTC m=+5.929780843 container died 5f9a2a0b2bbca015d43f4fd97dfd2c2a9c01ddd6a58b0c74fb585de1b318bb0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-03-11T22:33:32.518 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:32 smithi045 podman[228990]: 2024-03-11 22:33:32.235961297 +0000 UTC m=+6.388864114 container cleanup 5f9a2a0b2bbca015d43f4fd97dfd2c2a9c01ddd6a58b0c74fb585de1b318bb0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-11T22:33:32.518 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:32 smithi045 bash[228990]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0 2024-03-11T22:33:32.787 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:32 smithi045 podman[229012]: 2024-03-11 22:33:32.51267883 +0000 UTC m=+0.727804424 container remove 5f9a2a0b2bbca015d43f4fd97dfd2c2a9c01ddd6a58b0c74fb585de1b318bb0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-11T22:33:33.113 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:32 smithi045 podman[229109]: 2024-03-11 22:33:32.725688018 +0000 UTC m=+0.019372704 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:33:33.114 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:32 smithi045 podman[229109]: 2024-03-11 22:33:32.863692311 +0000 UTC m=+0.157376990 container create 3a38fadc5e65f5707a10959e340675b3240f3b9ed6a410c3fd41174ab6643a69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:33:33.537 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:33 smithi045 podman[229109]: 2024-03-11 22:33:33.107776095 +0000 UTC m=+0.401460773 container init 3a38fadc5e65f5707a10959e340675b3240f3b9ed6a410c3fd41174ab6643a69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T22:33:33.537 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:33 smithi045 podman[229109]: 2024-03-11 22:33:33.113740746 +0000 UTC m=+0.407425424 container start 3a38fadc5e65f5707a10959e340675b3240f3b9ed6a410c3fd41174ab6643a69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-11T22:33:33.537 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:33 smithi045 podman[229109]: 2024-03-11 22:33:33.274294377 +0000 UTC m=+0.567979075 container attach 3a38fadc5e65f5707a10959e340675b3240f3b9ed6a410c3fd41174ab6643a69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-11T22:33:34.892 INFO:journalctl@ceph.osd.0.smithi045.stdout:Mar 11 22:33:34 smithi045 podman[229109]: 2024-03-11 22:33:34.605205298 +0000 UTC m=+1.898889983 container died 3a38fadc5e65f5707a10959e340675b3240f3b9ed6a410c3fd41174ab6643a69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-11T22:33:35.188 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T22:33:35.232 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:33:35.232 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T22:33:35.232 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T22:33:35.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.1 2024-03-11T22:33:35.537 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:35 smithi045 systemd[1]: Stopping Ceph osd.1 for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:33:36.037 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:35 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1[50343]: 2024-03-11T22:33:35.542+0000 7ff2d746e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:33:36.037 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:35 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1[50343]: 2024-03-11T22:33:35.542+0000 7ff2d746e700 -1 osd.1 579 *** Got signal Terminated *** 2024-03-11T22:33:36.037 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:35 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1[50343]: 2024-03-11T22:33:35.542+0000 7ff2d746e700 -1 osd.1 579 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:33:41.521 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:41 smithi045 podman[229241]: 2024-03-11 22:33:41.165220571 +0000 UTC m=+5.717414380 container died da310046d383989904d7fa316430f9edb4f6ab04854e83a25bdd2d47ee0f933a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T22:33:41.787 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:41 smithi045 podman[229241]: 2024-03-11 22:33:41.714238599 +0000 UTC m=+6.266432413 container cleanup da310046d383989904d7fa316430f9edb4f6ab04854e83a25bdd2d47ee0f933a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-94a5397, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T22:33:41.787 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:41 smithi045 bash[229241]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1 2024-03-11T22:33:42.287 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:41 smithi045 podman[229264]: 2024-03-11 22:33:41.924128015 +0000 UTC m=+0.745170628 container remove da310046d383989904d7fa316430f9edb4f6ab04854e83a25bdd2d47ee0f933a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-03-11T22:33:42.600 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:42 smithi045 podman[229360]: 2024-03-11 22:33:42.243005004 +0000 UTC m=+0.017780118 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:33:42.600 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:42 smithi045 podman[229360]: 2024-03-11 22:33:42.349952756 +0000 UTC m=+0.124727911 container create 094d383370854841e26c5e94fa5e7ee0a2c070819be3aa3fa65f199e4aca44f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-11T22:33:42.907 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:42 smithi045 podman[229360]: 2024-03-11 22:33:42.594303186 +0000 UTC m=+0.369078298 container init 094d383370854841e26c5e94fa5e7ee0a2c070819be3aa3fa65f199e4aca44f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-11T22:33:42.907 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:42 smithi045 podman[229360]: 2024-03-11 22:33:42.600083448 +0000 UTC m=+0.374858563 container start 094d383370854841e26c5e94fa5e7ee0a2c070819be3aa3fa65f199e4aca44f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-11T22:33:42.907 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:42 smithi045 podman[229360]: 2024-03-11 22:33:42.685761911 +0000 UTC m=+0.460537024 container attach 094d383370854841e26c5e94fa5e7ee0a2c070819be3aa3fa65f199e4aca44f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:33:43.194 INFO:journalctl@ceph.osd.1.smithi045.stdout:Mar 11 22:33:42 smithi045 podman[229360]: 2024-03-11 22:33:42.9013058 +0000 UTC m=+0.676080908 container died 094d383370854841e26c5e94fa5e7ee0a2c070819be3aa3fa65f199e4aca44f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:33:43.576 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T22:33:43.618 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:33:43.619 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T22:33:43.619 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T22:33:43.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.2 2024-03-11T22:33:43.976 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:43 smithi045 systemd[1]: Stopping Ceph osd.2 for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:33:44.287 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:43 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2[56488]: 2024-03-11T22:33:43.969+0000 7fbcad829700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:33:44.287 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:43 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2[56488]: 2024-03-11T22:33:43.969+0000 7fbcad829700 -1 osd.2 579 *** Got signal Terminated *** 2024-03-11T22:33:44.287 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:43 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2[56488]: 2024-03-11T22:33:43.969+0000 7fbcad829700 -1 osd.2 579 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:33:49.674 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:49 smithi045 podman[229492]: 2024-03-11 22:33:49.388326706 +0000 UTC m=+5.544800332 container died 9864fac4c518e565863d20a2bfdd4f111dd008a04f560a4d040e807fa501860f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T22:33:50.037 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:49 smithi045 podman[229492]: 2024-03-11 22:33:49.850179685 +0000 UTC m=+6.006653314 container cleanup 9864fac4c518e565863d20a2bfdd4f111dd008a04f560a4d040e807fa501860f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T22:33:50.037 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:49 smithi045 bash[229492]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2 2024-03-11T22:33:50.480 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:50 smithi045 podman[229512]: 2024-03-11 22:33:50.160517441 +0000 UTC m=+0.773811527 container remove 9864fac4c518e565863d20a2bfdd4f111dd008a04f560a4d040e807fa501860f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-11T22:33:50.778 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:50 smithi045 podman[229609]: 2024-03-11 22:33:50.378004842 +0000 UTC m=+0.017941487 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:33:50.778 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:50 smithi045 podman[229609]: 2024-03-11 22:33:50.527533189 +0000 UTC m=+0.167469865 container create fdbfcf905babe7aad716317f3cf7040158d2e40a82e425227f96dfe0b709f2c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2-deactivate, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-11T22:33:51.037 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:50 smithi045 podman[229609]: 2024-03-11 22:33:50.772204902 +0000 UTC m=+0.412141536 container init fdbfcf905babe7aad716317f3cf7040158d2e40a82e425227f96dfe0b709f2c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:33:51.037 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:50 smithi045 podman[229609]: 2024-03-11 22:33:50.778036466 +0000 UTC m=+0.417973103 container start fdbfcf905babe7aad716317f3cf7040158d2e40a82e425227f96dfe0b709f2c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.license=GPLv2) 2024-03-11T22:33:51.037 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:50 smithi045 podman[229609]: 2024-03-11 22:33:50.88060203 +0000 UTC m=+0.520538674 container attach fdbfcf905babe7aad716317f3cf7040158d2e40a82e425227f96dfe0b709f2c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-11T22:33:51.374 INFO:journalctl@ceph.osd.2.smithi045.stdout:Mar 11 22:33:51 smithi045 podman[229609]: 2024-03-11 22:33:51.08629796 +0000 UTC m=+0.726234585 container died fdbfcf905babe7aad716317f3cf7040158d2e40a82e425227f96dfe0b709f2c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:33:51.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T22:33:51.800 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:33:51.800 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T22:33:51.800 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-11T22:33:51.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.3 2024-03-11T22:33:52.037 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:51 smithi045 systemd[1]: Stopping Ceph osd.3 for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:33:52.537 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-3[63074]: 2024-03-11T22:33:52.141+0000 7fac1f807700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:33:52.538 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-3[63074]: 2024-03-11T22:33:52.141+0000 7fac1f807700 -1 osd.3 579 *** Got signal Terminated *** 2024-03-11T22:33:52.538 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:52 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-3[63074]: 2024-03-11T22:33:52.141+0000 7fac1f807700 -1 osd.3 579 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:33:53.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:33:53 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:33:53.523+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:33:54.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:33:54 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:33:54.482+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:33:55.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:33:55 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:33:55.516+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:33:56.787 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:56 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-3[63074]: 2024-03-11T22:33:56.401+0000 7fac1bdfb700 -1 osd.3 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:30.180711+0000 front 2024-03-11T22:33:30.180663+0000 (oldest deadline 2024-03-11T22:33:56.080813+0000) 2024-03-11T22:33:56.848 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:33:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:33:56.487+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:33:57.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:33:56 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:33:56.842+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:33:57.822 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:33:57 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:33:57.529+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:33:57.823 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:33:57 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:33:57.675+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:33:58.095 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:57 smithi045 podman[229749]: 2024-03-11 22:33:57.807374934 +0000 UTC m=+5.780247062 container died 10c6ca2d39ebc41d3267060cc17beda6688fa5ee05dcc69c4af5574db91fed89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-3, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-11T22:33:58.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:33:57 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:33:57.815+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:33:58.101 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:33:57 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:33:57.841+0000 7fe88fbb5700 -1 osd.4 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.841107+0000 front 2024-03-11T22:33:28.841055+0000 (oldest deadline 2024-03-11T22:33:57.041552+0000) 2024-03-11T22:33:58.537 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:58 smithi045 podman[229749]: 2024-03-11 22:33:58.271765298 +0000 UTC m=+6.244637433 container cleanup 10c6ca2d39ebc41d3267060cc17beda6688fa5ee05dcc69c4af5574db91fed89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-3, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-11T22:33:58.537 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:58 smithi045 bash[229749]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-3 2024-03-11T22:33:58.833 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:33:58 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:33:58.533+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:33:58.834 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:33:58 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:33:58.684+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:33:58.897 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:58 smithi045 podman[229772]: 2024-03-11 22:33:58.547926816 +0000 UTC m=+0.735745067 container remove 10c6ca2d39ebc41d3267060cc17beda6688fa5ee05dcc69c4af5574db91fed89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-3, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:33:59.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:33:58 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:33:58.827+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:33:59.101 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:33:58 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:33:58.834+0000 7fe88fbb5700 -1 osd.4 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.841107+0000 front 2024-03-11T22:33:28.841055+0000 (oldest deadline 2024-03-11T22:33:57.041552+0000) 2024-03-11T22:33:59.181 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:58 smithi045 podman[229870]: 2024-03-11 22:33:58.794429172 +0000 UTC m=+0.017509076 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:33:59.181 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:58 smithi045 podman[229870]: 2024-03-11 22:33:58.898730752 +0000 UTC m=+0.121810694 container create 806228278c561296947d0ef4e45a01649ae602089b6a2ad23ba03f86fe84e8e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-03-11T22:33:59.181 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:59 smithi045 podman[229870]: 2024-03-11 22:33:59.093240057 +0000 UTC m=+0.316319954 container init 806228278c561296947d0ef4e45a01649ae602089b6a2ad23ba03f86fe84e8e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-11T22:33:59.181 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:59 smithi045 podman[229870]: 2024-03-11 22:33:59.098885078 +0000 UTC m=+0.321964976 container start 806228278c561296947d0ef4e45a01649ae602089b6a2ad23ba03f86fe84e8e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-3-deactivate, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:33:59.181 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:59 smithi045 podman[229870]: 2024-03-11 22:33:59.175538765 +0000 UTC m=+0.398618676 container attach 806228278c561296947d0ef4e45a01649ae602089b6a2ad23ba03f86fe84e8e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397) 2024-03-11T22:33:59.537 INFO:journalctl@ceph.osd.3.smithi045.stdout:Mar 11 22:33:59 smithi045 podman[229870]: 2024-03-11 22:33:59.397341569 +0000 UTC m=+0.620421457 container died 806228278c561296947d0ef4e45a01649ae602089b6a2ad23ba03f86fe84e8e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T22:33:59.840 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:33:59 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:33:59.568+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:33:59.840 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:33:59 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:33:59.711+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:33:59.840 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:33:59 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:33:59.799+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:00.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.3.service' 2024-03-11T22:34:00.101 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:33:59 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:33:59.834+0000 7fe88fbb5700 -1 osd.4 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.841107+0000 front 2024-03-11T22:33:28.841055+0000 (oldest deadline 2024-03-11T22:33:57.041552+0000) 2024-03-11T22:34:00.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:34:00.118 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-11T22:34:00.119 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-11T22:34:00.119 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.4 2024-03-11T22:34:00.451 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:00 smithi144 systemd[1]: Stopping Ceph osd.4 for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:34:00.749 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:00 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:00.535+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:00.749 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:00 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:34:00.445+0000 7fe8975c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:34:00.749 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:00 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:34:00.445+0000 7fe8975c9700 -1 osd.4 579 *** Got signal Terminated *** 2024-03-11T22:34:00.749 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:00 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:34:00.445+0000 7fe8975c9700 -1 osd.4 579 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:34:01.100 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:00 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:00.742+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:01.101 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:00 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:34:00.793+0000 7fe88fbb5700 -1 osd.4 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.841107+0000 front 2024-03-11T22:33:28.841055+0000 (oldest deadline 2024-03-11T22:33:57.041552+0000) 2024-03-11T22:34:01.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:00 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:00.762+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:01.786 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:01 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:01.522+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:02.101 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:01 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:34:01.780+0000 7fe88fbb5700 -1 osd.4 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.841107+0000 front 2024-03-11T22:33:28.841055+0000 (oldest deadline 2024-03-11T22:33:57.041552+0000) 2024-03-11T22:34:02.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:01 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:01.785+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:02.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:01 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:01.812+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:02.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:02 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:02.551+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:02.851 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:02 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:02.793+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:02.851 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:02 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:34:02.735+0000 7fe88fbb5700 -1 osd.4 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.841107+0000 front 2024-03-11T22:33:28.841055+0000 (oldest deadline 2024-03-11T22:33:57.041552+0000) 2024-03-11T22:34:03.351 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:02 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:02.856+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:03.835 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:03 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:03.571+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:03.835 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:03 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:03.817+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:03.835 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:03 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:34:03.717+0000 7fe88fbb5700 -1 osd.4 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.841107+0000 front 2024-03-11T22:33:28.841055+0000 (oldest deadline 2024-03-11T22:33:57.041552+0000) 2024-03-11T22:34:03.835 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:03 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:34:03.717+0000 7fe88fbb5700 -1 osd.4 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.142746+0000 front 2024-03-11T22:33:38.142655+0000 (oldest deadline 2024-03-11T22:34:02.742998+0000) 2024-03-11T22:34:04.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:03 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:03.828+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:04.847 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:04 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:04.577+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:04.847 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:04 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:34:04.719+0000 7fe88fbb5700 -1 osd.4 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.841107+0000 front 2024-03-11T22:33:28.841055+0000 (oldest deadline 2024-03-11T22:33:57.041552+0000) 2024-03-11T22:34:04.847 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:04 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4[39526]: 2024-03-11T22:34:04.719+0000 7fe88fbb5700 -1 osd.4 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.142746+0000 front 2024-03-11T22:33:38.142655+0000 (oldest deadline 2024-03-11T22:34:02.742998+0000) 2024-03-11T22:34:05.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:04 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:04.840+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:05.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:04 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:04.840+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:05.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:04 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:04.842+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:05.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:04 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:04.842+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:05.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:05.592+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:05.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:05.810+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:05.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:05.810+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:06.351 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:05.892+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:06.351 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:05 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:05.892+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:06.351 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:06 smithi144 podman[162819]: 2024-03-11 22:34:06.069954811 +0000 UTC m=+5.754900092 container died 9bdd4dd809cd23d1b8a4bfbe6d00a9d8fc7dd51268a7d15275478d3b0c7d33cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:34:06.708 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:06 smithi144 podman[162819]: 2024-03-11 22:34:06.434666963 +0000 UTC m=+6.119612243 container cleanup 9bdd4dd809cd23d1b8a4bfbe6d00a9d8fc7dd51268a7d15275478d3b0c7d33cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=) 2024-03-11T22:34:06.708 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:06 smithi144 bash[162819]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4 2024-03-11T22:34:06.709 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:06.592+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:07.020 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:06.940+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:07.020 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:06.940+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:07.020 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:06 smithi144 podman[162843]: 2024-03-11 22:34:06.702576024 +0000 UTC m=+0.634823866 container remove 9bdd4dd809cd23d1b8a4bfbe6d00a9d8fc7dd51268a7d15275478d3b0c7d33cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-11T22:34:07.020 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:06.770+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:07.020 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:06 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:06.770+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:07.312 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:07 smithi144 podman[162937]: 2024-03-11 22:34:06.917458861 +0000 UTC m=+0.018155305 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:34:07.313 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:07 smithi144 podman[162937]: 2024-03-11 22:34:07.046661766 +0000 UTC m=+0.147358237 container create 86ad21e7b7a526acf02c3c8bbf36e78fdc140064c9fb5bba394513e75592b847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:34:07.313 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:07 smithi144 podman[162937]: 2024-03-11 22:34:07.306730733 +0000 UTC m=+0.407427162 container init 86ad21e7b7a526acf02c3c8bbf36e78fdc140064c9fb5bba394513e75592b847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-11T22:34:07.596 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:07 smithi144 podman[162937]: 2024-03-11 22:34:07.312265543 +0000 UTC m=+0.412961968 container start 86ad21e7b7a526acf02c3c8bbf36e78fdc140064c9fb5bba394513e75592b847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T22:34:07.597 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:07 smithi144 podman[162937]: 2024-03-11 22:34:07.423143246 +0000 UTC m=+0.523839690 container attach 86ad21e7b7a526acf02c3c8bbf36e78fdc140064c9fb5bba394513e75592b847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T22:34:07.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:07 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:07.589+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:07.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:07 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:07.589+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.596976+0000 front 2024-03-11T22:33:38.596987+0000 (oldest deadline 2024-03-11T22:34:06.797503+0000) 2024-03-11T22:34:07.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:07 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:07.734+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:07.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:07 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:07.734+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:08.351 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:07 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:07.894+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:08.351 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:07 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:07.894+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:08.851 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:08 smithi144 podman[162937]: 2024-03-11 22:34:08.58798116 +0000 UTC m=+1.688677590 container died 86ad21e7b7a526acf02c3c8bbf36e78fdc140064c9fb5bba394513e75592b847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397) 2024-03-11T22:34:08.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:08 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:08.608+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:08.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:08 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:08.608+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.596976+0000 front 2024-03-11T22:33:38.596987+0000 (oldest deadline 2024-03-11T22:34:06.797503+0000) 2024-03-11T22:34:08.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:08 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:08.773+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:08.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:08 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:08.773+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:09.145 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:08 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:08.931+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:09.145 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:08 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:08.931+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:09.145 INFO:journalctl@ceph.osd.4.smithi144.stdout:Mar 11 22:34:09 smithi144 podman[162983]: 2024-03-11 22:34:09.118136739 +0000 UTC m=+0.520945219 container remove 86ad21e7b7a526acf02c3c8bbf36e78fdc140064c9fb5bba394513e75592b847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T22:34:09.150 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.4.service' 2024-03-11T22:34:09.191 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:34:09.191 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-11T22:34:09.191 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-11T22:34:09.191 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.5 2024-03-11T22:34:09.521 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:09 smithi144 systemd[1]: Stopping Ceph osd.5 for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:34:09.522 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:09 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:09.515+0000 7f8129483700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:34:09.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:09 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:09.756+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:09.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:09 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:09.756+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:09.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:09 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:09.515+0000 7f8129483700 -1 osd.5 579 *** Got signal Terminated *** 2024-03-11T22:34:09.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:09 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:09.515+0000 7f8129483700 -1 osd.5 579 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:34:09.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:09 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:09.573+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:09.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:09 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:09.573+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.596976+0000 front 2024-03-11T22:33:38.596987+0000 (oldest deadline 2024-03-11T22:34:06.797503+0000) 2024-03-11T22:34:10.351 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:09 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:09.927+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:10.351 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:09 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:09.927+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:10.942 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:10 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:10.612+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:10.942 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:10 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:10.612+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.596976+0000 front 2024-03-11T22:33:38.596987+0000 (oldest deadline 2024-03-11T22:34:06.797503+0000) 2024-03-11T22:34:10.942 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:10 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:10.750+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:10.942 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:10 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:10.750+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:10.942 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:10 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:10.750+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:11.351 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:10 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:10.935+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:11.351 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:10 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:10.935+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:11.917 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:11 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:11.658+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:11.917 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:11 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:11.658+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.596976+0000 front 2024-03-11T22:33:38.596987+0000 (oldest deadline 2024-03-11T22:34:06.797503+0000) 2024-03-11T22:34:11.917 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:11 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:11.751+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:11.917 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:11 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:11.751+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:11.917 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:11 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:11.751+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:12.351 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:11 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:11.911+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:12.351 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:11 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:11.911+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:12.351 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:11 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:11.911+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.258610+0000 front 2024-03-11T22:33:47.258536+0000 (oldest deadline 2024-03-11T22:34:11.358468+0000) 2024-03-11T22:34:13.101 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:12 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:12.669+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:13.101 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:12 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:12.670+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.596976+0000 front 2024-03-11T22:33:38.596987+0000 (oldest deadline 2024-03-11T22:34:06.797503+0000) 2024-03-11T22:34:13.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:12 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:12.880+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:13.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:12 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:12.880+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:13.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:12 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:12.880+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.258610+0000 front 2024-03-11T22:33:47.258536+0000 (oldest deadline 2024-03-11T22:34:11.358468+0000) 2024-03-11T22:34:13.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:12 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:12.784+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:13.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:12 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:12.784+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:13.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:12 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:12.784+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:14.101 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:13.670+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:28.596352+0000 front 2024-03-11T22:33:28.596281+0000 (oldest deadline 2024-03-11T22:33:53.296091+0000) 2024-03-11T22:34:14.101 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:13.670+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.596976+0000 front 2024-03-11T22:33:38.596987+0000 (oldest deadline 2024-03-11T22:34:06.797503+0000) 2024-03-11T22:34:14.101 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5[45689]: 2024-03-11T22:34:13.670+0000 7f8125a77700 -1 osd.5 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:46.798092+0000 front 2024-03-11T22:33:46.798143+0000 (oldest deadline 2024-03-11T22:34:12.698046+0000) 2024-03-11T22:34:14.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:13.851+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:14.102 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:13.851+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:14.102 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:13.851+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.258610+0000 front 2024-03-11T22:33:47.258536+0000 (oldest deadline 2024-03-11T22:34:11.358468+0000) 2024-03-11T22:34:14.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:13.791+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:14.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:13.791+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:14.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:13 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:13.791+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:15.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:14 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:14.854+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:15.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:14 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:14.854+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:15.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:14 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:14.854+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.258610+0000 front 2024-03-11T22:33:47.258536+0000 (oldest deadline 2024-03-11T22:34:11.358468+0000) 2024-03-11T22:34:15.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:14 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:14.779+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:15.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:14 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:14.779+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:15.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:14 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:14.779+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:15.486 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:15 smithi144 podman[163066]: 2024-03-11 22:34:15.171701927 +0000 UTC m=+5.764172596 container died c12a34e0bc52520c066212259e706704d61758ddcf633f4c09db62c69eb9812f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T22:34:15.746 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:15 smithi144 podman[163066]: 2024-03-11 22:34:15.579525026 +0000 UTC m=+6.171995749 container cleanup c12a34e0bc52520c066212259e706704d61758ddcf633f4c09db62c69eb9812f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-11T22:34:15.746 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:15 smithi144 bash[163066]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5 2024-03-11T22:34:16.101 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:15 smithi144 podman[163089]: 2024-03-11 22:34:15.788982162 +0000 UTC m=+0.616201175 container remove c12a34e0bc52520c066212259e706704d61758ddcf633f4c09db62c69eb9812f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T22:34:16.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:15 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:15.900+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:16.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:15 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:15.900+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:16.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:15 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:15.900+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.258610+0000 front 2024-03-11T22:33:47.258536+0000 (oldest deadline 2024-03-11T22:34:11.358468+0000) 2024-03-11T22:34:16.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:15 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:15.739+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:16.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:15 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:15.739+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:16.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:15 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:15.739+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:16.481 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:16 smithi144 podman[163188]: 2024-03-11 22:34:16.025457549 +0000 UTC m=+0.018780398 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:34:16.482 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:16 smithi144 podman[163188]: 2024-03-11 22:34:16.156849164 +0000 UTC m=+0.150172056 container create eb2de857b0d1584f680814c07ed5441db9d1c2ddc2ced21bc2950387868196d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-11T22:34:16.482 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:16 smithi144 podman[163188]: 2024-03-11 22:34:16.351501745 +0000 UTC m=+0.344824595 container init eb2de857b0d1584f680814c07ed5441db9d1c2ddc2ced21bc2950387868196d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:34:16.482 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:16 smithi144 podman[163188]: 2024-03-11 22:34:16.357178272 +0000 UTC m=+0.350501121 container start eb2de857b0d1584f680814c07ed5441db9d1c2ddc2ced21bc2950387868196d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:34:16.482 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:16 smithi144 podman[163188]: 2024-03-11 22:34:16.475544383 +0000 UTC m=+0.468867232 container attach eb2de857b0d1584f680814c07ed5441db9d1c2ddc2ced21bc2950387868196d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-11T22:34:16.851 INFO:journalctl@ceph.osd.5.smithi144.stdout:Mar 11 22:34:16 smithi144 podman[163188]: 2024-03-11 22:34:16.648037832 +0000 UTC m=+0.641360684 container died eb2de857b0d1584f680814c07ed5441db9d1c2ddc2ced21bc2950387868196d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-11T22:34:16.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:16.694+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:16.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:16.694+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:16.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:16.694+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:17.243 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:16.942+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:17.244 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:16.942+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:17.244 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:16 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:16.942+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.258610+0000 front 2024-03-11T22:33:47.258536+0000 (oldest deadline 2024-03-11T22:34:11.358468+0000) 2024-03-11T22:34:17.258 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.5.service' 2024-03-11T22:34:17.299 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:34:17.299 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-11T22:34:17.299 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-11T22:34:17.300 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.6 2024-03-11T22:34:17.601 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:17 smithi144 systemd[1]: Stopping Ceph osd.6 for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:34:17.919 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:17.741+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:17.919 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:17.741+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:17.919 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:17.741+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:17.920 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:17.643+0000 7f818734c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:34:17.920 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:17.643+0000 7f818734c700 -1 osd.6 579 *** Got signal Terminated *** 2024-03-11T22:34:17.920 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:17.643+0000 7f818734c700 -1 osd.6 579 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:34:17.920 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:17.912+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:17.920 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:17.912+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:17.920 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:17 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:17.912+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.258610+0000 front 2024-03-11T22:33:47.258536+0000 (oldest deadline 2024-03-11T22:34:11.358468+0000) 2024-03-11T22:34:19.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:18.743+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:19.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:18.743+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:19.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:18.743+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:19.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:18.939+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:19.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:18.939+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:19.102 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:18 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:18.939+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.258610+0000 front 2024-03-11T22:33:47.258536+0000 (oldest deadline 2024-03-11T22:34:11.358468+0000) 2024-03-11T22:34:20.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:19 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:19.909+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:20.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:19 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:19.909+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:20.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:19 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:19.909+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.258610+0000 front 2024-03-11T22:33:47.258536+0000 (oldest deadline 2024-03-11T22:34:11.358468+0000) 2024-03-11T22:34:20.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:19 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:19.731+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:20.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:19 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:19.731+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:20.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:19 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:19.731+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:20.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:19 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:19.731+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-03-11T22:33:55.874467+0000 front 2024-03-11T22:33:55.874525+0000 (oldest deadline 2024-03-11T22:34:19.374355+0000) 2024-03-11T22:34:21.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:20 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:20.899+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:21.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:20 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:20.899+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:21.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:20 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:20.899+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.258610+0000 front 2024-03-11T22:33:47.258536+0000 (oldest deadline 2024-03-11T22:34:11.358468+0000) 2024-03-11T22:34:21.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:20 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:20.764+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:21.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:20 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:20.764+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:21.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:20 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:20.764+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:21.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:20 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:20.764+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-03-11T22:33:55.874467+0000 front 2024-03-11T22:33:55.874525+0000 (oldest deadline 2024-03-11T22:34:19.374355+0000) 2024-03-11T22:34:22.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:21 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:21.865+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.555691+0000 front 2024-03-11T22:33:27.555591+0000 (oldest deadline 2024-03-11T22:33:57.555919+0000) 2024-03-11T22:34:22.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:21 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:21.865+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:38.056894+0000 front 2024-03-11T22:33:38.056756+0000 (oldest deadline 2024-03-11T22:34:03.857153+0000) 2024-03-11T22:34:22.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:21 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:21.865+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.258610+0000 front 2024-03-11T22:33:47.258536+0000 (oldest deadline 2024-03-11T22:34:11.358468+0000) 2024-03-11T22:34:22.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:21 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6[51605]: 2024-03-11T22:34:21.865+0000 7f8183940700 -1 osd.6 579 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-03-11T22:33:51.859365+0000 front 2024-03-11T22:33:51.859275+0000 (oldest deadline 2024-03-11T22:34:21.259780+0000) 2024-03-11T22:34:22.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:21 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:21.725+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:22.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:21 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:21.725+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:22.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:21 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:21.725+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:22.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:21 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:21.725+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-03-11T22:33:55.874467+0000 front 2024-03-11T22:33:55.874525+0000 (oldest deadline 2024-03-11T22:34:19.374355+0000) 2024-03-11T22:34:23.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:22.676+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:23.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:22.676+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:23.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:22.676+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:23.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:22 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:22.676+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-03-11T22:33:55.874467+0000 front 2024-03-11T22:33:55.874525+0000 (oldest deadline 2024-03-11T22:34:19.374355+0000) 2024-03-11T22:34:23.456 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:23 smithi144 podman[163321]: 2024-03-11 22:34:23.170691925 +0000 UTC m=+5.655084024 container died 24d14a40cee8e8f84c328815feb8bd4e6c7e58387e5ab9505b2a9943856b75c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T22:34:23.707 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:23 smithi144 podman[163321]: 2024-03-11 22:34:23.548905232 +0000 UTC m=+6.033297331 container cleanup 24d14a40cee8e8f84c328815feb8bd4e6c7e58387e5ab9505b2a9943856b75c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-11T22:34:23.707 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:23 smithi144 bash[163321]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6 2024-03-11T22:34:24.101 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:23 smithi144 podman[163343]: 2024-03-11 22:34:23.808972213 +0000 UTC m=+0.639681407 container remove 24d14a40cee8e8f84c328815feb8bd4e6c7e58387e5ab9505b2a9943856b75c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-11T22:34:24.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:23.699+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:24.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:23.699+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:24.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:23.699+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:24.102 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:23.699+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-03-11T22:33:55.874467+0000 front 2024-03-11T22:33:55.874525+0000 (oldest deadline 2024-03-11T22:34:19.374355+0000) 2024-03-11T22:34:24.102 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:23.716Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.45:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.45:8765: connect: connection refused" 2024-03-11T22:34:24.102 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:23.718Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.45:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.45:8765: connect: connection refused" 2024-03-11T22:34:24.102 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:23.722Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.45:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.45:8765: connect: connection refused" 2024-03-11T22:34:24.102 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:23 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:23.727Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.45:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.45:8765: connect: connection refused" 2024-03-11T22:34:24.366 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:24 smithi144 podman[163442]: 2024-03-11 22:34:24.016147984 +0000 UTC m=+0.026389634 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:34:24.366 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:24 smithi144 podman[163442]: 2024-03-11 22:34:24.142786802 +0000 UTC m=+0.153028439 container create 8adedb2234510ee578d4ad234660c87dfcea0c9749aede988a1c6203553df5a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T22:34:24.366 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:24 smithi144 podman[163442]: 2024-03-11 22:34:24.353750716 +0000 UTC m=+0.363992353 container init 8adedb2234510ee578d4ad234660c87dfcea0c9749aede988a1c6203553df5a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T22:34:24.366 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:24 smithi144 podman[163442]: 2024-03-11 22:34:24.3598268 +0000 UTC m=+0.370068439 container start 8adedb2234510ee578d4ad234660c87dfcea0c9749aede988a1c6203553df5a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-11T22:34:24.651 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:24 smithi144 podman[163442]: 2024-03-11 22:34:24.437004421 +0000 UTC m=+0.447246075 container attach 8adedb2234510ee578d4ad234660c87dfcea0c9749aede988a1c6203553df5a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:34:24.962 INFO:journalctl@ceph.osd.6.smithi144.stdout:Mar 11 22:34:24 smithi144 podman[163442]: 2024-03-11 22:34:24.646057117 +0000 UTC m=+0.656298757 container died 8adedb2234510ee578d4ad234660c87dfcea0c9749aede988a1c6203553df5a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T22:34:24.962 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:24 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:24.654+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:24.962 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:24 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:24.654+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:24.962 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:24 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:24.654+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:24.963 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:24 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:24.654+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-03-11T22:33:55.874467+0000 front 2024-03-11T22:33:55.874525+0000 (oldest deadline 2024-03-11T22:34:19.374355+0000) 2024-03-11T22:34:25.257 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.6.service' 2024-03-11T22:34:25.303 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:34:25.303 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-11T22:34:25.304 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-11T22:34:25.304 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.7 2024-03-11T22:34:25.551 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:25 smithi144 systemd[1]: Stopping Ceph osd.7 for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:34:25.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:25.638+0000 7f29aef13700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:34:25.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:25.638+0000 7f29aef13700 -1 osd.7 579 *** Got signal Terminated *** 2024-03-11T22:34:25.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:25.638+0000 7f29aef13700 -1 osd.7 579 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:34:25.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:25.644+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:25.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:25.644+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:25.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:25.644+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:25.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:25 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:25.644+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-03-11T22:33:55.874467+0000 front 2024-03-11T22:33:55.874525+0000 (oldest deadline 2024-03-11T22:34:19.374355+0000) 2024-03-11T22:34:26.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:26.609+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:26.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:26.609+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:26.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:26.609+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:26.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:26 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:26.609+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-03-11T22:33:55.874467+0000 front 2024-03-11T22:33:55.874525+0000 (oldest deadline 2024-03-11T22:34:19.374355+0000) 2024-03-11T22:34:27.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:27 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:27.569+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:27.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:27 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:27.569+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:27.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:27 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:27.569+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:27.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:27 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:27.569+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-03-11T22:33:55.874467+0000 front 2024-03-11T22:33:55.874525+0000 (oldest deadline 2024-03-11T22:34:19.374355+0000) 2024-03-11T22:34:28.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:28 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:28.554+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:28.852 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:28 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:28.554+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:28.852 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:28 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:28.554+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:28.852 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:28 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:28.554+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-03-11T22:33:55.874467+0000 front 2024-03-11T22:33:55.874525+0000 (oldest deadline 2024-03-11T22:34:19.374355+0000) 2024-03-11T22:34:29.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:29 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:29.594+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:29.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:29 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:29.594+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:29.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:29 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:29.594+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:29.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:29 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:29.594+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-03-11T22:33:55.874467+0000 front 2024-03-11T22:33:55.874525+0000 (oldest deadline 2024-03-11T22:34:19.374355+0000) 2024-03-11T22:34:29.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:29 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:29.594+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.144:6804 osd.4 since back 2024-03-11T22:34:05.076422+0000 front 2024-03-11T22:34:05.076320+0000 (oldest deadline 2024-03-11T22:34:28.576290+0000) 2024-03-11T22:34:30.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:30 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:30.555+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-03-11T22:33:27.270669+0000 front 2024-03-11T22:33:27.270579+0000 (oldest deadline 2024-03-11T22:33:56.071073+0000) 2024-03-11T22:34:30.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:30 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:30.555+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-03-11T22:33:40.072694+0000 front 2024-03-11T22:33:40.072681+0000 (oldest deadline 2024-03-11T22:34:04.772563+0000) 2024-03-11T22:34:30.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:30 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:30.555+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-03-11T22:33:47.673602+0000 front 2024-03-11T22:33:47.673676+0000 (oldest deadline 2024-03-11T22:34:09.973579+0000) 2024-03-11T22:34:30.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:30 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:30.555+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-03-11T22:33:55.874467+0000 front 2024-03-11T22:33:55.874525+0000 (oldest deadline 2024-03-11T22:34:19.374355+0000) 2024-03-11T22:34:30.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:30 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7[58138]: 2024-03-11T22:34:30.555+0000 7f29ab507700 -1 osd.7 579 heartbeat_check: no reply from 172.21.15.144:6804 osd.4 since back 2024-03-11T22:34:05.076422+0000 front 2024-03-11T22:34:05.076320+0000 (oldest deadline 2024-03-11T22:34:28.576290+0000) 2024-03-11T22:34:31.391 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:31 smithi144 podman[163574]: 2024-03-11 22:34:31.113461036 +0000 UTC m=+5.601308795 container died 6b91511115b1c3d8e0af2eac412a96bfcb1970fb2426fecc185748bb2244dda0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:34:31.791 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:31 smithi144 podman[163574]: 2024-03-11 22:34:31.468092834 +0000 UTC m=+5.955940595 container cleanup 6b91511115b1c3d8e0af2eac412a96bfcb1970fb2426fecc185748bb2244dda0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T22:34:31.791 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:31 smithi144 bash[163574]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7 2024-03-11T22:34:32.101 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:31 smithi144 podman[163608]: 2024-03-11 22:34:31.785888779 +0000 UTC m=+0.676353771 container remove 6b91511115b1c3d8e0af2eac412a96bfcb1970fb2426fecc185748bb2244dda0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, io.buildah.version=1.33.5) 2024-03-11T22:34:32.445 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:32 smithi144 podman[163708]: 2024-03-11 22:34:32.031431554 +0000 UTC m=+0.018056247 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:34:32.445 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:32 smithi144 podman[163708]: 2024-03-11 22:34:32.16181748 +0000 UTC m=+0.148442158 container create bbb51e1d1075d611e7dbde579d8a83c720d77085181290c8501c7da4db5eb1d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, GIT_BRANCH=HEAD) 2024-03-11T22:34:32.445 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:32 smithi144 podman[163708]: 2024-03-11 22:34:32.355770709 +0000 UTC m=+0.342395387 container init bbb51e1d1075d611e7dbde579d8a83c720d77085181290c8501c7da4db5eb1d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:34:32.445 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:32 smithi144 podman[163708]: 2024-03-11 22:34:32.362754196 +0000 UTC m=+0.349378874 container start bbb51e1d1075d611e7dbde579d8a83c720d77085181290c8501c7da4db5eb1d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-11T22:34:32.445 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:32 smithi144 podman[163708]: 2024-03-11 22:34:32.438959244 +0000 UTC m=+0.425583933 container attach bbb51e1d1075d611e7dbde579d8a83c720d77085181290c8501c7da4db5eb1d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-11T22:34:32.851 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:32 smithi144 podman[163708]: 2024-03-11 22:34:32.648752316 +0000 UTC m=+0.635377002 container died bbb51e1d1075d611e7dbde579d8a83c720d77085181290c8501c7da4db5eb1d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7-deactivate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:34:33.172 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.7.service' 2024-03-11T22:34:33.200 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:33 smithi144 podman[163754]: 2024-03-11 22:34:33.150286646 +0000 UTC m=+0.496788345 container remove bbb51e1d1075d611e7dbde579d8a83c720d77085181290c8501c7da4db5eb1d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, ceph=True) 2024-03-11T22:34:33.200 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:33 smithi144 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-11T22:34:33.200 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:33 smithi144 systemd[1]: Stopped Ceph osd.7 for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:34:33.200 INFO:journalctl@ceph.osd.7.smithi144.stdout:Mar 11 22:34:33 smithi144 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@osd.7.service: Consumed 16min 32.623s CPU time. 2024-03-11T22:34:33.215 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:34:33.215 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-11T22:34:33.215 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-11T22:34:33.215 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-11T22:34:33.495 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:34:33 smithi045 systemd[1]: Stopping Ceph rgw.foo.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:34:33.787 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:34:33 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-rgw-foo-a[68460]: 2024-03-11T22:34:33.487+0000 7f4c68abe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:34:33.787 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:34:33 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-rgw-foo-a[68460]: 2024-03-11T22:34:33.487+0000 7f4c7732eac0 -1 shutting down 2024-03-11T22:34:43.788 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:34:43 smithi045 bash[230002]: time="2024-03-11T22:34:43Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-11T22:34:43.788 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:34:43 smithi045 podman[230002]: 2024-03-11 22:34:43.516606873 +0000 UTC m=+10.109427339 container died 7a0d92e11abc9d7e1dbb25aeb3b1e7ada05f4a30cddec6f3cdbc9c67ced2e232 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-11T22:34:44.141 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:34:43 smithi045 podman[230002]: 2024-03-11 22:34:43.976468064 +0000 UTC m=+10.569288530 container cleanup 7a0d92e11abc9d7e1dbb25aeb3b1e7ada05f4a30cddec6f3cdbc9c67ced2e232 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:34:44.141 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:34:43 smithi045 bash[230002]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-11T22:34:44.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-11T22:34:44.392 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:34:44 smithi045 podman[230039]: 2024-03-11 22:34:44.135826104 +0000 UTC m=+0.597717383 container remove 7a0d92e11abc9d7e1dbb25aeb3b1e7ada05f4a30cddec6f3cdbc9c67ced2e232 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-rgw-foo-a, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-11T22:34:44.392 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:34:44 smithi045 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-11T22:34:44.393 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:34:44 smithi045 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-11T22:34:44.393 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:34:44 smithi045 systemd[1]: Stopped Ceph rgw.foo.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:34:44.393 INFO:journalctl@ceph.rgw.foo.a.smithi045.stdout:Mar 11 22:34:44 smithi045 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@rgw.foo.a.service: Consumed 3.757s CPU time. 2024-03-11T22:34:44.404 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:34:44.405 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-11T22:34:44.405 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-11T22:34:44.405 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@prometheus.a 2024-03-11T22:34:44.794 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 systemd[1]: Stopping Ceph prometheus.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:34:45.101 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:44.788Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:34:45.101 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:44.788Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T22:34:45.101 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:44.788Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T22:34:45.101 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:44.788Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T22:34:45.102 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:44.788Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T22:34:45.102 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:44.788Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T22:34:45.102 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:44.788Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T22:34:45.102 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:44.788Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T22:34:45.102 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:44.798Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T22:34:45.102 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:44.799Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T22:34:45.102 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:44.799Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T22:34:45.102 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a[72828]: ts=2024-03-11T22:34:44.799Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T22:34:45.102 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:44 smithi144 podman[163839]: 2024-03-11 22:34:44.809323176 +0000 UTC m=+0.117049168 container died 06382fce5ba2561ae5c5da17a2755856508b886dde7c480b11295a0f12eba4f8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:34:45.392 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:45 smithi144 podman[163839]: 2024-03-11 22:34:45.101498711 +0000 UTC m=+0.409224711 container cleanup 06382fce5ba2561ae5c5da17a2755856508b886dde7c480b11295a0f12eba4f8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:34:45.392 INFO:journalctl@ceph.prometheus.a.smithi144.stdout:Mar 11 22:34:45 smithi144 bash[163839]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T22:34:45.496 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-11T22:34:45.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:34:45.534 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-11T22:34:45.534 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:34:45.735 INFO:teuthology.orchestra.run.smithi045.stdout:Deleting cluster with fsid: f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:34:48.249 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:34:47 smithi045 systemd[1]: Stopping Ceph alertmanager.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:34:48.537 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:34:48 smithi045 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a[86801]: ts=2024-03-11T22:34:48.242Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:34:48.537 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:34:48 smithi045 podman[230522]: 2024-03-11 22:34:48.304228928 +0000 UTC m=+0.169805440 container died 668cd5003e5013eb7fc070440b520ba2a37bc2ebbd7501b99a35c6d69fd99bb0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:34:49.037 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:34:48 smithi045 podman[230522]: 2024-03-11 22:34:48.672940409 +0000 UTC m=+0.538516921 container cleanup 668cd5003e5013eb7fc070440b520ba2a37bc2ebbd7501b99a35c6d69fd99bb0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:34:49.037 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:34:48 smithi045 bash[230522]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T22:34:49.493 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:34:49 smithi045 podman[230541]: 2024-03-11 22:34:49.065746018 +0000 UTC m=+0.757792760 container remove 668cd5003e5013eb7fc070440b520ba2a37bc2ebbd7501b99a35c6d69fd99bb0 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:34:49.493 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:34:49 smithi045 podman[230541]: 2024-03-11 22:34:49.142312963 +0000 UTC m=+0.834359706 volume remove 9636ff0db1f0237a3ca036f8ef5164c6d30fd6f6fb34452095f79747947ec1e1 2024-03-11T22:34:49.493 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:34:49 smithi045 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T22:34:49.493 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:34:49 smithi045 systemd[1]: Stopped Ceph alertmanager.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:34:49.493 INFO:journalctl@ceph.alertmanager.a.smithi045.stdout:Mar 11 22:34:49 smithi045 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.888s CPU time. 2024-03-11T22:34:49.755 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:34:49 smithi045 systemd[1]: Stopping Ceph node-exporter.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:34:50.037 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:34:49 smithi045 podman[230658]: 2024-03-11 22:34:49.748961362 +0000 UTC m=+0.098716896 container died c47febbb96576283bc98c962f252f8126eb910c4af365179ef9610fc3558ce8f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:34:50.537 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:34:50 smithi045 podman[230658]: 2024-03-11 22:34:50.143569287 +0000 UTC m=+0.493324825 container cleanup c47febbb96576283bc98c962f252f8126eb910c4af365179ef9610fc3558ce8f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:34:50.537 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:34:50 smithi045 bash[230658]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T22:34:50.537 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:34:50 smithi045 podman[230677]: 2024-03-11 22:34:50.353443763 +0000 UTC m=+0.602467098 container remove c47febbb96576283bc98c962f252f8126eb910c4af365179ef9610fc3558ce8f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:34:50.537 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:34:50 smithi045 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:34:50.980 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:34:50 smithi045 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-11T22:34:50.980 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:34:50 smithi045 systemd[1]: Stopped Ceph node-exporter.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:34:50.981 INFO:journalctl@ceph.node-exporter.a.smithi045.stdout:Mar 11 22:34:50 smithi045 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 4.955s CPU time. 2024-03-11T22:34:51.264 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:34:51.458 INFO:teuthology.orchestra.run.smithi144.stdout:Deleting cluster with fsid: f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:34:53.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:34:53 smithi144 systemd[1]: Stopping Ceph iscsi.iscsi.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:34:54.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:34:53 smithi144 ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a[63667]: debug Shutdown received 2024-03-11T22:35:04.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:35:03 smithi144 bash[164259]: time="2024-03-11T22:35:03Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-11T22:35:04.101 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:35:03 smithi144 podman[164259]: 2024-03-11 22:35:03.772725212 +0000 UTC m=+10.126808391 container died 8976c83b2a7cd9359b9c290c75ed525c178cfb59d774390dd2b7b0f5b876fe8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T22:35:04.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:35:04 smithi144 podman[164259]: 2024-03-11 22:35:04.222310119 +0000 UTC m=+10.576393299 container cleanup 8976c83b2a7cd9359b9c290c75ed525c178cfb59d774390dd2b7b0f5b876fe8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:35:04.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:35:04 smithi144 bash[164259]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a 2024-03-11T22:35:04.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:35:04 smithi144 podman[164296]: 2024-03-11 22:35:04.43075802 +0000 UTC m=+0.654868034 container remove 8976c83b2a7cd9359b9c290c75ed525c178cfb59d774390dd2b7b0f5b876fe8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-iscsi-iscsi-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T22:35:04.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:35:04 smithi144 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-11T22:35:04.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:35:04 smithi144 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-11T22:35:04.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:35:04 smithi144 systemd[1]: Stopped Ceph iscsi.iscsi.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:35:04.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi144.stdout:Mar 11 22:35:04 smithi144 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Consumed 3.564s CPU time. 2024-03-11T22:35:06.490 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:35:06 smithi144 systemd[1]: Stopping Ceph grafana.a for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:35:06.780 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:35:06 smithi144 podman[164573]: 2024-03-11 22:35:06.484483377 +0000 UTC m=+0.108650497 container died 5d44c69f500708ad3bf7fc5a1006d65329450fbd6c66623e2bd4d4e5e5f468f3 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a, architecture=x86_64, distribution-scope=public, maintainer=Paul Cuzner , io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, name=ubi8, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, io.openshift.tags=base rhel8, release=239.1651231664, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-11T22:35:07.101 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:35:06 smithi144 podman[164573]: 2024-03-11 22:35:06.863001685 +0000 UTC m=+0.487168821 container cleanup 5d44c69f500708ad3bf7fc5a1006d65329450fbd6c66623e2bd4d4e5e5f468f3 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a, architecture=x86_64, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, vendor=Red Hat, Inc., maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, distribution-scope=public, io.buildah.version=1.32.0) 2024-03-11T22:35:07.101 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:35:06 smithi144 bash[164573]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a 2024-03-11T22:35:07.443 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:35:07 smithi144 podman[164592]: 2024-03-11 22:35:07.130855249 +0000 UTC m=+0.645589908 container remove 5d44c69f500708ad3bf7fc5a1006d65329450fbd6c66623e2bd4d4e5e5f468f3 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-grafana-a, distribution-scope=public, description=Ceph Grafana Container, maintainer=Paul Cuzner , version=8.5, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, name=ubi8, architecture=x86_64, io.openshift.expose-services=, com.redhat.component=ubi8-container, release=239.1651231664, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-03-11T22:35:07.443 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:35:07 smithi144 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:35:07.443 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:35:07 smithi144 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-11T22:35:07.443 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:35:07 smithi144 systemd[1]: Stopped Ceph grafana.a for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:35:07.443 INFO:journalctl@ceph.grafana.a.smithi144.stdout:Mar 11 22:35:07 smithi144 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@grafana.a.service: Consumed 13.292s CPU time. 2024-03-11T22:35:07.711 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:35:07 smithi144 systemd[1]: Stopping Ceph node-exporter.b for f75aafa4-dff2-11ee-95c8-87774f69a715... 2024-03-11T22:35:07.965 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:35:07 smithi144 podman[164709]: 2024-03-11 22:35:07.705400955 +0000 UTC m=+0.131077644 container died d0124b7a789784acc3c81361e9711f53310fc3c1996761a5620d4ecd9a1ea3dd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:35:08.257 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:35:08 smithi144 podman[164709]: 2024-03-11 22:35:08.041804147 +0000 UTC m=+0.467480841 container cleanup d0124b7a789784acc3c81361e9711f53310fc3c1996761a5620d4ecd9a1ea3dd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:35:08.257 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:35:08 smithi144 bash[164709]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b 2024-03-11T22:35:08.515 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:35:08 smithi144 podman[164729]: 2024-03-11 22:35:08.251300504 +0000 UTC m=+0.547160757 container remove d0124b7a789784acc3c81361e9711f53310fc3c1996761a5620d4ecd9a1ea3dd (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f75aafa4-dff2-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:35:08.515 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:35:08 smithi144 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:35:08.515 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:35:08 smithi144 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-11T22:35:08.515 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:35:08 smithi144 systemd[1]: Stopped Ceph node-exporter.b for f75aafa4-dff2-11ee-95c8-87774f69a715. 2024-03-11T22:35:08.516 INFO:journalctl@ceph.node-exporter.b.smithi144.stdout:Mar 11 22:35:08 smithi144 systemd[1]: ceph-f75aafa4-dff2-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 4.545s CPU time. 2024-03-11T22:35:09.158 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:35:09.193 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:35:09.222 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T22:35:09.225 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591957/remote/smithi045/crash 2024-03-11T22:35:09.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:35:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:tar: /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:35:09.278 INFO:teuthology.orchestra.run.smithi045.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:35:09.280 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591957/remote/smithi144/crash 2024-03-11T22:35:09.281 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:35:09.362 INFO:teuthology.orchestra.run.smithi144.stderr:tar: /var/lib/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:35:09.362 INFO:teuthology.orchestra.run.smithi144.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:35:09.363 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T22:35:09.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-11T22:35:09.449 INFO:teuthology.orchestra.run.smithi045.stdout:2024-03-11T22:20:00.001083+0000 mon.a (mon.0) 1459 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-11T22:35:09.449 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T22:35:09.450 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[SEC\]' /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-11T22:35:09.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[ERR\]' /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-11T22:35:09.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[WRN\]' /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-11T22:35:09.620 INFO:teuthology.orchestra.run.smithi045.stdout:2024-03-11T22:20:00.001083+0000 mon.a (mon.0) 1459 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-11T22:35:09.621 INFO:tasks.cephadm:Compressing logs... 2024-03-11T22:35:09.621 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:35:09.663 DEBUG:teuthology.orchestra.run.smithi144:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:35:09.709 INFO:teuthology.orchestra.run.smithi144.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-11T22:35:09.709 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:35:09.717 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-11T22:35:09.718 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:35:09.718 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-11T22:35:09.718 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-11T22:35:09.718 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-11T22:35:09.718 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T22:35:09.718 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/tcmu-runner.log 2024-03-11T22:35:09.718 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:35:09.718 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:35:09.718 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:35:09.718 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T22:35:09.728 INFO:teuthology.orchestra.run.smithi045.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-11T22:35:09.728 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:35:09.735 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T22:35:09.735 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:35:09.736 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-11T22:35:09.736 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mon.a.log: /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T22:35:09.736 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-11T22:35:09.736 INFO:teuthology.orchestra.run.smithi045.stderr: 57.7% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-11T22:35:09.738 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T22:35:09.738 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T22:35:09.738 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:35:09.738 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:35:09.738 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T22:35:09.739 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:35:09.739 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T22:35:09.748 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mon.c.log: 91.6% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:35:09.768 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mon.b.log: 81.5% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:35:09.831 INFO:teuthology.orchestra.run.smithi144.stderr: 89.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:35:09.853 INFO:teuthology.orchestra.run.smithi144.stderr: 67.0% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/tcmu-runner.log.gz 2024-03-11T22:35:09.872 INFO:teuthology.orchestra.run.smithi144.stderr: 92.4% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T22:35:09.876 INFO:teuthology.orchestra.run.smithi144.stderr: 90.1% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:35:09.883 INFO:teuthology.orchestra.run.smithi144.stderr: 91.1% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:35:09.923 INFO:teuthology.orchestra.run.smithi045.stderr: 91.2% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:35:09.935 INFO:teuthology.orchestra.run.smithi045.stderr: 94.0% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:35:10.242 INFO:teuthology.orchestra.run.smithi144.stderr: 85.1% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:35:10.243 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:35:10.353 INFO:teuthology.orchestra.run.smithi045.stderr: 95.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:35:10.814 INFO:teuthology.orchestra.run.smithi045.stderr: 89.5% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T22:35:12.157 INFO:teuthology.orchestra.run.smithi144.stderr: 91.1% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T22:35:17.230 INFO:teuthology.orchestra.run.smithi045.stderr: 91.1% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T22:35:28.988 INFO:teuthology.orchestra.run.smithi045.stderr: 90.3% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T22:35:45.091 INFO:teuthology.orchestra.run.smithi144.stderr: 94.7% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-11T22:37:35.368 INFO:teuthology.orchestra.run.smithi045.stderr: 93.0% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-11T22:37:51.063 INFO:teuthology.orchestra.run.smithi144.stderr: 93.6% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-11T22:38:05.587 INFO:teuthology.orchestra.run.smithi144.stderr: 94.5% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-11T22:39:05.020 INFO:teuthology.orchestra.run.smithi144.stderr: 93.0% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-11T22:39:05.022 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-03-11T22:39:05.022 INFO:teuthology.orchestra.run.smithi144.stderr:real 3m55.343s 2024-03-11T22:39:05.022 INFO:teuthology.orchestra.run.smithi144.stderr:user 4m10.771s 2024-03-11T22:39:05.022 INFO:teuthology.orchestra.run.smithi144.stderr:sys 0m11.007s 2024-03-11T22:49:00.919 INFO:teuthology.orchestra.run.smithi045.stderr: 93.2% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T22:50:08.792 INFO:teuthology.orchestra.run.smithi045.stderr: 93.3% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T22:51:04.430 INFO:teuthology.orchestra.run.smithi045.stderr: 92.8% -- replaced with /var/log/ceph/f75aafa4-dff2-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T22:51:04.432 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-03-11T22:51:04.432 INFO:teuthology.orchestra.run.smithi045.stderr:real 15m54.758s 2024-03-11T22:51:04.432 INFO:teuthology.orchestra.run.smithi045.stderr:user 7m23.809s 2024-03-11T22:51:04.432 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m20.376s 2024-03-11T22:51:04.433 INFO:tasks.cephadm:Archiving logs... 2024-03-11T22:51:04.434 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591957/remote/smithi045/log 2024-03-11T22:51:04.435 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:52:09.963 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591957/remote/smithi144/log 2024-03-11T22:52:09.965 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:52:47.968 INFO:tasks.cephadm:Removing cluster... 2024-03-11T22:52:47.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 --force 2024-03-11T22:52:48.207 INFO:teuthology.orchestra.run.smithi045.stdout:Deleting cluster with fsid: f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:52:49.013 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f75aafa4-dff2-11ee-95c8-87774f69a715 --force 2024-03-11T22:52:49.205 INFO:teuthology.orchestra.run.smithi144.stdout:Deleting cluster with fsid: f75aafa4-dff2-11ee-95c8-87774f69a715 2024-03-11T22:52:49.791 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T22:52:49.791 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:52:49.822 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:52:49.836 INFO:tasks.cephadm:Teardown complete 2024-03-11T22:52:49.837 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-11T22:52:49.847 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-11T22:52:49.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T22:52:49.864 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T22:52:49.969 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T22:52:49.979 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T22:52:49.979 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:52:49.982 DEBUG:teuthology.orchestra.run.smithi144:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:52:49.995 INFO:teuthology.orchestra.run.smithi144.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:52:50.001 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:52:50.020 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:52:50.020 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2024-03-11T22:52:50.020 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +4221us[+4221us] +/- 127ms 2024-03-11T22:52:50.020 INFO:teuthology.orchestra.run.smithi144.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -1267us[-1219us] +/- 65ms 2024-03-11T22:52:50.020 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -954us[ -954us] +/- 74ms 2024-03-11T22:52:50.020 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:52:50.054 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:52:50.054 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-03-11T22:52:50.054 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 +4280us[+4300us] +/- 127ms 2024-03-11T22:52:50.054 INFO:teuthology.orchestra.run.smithi045.stdout:^* hv02.front.sepia.ceph.com 3 6 377 60 -1237us[-1217us] +/- 65ms 2024-03-11T22:52:50.054 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 -889us[ -870us] +/- 74ms 2024-03-11T22:52:50.055 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:52:50.055 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T22:52:50.065 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T22:52:50.065 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T22:52:50.075 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:52:50.209 DEBUG:teuthology.orchestra.run.smithi144:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:52:50.331 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:52:50.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:52:50.364 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:52:50.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:52:50.393 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T22:52:50.403 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T22:52:50.413 INFO:teuthology.task.internal:Duration was 3486.264870 seconds 2024-03-11T22:52:50.414 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T22:52:50.425 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T22:52:50.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:52:50.431 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:52:50.505 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:52:50.510 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:52:50.857 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T22:52:50.857 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-03-11T22:52:50.857 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:52:50.878 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi144.front.sepia.ceph.com 2024-03-11T22:52:50.878 DEBUG:teuthology.orchestra.run.smithi144:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:52:50.908 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T22:52:50.908 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:52:50.921 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:52:50.977 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T22:52:50.977 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:52:50.991 DEBUG:teuthology.orchestra.run.smithi144:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:52:52.363 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T22:52:52.375 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T22:52:52.375 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:52:52.412 DEBUG:teuthology.orchestra.run.smithi144:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:52:52.519 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T22:52:52.531 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:52:52.533 DEBUG:teuthology.orchestra.run.smithi144:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:52:52.561 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = core 2024-03-11T22:52:52.563 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-03-11T22:52:52.597 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:52:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:52:52.651 DEBUG:teuthology.orchestra.run.smithi144:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:52:52.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:52:52.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T22:52:52.679 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T22:52:52.680 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591957/remote/smithi045 2024-03-11T22:52:52.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:52:52.768 DEBUG:teuthology.misc:Transferring archived files from smithi144:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591957/remote/smithi144 2024-03-11T22:52:52.768 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:52:52.808 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T22:52:52.808 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:52:52.811 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:52:52.856 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T22:52:52.868 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T22:52:52.868 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T22:52:52.885 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T22:52:52.886 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:52:52.888 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:52:52.900 INFO:teuthology.orchestra.run.smithi045.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:52 /home/ubuntu/cephtest 2024-03-11T22:52:52.911 INFO:teuthology.orchestra.run.smithi144.stdout: 265814 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:52 /home/ubuntu/cephtest 2024-03-11T22:52:52.912 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T22:52:52.923 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T22:52:52.945 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T22:52:52.968 DEBUG:teuthology.nuke:shortname: smithi045 2024-03-11T22:52:52.969 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:52:52.988 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591957', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:46:21.651062', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL8HCKAiUE5i57nH3luyguokdWX60uLTvKIIIoEY6+6zrotNR7XmIqhZMcMq9WIpidu+vlY72/Egm3+g/dj5fII='} 2024-03-11T22:52:52.992 DEBUG:teuthology.nuke:shortname: smithi144 2024-03-11T22:52:52.992 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:52:53.009 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7591957', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 21:46:21.652621', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEeSpawdyKHIPZnjnCakl3KMpXLEPUzEfx4UmuNci2etZ/uYwCAVLxMp2K//Tpoq62Ob4NrV0XeGWhHbNOZ3KE4='} 2024-03-11T22:52:53.037 INFO:teuthology.orchestra.console.smithi045:Power off 2024-03-11T22:52:53.037 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:52:53.053 INFO:teuthology.orchestra.console.smithi144:Power off 2024-03-11T22:52:53.054 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:52:53.062 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2024-03-11T22:52:53.063 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:52:53.075 DEBUG:teuthology.orchestra.console.smithi144:power off output: Chassis Power Control: Down/Off 2024-03-11T22:52:53.075 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:52:53.086 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-03-11T22:52:53.098 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2024-03-11T22:52:57.087 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:52:57.099 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:52:57.198 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-03-11T22:52:57.209 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2024-03-11T22:53:01.203 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:53:01.212 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:53:01.314 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-03-11T22:53:01.321 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is off 2024-03-11T22:53:01.416 INFO:teuthology.orchestra.console.smithi045:Power off completed 2024-03-11T22:53:01.422 INFO:teuthology.orchestra.console.smithi144:Power off completed 2024-03-11T22:53:01.526 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} duration: 3486.264870405197 failure_reason: '"2024-03-11T22:20:00.001083+0000 mon.a (mon.0) 1459 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-11T22:53:01.526 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:53:01.593 INFO:teuthology.run:FAIL